Where do you need the hardware performance?

[Dennis McEntire]

I am wondering if the "splitting" of the files that get entered into
the XtreemFS system occurs on the OSD or the MRC? Basically, we are
trying to figure out how to help out the speed of copying (cp) files
into the system. I am not sure where the delays occur, but it would be
nice to throw hardware at it to help the speed issue.

Also, I thought I read somewhere that a future feature would include
variable block sizes. So if we are going to copy a 1GB file, we could
use 1MB blocks, whereas a small text file would use 32K blocks. Did I
read a roadmap wrong somewhere or is this still something in the
works?

Thanks!

Dennis

[Minor Gordon]

Hi Dennis,

On Mon, Dec 14, 2009 at 11:29 PM, Dennis McEntire <dmce...@gmail.com> wrote:

I am wondering if the "splitting" of the files that get entered into
the XtreemFS system occurs on the OSD or the MRC? Basically, we are
trying to figure out how to help out the speed of copying (cp) files
into the system. I am not sure where the delays occur, but it would be
nice to throw hardware at it to help the speed issue.

By "splitting" I assume you mean "striping", the way files are distributed over multiple OSDs, and not the split between file metadata (on the MRC) and data (on OSDs). Adding more OSDs will increase total I/O bandwidth -- assuming you have programs that can make use of the bandwidth! A single instance of cp usually can't because it only does 1 I/O (read/write) at a time with relatively small buffer sizes.

Also, I thought I read somewhere that a future feature would include
variable block sizes. So if we are going to copy a 1GB file, we could
use 1MB blocks, whereas a small text file would use 32K blocks. Did I
read a roadmap wrong somewhere or is this still something in the
works?

I think you probably mean "stripe sizes". A stripe in XtreemFS (an object-based file system) is roughly analogous to a block in a block-based file system like ext3. Stripes are placed on different OSDs depending on their location in the file, e.g. if the stripe size is 8K then the first 8K of the file is placed on OSD 1, the second 8K on OSD 2, etc. in a round-robin manner. As of release 1.2 cp uses the stripe size as its buffer size, which means that it always reads/writes a full stripe. Before release 1.2 cp always used 4k buffers, which meant that it would write the first 4K to OSD 1, the second 4K to OSD 1, the third 4K to OSD 2, etc.

The stripe size is a tradeoff between latency, bandwidth, and space on the OSDs. If you have large files and OSDs connected by a LAN you should increase the stripe size above the default 128k. If you have small files then you should keep the default stripe size or lower it. If you have large files and OSDs connected by a WAN then you'll have to experiment a bit. It's worth testing different configurations anyway.

You can set the stripe size for a specific file with xtfs_sp (see the man page). Setting a stripe size with xtfs_sp on a directory causes all files created in that directory (but not subdirectories) to inherit that stripe size. You can set the default stripe size for an entire volume when you create the volume with xtfs_mkvol.

Hope this helps.

Minor

[Felix Hupfeld]

Hi everyone,

a quick clarification to make things even more complicated ...

Conceptually (and how things are in the architecture and implemented),
we have object and on top of that striping. The file system "splits"
files into equal-sized objects (the last object might be shorter),
hence the name object-based file system. These objects are stored on
OSDs (object storage devices).

The concept of objects is similar to blocks in block-based file
systems. The difference is that a block size is set for the whole file
system, whereas object size can be set per file at creation time (and
therefore be adapted to file sizes). In addition, OSDs not only know
about block numbers as block stores do, but know file ids, which is a
precondition to a few of XtreemFS' features.

So objects are stored on OSDs, and from an abstract point of view the
assignment of objects to OSD is where the concepts of striping and
replication come in. Files have replicas, each of which can be striped
across multiple OSDs. As they are concepts on top of objects, they
share the notion of object sizes, and the stripe size is equal to the
object size. Note that the manual and our tools work on striping
level, you'll find that you set the object size of an unstriped file
by setting the stripe size and no striping policy. And that's the
abstraction level where Minor's response hooks in...