Multiple OSD per host?

[wern...@gmail.com]

Heya -

Looking at multiple distributed FS's, and while most I've come across appear to be doing what I want them to do, they just don't provide the consistency or client access that I'm looking for sometimes.

I know in projects like Ceph and MooseFS, you have multiple disks, each acting as an OSD, and being served out to the grid.

XtreemFS offers a lot of what I'm looking for, especially the posix issue and the consistency issue, and even more so, the nonblocking redirection to another copy while the file's in use (epic btw).

Does XTFS offer the same choice, or do I need to provide a RAID backend for a singular OSD per storage node?

Obviously, looking to avoid that, since that way I gain more storage space per system, and avoid the duplication of effort (spreading data across nodes, and then raid handling the spread across disks)

Hopefully it's not a ridiculously stupid question, but I couldn't find any references to it in the manuals or online!

Thanks!

[Felix Hupfeld]

As you probably suspect, your options are:

* one OSD server per host with RAID

* several OSD servers per host, ie. one per hard disk. As XtreemFS' OSD does not support managing several hard disks in one process, you need to start several OSD per host.

2013/2/11 <wern...@gmail.com>

--
 
---
You received this message because you are subscribed to the Google Groups "XtreemFS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xtreemfs+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[wern...@gmail.com]

Felix - 

Thanks for the response, it's much appreciated!

That's sort of what I figured, and the more I dig into the config files, the more it's looking like having the RAID backend might make things easier.

Unless, of course, someone can point me where to configure multiple OSDs per host? I'm not seeing how to distinguish between different configs for different OSDs.

Any thoughts?

Thanks again!

Russ

[Carlo Daffara]

We prefer to use BTRFS as a way to aggregate different disks, so as to export a single OSD per host. However, if you want, you can run multiple xtreemfs-osd processes, using different configuration files- so you can have multiple ports and multiple storage units in a single host.

regards,
Carlo Daffara

CloudWeavers

--

[Michael Berlin]

Dear Russ,

I once posted a init.d script which allows you to start and stop
multiple OSDs with different configuration files:
https://groups.google.com/forum/#!msg/xtreemfs/12JsqO_rwMI/9nuWhK02uAQJ

I still use this script for one of our internal installations - however,
it is not updated to the latest changes of the init.d script, i.e. it
may require some more testing and minor modifications.

The multiple OSD configuration files must differ in the following settings:

a) uuid =

Every OSD needs a unique UUID. btw: they don't have to be randomized,
only unique and must never change - for example, sometimes I set the
UUID just to "[installation-name]-osd1", [installation-name]-osd2" and
so on. This way you can also use the UUIDs to keep track of the
different machines.

b) listen.port =

Port of the OSD.

c) http_port =

Port for the Webinterface.

d) object_dir =

Place where the data is stored.

Best regards,
Michael

[saltr...@gmail.com]

I have 12 disks connected to the osd server and wanted to add in 12 osds.

I was looking at this file "osdconfig.properties" to copy and modify for each osd, however

was concerned wh

It seems, I need to edit this one

"

# directory containing XtreemFS file content
object_dir = /var/lib/xtreemfs/objs/

"

Maybe create /var/lib/xtreemfs/osd.1, /var/lib/xtreemfs/osd.2 etc and then use each one per osd prop file.

But I don't see the hard disk information..

like where does the osd map to say /dev/sd5.

Does xtreemfs handle formatting the disk in to a particular fs or should I format it is xfs or something else ?

Please advice

Thank you

Regards

Ruchika

[Christoph Kleineweber]

Hi Ruchika!

> I have 12 disks connected to the osd server and wanted to add in 12 osds.
>
> I was looking at this file "osdconfig.properties" to copy and modify for each osd, however
> was concerned wh
> It seems, I need to edit this one
> "
> # directory containing XtreemFS file content
> object_dir = /var/lib/xtreemfs/objs/
>
> "
> Maybe create /var/lib/xtreemfs/osd.1, /var/lib/xtreemfs/osd.2 etc and then use each one per osd prop file.
>
> But I don't see the hard disk information..
> like where does the osd map to say /dev/sd5.
>
> Does xtreemfs handle formatting the disk in to a particular fs or should I format it is xfs or something else ?

You have to create a file system on each of the disks (ext4 for instance works well). In your case, the object_dir has to be the mount point of the disk’s file system. Mind that each of the OSDs have to run on a separate port and needs an unique UUID.

Kind regards,
Christoph

[saltr...@gmail.com]

Christoph,

Thank you.

Got it!!

Was curious if anyone tests xfs before or if there was higher probability of errors with xfs on the drives.

Thanks you have been most helpful.

I am trying to build my client on a PowerPC 32 b platform and hope it works.

Thank you

Regards

Ruchika 

[Christoph Kleineweber]

> Was curious if anyone tests xfs before or if there was higher
> probability of errors with xfs on the drives.
> Thanks you have been most helpful.

I think one of the special features of XFS is its good scalability to be
usable for large file systems, e.g. in a RAID, and the delayed
allocation. The first seems to be irrelevant while using one OSD per
disk. One of my former colleges measured that ext4 is faster under an
XtreemFS OSD than XFS, so I do not see any need to weaker consistency
that is caused by XFS' delayed allocation. Another advantage of delayed
allocation should be less fragmentation. I expect the performance
drawback you may experience by fragmentation on ext4 is negligible due
to the overhead of XtreemFS.

> I am trying to build my client on a PowerPC 32 b platform and hope it works.

We do not have PowerPC machines and never tried to build the XtreemFS
client on this platform, but I do not expect any issues. I would be
happy if you could share your experiences with PowerPC.

Kind regards,
Christoph