Experiences with large scale use?

[Miles Fidelman]

Hi Folks,

I've been running a small cluster, supporting a small service bureau,
and I'm getting ready to do a serious update - with the notion of both
preparing to scale, and adding some geographic redundancy.  Currently
running DRBD mirroring & Xen VMs with failover across 4 servers.

It looks like XtreemFS checks all the boxes that I'm looking for:

- block storage

- high availability & fault tolerance

- mirroring across more than 2 servers

- geographic distribution

But.. a few question:

- how stable & supported is it (the copyrights on documents seem to end
at 2015, the email list is not very active)

- how far down does it go into disk management (does one need to use a
logical volume manager with it, does it have any features for mixing
HDDs and SSDs, etc.)

- is anybody out there using it in large scale, production use?

Thanks for any comments anyone might have to offer.

Miles Fidelman

--
In theory, there is no difference between theory and practice.
In practice, there is. .... Yogi Berra

Theory is when you know everything but nothing works.
Practice is when everything works but no one knows why.
In our lab, theory and practice are combined:
nothing works and no one knows why. ... unknown

[Robert Schmidtke]

Hi Miles,

you're right, XtreemFS does support your features.

It is a research project and as such has seen a variety of maintainers/developers.

There is no dedicated support or feature development, and it is highly likely that this will not change, as most researchers have moved on to other topics.

We will make an official announcement regarding this issue soon, so the mailing list will mostly be operated by users (me being more or less the only one involved with XtreemFS, and I'm about to withdraw from the project too).

The project will remain open source on GitHub and can be freely developed by everyone.

XtreemFS does not do disk management, the easiest way to address the HDD/SDD issue would be to just start two OSDs per node, one using HDDs as data directories, the other using SSDs.

The underlying file system is not of importance, as XtreemFS stores data using regular file system operations (we mostly used ext* and xfs as underlying file systems).

You can choose whatever you like, though.

Concerning large scale deployments we do not have exact numbers, as anyone can download and use the file system.

During our tests we routinely use 60+ nodes.

However please be aware that while OSD failover works seamlessly, DIR and MRC failover suffers from a race condition and as such may lead to an unusable state in case of failure.

That's why we advise users to proceed with caution when using production deployments, as the DIR and MRC are single points of failure, effectively.

I hope this helps.

Cheers

Robert

[Miles Fidelman]

Thanks Robert,

Of course you neglected to mention that Quobyte is now shipping a proprietary, commercial version.

Re. mixing HDD and SSD OSDs - that doesn't really do anything without something to automatically allocate data across the two to optimize performance.

Cheers,

Miles

--

---
You received this message because you are subscribed to the Google Groups "XtreemFS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xtreemfs+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/xtreemfs/7c3564f0-656d-4848-888b-c845efd9c7e7%40googlegroups.com.