QFS as a HA filesystem?

[Dieter Plaetinck]

Hi,

I'm looking to build a HA storage platform of around 3PB for mostly cold (rarely accessed) video files.

the data is immutable so erasure coding seems like the better fit than replication.  QFS is the only open source based system I could find implementing erasure codes.

i'ld fill it with video files, avg filesize 150MB, max size 5GB (can increase later)

but I'm not sure if QFS is architecturally a good fit:

* i wouldn't do any hadoop or batch workload, no map/reduce. just storing and fetching files (which would get chunked up and erasure encoded), which is however, obviously, sequential reads/writes

* the system needs to be HA.  how does one typically recover from a namenode failure with QFS? and is effort being done to eliminate this SPOF?

I'm thinking of replacing the central namenode with a central cluster, relying on Cassandra. this could contain all files (and optionally all chunks. haven't thought that through) and would enable customizable placement. 

* from https://research.microsoft.com/pubs/179583/LRC12-cheng%20webpage.pdf I gather that LRC(12,2,2) is a very compelling alternative to Reed-Solomon, any thoughts on changing algorithms?

* is there any information on cpu/memory load for the 1)generation of encoded data and 2)reconstruction of content based on chunks, as a function of input/output load? i.e. what's the computational overhead of the erasure coding?

thanks.

Dieter

[Silvius Rus]

Hello Dieter,

We have a couple of production filesystem instances similar to what you describe (over 3 PB, file size around 100 MB, some single files of at least 10 GB each).

We are interested in developing HA but it is not high priority.  Last time I checked the file systems they had over 100 days of uptime each.  They are under heavy load (multi PB I/O every day).  We had an incident recently where the metaserver box got rebooted, but the metaserver came back after the reboot and the systems recovered.  There were a few minutes of outage though (below 10), and pending open files did get truncated.

Scaling the metaserver out and providing HA are nice goals, but large undertakings.  Without having experimented with it, my main concerns about using a cluster to store the metadata would be latency and consistency.  If you require a write quorum you'll be slower (which may or may not be acceptable).  If you go with single write master and async replication then you're not really HA.

It's certainly possible to go with a simpler solution for HA, such as a hot standby that gets passed the master log + a failover mechanism for chunkservers and clients.

We have not decided yet on HA goals or methods.  I believe we'll be looking into it before end of 2013.

Thank you for the LRC reference.  It could perhaps be interesting to offer LRC encoding as an option, but we have not looked into it yet.  If you want to go with 1.33x you can try Reed-Solomon 9+3 in QFS.  You save space but pay in recovery bandwidth.  We have only tested 6+3 at scale.

Erasure encoding goes at 2.1 GB/s and decoding 1.2 GB/s on a Westmere core at 2.93 GHz using vector instructions.  It may be significantly slower if the CPU does not provide the required vector instructions.

Silvius