beegfs, "cheap" NAS units and buddy groups

[ste...@fletcher-parker.com]

I've recently come into ownership of a small beeGFS setup. I personally have very little experience, apart from setting up a couple of little test clusters. All positive experiences so far...

Our current system leverages ZFS raid2z arrays in supermicro servers as storage targets. I have been asked to expand the array on a limited budget. 

The cost per available TB on the supermicros is quite high and the performance not ideal. Performance is poor because the beeGFS was tacked onto an existing NFS setup. ZFS Array design is primarily for redundancy, not performance. I am not allowed to break the ZFS arrays.

 A cost analysis suggests the use of a cheap commericial NAS such as a QNAP TS-463U-RP (8-bay HDD, 10Gbe) running raid 0, using buddy mirroring, would a) saturate the 10Gbe and b) rebuild an entire NAS array in around the same amount of time as a single disk resilver on the ZFS box (e.g. no parity). Basically using buddy to turn raid 0 into raid 10. Our compute has 40Gbe, which 8 NAS units should make a fair attempt at saturating. With full buddy mirroring and 8TB drives, it works out at around $50k AUD for 256Tb of storage.

My supermicro supplier is horrified. A quick google on the use of off-the-shelf NAS drives and beeGFS was not very revealing.

Can anyone share any experience with using low-cost NAS drives as storage targets, and any potential pitfalls? Any suggestions on a better way to spend the $50k which balances storage, redundancy and performance?

Cheers

Stu

[Nick Tan]

Hi Stu,

 

Buddy Mirroring is good for host failure, but not so good for disk failure.  If you have a disk failure in your raid-0, as long as the beegfs-storage process is still alive, then beegfs will think the disk target is fine and won’t do any failover.  The beegfs-storage process must be stopped (manually, or by node failure) for the mirror to be promoted to primary.  BeeGFS assumes some level of disk redundancy that is separate to buddy mirroring.

 

Also, buddy mirroring is a licensed feature, so you’ll need to take the cost of licensing into account within your $50k budget.

 

Are you planning on running metadata on the QNAP as well?  You’d want SSD or NVME for metadata, and if your application is metadata heavy you might want to consider InfiniBand to cut down on the latency.

 

I personally wouldn’t use QNAP but that’s just because I have had some pretty bad experiences with Qnap boxes, but I don’t see any reason why it wouldn’t work.

 

I’ve recently gone through a performance tuning exercise with BeeGFS - there are many options which can dramatically boost performance depending on where the bottleneck lies.  It could be that with some tuning you could drastically improve performance on your existing ZFS targets.

 

Hope this helps.

 

Nick

 

Nick Tan, Senior Systems Administrator

Spookfish Global Operations Pty Ltd 

m: +61 413 134 846

w: spookfish.com

--
You received this message because you are subscribed to the Google Groups "beegfs-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fhgfs-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

PRIVACY AND CONFIDENTIALITY NOTICE
The information contained in this message is intended for the named recipients only. It may contain confidential information and if you are not the intended recipient, you must not copy, distribute or take any action in reliance on it. If you have received this message in error please destroy it and reply to the sender immediately or contact us at the above telephone number.
VIRUS DISCLAIMER
While we take every precaution against presence of computer viruses on our system, we accept no responsibility for loss or damage arising from the transmission of viruses to e-mail recipients.

[Stewart Fletcher]

Thanks for the response Nick, 

> Buddy Mirroring is good for host failure, but not so good for disk failure.

Also, buddy mirroring is a licensed feature, so you’ll need to take the cost of licensing into account within your $50k budget.

Noted. My other option is to run raid5s and regular checkpointing.  This was to be my fallback if licensing breaks my budget. It might have to be my primary...

 

Are you planning on running metadata on the QNAP as well? 

No, I have a server with SSD for meta. I plan to duplicate if I push forward.

  

I personally wouldn’t use QNAP but that’s just because I have had some pretty bad experiences with Qnap boxes, but I don’t see any reason why it wouldn’t work.

I have a couple, which is what got me thinking buddy might be able to offset the low reliability. But as the drives seem to be the major cost, I'll throw some better quality NAS into my spreadsheet.

 

I’ve recently gone through a performance tuning exercise with BeeGFS - there are many options which can dramatically boost performance depending on where the bottleneck lies.  It could be that with some tuning you could drastically improve performance on your existing ZFS targets.

Current setup is not ideal. zfs/FreeBSD runing iSCSI targets with the metaserver as initiator running the storage service. Targets been mounted inside the production NFS/ZFS filesystem. If I break that, the pitchforks come out.  Once I have a stable filesystem in parallel I might be able to move everything, break the pools and start again.

 

Hope this helps.

Heaps, thanks for the input.

Cheers,

Stu