BeeGFS vs GlusterFS read performance. Do I expect too high for BeeGFS?

[John Lewis]

I'm doing 2 read benchmark test on GlusterFS and BeeGFS. Both are in default setting, no tuning or whatsoever:

GlusterFS, first benchmark:

root@node1:/var/www/upload/dd# time sh -c "dd if=/var/www/upload/dd/test.tmp of=/dev/null bs=4k"
56000000+0 records in
56000000+0 records out
229376000000 bytes (229 GB) copied, 437.076 s, 525 MB/s

GlusterFS, second benchmark:

root@node1:/var/www/upload/dd# time sh -c "dd if=/var/www/upload/dd/test.tmp of=/dev/null bs=4k"
56000000+0 records in
56000000+0 records out
229376000000 bytes (229 GB) copied, 477.413 s, 480 MB/s

BeeGFS, first benchmark:

root@node1:/var/www/beegfs/dd# time sh -c "dd if=/var/www/beegfs/dd/test.tmp of=/dev/null bs=4k"
56000000+0 records in
56000000+0 records out
229376000000 bytes (229 GB) copied, 1414.29 s, 162 MB/s

BeeGFS, second benchmark:

root@node1:/var/www/beegfs/dd# time sh -c "dd if=/var/www/beegfs/dd/test.tmp of=/dev/null bs=4k"
56000000+0 records in
56000000+0 records out
229376000000 bytes (229 GB) copied, 1430.03 s, 160 MB/s

As we can see, BeeGFS read performance is only about 1/3 GlusterFS.

BeeGFS is mounted using parameter like on the help file: --setpattern --chunksize=1m --numtargets=2
(not sure the "2" is the one should I use on --numtargets parameter, for sure I only have 2 node)

What did I do wrong here? Maybe I should mount the mirror using another parameter? or maybe need to tune some config?

Thanks a lot!

Best regards,
John.

[P Serocka]

Not sure what you're aiming at when doing streaming reads at 4KB blocksize...

Try dd bs=128k and dd bs=1M

My guess would be that with the given default configs GlusterFS
does a better job in prefetching, enabling it to hit the
the disks with much larger transfer sizes than 4KB.

(And assuming of course that the hardware of your two clusters
are basically identical, node/CPU/disk-wise).

Cheers

-- Peter

> --
> You received this message because you are subscribed to the Google Groups "beegfs-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to fhgfs-user+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Harry Mangalam]

I'm going to sort of ignore the benchmarky stuff, since it seems to be reiterating what I found previously.  Gluster and BeeGFS have similar performance metrics if tuned correctly on large files (I found that Gluster actually was slightly faster on single user, unloaded tests), but unless you're running a single-user systems, BeeGFS is a much better choice. Gluster falls apart under load, does not handle small files (especially small writes into files) or recursion well.  It is (from its own developers), essentially a high performance alternative to tape, so while it can do very admirable IO on a few streams, on an HPC system with lots of IO from lots of of sources and sinks, it just fails miserably.

This is now getting dated, but I think it may still be useful:

http://moo.nac.uci.edu/~hjm/fhgfs_vs_gluster.html

hjm