separate servers vs multiple JBODs hosted on 1 server.

[Harry Mangalam]

We are planning a small backup filesystem (up to ~0.5 PB) and we're debating the approach of multiple servers with a few arrays each on a 10GbE switch vs a single server with lots of JBOD-hosted arrays (also on 10GbE).  ie one active server will have 2-4 disk controller HBAs connected to 1 JBOD each.  In other words we would be using the PCI backplane as the switching fabric instead of an external switch.  All the BeGFS daemons (admon, meta, mgmnt, storage) would be running on the same server.

The system will be fed by a 10GbE channel from the primary storage.

The arrays will be BeeGFS, probably on ZFS, and doesn't have to be enormously high-perf, but we're wondering if there is a simple rationale for either one.  This would seem to remove the necessity for multiple intelligent servers, and therefore lower the cost a lot.  (and also, IIRC, reduce the cost of ThinkParq support :) )

ThinkParq produced this nice summation of optimal targets per array size: 

http://www.beegfs.com/docs/Picking_the_right_Number_of_Targets_per_Server_for_BeeGFS_by_ThinkParQ.pdf

and we would keep to that by splitting each 60-disk JBOD into 4 arrays.  That seems like a lot of traffic to shove down a PCI bus and disk controller, but it's going to be hitting those controllers anyway.  The question is whether it's acceptable to push this much data onto one PCI bus or spread it out on multiple servers.

Since the bottleneck will be the 10GbE channel from the primary storage, the data feed can't get too large, so a multi-JBOD fanout from one server may be fine.

Do other BeeGFSers (especially using a ZFS underlay) have feedback on this?

Thanks in advance

Harry

[Sven Breuner]

Hi Harry,

if I got this right, I guess it really just boils down to the question
of whether 10Gbps will be good enough for this use case, doesn't it?
Because just like you said, this will always be the limit in the setup
that you described, no matter if you attach 60 or n*60 disks to the server.

Or did I get this wrong?

Best regards,
Sven

> --
> You received this message because you are subscribed to the Google
> Groups "beegfs-user" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to fhgfs-user+...@googlegroups.com
> <mailto:fhgfs-user+...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

[harry mangalam]

Thanks for the input, Sven.

 

Yes, this is the upper limit (currently; we are trying to get dual 10GbE feeders to the system) but in testing on various combinations, we're seeing that while we can push about 1.3 GB/s to a single 24spindle RAIDZ2 via dd on the storage server, in real life we're seeing about half of that bc we assume that the mix of large and small files is causing too much overhead.

 

We'll be trying to chop up the single vdev into multiples to see if that helps, but we're still worried about pushing all the data thru a single PCIe3 bus. Theoretically it has plenty of bandwidth, but in Real Life?

 

So the question is:

 

If we add more JBODs to the storage server, each on its own HBA, will total IO increase or will there still be a bottleneck in either the ZFS/BeeGFS layer or via contention in the PCIe3 bus?

 

'It depends' is almost certainly the answer, but as a general rule of thumb..?

 

Regardless, we'll try it out and report back.

 

Harry

---

Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine

[m/c 2225] / 92697 Google Voice Multiplexer: (949) 478-4487

415 South Circle View Dr, Irvine, CA, 92697 [shipping]

XSEDE 'Campus Champion' - ask me about your research computing needs.

Map to Office | Map to Data Center Gate

[the command line is the new black]

---

 

[David Ramírez]

Hi. I'll testing new XenServer 7.0. Somebody are tested BeeGFS Client to
VM Storage?? It works?? I prefer it than a ISCSI for example. I hope
works fine with 10T connection

Thanks

--


Este correo y sus archivos asociados son privados y confidenciales y va
dirigido exclusivamente a su destinatario. Si recibe este correo sin ser el
destinatario del mismo, le rogamos proceda a su eliminación y lo ponga en
conocimiento del emisor. La difusión por cualquier medio del contenido de
este correo podría ser sancionada conforme a lo previsto en las leyes
españolas. No se autoriza la utilización con fines comerciales o para su
incorporación a ficheros automatizados de las direcciones del emisor o del
destinatario .

This mail and its attached files are confidential and are exclusively
intended to their addressee. In case you may receive this mail not being
its addressee, we beg you to let us know the error by reply and to proceed
to delete it. The circulation by any mean of this mail could be penalised
in accordance with the Spanish legislation. The use of both the transmitter
and the addressee’s address with a commercial aim, or in order to be
incorporated to automated files, is not authorised.