QLogic NICs disabled on Clemson nodes

[Edward Tremel]

Hi admins,

I'm trying to test an RDMA-accelerated distributed storage system, and the c8220 nodes at Clemson seem like they could be used for this because they have QLogic 40GBps InfiniBand NICs (according to https://docs.cloudlab.us/hardware.html#(part._cloudlab-clemson)), but whenever I provision an experiment using those nodes, their QLogic NICs are not available for use. Specifically, the ibv_devinfo command always shows one RDMA device, qib0, with the state PORT_DOWN, and the fi_info command (from LibFabrics) cannot find any devices with the "Verbs" capability.

I see that these nodes have two NICs, one Ethernet and one RDMA, so is there something I need to do when provisioning my experiment to request that the LAN between these nodes uses the RDMA network? Or are the QLogic NICs physically disconnected right now, and I need to use a different node type if I want to use RDMA?

For reference, here's a link to my current experiment: https://www.cloudlab.us/status.php?uuid=a270155e-f16f-11f0-bc80-e4434b2381fc

Thanks,

Edward

[ajma...@gmail.com]

Hi Edward,

The Clemson Infiniband fabric was decommissioned last year (see https://groups.google.com/g/cloudlab-users/c/K4YR557a98s/m/1yCBqYfcBAAJ), so the QLogic interfaces are no longer functional.  If you want specifically Infiniband, then the only remaining option is to use the Apt cluster, but if you are fine using RoCE, then there are plenty of newer hardware types that should be able to support your work.

Best,

 - Aleks

[Edward Tremel]

Hi Aleks,

Thanks for your prompt reply. I didn't know about the InfiniBand fabric being decommissioned, since I rarely check this Google Group; you might want to update the Cloudlab Manual page so that it no longer says the Cloudlab Clemson site has an InfiniBand network. Fortunately, RoCE works fine for me, so I can just use a different node type. (I was just hoping to use the c8220 nodes since they are available more often than the newer ones).

Best regards,

Edward

[ajma...@gmail.com]

Yeah, I brought that up as soon as I saw your original email.  We'll work on removing that from the manual to avoid any ambiguity going forward.