ROCE support on Mellanox switch

[paridhika kayal]

Hi,

I wanted to enable the switch for ROCE. But the switch I am.  using does not provide the option to enable it.

I was trying to follow the following manual: https://docs.nvidia.com/networking/display/ONYX382204/RoCE+Commands 

Can you please tell me which do you have any other switch which I can use to enable RDMA traffic?

Here is the link to my experiment: https://www.cloudlab.us/status.php?uuid=518a7cb6-c025-11ec-b318-e4434b2381fc

Thanks,

Paridhika

[Leigh Stoller]

> Can you please tell me which do you have any other switch which I can use to enable RDMA traffic?

Hi. There are 5 dell-s4048 user allocatable switches currently free.
I do not know if they do what you need but Google will.

Leigh

[paridhika kayal]

Hi Leigh,

I tried to use those switches but I couldn't enter their shell. Can you please tell me if they are configurable?

Thanks,

[Mike Hibler]

We do not have ONYX for those switches, they run MLNX-OS (or other open
network OSes). So those commands won't work. Here is one page related to
MLNX-OS:

https://support.mellanox.com/s/article/howto-run-roce-over-l2-enabled-with-pfc

> --
> You received this message because you are subscribed to the Google Groups
> "cloudlab-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to cloudlab-user...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/
> cloudlab-users/2371f522-b12c-4480-a282-a0984998b0f5n%40googlegroups.com.

[Leigh Stoller]

> I tried to use those switches but I couldn't enter their shell. Can you please tell me if they are configurable?

Hi. So you had an experiment running and could not log into the switch?
I don’t remember if you told us (sent us a link to a running experiment
with this problem).

They are fully configurable by the user.

Leigh

[paridhika kayal]

Hi Mike,

Thank you for the reference, it helps. 

Can you also tell me the nodes that I can use to run RDMA traffic? I am currently using the xl170 type nodes but it does not have a RoCE device.

It gives me the following error. 

node1:~> ib_send_bw
 Did not detect devices
 If device exists, check if driver is up
 Unable to find the Infiniband/RoCE device

Thanks,

Paridhika

[paridhika kayal]

Hi Leigh,

Currently, I am using mlnx-sn2410 switch and I can configure it. I tried dell-s4048 earlier which did not have an option to open the shell.

[paridhika kayal]

Hi,

Thank you for the help. 

I am able to configure  mlnx-sn2410 switch for RDMA traffic. But I am not able to install  Mellanox OFED on xl170 type nodes through the shell.

Mike suggested in another post to log in through the console to install the driver. But at the console, it asks me for login and password. 

Can you please help me with getting the login and password? I tried to use my cloudlab password but it is not working.

[Mike Hibler]

Sorry, I didn't reply earlier. If you choose "Console" from the dropdown
for any node, it will take you to a console tab. At the top of the pane
is "Password". Click on that and login as root using that password.

> --
> You received this message because you are subscribed to the Google Groups
> "cloudlab-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to cloudlab-user...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/

> cloudlab-users/3e56a08b-a007-4ab7-9266-ce1e799e548cn%40googlegroups.com.

[paridhika kayal]

Hi Mike,

That works. Thanks a lot for your help. 

Best,

Paridhika