Physical connection between connectx cards

[Nurlan Nazaraliyev]

Hi,

I have 2 r7525 nodes with Connectx-5 and Connectx-6 cards. I think they are not physically connected to each other. Is it possible to make such a connection?

Best,

Nurlan

[Leigh Stoller]

> I have 2 r7525 nodes with Connectx-5 and Connectx-6 cards. I think they are not physically connected to each other. Is it possible to make such a connection?

See the Hardware chapter of the Cloudlab manual for a description of the
r7525 nodes. Click on the Docs link in the upper right of the web UI.

There is a single Connectx-5 card that is wired to the switch fabric.
There two Connectx-6 cards are Bluefield-2 and also wired the fabric.
Your profile will need to request the topology you want. For example:

"""Bluefield2 - This topology uses two Bluefield2 enabled hosts @ Clemson. They are connected back to back with 3 links; 2 of them forced to be 40G to enforce SmartNIC connectivity. The other link is only required as a common 1G interfaces have the machines connected normally, too"""

# Import the Portal object.
import geni.portal as portal
# Import the ProtoGENI library.
import geni.rspec.pg as pg
# Import the Emulab specific extensions.
import geni.rspec.emulab as emulab
# Create a portal object,
pc = portal.Context()
# Create a Request object to start building the RSpec.
request = pc.makeRequestRSpec()

# Node bf1
node_bf1 = request.RawPC('bf1')
node_bf1.hardware_type = 'r7525'
node_bf1.disk_image = 'urn:publicid:IDN+emulab.net+image+emulab-ops//UBUNTU20-64-STD'
iface0 = node_bf1.addInterface('interface-0')
iface1 = node_bf1.addInterface('interface-2')
iface2 = node_bf1.addInterface('interface-4')

# Node bf2
node_bf2 = request.RawPC('bf2')
node_bf2.hardware_type = 'r7525'
node_bf2.disk_image = 'urn:publicid:IDN+emulab.net+image+emulab-ops//UBUNTU20-64-STD'
iface4 = node_bf2.addInterface('interface-1')
iface5 = node_bf2.addInterface('interface-3')
iface6 = node_bf2.addInterface('interface-5')

# Link link-0
link_0 = request.Link('link-0')
link_0.Site('undefined')
iface4.bandwidth = 40000000
link_0.addInterface(iface4)
iface0.bandwidth = 40000000
link_0.addInterface(iface0)

# Link link-1
link_1 = request.Link('link-1')
link_1.Site('undefined')
iface1.bandwidth = 40000000
link_1.addInterface(iface1)
iface5.bandwidth = 40000000
link_1.addInterface(iface5)

# Link link-2
link_2 = request.Link('link-2')
link_2.Site('undefined')
iface2.bandwidth = 1000000
link_2.addInterface(iface2)
iface6.bandwidth = 1000000
link_2.addInterface(iface6)

# Print the generated rspec
pc.printRequestRSpec(request)

[Nurlan Nazaraliyev]

Is there any way to assign them static IPs? I cannot still see the interfaces corresponding to Connectx cards under ifconfig. 

Best,

Nurlan

[Leigh Stoller]

> Is there any way to assign them static IPs? I cannot still see the interfaces corresponding to Connectx cards under ifconfig.

Hi. Have you started a new experiment using the profile I mentioned?
Without the proper profile running, there is no way to use those interfaces,
the switch ports are held in a shutdown state while not “in use”.

Leigh

[Nurlan Nazaraliyev]

Hi,

I just run the profile as a python file on existing experiment. 

How do I create a new experiment with this profile? I could not find an option to select the necessary profile or copy&paste the given profile when I create new experiment.

Best,

Nurlan

[Leigh Stoller]

> I just run the profile as a python file on existing experiment.
> How do I create a new experiment with this profile? I could not find an option to select the necessary profile or copy&paste the given profile when I create new experiment.

Hi. Click on the Docs link and take a look at the Cloudlab manual.
You need to create a new profile, and then instantiate that profile
(a new experiment). This is all described with examples in the manual.

Thanks
Leigh

[Nurlan Nazaraliyev]

I was able to create new experiment with the given profile. Thanks a lot for the help.

I installed mlnx ofed driver and ifconfig showed all the networks available. Then I build&installed mlnx-ofed-kernel part manually using dpkg. After this, the interfaces corresponding to Connectx cards disappeared. 

Why is this the case? Are we not allowed to install custom drivers?

Note: the custom mlnx-ofed-kernel is working with no problem,

Best,

Nurlan

[David M Johnson]

On 5/29/24 17:18, 'Nurlan Nazaraliyev' via cloudlab-users wrote:
> I was able to create new experiment with the given profile. Thanks a lot
> for the help.
> I installed mlnx ofed driver and ifconfig showed all the networks
> available. Then I build&installed mlnx-ofed-kernel part manually using
> dpkg. After this, the interfaces corresponding to Connectx cards
> disappeared.
> Why is this the case? Are we not allowed to install custom drivers?

To clarify:

* Are you trying to use the connectx-6 (Bluefield) NICs?
* You first installed an updated ofed kernel driver via apt-get,
rebooted, and saw the connectx-6 NICs?
* Then you rebuilt the binary .deb packages from dpkg source, and
installed the custom-built .debs?

Many things could have gone wrong in the last step. Did you verify that
newer kernel modules were installed, e.g. via `modinfo`? If so, you
will want to diff the boot logs from a successful boot (e.g. with the
binary driver package from mellanox, vs the custom-built package boot),
so you can see why your custom drivers did not boot. I'm not familiar
with the OFED source dpkg, so it's hard for me to say more.

> Note: the custom mlnx-ofed-kernel is working with no problem,

This confuses me -- you're saying that the kernel boots with this custom
package, but that you don't see interfaces when booted with the custom
kernel?

> Best,
> Nurlan

David

> On Wednesday, May 29, 2024 at 11:48:55 AM UTC-7 Leigh Stoller wrote:
>
>
> > I just run the profile as a python file on existing experiment.
> > How do I create a new experiment with this profile? I could not
> find an option to select the necessary profile or copy&paste the
> given profile when I create new experiment.
>
> Hi. Click on the Docs link and take a look at the Cloudlab manual.
> You need to create a new profile, and then instantiate that profile
> (a new experiment). This is all described with examples in the manual.
>
> Thanks
> Leigh
>

> --
> You received this message because you are subscribed to the Google
> Groups "cloudlab-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to cloudlab-user...@googlegroups.com
> <mailto:cloudlab-user...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/cloudlab-users/0d845c8f-0a1d-47c5-af25-42a3b2a7094cn%40googlegroups.com <https://groups.google.com/d/msgid/cloudlab-users/0d845c8f-0a1d-47c5-af25-42a3b2a7094cn%40googlegroups.com?utm_medium=email&utm_source=footer>.