[Mellanox ConnectX-6] Network port Physical state: Disabled on Wisconsin node[d8545]

[cheng wendy]

I am currently working on a CloudLab node (Wisconsin node[d8545]) equipped with a Mellanox ConnectX-6 NIC. However, the InfiniBand/Ethernet interface cannot come up — the port remains in Physical state: Disabled. Below are the details of my environment and the troubleshooting steps I have tried:

1. Hardware / Environment

• Node type: d8545 (AMD EPYC Rome, NVIDIA A100 GPUs)

• NIC: Mellanox Technologies MT28908 Family [ConnectX-6] (PCI address: a1:00.0)

• OS: Ubuntu 22.04 (Kernel 5.15)

• MLNX_OFED: 5.8-7.0.6.1

2. Observations

• ibstat shows:

  Port 1: State: Down Physical state: Disabled Link layer: Ethernet

• ethtool enp161s0np0 shows:

  Speed: Unknown! Link detected: no

• mst status -v detects the device correctly:

  ConnectX6(rev:0) /dev/mst/mt4123_pciconf0 a1:00.0 mlx5_0 net-enp161s0np0

• Kernel modules (mlx5_core, mlx5_ib) are loaded.

• RDMA device (mlx5_0) is listed by ibv_devices.

3. Questions

1. Is the InfiniBand/Ethernet fabric connected in the CloudLab d8545 nodes?

2. Should I expect the ConnectX-6 physical ports to be usable (e.g., for RoCE or IB experiments), or are they administratively disabled at the testbed level?

3. If the ports are expected to be functional, could you please advise on the correct configuration steps (firmware, OFED, link bring-up, etc.)?

Thank you for your support!

[cheng wendy]

https://www.cloudlab.us/status.php?uuid=af242e28-7f1e-11f0-bc80-e4434b2381fc

[Mike Hibler]

You only have one node in your experiment so there are no active Ethernet
interfaces. You need to configure at least two nodes and a link between
them in order for the interface to be activated (i.e., the connected switch
port enabled).

There is no IB, it is connected to an Ethernet switch. ROCE should work
just fine.

There is no global experiment fabric. You will only be able to communicate
with nodes in your own experiment. You can communicate with all nodes (and
non-Cloudlab sites) via the public facing control network, but you should
not use that for experimentation.

On Fri, Aug 22, 2025 at 12:54:40AM -0700, cheng wendy wrote:
> https://www.cloudlab.us/status.php?uuid=af242e28-7f1e-11f0-bc80-e4434b2381fc
>
> 在2025年8月22日星期五 UTC+8 15:48:44<cheng wendy> 写道：
>
>
> I am currently working on a CloudLab node (Wisconsin node[d8545]) equipped
> with a Mellanox ConnectX-6 NIC. However, the InfiniBand/Ethernet interface
> cannot come up — the port remains in Physical state: Disabled. Below are
> the details of my environment and the troubleshooting steps I have tried:
>
> 1. Hardware / Environment
> □ Node type: d8545 (AMD EPYC Rome, NVIDIA A100 GPUs)
>
> □ NIC: Mellanox Technologies MT28908 Family [ConnectX-6] (PCI address:
> a1:00.0)
>
> □ OS: Ubuntu 22.04 (Kernel 5.15)
>
> □ MLNX_OFED: 5.8-7.0.6.1
>
> 2. Observations
> □ ibstat shows:
>
> Port 1: State: Down Physical state: Disabled Link layer: Ethernet
> □ ethtool enp161s0np0 shows:
>
> Speed: Unknown! Link detected: no
> □ mst status -v detects the device correctly:
>
> ConnectX6(rev:0) /dev/mst/mt4123_pciconf0 a1:00.0 mlx5_0
> net-enp161s0np0
> □ Kernel modules (mlx5_core, mlx5_ib) are loaded.
>
> □ RDMA device (mlx5_0) is listed by ibv_devices.
>
> 3. Questions
>

> 1. Is the InfiniBand/Ethernet fabric connected in the CloudLab d8545
> nodes?
>
> 2. Should I expect the ConnectX-6 physical ports to be usable (e.g., for

> RoCE or IB experiments), or are they administratively disabled at the
> testbed level?
>

> 3. If the ports are expected to be functional, could you please advise on

> the correct configuration steps (firmware, OFED, link bring-up, etc.)?
>
> Thank you for your support!
>

> --
> You received this message because you are subscribed to the Google Groups
> "cloudlab-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to cloudlab-user...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/cloudlab-users/
> 09279f61-2bd2-4d1d-904d-c082500b283bn%40googlegroups.com.