No SR-IOV capability on ConnectX6DX

[David Lin]

Hi all,

I am using Clemson r6525 nodes 304 and 298 to configure SR-IOV. On node 304, everything works as expected, but node 298 appears to have an issue.

Below is the relevant information. Does anyone know whether this might be related to a BIOS configuration or some other system setting?

The NIC configuration shows that SR-IOV is enabled:

sudo mlxconfig -d /dev/mst/mt4125_pciconf0 q SRIOV_EN NUM_OF_VFS

Device type:  ConnectX6DX
Description:  Mellanox ConnectX-6 Dx Dual Port 100GbE QSFP56 Network Adapter

Configurations:
   SRIOV_EN    True(1)
   NUM_OF_VFS  4

However, when checking the PCI devices:

for dev in 63:00.0 63:00.1 81:00.0 81:00.1; do
 echo "==== $dev ===="
 sudo lspci -s $dev -vv | grep -A15 -i "Single Root I/O Virtualization" || echo "no SR-IOV capability"
done

The output shows that 63:00.x supports SR-IOV, but 81:00.x does not report SR-IOV capability:

==== 63:00.0 ====
Capabilities: Single Root I/O Virtualization (SR-IOV)
...

==== 63:00.1 ====
Capabilities: Single Root I/O Virtualization (SR-IOV)
...

==== 81:00.0 ====
no SR-IOV capability

==== 81:00.1 ====
no SR-IOV capability

This behavior is unexpected because both ports belong to the same ConnectX-6 Dx adapter.

Has anyone encountered a similar issue before? Could this be caused by a BIOS configuration, firmware setting, or PCIe configuration difference between the nodes?

Thanks!