Unable to use TRex in an Ubuntu VM using Broadcom NetXtreme NICs

[Ravi Kumar Reddy]

Hi All,

I am unable to use TRex  in an Ubuntu VM using Broadcom NetXtreme NICs. Please find the logs below:

root@rreddyk-ubuntu-trex:/opt/trex/v2.99# ethtool -i ens4
driver: bnxt_en
version: 5.15.0-1013-oracle
firmware-version: 20.8.187.0/pkg 20.8.29.0
expansion-rom-version:
bus-info: 0000:00:03.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no
root@rreddyk-ubuntu-trex:/opt/trex/v2.99#

root@rreddyk-ubuntu-trex:/opt/trex/v2.99# ./t-rex-64 -f ./cap2/cur_flow_single.yaml -m 150 -c 1 -d 20
The ports are bound/configured.
Starting  TRex v2.99 please wait  ...
bnxt_setup_mac_addr(): Failed to allocate multicast addr table
ethdev initialisation failed
EAL: Requested device 0000:00:04.0 cannot be used
bnxt_setup_mac_addr(): Failed to allocate multicast addr table
ethdev initialisation failed
EAL: Requested device 0000:00:05.0 cannot be used
EAL: Bus (pci) probe failed.
ERROR in DPDK map
Could not find requested interface 0000:00:04.0
root@rreddyk-ubuntu-trex:/opt/trex/v2.99#
root@rreddyk-ubuntu-trex:/opt/trex/v2.99# python3 dpdk_setup_ports.py -s

Network devices using DPDK-compatible driver
============================================
0000:00:04.0 'NetXtreme-E Ethernet Virtual Function' drv=igb_uio unused=bnxt_en,vfio-pci,uio_pci_generic
0000:00:05.0 'NetXtreme-E Ethernet Virtual Function' drv=igb_uio unused=bnxt_en,vfio-pci,uio_pci_generic

Network devices using kernel driver
===================================
0000:00:03.0 'NetXtreme-E Ethernet Virtual Function' if=ens3 drv=bnxt_en unused=igb_uio,vfio-pci,uio_pci_generic *Active*

Other network devices
=====================
<none>
root@rreddyk-ubuntu-trex:/opt/trex/v2.99#

It looks like a malloc call is failing in DPDK. Can you please let me know a way to resolve this issue?

Thanks,

Ravi Kumar Reddy K

[Цветков Андрей]

Hi.

What chip is using in you card ?

понедельник, 8 августа 2022 г. в 09:42:54 UTC+3, Ravi Kumar Reddy:

[Ravi Kumar Reddy]

Hi,

Please find the detail below:

ubuntu@rreddyk-ubuntu-trex:~$ sudo lshw -c network
  *-network:0              
       description: Ethernet interface
       product: NetXtreme-E Ethernet Virtual Function
       vendor: Broadcom Inc. and subsidiaries
       physical id: 3
       bus info: pci@0000:00:03.0
       logical name: ens3
       logical name: /dev/fb0
       version: 00
       serial: 02:00:17:0a:a0:7b
       capacity: 25Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: msix pciexpress bus_master cap_list ethernet physical fibre 1000bt-fd 25000bt-fd autonegotiation fb
       configuration: autonegotiation=on broadcast=yes depth=32 driver=bnxt_en driverversion=5.15.0-1013-oracle duplex=full firmware=20.8.187.0/pkg 20.8.29.0 ip=10.100.28.117 latency=0 link=yes mode=1024x768 multicast=yes visual=truecolor xres=1024 yres=768
       resources: iomemory:200-1ff iomemory:200-1ff iomemory:200-1ff irq:0 memory:2000400000-2000403fff memory:2000000000-20000fffff memory:2000404000-2000407fff
  *-network:1
       description: Ethernet controller
       product: NetXtreme-E Ethernet Virtual Function
       vendor: Broadcom Inc. and subsidiaries
       physical id: 4
       bus info: pci@0000:00:04.0
       version: 00
       width: 64 bits
       clock: 33MHz
       capabilities: msix pciexpress cap_list
       configuration: driver=igb_uio latency=0
       resources: iomemory:200-1ff iomemory:200-1ff iomemory:200-1ff irq:0 memory:2000408000-200040bfff memory:2000100000-20001fffff memory:200040c000-200040ffff
  *-network:2
       description: Ethernet controller
       product: NetXtreme-E Ethernet Virtual Function
       vendor: Broadcom Inc. and subsidiaries
       physical id: 5
       bus info: pci@0000:00:05.0
       version: 00
       width: 64 bits
       clock: 33MHz
       capabilities: msix pciexpress cap_list
       configuration: driver=igb_uio latency=0
       resources: iomemory:200-1ff iomemory:200-1ff iomemory:200-1ff irq:0 memory:2000410000-2000413fff memory:2000200000-20002fffff memory:2000414000-2000417fff
ubuntu@rreddyk-ubuntu-trex:~$

ubuntu@rreddyk-ubuntu-trex:~$ lspci -vvv | grep -i net
00:03.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme-E Ethernet Virtual Function
    Subsystem: Oracle/SUN NetXtreme-E Ethernet Virtual Function
00:04.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme-E Ethernet Virtual Function
    Subsystem: Oracle/SUN NetXtreme-E Ethernet Virtual Function
00:05.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme-E Ethernet Virtual Function
    Subsystem: Oracle/SUN NetXtreme-E Ethernet Virtual Function
ubuntu@rreddyk-ubuntu-trex:~$

Thanks,

Ravi

[Цветков Андрей]

Ethernet Virtual - Hmmm...

https://doc.dpdk.org/guides/nics/bnxt.html
Try   vfio-pci driver.

понедельник, 8 августа 2022 г. в 11:49:48 UTC+3, raviku...@gmail.com:

[Ravi Kumar Reddy]

Trex worked with devices bound to igb_uio in Jan 2021 Timeframe. I am not sure what changed. I am using v2.99 only.

Further I tried binding them to vfio-pci. Following is the issue I face:

root@rreddyk-ubuntu-trex:/home/ubuntu# lsmod | grep -i vfio
root@rreddyk-ubuntu-trex:/home/ubuntu# modprobe vfio-pci
root@rreddyk-ubuntu-trex:/home/ubuntu# lsmod | grep -i vfio
root@rreddyk-ubuntu-trex:/home/ubuntu# echo -n "0000:00:04.0" > /sys/bus/pci/drivers/vfio-pci/bind
bash: echo: write error: No such device
root@rreddyk-ubuntu-trex:/home/ubuntu#

Thanks,

Ravi

[Ravi Kumar Reddy]

Hi Bes and Shlomo,

Thank you for your time and quick support on this issue. As discussed, I will use an TRex v2.81 or older to resolve this issue and move forward. I will raise a query with DPDK on the same. 

Thanks,

Ravi