TRex v2.88 and v2.98 are unable to start with CentOS 7.9 and ConnectX-5 NICs

[btjo...@gmail.com]

Hello,

I'm trying to bring up TRex on a new server and I'm seeing the same error and am unable to start an 8-port TRex instance on CentOS 7.9 with TRex v2.88 as well as v2.98.

EAL: /lib64/libmlx5.so.1: version `MLX5_OFED' not found (required by so/x86_64/libmlx5-64.so)

TRex is being started via systemd which has worked fine for other TRex instances before.

[root@htrex-9112 htrex-9112]# systemctl status trex_bm1
● trex_bm1.service - Physical TRex Server Instance 1 Service
   Loaded: loaded (/usr/lib/systemd/system/trex_bm1.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2022-06-11 17:33:08 PDT; 29s ago
  Process: 3362 ExecStartPre=/bin/sleep 30 (code=exited, status=0/SUCCESS)
 Main PID: 3375 (t-rex-64)
    Tasks: 6
   CGroup: /system.slice/trex_bm1.service
           ├─3375 /bin/bash /corp/sta/opt/TRex/servers/htrex-9112/core/t-rex-64 -i --cfg /corp/sta/opt/TRex/servers/htrex-9112/trex_cfg_bm1.yaml --mb...
           ├─3468 /usr/bin/python3 -m trex.scapy_server.scapy_zmq_server -p 4507
           ├─3556 python3 ./dpdk_setup_ports.py --cleanup-servers --parent=-i --cfg /corp/sta/opt/TRex/servers/htrex-9112/trex_cfg_bm1.yaml --mbuf-fa...
           └─3559 /usr/bin/python3 general_daemon_server stop -n Scapy

Jun 11 17:32:38 htrex-9112 systemd[1]: Starting Physical TRex Server Instance 1 Service...
Jun 11 17:33:08 htrex-9112 systemd[1]: Started Physical TRex Server Instance 1 Service.
Jun 11 17:33:37 htrex-9112 t-rex-64[3375]: EAL: /lib64/libmlx5.so.1: version `MLX5_OFED' not found (required by so/x86_64/libmlx5-64.so)
Jun 11 17:33:37 htrex-9112 t-rex-64[3375]: EAL: FATAL: Cannot init plugins
Jun 11 17:33:37 htrex-9112 t-rex-64[3375]: EAL: Cannot init plugins
Jun 11 17:33:37 htrex-9112 t-rex-64[3375]: EAL: Error - exiting with code: 1
Jun 11 17:33:37 htrex-9112 t-rex-64[3375]: Cause: Invalid EAL arguments

 cat /etc/redhat-release
CentOS Linux release 7.9.2009 (Core)

uname -a
Linux htrex-9112 3.10.0-1160.el7.x86_64 #1 SMP Mon Oct 19 16:18:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

I installed the MLNX_OFED 5.6-2.0.9.0 drivers specified by the NVIDIA matrix on the download tab of this page:

https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/

This is the file I pulled down:

http://www.mellanox.com/page/mlnx_ofed_eula?mtag=linux_sw_drivers&mrequest=downloads&mtype=ofed&mver=MLNX_OFED-5.6-2.0.9.0&mname=MLNX_OFED_LINUX-5.6-2.0.9.0-rhel7.9-x86_64.tgz

I installed per https://trex-tgn.cisco.com/trex/doc/trex_appendix_mellanox.html

I do see a note on that page with respect to upgrading that might provide a clue.

Table 2 indicates that for TRex version v2.89 and above, the CentOS 7.9 distro is needed and the OFED release must be only GA 5.3-1.

I'll try the GA 5.3-1 release of MLNX_OFED and see if that helps.

Thanks,

Bruce.

[btjo...@gmail.com]

After installing MLNX_OFED 5.3_LINUX-5.3-1.0.0.1-rhel7.9-x86_64, TRex is able to start successfully with both v2.88 and v2.98.

Hopefully this helps someone else get recent TRex versions working with the CentOS 7.9 distribution.

Bruce. 

[Nathan Hancock]

Just adding my experience here as well.  I also am using Centos 7.9, and had the same error message of 

" EAL: /lib64/libmlx5.so.1: version `MLX5_OFED' not found (required by so/x86_64/libmlx5-64.so)"

when trying to run trex for the first time after installing MLNX OFED drivers 5.6-2.0.9, which were the latest drivers from Nvidia/Mellanox at the time of this posting.  

Following the advice here to install the MLNX_OFED 5.3-1.0.0.1 drivers instead, I was able to get trex to start up without issue.  

Note that when installing my Mellanox OFED drivers, I encountered an error that I did not have the nesecary kernel support for installing the ofed driver.  So I had to install the driver using an extra flag on the install script.
/home/nate/Downloads/mlnx-en-5.3-1.0.0.1-rhel7.9-x86_64/install --with-mft --with-mstflint --dpdk --upstream-libs --add-kernel-support

Following these install instructions exactly is critical to make this work on Centos 7.9
https://trex-tgn.cisco.com/trex/doc/trex_appendix_mellanox.html#_ofed_installation