Azure Mellanox issue

[Vijay Ram]

Hi Team,

I have installed OFED "mlnx-en-5.2-1.0.4.0-rhel8.3-x86_64.iso in Cent OS (8.3 kernel version) successfully with no errors. ConnectX-4 NIC installed. 

but when I try to run ASTF

./t-rex-64 -i --astf --cfg /etc/trex_cfg.yaml --astf-server-only I get the below error

./t-rex-64 -i --astf --cfg /etc/trex_cfg.yaml --astf-server-only -c 0x1
Warning: Mellanox NICs were tested only with RedHat/CentOS 7.9
Correct usage with other Linux distributions is not guaranteed.
The ports are bound/configured.
Starting  TRex v2.88 please wait  ...
EAL: /lib64/libmlx5.so.1: version `MLX5_1.15' not found (required by so/x86_64/libmlx5-64.so)
EAL: FATAL: Cannot init plugins
EAL: Cannot init plugins
 You might need to run ./trex-cfg  once
EAL: Error - exiting with code: 1
  Cause: Invalid EAL arguments

How to fix the issue?

thx,

Vijayram

[hanoh haim]

Hi Vijay, 

try to work with latest trex package (v2.89) and OFED 5.3.1 (see Mellanox appendix)

Thanks

Hanoh

[Vijay Ram]

Hi Hanoh,

Thanks much for your response.

I followed the below link

https://trex-tgn.cisco.com/trex/doc/trex_appendix_mellanox.html

In Azure, I spinned up CentOS 7.9 VM (kernel version - 3.10.0-1160.6.1.e17.x86_64) with OFED 5.2-1.0.4.0. 

Installation was successful and also ./trex-cfg could bind the ports 

[root@trafficgencentos7 v2.89]# ./dpdk_setup_ports.py -t
+----+------+--------------+-------------------+-------------------------------------------------+-----------+----------+--------+
| ID | NUMA |     PCI      |        MAC        |                      Name                       |  Driver   | Linux IF | Active |
+====+======+==============+===================+=================================================+===========+==========+========+
| 0  | 0    | dded:00:02.0 | 00:0d:3a:b1:be:8f | MT27710 Family [ConnectX-4 Lx Virtual Function] | mlx5_core | eth2     |        |
+----+------+--------------+-------------------+-------------------------------------------------+-----------+----------+--------+
[root@trafficgencentos7 v2.89]#

I could see eth0 and eth2 having same MAC

[root@trafficgencentos7 v2.89]# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.11.29  netmask 255.255.255.240  broadcast 192.168.11.31
        inet6 fe80::20d:3aff:feb1:be8f  prefixlen 64  scopeid 0x20<link>
        ether 00:0d:3a:b1:be:8f  txqueuelen 1000  (Ethernet)
        RX packets 394160  bytes 178414091 (170.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 432248  bytes 104356473 (99.5 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth2: flags=6211<UP,BROADCAST,RUNNING,SLAVE,MULTICAST>  mtu 9238
        ether 00:0d:3a:b1:be:8f  txqueuelen 1000  (Ethernet)
        RX packets 3524  bytes 3757291 (3.5 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 452531  bytes 105698311 (100.8 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

I have configured below in trex_cfg.yaml with my PCI address

#List of interfaces. Change to suit your setup. Use ./dpdk_setup_ports.py -s to see available options
  interfaces    : ["dummy","dded:00:02.0"]
  low_end: true
  port_info       :  # Port IPs. Change to suit your needs. In case of loopback, you can leave as is.
          - ip         : 1.1.1.1
            default_gw : 2.2.2.2
          - ip         : 192.168.11.29
            default_gw : 192.168.11.17

But when I start to ./t-rex-64 -i --astf --cfg /etc/trex_cfg.yaml 

My dest (GW) MAC is NOT resolved but whereas if I change my trex_cfg.yaml to contain as below

  interfaces    : ["dummy","eth0"]
  low_end: true
  port_info       :  # Port IPs. Change to suit your needs. In case of loopback, you can leave as is.
          - ip         : 1.1.1.1
            default_gw : 2.2.2.2
          - ip         : 192.168.11.29
            default_gw : 192.168.11.17

Then the ./t-rex-64 is able to resolve GW MAC.

I have same GW and SRC IP in both cfg files but with DPDK port I couldn't resolve MAC.

Any further config mapping required here?

thx,

Vijayram

--

with regards
Vijayram.P

[hanoh haim]

Hi Vijay, 

If this is for Azure, it is better to follow this wiki as it is not just mlx4/5 driver. 

https://github.com/cisco-system-traffic-generator/trex-core/wiki/Azure-with-netvsc-DPDK-driver-(v2.89)

If it is just a mlx5 PF, follow the appendix 

Thanks

Hanoh

[Vijay Ram]

Thanks Hanoh.

I followed the  below link exactly and successfully built v2.89 TRex in Azure  Ubuntu VM 18.04

https://github.com/cisco-system-traffic-generator/trex-core/wiki/Azure-with-netvsc-DPDK-driver-(v2.89)

Below is my port detail

lspci output below

I'm using the setup as client and server. This Azure VM acts as TRex server and so I added "dumm" in client

I get the error while adding "dummy". Hope we can use TRex in server only mode in Azure setup.

How do I fix this issue?

thx,

Vijayram

--

with regards
Vijayram.P

[Vijay Ram]

Hi Hanoh,

Could you please share your feedback to fix ASTF in client/server execution mode in Azure?

I get error for adding "dummy" port as required for client/server mode.

thx,

Vijayram

--

with regards
Vijayram.P

[hanoh haim]

You should remove the "dummy" from the ext_dpdk_opt and interfaces_vdevs

--

Hanoh

Sent from my iPhone

[Vijay Ram]

Thanks much Hanoh.

As per your suggestion I ignored "dummy2 in the code and executed the server script but I get error again

I added a printf in  "device_probe_init" function but it was never hit

Seems like CTVport was NOT initialized with port 0 "dummy". 

In my VM I have DPDK port with driver "mlx5" and eth1 being "mgmt port" with hyperV driver

But in /etc/trex_cfg.yaml I have mentioned as only "dummy" and "dpdk pci" and nowhere mentioned eth1.

So not sure why "dummy" is NOT initialized or fetched from cfg file. 

Thx,

Vijayram

On Tue, Apr 20, 2021 at 10:48 AM hanoh haim <hhaim...@gmail.com> wrote:

COLD_FUNC void reorder_dpdk_ports() {

CTRexPortMapper * lp=CTRexPortMapper::Ins();

CPlatformYamlInfo *cg=&global_platform_cfg_info;

if ( cg->m_if_list_vdevs.size() > 0 ) {

if ( isVerbose(0) ){

printf(" size of interfaces_vdevs %d",(int)cg->m_if_list_vdevs.size());

}

int if_index = 0;

for (std::string &opts : cg->m_if_list_vdevs) {

uint16_t port_id;

int ret = rte_eth_dev_get_port_by_name((const char *)opts.c_str(), &port_id);

if (ret) {

printf("Failed to find %s in DPDK vdev ", opts.c_str());

dump_dpdk_devices();

exit(1);

}

if ( isVerbose(0) ){

printf(" ===>>>found %s %d \n",opts.c_str(),port_id);

}

lp->set_map(if_index,port_id);

if_index++;

}

return;

}

I see the issue, "dummy" is not supported in case of vdev falisafe jump, try to skip in case of "dummy" in this loop

Thanks

Hanoh

On Tue, 20 Apr 2021 at 12:00, Vijay Ram <vijayr...@gmail.com> wrote:

Ok, thanks Hanoh.

As per your suggestion I removed "dummy" but I still get the below error

Port 1 is the DPDK port. error states that I should stop any service before I start the test?

thx,

Vijayram

--

with regards
Vijayram.P

[hanoh haim]

Hi, 

Can you check a setup with 2 ports as described in the wiki ? maybe you have two issues. "dummy" with "interface_vdev" is not supported right now.

[Vijay Ram]

Thanks much Hanoh.

The diff you shared fixed the issue and I could able to run the t-rex-64 server script. 

I have  an asymmetry issue in Azure VM which I'm working on and post that I will update end to end of this test.

thx,

Vijayram

On Tue, Apr 20, 2021 at 1:33 PM hanoh haim <hhaim...@gmail.com> wrote:

diff --git a/src/main_dpdk.cpp b/src/main_dpdk.cpp
index a7779b822..643eb61c0 100644
--- a/src/main_dpdk.cpp
+++ b/src/main_dpdk.cpp
@@ -6754,14 +6754,22 @@ COLD_FUNC void reorder_dpdk_ports() {

 
     if ( cg->m_if_list_vdevs.size()  > 0 ) {
         if ( isVerbose(0) ){

-           printf(" size of interfaces_vdevs %d",(int)cg->m_if_list_vdevs.size());
+           printf(" size of interfaces_vdevs %d \n",(int)cg->m_if_list_vdevs.size());

         }
         int if_index = 0;
         for (std::string &opts : cg->m_if_list_vdevs) {

+            if ( CTVPort(if_index).is_dummy() ) {
+                if ( isVerbose(0) ){
+                    printf(" dummy interface skipped \n");
+                }
+                if_index++;
+                continue;
+            }
+

             uint16_t port_id;
             int ret = rte_eth_dev_get_port_by_name((const char *)opts.c_str(), &port_id);
                if (ret) {

-                 printf("Failed to find  %s in DPDK vdev ", opts.c_str());
+                 printf("Failed to find  %s in DPDK vdev \n", opts.c_str());
                  dump_dpdk_devices();
                  exit(1);
                }
@@ -6772,18 +6780,19 @@ COLD_FUNC void reorder_dpdk_ports() {
             if_index++;
         }
         return;
-    }
+    }else{
   
-    if ( CGlobalInfo::m_options.m_is_vdev ) {
-        uint8_t if_index = 0;
-        for (int i=0; i<global_platform_cfg_info.m_if_list.size(); i++) {
-            if ( CTVPort(i).is_dummy() ) {
-                continue;
+        if ( CGlobalInfo::m_options.m_is_vdev ) {
+            uint8_t if_index = 0;
+            for (int i=0; i<global_platform_cfg_info.m_if_list.size(); i++) {
+                if ( CTVPort(i).is_dummy() ) {
+                    continue;
+                }
+                lp->set_map(i, if_index);
+                if_index++;
             }
-            lp->set_map(i, if_index);
-            if_index++;
+            return;
         }
-        return;
     }
 
     #define BUF_MAX 200

try this diff

--

with regards
Vijayram.P

[hanoh haim]

Thanks for the update, I will push them to master 

--