dpdk_setup_ports.py fails to create a config file depending on device config

[kri...@redhat.com]

Hi

When we run dpdk_setup_ports.py to create a new config file, it is failing in recent versions of TRex.  I believe this is ultimately caused by the fact that dpdk_nic_bind.py is calling t-rex-64 with the --dump-interfaces argument to determine information about the devices.  Depending on the current device configuration (ie. how many devices are bound to vfio-pci, their order, etc.) this only succeeds if the devices being requested are among the first 4 devices that TRex is finding.  If the device(s) are not one of the first 4 found it ends up not getting a TRex_Driver assigned to it so it does not match the other devices which causes the script to error.

I think this is caused by the default m_max_ports setting in main_dpdk.cpp which defaults to 4 (and is used to control the loop in dump_interfaces_info()).  Near as I can tell, the only way to change the m_max_ports value is via a config file.  The fact that this only exists in a config file which is what we are trying to create gets messy -- I have not been able to figure out how to give this information to dpdk_setup_ports.py where it will still generate a new config file.

Am I missing something?  Is there a way around this?  Or is this just a bug?

Here are some details.  On my system, I can simulate the problem by calling t-rex-64 --dump-interfaces the way dpdk_setup_ports.py does:

# show the vfio-pci bound devices...

$ pwd

/sys/module/vfio_pci/drivers/pci:vfio-pci

$ ls -l 00* | sed -e "s/^.*:[0-9]\+\s//"

0000:3b:00.0 -> ../../../../devices/pci0000:3a/0000:3a:00.0/0000:3b:00.0

0000:3b:00.1 -> ../../../../devices/pci0000:3a/0000:3a:00.0/0000:3b:00.1

0000:5e:00.0 -> ../../../../devices/pci0000:5d/0000:5d:00.0/0000:5e:00.0

0000:5e:00.1 -> ../../../../devices/pci0000:5d/0000:5d:00.0/0000:5e:00.1

0000:af:00.0 -> ../../../../devices/pci0000:ae/0000:ae:00.0/0000:af:00.0

0000:af:00.1 -> ../../../../devices/pci0000:ae/0000:ae:00.0/0000:af:00.1

$ ./t-rex-64 --dump-interfaces 0000:3b:00.0 0000:3b:00.1 0000:5e:00.0 0000:5e:00.1 0000:af:00.0 0000:af:00.1

Starting  TRex v2.87 please wait  ... 

Showing interfaces info.

PCI: 0000:3b:00.0 - MAC: 3C:FD:FE:EE:4C:D0 - Driver: net_i40e

PCI: 0000:3b:00.1 - MAC: 3C:FD:FE:EE:4C:D1 - Driver: net_i40e

PCI: 0000:5e:00.0 - MAC: F8:F2:1E:52:59:98 - Driver: net_ixgbe

PCI: 0000:5e:00.1 - MAC: F8:F2:1E:52:59:99 - Driver: net_ixgbe

The 0000:af:00.0|1 devices (which are the same as 0000:3b:00.0|1 -- Intel XXV710 dual port 25Gb) are always missed.  I can get t-rex-64 to find them if I build a simple config file and use it instead of specifying the devices on the command line (but that isn't how dpdk_setup_ports.py/dpdk_nic_bind.py does it):

$ cat trex_cfg.yaml

- version: 2

  interfaces: ["0000:3b:00.0","0000:3b:00.1","0000:5e:00.0","0000:5e:00.1","0000:af:00.0","0000:af:00.1"]

$ ./t-rex-64 --dump-interfaces --cfg trex_cfg.yaml

Starting  TRex v2.87 please wait  ... 

Showing interfaces info.

PCI: 0000:3b:00.0 - MAC: 3C:FD:FE:EE:4C:D0 - Driver: net_i40e

PCI: 0000:3b:00.1 - MAC: 3C:FD:FE:EE:4C:D1 - Driver: net_i40e

PCI: 0000:5e:00.0 - MAC: F8:F2:1E:52:59:98 - Driver: net_ixgbe

PCI: 0000:5e:00.1 - MAC: F8:F2:1E:52:59:99 - Driver: net_ixgbe

PCI: 0000:af:00.0 - MAC: 3C:FD:FE:A9:CF:34 - Driver: net_i40e

PCI: 0000:af:00.1 - MAC: 3C:FD:FE:A9:CF:35 - Driver: net_i40e

Karl

[hanoh haim]

Hi Karl, 

This is a relatively new feature, thanks for reporting it. 

I will have a look 

Thanks

Hanoh

--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/38e8ce18-2e94-4c17-8efd-7b7e32c442dcn%40googlegroups.com.

--

Hanoh

Sent from my iPhone

[hanoh haim]

Hi, I've removed trex_cfg.yaml /etc/ and add this 

def get_info_from_trex(pci_addr_list):

print(" --> get_info_from_trex {} ".format(pci_addr_list)) <<<

if not pci_addr_list:

return {}

pci_info_dict = {}

run_command = './t-rex-64 --dump-interfaces %s' % ' '.join(pci_addr_list)

print(run_command) <<<

[csi-trex-12]> sudo ./dpdk_setup_ports.py -t
 --> get_info_from_trex ['0000:00:0e.0', '0000:00:0d.0', '0000:00:08.0', '0000:00:0f.0', '0000:00:10.0', '0000:00:11.0', '0000:00:12.0', '0000:00:07.0', '0000:00:09.0', '0000:00:0a.0', '0000:00:0b.0', '0000:00:0c.0']
./t-rex-64 --dump-interfaces 0000:00:0e.0 0000:00:0d.0 0000:00:08.0 0000:00:0f.0 0000:00:10.0 0000:00:11.0 0000:00:12.0 0000:00:07.0 0000:00:09.0 0000:00:0a.0 0000:00:0b.0 0000:00:0c.0

so I don't see an issue with latest version v2.88 

[Karl Rister]

I'm still seeing the same behavior (only listing the first 4 interfaces) which causes failure:

 

$ ./t-rex-64 --dump-interfaces 0000:3b:00.0 0000:3b:00.1 0000:5e:00.0 0000:5e:00.1 0000:af:00.0 0000:af:00.1

Starting  TRex v2.88 please wait  ...

Showing interfaces info.
PCI: 0000:3b:00.0 - MAC: 3C:FD:FE:EE:4C:D0 - Driver: net_i40e
PCI: 0000:3b:00.1 - MAC: 3C:FD:FE:EE:4C:D1 - Driver: net_i40e
PCI: 0000:5e:00.0 - MAC: F8:F2:1E:52:59:98 - Driver: net_ixgbe
PCI: 0000:5e:00.1 - MAC: F8:F2:1E:52:59:99 - Driver: net_ixgbe

$ ./dpdk_setup_ports.py -c  0000:3b:00.0 0000:3b:00.1 0000:af:00.0 0000:af:00.1 --cores-include  4 5 6 7 8 9 10 11 12 13 14 15 16 17 -o /tmp/foo.out
 --> get_info_from_trex ['0000:3b:00.0', '0000:3b:00.1', '0000:5e:00.0', '0000:5e:00.1', '0000:af:00.0', '0000:af:00.1']

./t-rex-64 --dump-interfaces 0000:3b:00.0 0000:3b:00.1 0000:5e:00.0 0000:5e:00.1 0000:af:00.0 0000:af:00.1

+----+------+---------+-------------------+------------------------------------------------+----------+----------+----------+
| ID | NUMA |   PCI   |        MAC        |                      Name                      |  Driver  | Linux IF |  Active  |
+====+======+=========+===================+================================================+==========+==========+==========+
| 0  | 0    | 19:00.0 | e4:43:4b:f1:e0:6c | Ethernet Controller 10G X550T                  | ixgbe    | eno1     |          |
+----+------+---------+-------------------+------------------------------------------------+----------+----------+----------+
| 1  | 0    | 19:00.1 | e4:43:4b:f1:e0:6d | Ethernet Controller 10G X550T                  | ixgbe    | eno2     |          |
+----+------+---------+-------------------+------------------------------------------------+----------+----------+----------+
| 2  | 0    | 1a:00.0 | e4:43:4b:f1:e0:6e | I350 Gigabit Network Connection                | igb      | eno3     | *Active* |
+----+------+---------+-------------------+------------------------------------------------+----------+----------+----------+
| 3  | 0    | 1a:00.1 | e4:43:4b:f1:e0:6f | I350 Gigabit Network Connection                | igb      | eno4     |          |
+----+------+---------+-------------------+------------------------------------------------+----------+----------+----------+
| 4  | 0    | 3b:00.0 | 3c:fd:fe:ee:4c:d0 | Ethernet Controller XXV710 for 25GbE SFP28     | vfio-pci |          |          |
+----+------+---------+-------------------+------------------------------------------------+----------+----------+----------+
| 5  | 0    | 3b:00.1 | 3c:fd:fe:ee:4c:d1 | Ethernet Controller XXV710 for 25GbE SFP28     | vfio-pci |          |          |
+----+------+---------+-------------------+------------------------------------------------+----------+----------+----------+
| 6  | 0    | 5e:00.0 | f8:f2:1e:52:59:98 | 82599ES 10-Gigabit SFI/SFP+ Network Connection | vfio-pci |          |          |
+----+------+---------+-------------------+------------------------------------------------+----------+----------+----------+
| 7  | 0    | 5e:00.1 | f8:f2:1e:52:59:99 | 82599ES 10-Gigabit SFI/SFP+ Network Connection | vfio-pci |          |          |
+----+------+---------+-------------------+------------------------------------------------+----------+----------+----------+
| 8  | 1    | af:00.0 |                   | Ethernet Controller XXV710 for 25GbE SFP28     | vfio-pci |          |          |
+----+------+---------+-------------------+------------------------------------------------+----------+----------+----------+
| 9  | 1    | af:00.1 |                   | Ethernet Controller XXV710 for 25GbE SFP28     | vfio-pci |          |          |
+----+------+---------+-------------------+------------------------------------------------+----------+----------+----------+
Interfaces should be of same type, got:
        * net_i40e
        * not_valid_drv

[hanoh haim]

Hi Karl, 

You are right, could you try this? didn't check it yet. 

diff --git a/src/main_dpdk.cpp b/src/main_dpdk.cpp
index 38c868a1f..e5e77f381 100644
--- a/src/main_dpdk.cpp
+++ b/src/main_dpdk.cpp
@@ -995,6 +995,10 @@ COLD_FUNC static int parse_options(int argc, char *argv[], bool first_time ) {
 
 
     if ( first_time ){
+        if ( cg->m_if_list.size() > CGlobalInfo::m_options.m_expected_portd ) {
+            CGlobalInfo::m_options.m_expected_portd = (uint16)cg->m_if_list.size();
+        }
+
         /* only first time read the configuration file */
         if ( po->platform_cfg_file.length() >0  ) {
             if ( node_dump ){
@@ -1015,12 +1019,7 @@ COLD_FUNC static int parse_options(int argc, char *argv[], bool first_time ) {
                 }
             }
         }
-    } else {
-        if ( cg->m_if_list.size() > CGlobalInfo::m_options.m_expected_portd ) {
-            cg->m_if_list.resize(CGlobalInfo::m_options.m_expected_portd);
-        }
-    }
-
+    }
     if ( get_is_interactive() ) {
         if ( opt_vlan_was_set ) {
             // Only purpose of this in interactive is for configuring the 82599 rules correctly

[Karl Rister]

Hi Hanoh

That seems to fix this particular issue:

$ ./t-rex-64 --dump-interfaces 0000:3b:00.0 0000:3b:00.1 0000:5e:00.0 0000:5e:00.1 0000:af:00.0 0000:af:00.1

Starting  TRex v2.88 please wait  ...
Showing interfaces info.
PCI: 0000:3b:00.0 - MAC: 3C:FD:FE:EE:4C:D0 - Driver: net_i40e
PCI: 0000:3b:00.1 - MAC: 3C:FD:FE:EE:4C:D1 - Driver: net_i40e
PCI: 0000:5e:00.0 - MAC: F8:F2:1E:52:59:98 - Driver: net_ixgbe
PCI: 0000:5e:00.1 - MAC: F8:F2:1E:52:59:99 - Driver: net_ixgbe

PCI: 0000:af:00.0 - MAC: 3C:FD:FE:A9:CF:34 - Driver: net_i40e
PCI: 0000:af:00.1 - MAC: 3C:FD:FE:A9:CF:35 - Driver: net_i40e

--
Karl Rister <kri...@redhat.com>

[hanoh haim]

Hi Karl, 

Actually, this commit fixes the issue. 

https://github.com/cisco-system-traffic-generator/trex-core/commit/e927224e14fc3ed27a5952547bd98a4137792bcb

Thanks for reporting. Will be fixed in next version

Thanks

Hanoh