TRex - E810SRIOV VF

[Haklat Haklat]

Hi,

I have now tried running TRex 2.92 with  E810 SRIOV VFs. But it fails with ERROR in DPDK map (see below). I can launch other DPDK app (VPP) on same host (but different DPDK EAL args: DPDK EAL init args: -c aaa8000000aaaa0 -n 4 --in-memory --no-telemetry --file-prefix vpp -a 0000:d8:01.0 --main-lcore 5).

Maybe below failing start output can give some clue to why it is failing with TRex:

[root@trex-sriov-intel-10-0 v2.92]# ./t-rex-64 -i -c 2 -v 7
Starting Scapy server.... Scapy server is started
The ports are bound/configured.
Starting  TRex v2.92 please wait  ...
Using configuration file /etc/trex_cfg.yaml
 port limit     :  not configured
 port_bandwidth_gb    :  10
 port_speed           :  0
 port_mtu             :  0
 if_mask        : None
 prefix              : trex1
 is low-end : 0
 stack type : linux_based
 limit_memory        : 2048
 thread_per_dual_if      : 1
 if        :  0000:d8:01.2, 0000:d8:11.5,
 enable_zmq_pub :  1
 zmq_pub_port   :  4500
 m_zmq_rpc_port    :  4501
 m_rx_desc    :  4096
 m_tx_desc    :  4096
 src     : 00:00:00:00:00:00
 dest    : 00:00:00:00:00:00
 src     : 00:00:00:00:00:00
 dest    : 00:00:00:00:00:00
 memory per 2x10G ports
 MBUF_64                                   : 16380
 MBUF_128                                  : 8190
 MBUF_256                                  : 8190
 MBUF_512                                  : 8190
 MBUF_1024                                 : 8190
 MBUF_2048                                 : 4095
 MBUF_4096                                 : 128
 MBUF_9K                                   : 512
 TRAFFIC_MBUF_64                           : 65520
 TRAFFIC_MBUF_128                          : 32760
 TRAFFIC_MBUF_256                          : 8190
 TRAFFIC_MBUF_512                          : 8190
 TRAFFIC_MBUF_1024                         : 8190
 TRAFFIC_MBUF_2048                         : 32760
 TRAFFIC_MBUF_4096                         : 128
 TRAFFIC_MBUF_9K                           : 512
 MBUF_DP_FLOWS                             : 524288
 MBUF_GLOBAL_FLOWS                         : 5120
 master   thread  : 5
 rx  thread  : 45
 dual_if : 0
    socket  : 1
   [   7   47     ]
CTimerWheelYamlInfo does not exist
 flags           : 18020f00
 write_file      : 0
 verbose         : 7
 realtime        : 1
 flip            : 0
 cores           : 2
 single core     : 0
 flow-flip       : 0
 no clean close  : 0
 zmq_publish     : 1
 vlan mode       : 1
 client_cfg      : 0
 mbuf_cache_disable  : 0
 cfg file        :
 mac file        :
 out file        :
 client cfg file :
 duration        : 0
 factor          : 1
 mbuf_factor     : 1
 latency         : 0 pkt/sec
 zmq_port        : 4500
 telnet_port     : 4501
 expected_ports  : 2
 tw_bucket_usec  : 20.000000 usec
 tw_buckets      : 1024 usec
 tw_levels       : 3 usec
 port : 0 dst:00:00:00:00:00:00  src:00:00:00:00:00:00 vlan:839
 port : 1 dst:00:00:00:00:00:00  src:00:00:00:00:00:00 vlan:859
 port : 2 dst:00:00:00:01:00:00  src:00:00:00:00:00:00 vlan:0
 port : 3 dst:00:00:00:01:00:00  src:00:00:00:00:00:00 vlan:0
 port : 4 dst:00:00:00:01:00:00  src:00:00:00:00:00:00 vlan:0
 port : 5 dst:00:00:00:01:00:00  src:00:00:00:00:00:00 vlan:0
 port : 6 dst:00:00:00:01:00:00  src:00:00:00:00:00:00 vlan:0
 port : 7 dst:00:00:00:01:00:00  src:00:00:00:00:00:00 vlan:0
 port : 8 dst:00:00:00:01:00:00  src:00:00:00:00:00:00 vlan:0
 port : 9 dst:00:00:00:01:00:00  src:00:00:00:00:00:00 vlan:0
 port : 10 dst:00:00:00:01:00:00  src:00:00:00:00:00:00 vlan:0
 port : 11 dst:00:00:00:01:00:00  src:00:00:00:00:00:00 vlan:0
 port : 12 dst:00:00:00:01:00:00  src:00:00:00:00:00:00 vlan:0
 port : 13 dst:00:00:00:01:00:00  src:00:00:00:00:00:00 vlan:0
 port : 14 dst:00:00:00:01:00:00  src:00:00:00:00:00:00 vlan:0
 port : 15 dst:00:00:00:01:00:00  src:00:00:00:00:00:00 vlan:0
 port : 16 dst:00:00:00:01:00:00  src:00:00:00:00:00:00 vlan:0
 port : 17 dst:00:00:00:01:00:00  src:00:00:00:00:00:00 vlan:0
 port : 18 dst:00:00:00:01:00:00  src:00:00:00:00:00:00 vlan:0
 port : 19 dst:00:00:00:01:00:00  src:00:00:00:00:00:00 vlan:0
 port : 20 dst:00:00:00:01:00:00  src:00:00:00:00:00:00 vlan:0
 port : 21 dst:00:00:00:01:00:00  src:00:00:00:00:00:00 vlan:0
 port : 22 dst:00:00:00:01:00:00  src:00:00:00:00:00:00 vlan:0
 port : 23 dst:00:00:00:01:00:00  src:00:00:00:00:00:00 vlan:0
 port : 24 dst:00:00:00:01:00:00  src:00:00:00:00:00:00 vlan:0
 port : 25 dst:00:00:00:01:00:00  src:00:00:00:00:00:00 vlan:0
 port : 26 dst:00:00:00:01:00:00  src:00:00:00:00:00:00 vlan:0
 port : 27 dst:00:00:00:01:00:00  src:00:00:00:00:00:00 vlan:0
 port : 28 dst:00:00:00:01:00:00  src:00:00:00:00:00:00 vlan:0
 port : 29 dst:00:00:00:01:00:00  src:00:00:00:00:00:00 vlan:0
 port : 30 dst:00:00:00:01:00:00  src:00:00:00:00:00:00 vlan:0
 port : 31 dst:00:00:00:01:00:00  src:00:00:00:00:00:00 vlan:0
 Total Memory :
 MBUF_64                                   : 81900
 MBUF_128                                  : 40950
 MBUF_256                                  : 16380
 MBUF_512                                  : 16380
 MBUF_1024                                 : 16380
 MBUF_2048                                 : 36855
 MBUF_4096                                 : 2048
 MBUF_DP_FLOWS                             : 524288
 MBUF_GLOBAL_FLOWS                         : 5120
 get_each_core_dp_flows                    : 262144
 Total memory                              :     252.59 Mbytes
 core_list : 5,45,7,47
 sockets : 1
 active sockets : 1
 ports_sockets : 1
1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
 phy   |   virt
 7      1
 47      2
DPDK args
 xx  -l  5,45,7,47  -n  4  --log-level  8  --main-lcore  5  -a  0000:d8:01.2  -a  0000:d8:11.5  --legacy-mem  --file-prefix  trex1  --socket-mem  0,2048
EAL: Detected 80 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/trex1/mp_socket
EAL: Selected IOVA mode 'VA'
 EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL:   using IOMMU type 1 (Type 1)
EAL: Probe PCI driver: net_ice_dcf (8086:1889) device: 0000:d8:01.2 (socket 1)
EAL: Releasing pci mapped resource for 0000:d8:01.2
EAL: Calling pci_unmap_resource for 0000:d8:01.2 at 0x19c0000000
EAL: Calling pci_unmap_resource for 0000:d8:01.2 at 0x19c0020000
EAL:   using IOMMU type 1 (Type 1)
EAL: Probe PCI driver: net_ice_dcf (8086:1889) device: 0000:d8:11.5 (socket 1)
EAL: Releasing pci mapped resource for 0000:d8:11.5
EAL: Calling pci_unmap_resource for 0000:d8:11.5 at 0x19c0024000
EAL: Calling pci_unmap_resource for 0000:d8:11.5 at 0x19c0044000
EAL: No legacy callbacks, legacy socket not created
ERROR in DPDK map
Could not find requested interface 0000:d8:01.2
Killing Scapy server... Scapy server is killed
[root@trex-sriov-intel-10-0 v2.92]#

BR//Håkan

[Haklat Haklat]

Hi,

is it maybe probing for wrong driver? In i40e NIC server it starts up fine using same TRex image with:

EAL: Probe PCI driver: net_i40e_vf (8086:154c) device: 0000:88:02.5 (socket 1)

But with E810 and SRIOV it looks like it is probing for physical function ice driver?

EAL: Probe PCI driver: net_ice_dcf (8086:1889) device: 0000:d8:01.2 (socket 1)

[root@trex-tgen-dpdk v2.92]#  lspci -Dvmmnnk -d 8086:1889
Slot:   0000:d8:01.0
Class:  Ethernet controller [0200]
Vendor: Intel Corporation [8086]
Device: Ethernet Adaptive Virtual Function [1889]
SVendor:        Intel Corporation [8086]
SDevice:        Device [0000]
Rev:    02
Driver: vfio-pci
Module: iavf
NUMANode:       1

Slot:   0000:d8:01.1
Class:  Ethernet controller [0200]
Vendor: Intel Corporation [8086]
Device: Ethernet Adaptive Virtual Function [1889]
SVendor:        Intel Corporation [8086]
SDevice:        Device [0000]
Rev:    02
Driver: vfio-pci
Module: iavf
NUMANode:       1

Slot:   0000:d8:01.2
Class:  Ethernet controller [0200]
Vendor: Intel Corporation [8086]
Device: Ethernet Adaptive Virtual Function [1889]
SVendor:        Intel Corporation [8086]
SDevice:        Device [0000]
Rev:    02
Driver: vfio-pci
Module: iavf
NUMANode:       1

A physical funcion print (device being 159b):
Slot:   0000:d8:00.0
Class:  Ethernet controller [0200]
Vendor: Intel Corporation [8086]
Device: Ethernet Controller E810-XXV for SFP [159b]
SVendor:        Intel Corporation [8086]
SDevice:        Ethernet 25G 2P E810-XXV Adapter [0002]
Rev:    02
Driver: ice
Module: ice
NUMANode:       1

BR//Håkan

[Haklat Haklat]

Hi,

I think in e810 case , net_iavf poll mode driver is maybe not part of TRex package. This one driver probably needed to get e810 SRIOV running on TRex.

Are there any plans when to upgrade DPDK version next time? think then maybe iavf driver have to be in there anyway for the i40e SRIOV to continue working. 

Think i40e_vf driver was removed in DPDK version 21.11.

BR//Håkan

[hanoh haim]

Hi Haklat,

We are working to integrate the latest DPDK 

Thanks

Hanoh

[Xiaoping Yan]

Hi Haim & Haklat,

Do you have some update for this?

I met exactly same issue, and I'm using latest trex version v2.99

I see in release note: https://trex-tgn.cisco.com/trex/doc/release_notes.html

V2.98:

• Upgrade to DPDK 22.03

  • i40evf was removed by DPDK but due to an issue with the general driver (iavf) it was return

  • E810 - still not supported

V2.99:

• E810 add drop mode, latency stream filter does not work yet.

What this "drop mode" mean?

Is E810 supported in v2.99, I would use it in stateless mode.

[hanoh haim]

Hi Xiaoping, 

What is your issue. drop mode means that we drop all the packets in the rx side and only count them ( this is how trex works in STL mode by default).

This does not work due to a filter issue, you can try adding --software  to CLI (this has a performance cost)

Thanks

Hanoh

[Xiaoping Yan]

Hi Haim,

I tested trex 2.99, it seems it still not support E810 SRIOV VF.

I got same error as Haklat described (probing for wrong driver net_ice_dcf and failed with ERROR in DPDK map).

Is E810 already supported with trex 2.99?

For now, I have found another device with XXV710, and it works with trex 2.87.

It would be better if E810 is also supported, becasue then we will have more devices available.

[Jose Gato]

I had the same error, after trying with 3.00 worked out of the box.

[Haklat haklat]

Yes,

here fix is mentioned:
https://github.com/cisco-system-traffic-generator/trex-core/issues/861
BR//Håkan

[Xiaoping Yan]

Yes, latest v3.00 works. Thank you for the information.

Did anyone tried stream & latency stats, in my case in stream and latency stats there is only tx packet, no rx packet (while on port stats, all sent packets are received.)