FATAL errors when starting TRex v3.06 in stateless mode

199 views

Skip to first unread message

btjo...@gmail.com

unread,

Nov 27, 2024, 7:06:45 PM11/27/24

to TRex Traffic Generator

I've been trying to uplift a TRex configuration that runs fine with v2.81 to v3.06.

The system is as follows:

Lanner NCA-5710C chassis with:
- 2x Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz with a total of 64 vCPUs
- 128G RAM
- 32 NICs organized as 4x 8-port X710
- 32x 1GB huge pages setup in grub and mounted by /etc/fstab.
- CentOS 7
- Kernel 3.10.0-1160.83.1.el7.x86_64

With v2.81, I was running with 2MB huge pages, but v3.06 wasn't recognizing them so I changed to using 1GB.

I' was running with 2 16-port instances with v2.81, but to start am only trying to get 1 working with v3.06. Eventually, I'd like to enable a single 32-port instance.

The config is as follows:

- version: 2
interfaces: ['19:00.3', '19:00.1', '18:00.3', '18:00.1', '1b:00.3', '1b:00.1', '1a:00.3', '1a:00.1', '19:00.2', '19:00.0', '18:00.2', '18:00.0', '1b:00.2', '1b:00.0', '1a:00.2', '1a:00.0']
prefix: instance1
port_bandwidth_gb: 10
limit_memory: 2048
zmq_pub_port: 4500
zmq_rpc_port: 4501
port_info:
- dest_mac: 00:00:00:00:01:00
src_mac: 00:90:0b:9e:14:df
- dest_mac: 00:00:00:00:01:01
src_mac: 00:90:0b:9e:14:dd

- dest_mac: 00:00:00:00:01:02
src_mac: 00:90:0b:9e:14:db
- dest_mac: 00:00:00:00:01:03
src_mac: 00:90:0b:9e:14:d9

- dest_mac: 00:00:00:00:01:04
src_mac: 00:90:0b:a0:c6:0c
- dest_mac: 00:00:00:00:01:05
src_mac: 00:90:0b:a0:c6:0a

- dest_mac: 00:00:00:00:01:06
src_mac: 00:90:0b:a0:c6:08
- dest_mac: 00:00:00:00:01:07
src_mac: 00:90:0b:a0:c6:06

- dest_mac: 00:00:00:00:01:08
src_mac: 00:90:0b:9e:14:de
- dest_mac: 00:00:00:00:01:09
src_mac: 00:90:0b:9e:14:dc

- dest_mac: 00:00:00:00:01:0a
src_mac: 00:90:0b:9e:14:da
- dest_mac: 00:00:00:00:01:0b
src_mac: 00:90:0b:9e:14:d8

- dest_mac: 00:00:00:00:01:0c
src_mac: 00:90:0b:a0:c6:0b
- dest_mac: 00:00:00:00:01:0d
src_mac: 00:90:0b:a0:c6:09

- dest_mac: 00:00:00:00:01:0e
src_mac: 00:90:0b:a0:c6:07
- dest_mac: 00:00:00:00:01:0f
src_mac: 00:90:0b:a0:c6:05

platform:
master_thread_id: 17
latency_thread_id: 16
dual_if:
- socket: 0
threads: [8,40]

- socket: 0
threads: [9,41]

- socket: 0
threads: [10,42]

- socket: 0
threads: [11,43]

- socket: 0
threads: [12,44]

- socket: 0
threads: [13,45]

- socket: 0
threads: [14,46]

- socket: 0
threads: [15,47]

When trying to start TRex with debug enabled, I see the following:

./t-rex-64 -i --cfg /etc/trex_cfg.yaml --mbuf-factor 0.01 -v 100
Starting Scapy server...... Scapy server is started
The ports are bound/configured.
Starting TRex v3.06 please wait ...
Using configuration file /etc/trex_cfg.yaml
port limit : not configured
port_bandwidth_gb : 10
port_speed : 0
port_mtu : 0
if_mask : None
prefix : instance1
is low-end : 0
stack type :
limit_memory : 2048
thread_per_dual_if : 1
if : 19:00.3, 19:00.1, 18:00.3, 18:00.1, 1b:00.3, 1b:00.1, 1a:00.3, 1a:00.1, 19:00.2, 19:00.0, 18:00.2, 18:00.0, 1b:00.2, 1b:00.0, 1a:00.2, 1a:00.0,
enable_zmq_pub : 1
zmq_pub_port : 4500
m_zmq_rpc_port : 4501
src : 00:90:0b:9e:14:df
dest : 00:00:00:00:01:00
src : 00:90:0b:9e:14:dd
dest : 00:00:00:00:01:01
src : 00:90:0b:9e:14:db
dest : 00:00:00:00:01:02
src : 00:90:0b:9e:14:d9
dest : 00:00:00:00:01:03
src : 00:90:0b:a0:c6:0c
dest : 00:00:00:00:01:04
src : 00:90:0b:a0:c6:0a
dest : 00:00:00:00:01:05
src : 00:90:0b:a0:c6:08
dest : 00:00:00:00:01:06
src : 00:90:0b:a0:c6:06
dest : 00:00:00:00:01:07
src : 00:90:0b:9e:14:de
dest : 00:00:00:00:01:08
src : 00:90:0b:9e:14:dc
dest : 00:00:00:00:01:09
src : 00:90:0b:9e:14:da
dest : 00:00:00:00:01:0a
src : 00:90:0b:9e:14:d8
dest : 00:00:00:00:01:0b
src : 00:90:0b:a0:c6:0b
dest : 00:00:00:00:01:0c
src : 00:90:0b:a0:c6:09
dest : 00:00:00:00:01:0d
src : 00:90:0b:a0:c6:07
dest : 00:00:00:00:01:0e
src : 00:90:0b:a0:c6:05
dest : 00:00:00:00:01:0f
memory per 2x10G ports
MBUF_64 : 16380
MBUF_128 : 8190
MBUF_256 : 8190
MBUF_512 : 8190
MBUF_1024 : 8190
MBUF_2048 : 4095
MBUF_4096 : 128
MBUF_9K : 512
TRAFFIC_MBUF_64 : 65520
TRAFFIC_MBUF_128 : 32760
TRAFFIC_MBUF_256 : 8190
TRAFFIC_MBUF_512 : 8190
TRAFFIC_MBUF_1024 : 8190
TRAFFIC_MBUF_2048 : 32760
TRAFFIC_MBUF_4096 : 128
TRAFFIC_MBUF_9K : 512
MBUF_DP_FLOWS : 524288
MBUF_GLOBAL_FLOWS : 5120
master thread : 17
rx thread : 16
dual_if : 0
socket : 0
[ 8 40 ]
dual_if : 1
socket : 0
[ 9 41 ]
dual_if : 2
socket : 0
[ 10 42 ]
dual_if : 3
socket : 0
[ 11 43 ]
dual_if : 4
socket : 0
[ 12 44 ]
dual_if : 5
socket : 0
[ 13 45 ]
dual_if : 6
socket : 0
[ 14 46 ]
dual_if : 7
socket : 0
[ 15 47 ]
CTimerWheelYamlInfo does not exist
flags : 8010c00
write_file : 0
verbose : 4
realtime : 1
flip : 0
cores : 1
single core : 0
flow-flip : 0
no clean close : 0
zmq_publish : 1
vlan mode : 0
client_cfg : 0
mbuf_cache_disable : 0
cfg file :
mac file :
out file :
client cfg file :
duration : 0
factor : 1
mbuf_factor : 0
latency : 0 pkt/sec
zmq_port : 4500
telnet_port : 4501
expected_ports : 16
tw_bucket_usec : 20.000000 usec
tw_buckets : 1024 usec
tw_levels : 3 usec
port : 0 dst:00:00:00:00:01:00 src:00:90:0b:9e:14:df
port : 1 dst:00:00:00:00:01:01 src:00:90:0b:9e:14:dd
port : 2 dst:00:00:00:00:01:02 src:00:90:0b:9e:14:db
port : 3 dst:00:00:00:00:01:03 src:00:90:0b:9e:14:d9
port : 4 dst:00:00:00:00:01:04 src:00:90:0b:a0:c6:0c
port : 5 dst:00:00:00:00:01:05 src:00:90:0b:a0:c6:0a
port : 6 dst:00:00:00:00:01:06 src:00:90:0b:a0:c6:08
port : 7 dst:00:00:00:00:01:07 src:00:90:0b:a0:c6:06
port : 8 dst:00:00:00:00:01:08 src:00:90:0b:9e:14:de
port : 9 dst:00:00:00:00:01:09 src:00:90:0b:9e:14:dc
port : 10 dst:00:00:00:00:01:0a src:00:90:0b:9e:14:da
port : 11 dst:00:00:00:00:01:0b src:00:90:0b:9e:14:d8
port : 12 dst:00:00:00:00:01:0c src:00:90:0b:a0:c6:0b
port : 13 dst:00:00:00:00:01:0d src:00:90:0b:a0:c6:09
port : 14 dst:00:00:00:00:01:0e src:00:90:0b:a0:c6:07
port : 15 dst:00:00:00:00:01:0f src:00:90:0b:a0:c6:05
port : 16 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 17 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 18 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 19 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 20 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 21 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 22 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 23 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 24 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 25 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 26 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 27 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 28 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 29 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 30 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 31 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
Total Memory :
MBUF_64 : 66830
MBUF_128 : 33415
MBUF_256 : 8845
MBUF_512 : 8845
MBUF_1024 : 8845
MBUF_2048 : 33087
MBUF_4096 : 8192
MBUF_DP_FLOWS : 41943
MBUF_GLOBAL_FLOWS : 409
get_each_core_dp_flows : 5242
Total memory : 158.35 Mbytes
core_list : 17,16,8,9,10,11,12,13,14,15
sockets : 0
active sockets : 1
ports_sockets : 1
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
phy | virt
8 1
9 2
10 3
11 4
12 5
13 6
14 7
15 8
DPDK args
xx -l 17,16,8,9,10,11,12,13,14,15 -n 4 --log-level 5 --main-lcore 17 -a 0000:19:00.3 -a 0000:19:00.1 -a 0000:18:00.3 -a 0000:18:00.1 -a 0000:1b:00.3 -a 0000:1b:00.1 -a 0000:1a:00.3 -a 0000:1a:00.1 -a 0000:19:00.2 -a 0000:19:00.0 -a 0000:18:00.2 -a 0000:18:00.0 -a 0000:1b:00.2 -a 0000:1b:00.0 -a 0000:1a:00.2 -a 0000:1a:00.0 --legacy-mem --file-prefix instance1 --socket-mem 2048,0
EAL: WARNING: Main core has no memory on local socket!
ETHDEV: Cannot allocate ethdev shared data
EAL: Requested device 0000:18:00.0 cannot be used
ETHDEV: Cannot allocate ethdev shared data
EAL: Requested device 0000:18:00.1 cannot be used
ETHDEV: Cannot allocate ethdev shared data
EAL: Requested device 0000:18:00.2 cannot be used
ETHDEV: Cannot allocate ethdev shared data
EAL: Requested device 0000:18:00.3 cannot be used
ETHDEV: Cannot allocate ethdev shared data
EAL: Requested device 0000:19:00.0 cannot be used
ETHDEV: Cannot allocate ethdev shared data
EAL: Requested device 0000:19:00.1 cannot be used
ETHDEV: Cannot allocate ethdev shared data
EAL: Requested device 0000:19:00.2 cannot be used
ETHDEV: Cannot allocate ethdev shared data
EAL: Requested device 0000:19:00.3 cannot be used
ETHDEV: Cannot allocate ethdev shared data
EAL: Requested device 0000:1a:00.0 cannot be used
ETHDEV: Cannot allocate ethdev shared data
EAL: Requested device 0000:1a:00.1 cannot be used
ETHDEV: Cannot allocate ethdev shared data
EAL: Requested device 0000:1a:00.2 cannot be used
ETHDEV: Cannot allocate ethdev shared data
EAL: Requested device 0000:1a:00.3 cannot be used
ETHDEV: Cannot allocate ethdev shared data
EAL: Requested device 0000:1b:00.0 cannot be used
ETHDEV: Cannot allocate ethdev shared data
EAL: Requested device 0000:1b:00.1 cannot be used
ETHDEV: Cannot allocate ethdev shared data
EAL: Requested device 0000:1b:00.2 cannot be used
ETHDEV: Cannot allocate ethdev shared data
EAL: Requested device 0000:1b:00.3 cannot be used
EAL: Bus (pci) probe failed.
TELEMETRY: No legacy callbacks, legacy socket not created
ERROR in DPDK map
Could not find requested interface 19:00.3
Killing Scapy server... Scapy server is killed

I have tried other released between v2.81 and v3.06 and have seen similar error messages indicating that 'ethdev shared data' cannot be allocated.