trex_daemon_server fails to start TRex
672 views
Skip to first unread message
Andreas Bourges
May 21, 2021, 12:34:06 AM5/21/21
to TRex Traffic Generator
Hi,
...I'm trying to start trex via trex_daemon_server, but it seems that there's a timeout when waiting for trex to come up. The problem happens most of the time when the server has been rebootet and trex needs to compile the nic module and set up its interfaces (12 x X710). After trying several times, trex would occasionally come up. Another way to start trex is to manually start it via cli ("t-rex -c 4 --software -i"). Once it has been up and running for a first time, it can be started via trex_daemon_server without problems.
Here's the systemctl log from trex_daemon_server and below you can find the logs received from trex_daemon_server via API:
journalctl -u trex_daemon_server --follow
-- Logs begin at Tue 2021-04-06 12:29:47 CEST. --
May 21 06:18:12 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:18:12 trex_server INFO Processing get_trex_version() command.
May 21 06:18:12 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:18:12 trex_server INFO Version : v2.87
May 21 06:18:12 mndelabtx01.mn-man.biz trex_daemon_server[137581]: DPDK version : DPDK 20.02.0
May 21 06:18:12 mndelabtx01.mn-man.biz trex_daemon_server[137581]: User : hhaim
May 21 06:18:12 mndelabtx01.mn-man.biz trex_daemon_server[137581]: Date : Dec 30 2020 , 12:31:52
May 21 06:18:12 mndelabtx01.mn-man.biz trex_daemon_server[137581]: Uuid : f4705f66-4a89-11eb-83d1-641225363740
May 21 06:18:12 mndelabtx01.mn-man.biz trex_daemon_server[137581]: Git SHA : 60683dd8b91e1cf5e7ddeae3c0d47b6d2990504d
May 21 06:18:12 mndelabtx01.mn-man.biz trex_daemon_server[137581]: Compiled with GCC : 6.2.0
May 21 06:18:12 mndelabtx01.mn-man.biz trex_daemon_server[137581]: Compiled with glibc : 2.16 (host: 2.28)
May 21 06:18:12 mndelabtx01.mn-man.biz trex_daemon_server[137581]: Sanitized image : no
May 21 06:18:50 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:18:50 trex_server INFO Processing get_running_status() command. Running status is: TRexStatus.Idle
May 21 06:18:50 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:18:50 trex_server INFO Processing start_trex() command.
May 21 06:18:50 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:18:50 trex_server INFO Processing is_reserved() command.
May 21 06:18:50 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:18:50 trex_server INFO TREX FULL COMMAND: nice -n -19 ./t-rex-64 --iom 0 -c 4 -i --emu --software --no-key
May 21 06:18:50 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:18:50 trex_server INFO TRex running state changed to 'Starting'.
May 21 06:18:50 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:18:50 trex_server INFO Starting ZMQ monitor on port 4500
May 21 06:18:50 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:18:50 zmq_monitor_thread INFO ZMQ monitor initialization finished
May 21 06:18:50 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:18:50 trex_server INFO TRex session has been successfully initiated.
May 21 06:18:50 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:18:50 zmq_monitor_thread INFO ZMQ monitor started listening @ tcp://localhost:4500
May 21 06:18:50 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:18:50 trex_server INFO Processing get_running_status() command. Running status is: TRexStatus.Starting
May 21 06:18:50 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:18:50 trex_launch_thread INFO TRex session initialized successfully, Parent process pid is 137637.
May 21 06:18:51 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:18:51 trex_server INFO Processing get_running_status() command. Running status is: TRexStatus.Starting
May 21 06:18:51 mndelabtx01.mn-man.biz sudo[137688]: root : TTY=unknown ; PWD=/data/TREX/v2.87 ; USER=root ; COMMAND=/bin/taskset -c 0 ./trex-emu -p 4510
May 21 06:18:51 mndelabtx01.mn-man.biz sudo[137688]: pam_unix(sudo:session): session opened for user root by (uid=0)
May 21 06:18:52 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:18:52 trex_server INFO Processing get_running_status() command. Running status is: TRexStatus.Starting
May 21 06:18:53 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:18:53 trex_server INFO Processing get_running_status() command. Running status is: TRexStatus.Starting
May 21 06:18:54 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:18:54 trex_server INFO Processing get_running_status() command. Running status is: TRexStatus.Starting
May 21 06:18:55 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:18:55 trex_server INFO Processing get_running_status() command. Running status is: TRexStatus.Starting
May 21 06:18:56 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:18:56 trex_server INFO Processing get_running_status() command. Running status is: TRexStatus.Starting
May 21 06:18:57 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:18:57 trex_server INFO Processing get_running_status() command. Running status is: TRexStatus.Starting
May 21 06:18:58 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:18:58 trex_server INFO Processing get_running_status() command. Running status is: TRexStatus.Starting
May 21 06:18:59 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:18:59 trex_server INFO Processing get_running_status() command. Running status is: TRexStatus.Starting
May 21 06:19:00 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:19:00 trex_server INFO Processing get_running_status() command. Running status is: TRexStatus.Starting
May 21 06:19:01 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:19:01 trex_server INFO Processing get_running_status() command. Running status is: TRexStatus.Starting
May 21 06:19:02 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:19:02 trex_server INFO Processing get_running_status() command. Running status is: TRexStatus.Starting
May 21 06:19:03 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:19:03 trex_server INFO Processing get_running_status() command. Running status is: TRexStatus.Starting
May 21 06:19:04 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:19:04 trex_server INFO Processing get_running_status() command. Running status is: TRexStatus.Starting
May 21 06:19:05 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:19:05 trex_server INFO Processing get_running_status() command. Running status is: TRexStatus.Starting
May 21 06:19:06 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:19:06 trex_server INFO Processing get_running_status() command. Running status is: TRexStatus.Starting
May 21 06:19:07 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:19:07 trex_server INFO Processing get_running_status() command. Running status is: TRexStatus.Starting
May 21 06:19:08 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:19:08 trex_server INFO Processing get_running_status() command. Running status is: TRexStatus.Starting
May 21 06:19:09 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:19:09 trex_server INFO Processing get_running_status() command. Running status is: TRexStatus.Starting
May 21 06:19:10 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:19:10 trex_server INFO Processing get_running_status() command. Running status is: TRexStatus.Starting
May 21 06:19:11 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:19:11 trex_server INFO Processing get_running_status() command. Running status is: TRexStatus.Starting
May 21 06:19:12 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:19:12 trex_server INFO Processing get_running_status() command. Running status is: TRexStatus.Starting
May 21 06:19:13 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:19:13 trex_server INFO Processing get_running_status() command. Running status is: TRexStatus.Starting
May 21 06:19:14 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:19:14 trex_server INFO Processing get_running_status() command. Running status is: TRexStatus.Starting
May 21 06:19:15 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:19:15 trex_server INFO Processing get_running_status() command. Running status is: TRexStatus.Starting
May 21 06:19:16 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:19:16 trex_server INFO Processing get_running_status() command. Running status is: TRexStatus.Starting
May 21 06:19:16 mndelabtx01.mn-man.biz xx[138639]: i40e_phy_conf_link(): Failed to get PHY capabilities: -7
May 21 06:19:17 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:19:17 trex_server INFO Processing get_running_status() command. Running status is: TRexStatus.Starting
May 21 06:19:18 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:19:18 trex_server INFO Processing get_running_status() command. Running status is: TRexStatus.Starting
May 21 06:19:19 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:19:19 trex_server INFO Processing get_running_status() command. Running status is: TRexStatus.Starting
May 21 06:19:20 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:19:20 trex_server INFO Processing get_running_status() command. Running status is: TRexStatus.Starting
May 21 06:19:21 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:19:21 trex_server INFO Processing get_running_status() command. Running status is: TRexStatus.Starting
May 21 06:19:22 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:19:22 trex_server INFO Processing get_running_status() command. Running status is: TRexStatus.Starting
May 21 06:19:23 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:19:23 trex_server INFO Processing get_running_status() command. Running status is: TRexStatus.Starting
May 21 06:19:24 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:19:24 trex_server INFO Processing get_running_status() command. Running status is: TRexStatus.Starting
May 21 06:19:25 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:19:25 trex_server INFO Processing get_running_status() command. Running status is: TRexStatus.Starting
May 21 06:19:26 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:19:26 trex_server INFO Processing get_running_status() command. Running status is: TRexStatus.Starting
May 21 06:19:26 mndelabtx01.mn-man.biz systemd-coredump[138959]: Process 138639 (_t-rex-64) of user 0 dumped core.
May 21 06:19:27 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:19:27 trex_launch_thread WARNING TRex run was terminated unexpectedly by outer process or by the hosting OS
May 21 06:19:27 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:19:27 trex_launch_thread INFO TRex running state changed to 'Idle'.
May 21 06:19:27 mndelabtx01.mn-man.biz trex_daemon_server[137581]: 05-21 06:19:27 trex_server INFO Processing get_running_status() command. Running status is: TRexStatus.Idle
logs from my script incl. messages recevied via API from trex_daemon_server:
2021-05-21 06:18:50,278 [ INFO] [trexclient.stream_manager: 336] : starting
2021-05-21 06:18:50,281 [DEBUG] [trexclient.stream_manager: 343] : current trex status: {'state': TRexStatus.Idle, 'verbose': 'TRex is Idle'}
2021-05-21 06:18:50,281 [DEBUG] [trexclient.stream_manager: 344] : TYPE: <class 'simple_enum._SimpleEnumVar'>
2021-05-21 06:18:50,281 [DEBUG] [trexclient.stream_manager: 376] : trex is idle...
2021-05-21 06:18:50,281 [ INFO] [trexclient.stream_manager: 377] : starting TRex server using options. {'c': '4', 'i': True, '--emu': True, '--software': True}
2021-05-21 06:19:27,386 [ERROR] [trexclient: 256] : trex_client seems to be already running: TRex is back to Idle state, verbose output:
TRex run was terminated unexpectedly by outer process or by the hosting OS.
Run output:
Starting Scapy server.... Scapy server is started
Starting Emu server.... Emu server is started
Trying to bind to vfio-pci ...
Trying to compile and bind to igb_uio ...
ERROR: We don't have precompiled igb_uio.ko module for your kernel version.
Will try compiling automatically...
Success.
/usr/bin/python dpdk_nic_bind.py --bind=igb_uio 0000:12:00.0 0000:12:00.1 0000:12:00.2 0000:12:00.3 0000:13:00.0 0000:13:00.1 0000:13:00.2 0000:13:00.3 0000:af:00.0 0000:af:00.1 0000:af:00.2 0000:af:00.3
The ports are bound/configured.
Starting TRex v2.87 please wait ...
set driver name net_i40e
driver capability : TCP_UDP_OFFLOAD TSO
set dpdk queues mode to MULTI_QUE
Number of ports found: 12
Port (0) firmware version 0x5005 is not eligible for DDP upgrade
Port (1) firmware version 0x5005 is not eligible for DDP upgrade
Port (2) firmware version 0x5005 is not eligible for DDP upgrade
Port (3) firmware version 0x5005 is not eligible for DDP upgrade
Port (4) firmware version 0x5005 is not eligible for DDP upgrade
Port (5) firmware version 0x5005 is not eligible for DDP upgrade
Port (6) firmware version 0x5005 is not eligible for DDP upgrade
Port (7) firmware version 0x5005 is not eligible for DDP upgrade
Port (8) firmware version 0x5005 is not eligible for DDP upgrade
Port (9) firmware version 0x5005 is not eligible for DDP upgrade
Port (10) firmware version 0x5005 is not eligible for DDP upgrade
Port (11) firmware version 0x5005 is not eligible for DDP upgrade
zmq publisher at: tcp://*:4500
i40e_phy_conf_link(): Failed to get PHY capabilities: -7
wait 2 sec ..
WARNING : there is no link on one of the ports, interactive mode can continue
port : 0
------------
link : link : Link Up - speed 10000 Mbps - full-duplex
promiscuous : 0
port : 1
------------
link : link : Link Up - speed 10000 Mbps - full-duplex
promiscuous : 0
port : 2
------------
link : link : Link Up - speed 10000 Mbps - full-duplex
promiscuous : 0
port : 3
------------
link : link : Link Up - speed 10000 Mbps - full-duplex
promiscuous : 0
port : 4
------------
link : link : Link Up - speed 10000 Mbps - full-duplex
promiscuous : 0
port : 5
------------
link : link : Link Up - speed 10000 Mbps - full-duplex
promiscuous : 0
port : 6
------------
link : Link Down
promiscuous : 0
port : 7
------------
link : Link Down
promiscuous : 0
port : 8
------------
link : Link Down
promiscuous : 0
port : 9
------------
link : Link Down
promiscuous : 0
port : 10
------------
link : Link Down
promiscuous : 0
port : 11
------------
link : Link Down
promiscuous : 0
number of ports : 12
max cores for 2 ports : 4
tx queues per port : 6
-------------------------------
RX core uses TX queue number 65535 on all ports
core, c-port, c-queue, s-port, s-queue, lat-queue
------------------------------------------
1 0 0 1 0 0
2 2 0 3 0 0
3 4 0 5 0 0
4 6 0 7 0 0
5 8 0 9 0 0
6 10 0 11 0 0
7 0 1 1 1 255
8 2 1 3 1 255
9 4 1 5 1 255
10 6 1 7 1 255
11 8 1 9 1 255
12 10 1 11 1 255
13 0 2 1 2 255
14 2 2 3 2 255
15 4 2 5 2 255
16 6 2 7 2 255
17 8 2 9 2 255
18 10 2 11 2 255
19 0 3 1 3 255
20 2 3 3 3 255
21 4 3 5 3 255
22 6 3 7 3 255
23 8 3 9 3 255
24 10 3 11 3 255
-------------------------------
Cleanup of old namespaces related to Linux-based stack
Cleanup Done
terminate called after throwing an instance of 'TrexException'
what(): Error: cound not create stack with following reason:
Timeout on waiting for results for of ticket 0
./t-rex-64: line 106: 138639 Aborted (core dumped) ./_$(basename $0) $INPUT_ARGS $EXTRA_INPUT_ARGS
Killing Scapy server... Scapy server is killed
Killing Emu server... Emu server is killed
2021-05-21 06:19:27,386 [ INFO] [trexclient: 259] : finished. exiting.
Any ideas ?
Thanks,
Andreas
Andreas Bourges
May 21, 2021, 6:33:55 AM5/21/21
to TRex Traffic Generator
below this is the output I receive from API when starting with "-v 7".
At the very end I see this (but I'm not sure if this might be the root-cause?!):
terminate called after throwing an instance of 'TrexException'
what(): Error: cound not create stack with following reason:
Timeout on waiting for results for of ticket 0
./t-rex-64: line 106: 164461 Aborted (core dumped) ./_$(basename $0) $INPUT_ARGS $EXTRA_INPUT_ARGS
Killing Scapy server... Scapy server is killed
Killing Emu server... Emu server is killed
complete output:
verbose output:
TRex run was terminated unexpectedly by outer process or by the hosting OS.
Run output:
Starting Scapy server.... Scapy server is started
Starting Emu server.... Emu server is started
The ports are bound/configured.
Starting TRex v2.87 please wait ...
Using configuration file /etc/trex_cfg.yaml
port limit : not configured
port_bandwidth_gb : 10
if_mask : None
is low-end : 0
stack type : linux_based
thread_per_dual_if : 1
if : 12:00.0, 12:00.1, 12:00.2, 12:00.3, 13:00.0, 13:00.1, 13:00.2, 13:00.3, af:00.0, af:00.1, af:00.2, af:00.3,
enable_zmq_pub : 1
zmq_pub_port : 4500
m_zmq_rpc_port : 4501
memory per 2x10G ports
MBUF_64 : 16380
MBUF_128 : 8190
MBUF_256 : 8190
MBUF_512 : 8190
MBUF_1024 : 8190
MBUF_2048 : 4095
MBUF_4096 : 128
MBUF_9K : 512
TRAFFIC_MBUF_64 : 65520
TRAFFIC_MBUF_128 : 32760
TRAFFIC_MBUF_256 : 8190
TRAFFIC_MBUF_512 : 8190
TRAFFIC_MBUF_1024 : 8190
TRAFFIC_MBUF_2048 : 32760
TRAFFIC_MBUF_4096 : 128
TRAFFIC_MBUF_9K : 512
MBUF_DP_FLOWS : 524288
MBUF_GLOBAL_FLOWS : 5120
master thread : 0
rx thread : 15
dual_if : 0
socket : 0
[ 1 2 3 32 33 34 35 ]
dual_if : 1
socket : 0
[ 4 5 6 7 36 37 38 ]
dual_if : 2
socket : 0
[ 8 9 10 39 40 41 42 ]
dual_if : 3
socket : 0
[ 11 12 13 14 43 44 45 ]
dual_if : 4
socket : 1
[ 16 17 18 19 48 49 50 ]
dual_if : 5
socket : 1
[ 20 21 22 51 52 53 54 ]
CTimerWheelYamlInfo does not exist
flags : 8040f00
write_file : 0
verbose : 7
realtime : 1
flip : 0
cores : 4
single core : 0
flow-flip : 0
no clean close : 0
zmq_publish : 1
vlan mode : 0
client_cfg : 0
mbuf_cache_disable : 0
cfg file :
mac file :
out file :
client cfg file :
duration : 0
factor : 1
mbuf_factor : 1
latency : 0 pkt/sec
zmq_port : 4500
telnet_port : 4501
expected_ports : 12
tw_bucket_usec : 20.000000 usec
tw_buckets : 1024 usec
tw_levels : 3 usec
port : 0 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 1 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 2 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 3 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 4 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 5 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 6 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 7 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 8 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 9 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 10 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 11 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 12 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 13 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 14 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 15 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 16 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 17 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 18 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 19 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 20 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 21 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 22 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 23 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 24 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 25 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 26 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 27 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 28 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 29 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 30 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 31 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
Total Memory :
MBUF_64 : 163800
MBUF_128 : 81900
MBUF_256 : 57330
MBUF_512 : 57330
MBUF_1024 : 57330
MBUF_2048 : 57330
MBUF_4096 : 24576
MBUF_DP_FLOWS : 3145728
MBUF_GLOBAL_FLOWS : 30720
get_each_core_dp_flows : 131072
Total memory : 993.42 Mbytes
core_list : 0,15,1,2,3,4,5,6,7,8,9,10,11,12,13,14,16,17,18,19,20,21,22,32,39,51
sockets : 0 1
active sockets : 2
ports_sockets : 2
0,0,0,0,0,0,0,0,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
phy | virt
1 1
2 7
3 13
4 2
5 8
6 14
7 20
8 3
9 9
10 15
11 4
12 10
13 16
14 22
16 5
17 11
18 17
19 23
20 6
21 12
22 18
32 19
39 21
51 24
DPDK args
xx -l 0,15,1,2,3,4,5,6,7,8,9,10,11,12,13,14,16,17,18,19,20,21,22,32,39,51 -n 4 --log-level 8 --master-lcore 0 -w 0000:12:00.0 -w 0000:12:00.1 -w 0000:12:00.2 -w 0000:12:00.3 -w 0000:13:00.0 -w 0000:13:00.1 -w 0000:13:00.2 -w 0000:13:00.3 -w 0000:af:00.0 -w 0000:af:00.1 -w 0000:af:00.2 -w 0000:af:00.3 --legacy-mem
EAL: Detected 64 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Static memory layout is selected, amount of reserved memory can be adjusted with -m or --socket-mem
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: No available hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: PCI device 0000:12:00.0 on NUMA socket 0
EAL: probe driver: 8086:1572 net_i40e
EAL: PCI device 0000:12:00.1 on NUMA socket 0
EAL: probe driver: 8086:1572 net_i40e
EAL: PCI device 0000:12:00.2 on NUMA socket 0
EAL: probe driver: 8086:1572 net_i40e
EAL: PCI device 0000:12:00.3 on NUMA socket 0
EAL: probe driver: 8086:1572 net_i40e
EAL: PCI device 0000:13:00.0 on NUMA socket 0
EAL: probe driver: 8086:1572 net_i40e
EAL: PCI device 0000:13:00.1 on NUMA socket 0
EAL: probe driver: 8086:1572 net_i40e
EAL: PCI device 0000:13:00.2 on NUMA socket 0
EAL: probe driver: 8086:1572 net_i40e
i40e_dev_sync_phy_type(): Failed to sync phy type: status=-7
EAL: PCI device 0000:13:00.3 on NUMA socket 0
EAL: probe driver: 8086:1572 net_i40e
i40e_dev_sync_phy_type(): Failed to sync phy type: status=-7
EAL: PCI device 0000:af:00.0 on NUMA socket 1
EAL: probe driver: 8086:1572 net_i40e
i40e_dev_sync_phy_type(): Failed to sync phy type: status=-7
EAL: PCI device 0000:af:00.1 on NUMA socket 1
EAL: probe driver: 8086:1572 net_i40e
i40e_dev_sync_phy_type(): Failed to sync phy type: status=-7
EAL: PCI device 0000:af:00.2 on NUMA socket 1
EAL: probe driver: 8086:1572 net_i40e
i40e_dev_sync_phy_type(): Failed to sync phy type: status=-7
EAL: PCI device 0000:af:00.3 on NUMA socket 1
EAL: probe driver: 8086:1572 net_i40e
i40e_dev_sync_phy_type(): Failed to sync phy type: status=-7
input : [12:00.0, 12:00.1, 12:00.2, 12:00.3, 13:00.0, 13:00.1, 13:00.2, 13:00.3, af:00.0, af:00.1, af:00.2, af:00.3]
dpdk : [0000:12:00.0, 0000:12:00.1, 0000:12:00.2, 0000:12:00.3, 0000:13:00.0, 0000:13:00.1, 0000:13:00.2, 0000:13:00.3, 0000:af:00.0, 0000:af:00.1, 0000:af:00.2, 0000:af:00.3]
pci_scan : [0000:12:00.0, 0000:12:00.1, 0000:12:00.2, 0000:12:00.3, 0000:13:00.0, 0000:13:00.1, 0000:13:00.2, 0000:13:00.3, 0000:af:00.0, 0000:af:00.1, 0000:af:00.2, 0000:af:00.3]
map : [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
TRex port mapping
-----------------
TRex vport: 0 dpdk_rte_eth: 0
TRex vport: 1 dpdk_rte_eth: 1
TRex vport: 2 dpdk_rte_eth: 2
TRex vport: 3 dpdk_rte_eth: 3
TRex vport: 4 dpdk_rte_eth: 4
TRex vport: 5 dpdk_rte_eth: 5
TRex vport: 6 dpdk_rte_eth: 6
TRex vport: 7 dpdk_rte_eth: 7
TRex vport: 8 dpdk_rte_eth: 8
TRex vport: 9 dpdk_rte_eth: 9
TRex vport: 10 dpdk_rte_eth: 10
TRex vport: 11 dpdk_rte_eth: 11
set driver name net_i40e
driver capability : TCP_UDP_OFFLOAD TSO
set dpdk queues mode to MULTI_QUE
DPDK devices 12 : 12
-----
0 : vdev 0000:12:00.0
1 : vdev 0000:12:00.1
2 : vdev 0000:12:00.2
3 : vdev 0000:12:00.3
4 : vdev 0000:13:00.0
5 : vdev 0000:13:00.1
6 : vdev 0000:13:00.2
7 : vdev 0000:13:00.3
8 : vdev 0000:af:00.0
9 : vdev 0000:af:00.1
10 : vdev 0000:af:00.2
11 : vdev 0000:af:00.3
-----
Number of ports found: 12
if_index : 0
driver name : net_i40e
min_rx_bufsize : 1024
max_rx_pktlen : 9728
max_rx_queues : 192
max_tx_queues : 192
max_mac_addrs : 64
rx_offload_capa : 0x92e6f
tx_offload_capa : 0x19fbf
rss reta_size : 512
flow_type_rss : 0x7ef8
tx_desc_max : 4096
tx_desc_min : 64
rx_desc_max : 4096
rx_desc_min : 64
Port (0) firmware version 0x5005 is not eligible for DDP upgrade
port 0: FW ver 05.00.05
Port (1) firmware version 0x5005 is not eligible for DDP upgrade
port 1: FW ver 05.00.05
Port (2) firmware version 0x5005 is not eligible for DDP upgrade
port 2: FW ver 05.00.05
Port (3) firmware version 0x5005 is not eligible for DDP upgrade
port 3: FW ver 05.00.05
Port (4) firmware version 0x5005 is not eligible for DDP upgrade
port 4: FW ver 05.00.05
Port (5) firmware version 0x5005 is not eligible for DDP upgrade
port 5: FW ver 05.00.05
Port (6) firmware version 0x5005 is not eligible for DDP upgrade
port 6: FW ver 05.00.05
Port (7) firmware version 0x5005 is not eligible for DDP upgrade
port 7: FW ver 05.00.05
Port (8) firmware version 0x5005 is not eligible for DDP upgrade
port 8: FW ver 05.00.05
Port (9) firmware version 0x5005 is not eligible for DDP upgrade
port 9: FW ver 05.00.05
Port (10) firmware version 0x5005 is not eligible for DDP upgrade
port 10: FW ver 05.00.05
Port (11) firmware version 0x5005 is not eligible for DDP upgrade
port 11: FW ver 05.00.05
zmq publisher at: tcp://*:4500
rx_data_q_num : 0
rx_drop_q_num : 0
rx_dp_q_num : 4
rx_que_total : 4
--
rx_desc_num_data_q : 4096
rx_desc_num_drop_q : 4096
rx_desc_num_dp_q : 4096
total_desc : 16384
--
tx_desc_num : 1024
port 0 desc: Ethernet Controller X710 for 10GbE SFP+
rx_qid: 0 (4096)
rx_qid: 1 (4096)
rx_qid: 2 (4096)
rx_qid: 3 (4096)
port 1 desc: Ethernet Controller X710 for 10GbE SFP+
rx_qid: 0 (4096)
rx_qid: 1 (4096)
rx_qid: 2 (4096)
rx_qid: 3 (4096)
port 2 desc: Ethernet Controller X710 for 10GbE SFP+
rx_qid: 0 (4096)
rx_qid: 1 (4096)
rx_qid: 2 (4096)
rx_qid: 3 (4096)
port 3 desc: Ethernet Controller X710 for 10GbE SFP+
rx_qid: 0 (4096)
rx_qid: 1 (4096)
rx_qid: 2 (4096)
rx_qid: 3 (4096)
port 4 desc: Ethernet Controller X710 for 10GbE SFP+
rx_qid: 0 (4096)
rx_qid: 1 (4096)
rx_qid: 2 (4096)
rx_qid: 3 (4096)
port 5 desc: Ethernet Controller X710 for 10GbE SFP+
rx_qid: 0 (4096)
rx_qid: 1 (4096)
rx_qid: 2 (4096)
rx_qid: 3 (4096)
i40e_phy_conf_link(): Failed to get PHY capabilities: -7
port 6 desc: Ethernet Controller X710 for 10GbE SFP+
i40e_dev_sync_phy_type(): Failed to sync phy type: status=-7
rx_qid: 0 (4096)
rx_qid: 1 (4096)
rx_qid: 2 (4096)
rx_qid: 3 (4096)
port 7 desc: Ethernet Controller X710 for 10GbE SFP+
rx_qid: 0 (4096)
rx_qid: 1 (4096)
rx_qid: 2 (4096)
rx_qid: 3 (4096)
port 8 desc: Ethernet Controller X710 for 10GbE SFP+
rx_qid: 0 (4096)
rx_qid: 1 (4096)
rx_qid: 2 (4096)
rx_qid: 3 (4096)
port 9 desc: Ethernet Controller X710 for 10GbE SFP+
rx_qid: 0 (4096)
rx_qid: 1 (4096)
rx_qid: 2 (4096)
rx_qid: 3 (4096)
port 10 desc: Ethernet Controller X710 for 10GbE SFP+
rx_qid: 0 (4096)
rx_qid: 1 (4096)
rx_qid: 2 (4096)
rx_qid: 3 (4096)
port 11 desc: Ethernet Controller X710 for 10GbE SFP+
rx_qid: 0 (4096)
rx_qid: 1 (4096)
rx_qid: 2 (4096)
rx_qid: 3 (4096)
base stack ctor
stack going to run: ip -V
stack going to run: sysctl -V
stack going to run: ethtool --version
Locking cleanup file /var/lock/trex_cleanup
Cleanup of old namespaces related to Linux-based stack
Cleaning IFs with postfixes T
stack going to run: ip link delete trex-a-b-0-T
stack going to run: ip link delete trex-a-6-0-T
stack going to run: ip link delete trex-a-7-0-T
stack going to run: ip link delete trex-a-8-0-T
stack going to run: ip link delete trex-a-0-0-T
stack going to run: ip link delete trex-a-a-0-T
stack going to run: ip link delete trex-a-4-0-T
stack going to run: ip link delete trex-a-9-0-T
stack going to run: ip link delete trex-a-1-0-T
stack going to run: ip link delete trex-a-5-0-T
stack going to run: ip link delete trex-a-2-0-T
stack going to run: ip link delete trex-a-3-0-T
Cleaning IFs with postfixes L
stack going to run: ip netns delete trex-a-1-0
stack going to run: ip netns delete trex-a-3-0
stack going to run: ip netns delete trex-a-4-0
stack going to run: ip netns delete trex-a-2-0
stack going to run: ip netns delete trex-a-5-0
stack going to run: ip netns delete trex-a-0-0
stack going to run: ip netns delete trex-a-8-0
stack going to run: ip netns delete trex-a-7-0
stack going to run: ip netns delete trex-a-9-0
stack going to run: ip netns delete trex-a-6-0
stack going to run: ip netns delete trex-a-a-0
stack going to run: ip netns delete trex-a-b-0
Cleanup Done
Using netns prefix trex-a-
add port node
add node
Run pending tasks for ticket 0
Running in BG
base stack ctor
add_node_internal 48:df:37:e0:3e:43
Namespaced node ctor
Linux node ctor
stack going to run: ip netns add trex-a-b-0
add port node
add node
Run pending tasks for ticket 0
Running in BG
base stack ctor
add port node
add node
Run pending tasks for ticket 0
Running in BG
base stack ctor
add_node_internal 48:df:37:e0:3e:41
Namespaced node ctor
Linux node ctor
stack going to run: ip netns add trex-a-9-0
add port node
add node
Run pending tasks for ticket 0
Running in BG
base stack ctor
add_node_internal 48:df:37:e0:3e:42
Namespaced node ctor
Linux node ctor
stack going to run: ip netns add trex-a-a-0
add port node
add node
Run pending tasks for ticket 0
Running in BG
base stack ctor
add_node_internal 48:df:37:e0:3e:40
add port node
add node
Run pending tasks for ticket 0
Running in BG
base stack ctor
Namespaced node ctor
Linux node ctor
stack going to run: ip netns add trex-a-8-0
add port node
add node
Run pending tasks for ticket 0
Running in BG
base stack ctor
add port node
add node
Run pending tasks for ticket 0
Running in BG
base stack ctor
add port node
add node
Run pending tasks for ticket 0
Running in BG
base stack ctor
add port node
add node
Run pending tasks for ticket 0
Running in BG
base stack ctor
add port node
add node
Run pending tasks for ticket 0
Running in BG
base stack ctor
add port node
add node
Run pending tasks for ticket 0
Running in BG
stack going to run: ip link add trex-a-b-0-T type veth peer name trex-a-b-0-L
add_node_internal 48:df:37:e0:3f:a3
Namespaced node ctor
Linux node ctor
stack going to run: ip netns add trex-a-7-0
add_node_internal 48:df:37:e0:3f:a2
Namespaced node ctor
Linux node ctor
stack going to run: ip netns add trex-a-6-0
stack going to run: ip link add trex-a-9-0-T type veth peer name trex-a-9-0-L
add_node_internal 48:df:37:e0:35:11
Namespaced node ctor
Linux node ctor
stack going to run: ip netns add trex-a-1-0
add_node_internal 48:df:37:e0:35:10
Namespaced node ctor
Linux node ctor
stack going to run: ip netns add trex-a-0-0
add_node_internal 48:df:37:e0:35:12
Namespaced node ctor
Linux node ctor
stack going to run: ip netns add trex-a-2-0
add_node_internal 48:df:37:e0:35:13
Namespaced node ctor
Linux node ctor
stack going to run: ip netns add trex-a-3-0
add_node_internal 48:df:37:e0:3f:a0
Namespaced node ctor
Linux node ctor
stack going to run: ip netns add trex-a-4-0
add_node_internal 48:df:37:e0:3f:a1
Namespaced node ctor
Linux node ctor
stack going to run: ip netns add trex-a-5-0
stack going to run: ip link add trex-a-a-0-T type veth peer name trex-a-a-0-L
stack going to run: ip link add trex-a-7-0-T type veth peer name trex-a-7-0-L
stack going to run: ip link add trex-a-8-0-T type veth peer name trex-a-8-0-L
stack going to run: ip link add trex-a-6-0-T type veth peer name trex-a-6-0-L
stack going to run: ip link add trex-a-4-0-T type veth peer name trex-a-4-0-L
stack going to run: ip link add trex-a-2-0-T type veth peer name trex-a-2-0-L
stack going to run: ip link add trex-a-5-0-T type veth peer name trex-a-5-0-L
stack going to run: sysctl net.ipv6.conf.trex-a-b-0-T.disable_ipv6=1
stack going to run: ip link add trex-a-0-0-T type veth peer name trex-a-0-0-L
stack going to run: sysctl net.ipv6.conf.trex-a-9-0-T.disable_ipv6=1
stack going to run: ip link add trex-a-1-0-T type veth peer name trex-a-1-0-L
stack going to run: ip link add trex-a-3-0-T type veth peer name trex-a-3-0-L
stack going to run: sysctl net.ipv6.conf.trex-a-a-0-T.disable_ipv6=1
stack going to run: sysctl net.ipv6.conf.trex-a-7-0-T.disable_ipv6=1
stack going to run: sysctl net.ipv6.conf.trex-a-8-0-T.disable_ipv6=1
stack going to run: sysctl net.ipv6.conf.trex-a-5-0-T.disable_ipv6=1
stack going to run: sysctl net.ipv6.conf.trex-a-2-0-T.disable_ipv6=1
stack going to run: sysctl net.ipv6.conf.trex-a-4-0-T.disable_ipv6=1
stack going to run: ip link set trex-a-b-0-T mtu 9280 up
stack going to run: ip link set trex-a-9-0-T mtu 9280 up
stack going to run: sysctl net.ipv6.conf.trex-a-0-0-T.disable_ipv6=1
stack going to run: sysctl net.ipv6.conf.trex-a-6-0-T.disable_ipv6=1
stack going to run: sysctl net.ipv6.conf.trex-a-3-0-T.disable_ipv6=1
stack going to run: sysctl net.ipv6.conf.trex-a-1-0-T.disable_ipv6=1
stack going to run: ip link set trex-a-a-0-T mtu 9280 up
stack going to run: ip link set trex-a-5-0-T mtu 9280 up
stack going to run: ip link set trex-a-7-0-T mtu 9280 up
stack going to run: ip link set trex-a-8-0-T mtu 9280 up
stack going to run: ip link set trex-a-2-0-T mtu 9280 up
stack going to run: ip link set trex-a-4-0-T mtu 9280 up
stack going to run: ip link set trex-a-0-0-T mtu 9280 up
stack going to run: ip link set trex-a-6-0-T mtu 9280 up
stack going to run: ip link set trex-a-b-0-L netns trex-a-b-0
stack going to run: ip link set trex-a-9-0-L netns trex-a-9-0
stack going to run: ip link set trex-a-3-0-T mtu 9280 up
stack going to run: ip link set trex-a-1-0-T mtu 9280 up
stack going to run: ip link set trex-a-a-0-L netns trex-a-a-0
stack going to run: ip link set trex-a-7-0-L netns trex-a-7-0
stack going to run: ip link set trex-a-8-0-L netns trex-a-8-0
stack going to run: ip link set trex-a-5-0-L netns trex-a-5-0
stack going to run: ip link set trex-a-4-0-L netns trex-a-4-0
stack going to run: ip link set trex-a-0-0-L netns trex-a-0-0
stack going to run: ip link set trex-a-2-0-L netns trex-a-2-0
stack going to run: ip netns exec trex-a-9-0 sysctl net.ipv6.conf.trex-a-9-0-L.disable_ipv6=1
stack going to run: ip link set trex-a-6-0-L netns trex-a-6-0
stack going to run: ip link set trex-a-3-0-L netns trex-a-3-0
stack going to run: ip netns exec trex-a-b-0 sysctl net.ipv6.conf.trex-a-b-0-L.disable_ipv6=1
stack going to run: ip link set trex-a-1-0-L netns trex-a-1-0
stack going to run: ip netns exec trex-a-5-0 sysctl net.ipv6.conf.trex-a-5-0-L.disable_ipv6=1
stack going to run: ip netns exec trex-a-4-0 sysctl net.ipv6.conf.trex-a-4-0-L.disable_ipv6=1
stack going to run: ip netns exec trex-a-7-0 sysctl net.ipv6.conf.trex-a-7-0-L.disable_ipv6=1
stack going to run: ip netns exec trex-a-a-0 sysctl net.ipv6.conf.trex-a-a-0-L.disable_ipv6=1
stack going to run: ip netns exec trex-a-8-0 sysctl net.ipv6.conf.trex-a-8-0-L.disable_ipv6=1
stack going to run: ip netns exec trex-a-2-0 sysctl net.ipv6.conf.trex-a-2-0-L.disable_ipv6=1
stack going to run: ip netns exec trex-a-0-0 sysctl net.ipv6.conf.trex-a-0-0-L.disable_ipv6=1
stack going to run: ip netns exec trex-a-3-0 sysctl net.ipv6.conf.trex-a-3-0-L.disable_ipv6=1
stack going to run: ip netns exec trex-a-1-0 sysctl net.ipv6.conf.trex-a-1-0-L.disable_ipv6=1
stack going to run: ip netns exec trex-a-6-0 sysctl net.ipv6.conf.trex-a-6-0-L.disable_ipv6=1
stack going to run: ip netns exec trex-a-7-0 ip link set trex-a-7-0-L mtu 9280 up
stack going to run: ip netns exec trex-a-4-0 ip link set trex-a-4-0-L mtu 9280 up
stack going to run: ip netns exec trex-a-8-0 ip link set trex-a-8-0-L mtu 9280 up
stack going to run: ip netns exec trex-a-2-0 ip link set trex-a-2-0-L mtu 9280 up
stack going to run: ip netns exec trex-a-a-0 ip link set trex-a-a-0-L mtu 9280 up
stack going to run: ip netns exec trex-a-0-0 ip link set trex-a-0-0-L mtu 9280 up
stack going to run: ip netns exec trex-a-3-0 ip link set trex-a-3-0-L mtu 9280 up
stack going to run: ip netns exec trex-a-6-0 ip link set trex-a-6-0-L mtu 9280 up
stack going to run: ip netns exec trex-a-1-0 ip link set trex-a-1-0-L mtu 9280 up
stack going to run: ip netns exec trex-a-7-0 ip link set dev trex-a-7-0-L address 48:df:37:e0:3f:a3
stack going to run: ip netns exec trex-a-4-0 ip link set dev trex-a-4-0-L address 48:df:37:e0:3f:a0
stack going to run: ip netns exec trex-a-8-0 ip link set dev trex-a-8-0-L address 48:df:37:e0:3e:40
stack going to run: ip netns exec trex-a-2-0 ip link set dev trex-a-2-0-L address 48:df:37:e0:35:12
stack going to run: ip netns exec trex-a-a-0 ip link set dev trex-a-a-0-L address 48:df:37:e0:3e:42
stack going to run: ip netns exec trex-a-0-0 ip link set dev trex-a-0-0-L address 48:df:37:e0:35:10
stack going to run: ip netns exec trex-a-6-0 ip link set dev trex-a-6-0-L address 48:df:37:e0:3f:a2
stack going to run: ip netns exec trex-a-3-0 ip link set dev trex-a-3-0-L address 48:df:37:e0:35:13
stack going to run: ip netns exec trex-a-1-0 ip link set dev trex-a-1-0-L address 48:df:37:e0:35:11
Canceling running tasks
Namespaced node dtor
terminate called after throwing an instance of 'TrexException'
what(): Error: cound not create stack with following reason:
Timeout on waiting for results for of ticket 0
./t-rex-64: line 106: 164461 Aborted (core dumped) ./_$(basename $0) $INPUT_ARGS $EXTRA_INPUT_ARGS
Killing Scapy server... Scapy server is killed
Killing Emu server... Emu server is killed
2021-05-21 12:25:27,160 [ INFO] [trexclient: 259] : finished. exiting.
hanoh haim
May 22, 2021, 2:46:12 PM5/22/21
to Andreas Bourges, TRex Traffic Generator
It looks like two different issues, the first is a timeout issue
--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/51d5636f-06bc-4031-a60d-f8a2e621ee8en%40googlegroups.com.
Hanoh
Sent from my iPhoneAndreas Bourges
May 25, 2021, 12:36:08 AM5/25/21
to TRex Traffic Generator
Hi Hanoh,
..still trying to get this running. I wrote a minimal script to start trex with increased timeout:
#!/usr/bin/env python3
import sys
import os
sys.path.extend(os.environ['TREX_PATH'].split(":"))
import trex_stf_lib
import trex_stf_lib.trex_client
import logging
from trex_stf_lib.trex_status_e import TRexStatus
formatter = logging.Formatter( fmt = '%(asctime)s [%(levelname)5s] [%(name)s:%(lineno)4s] : %(message)s' )
handler = logging.StreamHandler()
handler.setFormatter(formatter)
logger = logging.getLogger('trexclient')
logger.addHandler(handler)
logger.setLevel(logging.DEBUG)
logger.info("starting")
rpc_client = trex_stf_lib.trex_client.CTRexClient('localhost')
status = rpc_client.get_running_status()
logger.debug("current trex status: {}".format(status['state']))
if status['state'] == TRexStatus.Running:
logger.debug("killing trex ...")
rpc_client.force_kill(confirm=False)
trex_options = {
'c': "4",
'i': True,
'--emu': True,
'--software': True
}
logger.debug("starting trex with options {}".format(trex_options))
try:
ret = rpc_client.start_stateless(user='abourges', timeout=120, **trex_options)
except Exception as e:
logger.error("got exception during trex-start: {}".format(e.args))
logger.debug("final status: {}".format(rpc_client.get_running_status()['state']))
Script fails after ~30s - changing the timeout does not make any difference:
[j0578@mndelabtx01 trex_daemon_problem]$ ./start_trex.py
2021-05-25 06:22:56,037 [ INFO] [trexclient: 40] : starting
2021-05-25 06:22:56,042 [DEBUG] [trexclient: 45] : current trex status: TRexStatus.Idle
2021-05-25 06:22:56,042 [DEBUG] [trexclient: 59] : starting trex with options {'c': '4', 'i': True, '--emu': True, '--software': True}
2021-05-25 06:23:25,116 [ERROR] [trexclient: 63] : got exception during trex-start: ("TRex is back to Idle state, verbose output:\nTRex run was terminated unexpectedly by outer process or by the hosting OS.\n\nRun output:\nStarting Scapy server.... Scapy server is started\nStarting Emu server.... Emu server is started\nThe ports are bound/configured.\nStarting TRex v2.87 please wait ... \n set driver name net_i40e \n driver capability : TCP_UDP_OFFLOAD TSO \n set dpdk queues mode to MULTI_QUE \n Number of ports found: 12\nPort (0) firmware version 0x5005 is not eligible for DDP upgrade \nPort (1) firmware version 0x5005 is not eligible for DDP upgrade \nPort (2) firmware version 0x5005 is not eligible for DDP upgrade \nPort (3) firmware version 0x5005 is not eligible for DDP upgrade \nPort (4) firmware version 0x5005 is not eligible for DDP upgrade \nPort (5) firmware version 0x5005 is not eligible for DDP upgrade \nPort (6) firmware version 0x5005 is not eligible for DDP upgrade \nPort (7) firmware version 0x5005 is not eligible for DDP upgrade \nPort (8) firmware version 0x5005 is not eligible for DDP upgrade \nPort (9) firmware version 0x5005 is not eligible for DDP upgrade \nPort (10) firmware version 0x5005 is not eligible for DDP upgrade \nPort (11) firmware version 0x5005 is not eligible for DDP upgrade \nzmq publisher at: tcp://*:4500\n wait 2 sec ..\n WARNING : there is no link on one of the ports, interactive mode can continue\nport : 0 \n------------\nlink : link : Link Up - speed 10000 Mbps - full-duplex\npromiscuous : 0 \nport : 1 \n------------\nlink : link : Link Up - speed 10000 Mbps - full-duplex\npromiscuous : 0 \nport : 2 \n------------\nlink : link : Link Up - speed 10000 Mbps - full-duplex\npromiscuous : 0 \nport : 3 \n------------\nlink : link : Link Up - speed 10000 Mbps - full-duplex\npromiscuous : 0 \nport : 4 \n------------\nlink : link : Link Up - speed 10000 Mbps - full-duplex\npromiscuous : 0 \nport : 5 \n------------\nlink : link : Link Up - speed 10000 Mbps - full-duplex\npromiscuous : 0 \nport : 6 \n------------\nlink : Link Down\npromiscuous : 0 \nport : 7 \n------------\nlink : Link Down\npromiscuous : 0 \nport : 8 \n------------\nlink : Link Down\npromiscuous : 0 \nport : 9 \n------------\nlink : Link Down\npromiscuous : 0 \nport : 10 \n------------\nlink : Link Down\npromiscuous : 0 \nport : 11 \n------------\nlink : Link Down\npromiscuous : 0 \n number of ports : 12 \n max cores for 2 ports : 4 \n tx queues per port : 6 \n -------------------------------\nRX core uses TX queue number 65535 on all ports\n core, c-port, c-queue, s-port, s-queue, lat-queue\n ------------------------------------------\n 1 0 0 1 0 0 \n 2 2 0 3 0 0 \n 3 4 0 5 0 0 \n 4 6 0 7 0 0 \n 5 8 0 9 0 0 \n 6 10 0 11 0 0 \n 7 0 1 1 1 255 \n 8 2 1 3 1 255 \n 9 4 1 5 1 255 \n 10 6 1 7 1 255 \n 11 8 1 9 1 255 \n 12 10 1 11 1 255 \n 13 0 2 1 2 255 \n 14 2 2 3 2 255 \n 15 4 2 5 2 255 \n 16 6 2 7 2 255 \n 17 8 2 9 2 255 \n 18 10 2 11 2 255 \n 19 0 3 1 3 255 \n 20 2 3 3 3 255 \n 21 4 3 5 3 255 \n 22 6 3 7 3 255 \n 23 8 3 9 3 255 \n 24 10 3 11 3 255 \n -------------------------------\nCleanup of old namespaces related to Linux-based stack\nCleanup Done\nterminate called after throwing an instance of 'TrexException'\n what(): Error: cound not create stack with following reason:\nTimeout on waiting for results for of ticket 0\n./t-rex-64: line 106: 310676 Aborted (core dumped) ./_$(basename $0) $INPUT_ARGS $EXTRA_INPUT_ARGS\nKilling Scapy server... Scapy server is killed\nKilling Emu server... Emu server is killed\n",)
2021-05-25 06:23:25,117 [DEBUG] [trexclient: 65] : final status: TRexStatus.Idle
[j0578@mndelabtx01 trex_daemon_problem]$
When omitting the "--emu" option, things work better most of the time, but then the start task takes less than 28/29s - every now and then I still receive the fault, though, when the start-time crosses the 30s limit.
Thanks,
Andreas
hanoh haim
May 25, 2021, 2:42:41 AM5/25/21
to Andreas Bourges, TRex Traffic Generator
Hi Andreas,
This is a good input. The old exception, you had was something else related to Linux namespace.
It seems the timeout does not forward to the server (from the API)
Thanks
Hanoh
Andreas Bourges
May 25, 2021, 2:53:27 AM5/25/21
to TRex Traffic Generator
...alright - then let's focus on this problem first and I'll check the other one later :-)
Thanks,
Andreas
hanoh haim
May 25, 2021, 6:59:33 AM5/25/21
to Andreas Bourges, TRex Traffic Generator
Hi Andreas,
Just talked to Yaroslav and he pointed me to the log and I think he managed to explain things here:
1) There is no issue with the timeout -- your trex process crashes and the daemon understand that so the timeout won't help here ..
"Cleanup of old namespaces related to Linux-based stack\nCleanup Done\nterminate called after throwing an instance of 'TrexException'\n what(): Error: cound not create stack with following reason:\nTimeout on waiting for results for of ticket 0\"
2) EMU is not related to this
3) The question why it is crashing is simple too, you created many trex linux namespace and killed the trex, in your trex_cfg.yaml you use linux_namepsace.
trex try to remove the old namespace at startup and there is a short timeout to do that of 5 sec (you can't change it, hardcoded).
My suggestions:
1) Can you re-run your script with a change? try to remove the **old** trex linux namespaces using a script (the names can be taken from -v 7 output log) before you load a new trex server
2) Out of curiosity, why are you using EMU and linux-namesapces ? can you use only EMU?
Thanks
Hanoh
--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/e0ca16a3-5e5f-4c0f-a7fb-f54e00fa3b9an%40googlegroups.com.
andyb...@googlemail.com
May 25, 2021, 7:17:28 AM5/25/21
to hanoh haim, TRex Traffic Generator
Hi Hanoh,
…sorry – (again) my fault ☹ - removing the namespaces before starting did not solve the problem (still this shouldn’t happen?!), but the linux-namespace config was a left-over that is actually not used in my code ☹ So I removed the stack: linux_based from the configuration and everything seems to work fine 😊
Sorry for keeping you busy and thanks (at Yaroslav, too!) for looking into that.
Best regards,
Andreas
Reply all
Reply to author
Forward
0 new messages