Crashed while starting the trex instance

[ethan...@gmail.com]

Hi,

I have met a problem after updating the trex to v3.03.

OS:

Linux version 4.4.0-142-generic (buildd@lgw01-amd64-033) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.10) ) #168-Ubuntu SMP Wed Jan 16 21:00:45 UTC 2019

TRex configuration:

root@ubuntu:~# cat /etc/trex_qos_std_police_v4.yaml
### Config file generated by dpdk_setup_ports.py ###

- version: 2
  interfaces: ['04:00.0', '0c:00.0']
  stack: linux_based
  prefix: qos_std_police_v4_1690887282
  zmq_pub_port: 4501
  zmq_rpc_port: 4500
  port_info:
      - ip: 11.0.6.3
        default_gw: 11.0.6.1
      - ip: 12.0.6.3
        default_gw: 12.0.6.1

  platform:
      master_thread_id: 0
      latency_thread_id: 1
      dual_if:
        - socket: 0
          threads: [2]

It crashed when I started the trex instance with astf mode，shown as following:

root@ubuntu:/home/hillstone/v3.03# ./t-rex-64 -i --astf --cfg /etc/trex_qos_std_police_v4.yaml
Starting Cmds server.... Cmds server is started
The ports are bound/configured.
Starting  TRex v3.03 please wait  ...
 set driver name net_vmxnet3
 driver capability  : TSO  LRO
 set dpdk queues mode to ONE_QUE
 Number of ports found: 2
zmq publisher at: tcp://*:4501
 wait 1 sec .
port : 0
------------
link         :  link : Link Up - speed 10000 Mbps - full-duplex
promiscuous  : 0
port : 1
------------
link         :  link : Link Up - speed 10000 Mbps - full-duplex
promiscuous  : 0
 number of ports         : 2
 max cores for 2 ports   : 1
 tx queues per port      : 3
 -------------------------------
RX core uses TX queue number 65535 on all ports
 core, c-port, c-queue, s-port, s-queue, lat-queue
 ------------------------------------------
 1        0      0       1       0      0  
 -------------------------------
Cleanup of old namespaces related to Linux-based stack
Cleanup Done
assert: ../../src/stx/common/trex_stack_base.cpp:387 bool CStackBase::has_port(const string&) Assertion 'm_port_node!=nullptr' failed.

*** traceback follows ***

1       0x55825d5626b5 __assert_fail + 741
2       0x55825d6e4919 ./_t-rex-64-o(+0x37b919) [0x55825d6e4919]
3       0x55825d6d464b CRxCore::has_port(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 75
4       0x55825d6b6ec5 TrexRxSetL3Mode::handle(CRxCore*) + 101
5       0x55825d6d3383 CRxCore::periodic_check_for_cp_messages() + 195
6       0x55825d6d59fa CRxCore::work_tick() + 426
7       0x55825d794706 CRxAstfCore::_do_start() + 438
8       0x55825d6d246f CRxCore::start() + 399
9       0x55825d4bedfd CGlobalTRex::run_in_rx_core() + 117
10      0x55825d4e624a ./_t-rex-64-o(+0x17d24a) [0x55825d4e624a]
11      0x55825d90a791 eal_thread_loop + 433
12      0x7f615e1136ba /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f615e1136ba]
13      0x7f615d18a41d clone + 109


*** addr2line information follows ***

??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0


terminate called after throwing an instance of 'TrexException'
  what():  internal error: failed to get response from remote core for more than '1000' ms
./t-rex-64: line 106:  9043 Aborted                 ./_t-rex-64-o $INPUT_ARGS $EXTRA_INPUT_ARGS
Killing Cmds server... Cmds server is killed

And I have tried v2.93, and it worked normally. Any idea with this?

Thanks

[ethan...@gmail.com]

Hi Hanoch,

Could you help me with this? It happens very frequently, and I have to reboot the linux pc to get it work again.

Thanks

[hanoh haim]

Hi Etan, 

Can you check without the "stack: linux_based", is it required ?

thanks

Hanoh

[ethan...@gmail.com]

Hi Hanoch,

I have tried without "stack: linux_based" and it started normally. 

And yes, I also need ipv6 traffic, which requires linux_based stack

[hanoh haim]

Hi Ethan, 

"linux_stack" used the LXE linux support, I would try with CentOS Or redhat or older Ubuntu version 

Hanoh

--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/1e2eeffb-4cfc-4494-9dd2-ae39f972523cn%40googlegroups.com.

--

Hanoh

Sent from my iPhone

[ethan...@gmail.com]

Hi,

But v2.93 works well on the same machine. If this is an os related problem, it would also crash in v2.93,  right?

Thanks

[hanoh haim]

Hi Ethan, 

Sorry for the late replay, 

You are right. we didn't change anything that related to this in the code. 

Thanks

Hanoh

[ethan...@gmail.com]

Hi Hanoch,

Recently I looked into this again, and I found that the problem might be related to this commit  fix #746 : added a command executor process for linux based stack · cisco-system-traffic-generator/trex-core@64e4666 (github.com)

I revert the commit and compile TRex locally, and it works well.

Hope you can take a look.

Thanks

[hanoh haim]

Hi Ethan

thanks you for identifying the commit that initiated the problem, 

I've opened an issue 

https://github.com/cisco-system-traffic-generator/trex-core/issues/1065

I looked into the code and can't see right away what is the issue. 

Once we will have a time we will look into this and reproduce 

Thanks

Hanoh