t-rex process using 100% CPU when idle (no traffic)

739 views
Skip to first unread message

Alec Hothan

unread,
Apr 27, 2017, 12:23:11 PM4/27/17
to TRex Traffic Generator

Hi,

Is there a way to quiesce the TRex process when there is no more traffic being generated (other than killing the Trex process)?


Tasks: 669 total,   1 running, 668 sleeping,   0 stopped,   0 zombie
%Cpu(s):  2.7 us,  0.1 sy,  0.0 ni, 96.9 id,  0.3 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 26392556+total, 17957105+free, 14804284 used, 69550216 buff/cache
KiB Swap: 33554428 total, 33554428 free,        0 used. 24791838+avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
25234 root      20   0 8992872  10228   3552 S 104.3  0.0   1:05.42 _t-rex-64
13262 2020      20   0 24.796g 2.904g  40836 S  14.0  1.2 240:45.92 java
13202 2021      20   0 16.544g 911468  16044 S   9.3  0.3 176:47.63 java
12580 root      20   0 2051780  35620   8452 S   4.7  0.0   1:15.81 docker-current
  252 root      20   0       0      0      0 S   0.3  0.0   0:00.20 khungtaskd

Unclear to me if that is a DPDK PMD thread using all the CPU or another TRex specific thread?

We're planning to run TRex in a container on long-lived production servers that are shared to do other important monitoring work so it is better to relinquish the CPU when there is no more traffic being generated.

Thanks

  Alec

hanoh haim

unread,
Apr 27, 2017, 12:38:07 PM4/27/17
to Alec Hothan, TRex Traffic Generator
Hi Alec,
Can you send the output of 
$ top -p <trex-pid> -H 

Hanoh
--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+u...@googlegroups.com.
To post to this group, send email to trex...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/b5b4b764-d0ef-4680-9f17-7408beb783b3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
Hanoh
Sent from my iPhone

Itay Marom

unread,
Apr 27, 2017, 12:56:19 PM4/27/17
to hanoh haim, Alec Hothan, TRex Traffic Generator
Hi Alec,

another thing:
can you please connect through the console and reset the ports:

./trex-console
reset

I am guessing this is the RX core and even if there is no traffic, there are some latency streams from previous run
causing the RX core to run at full speed not to have a jitter when a RX packet arrives.

please see:
https://trex-tgn.cisco.com/youtrack/issue/trex-380

we can verify this if you log and simply reset the ports.

Thanks,
Itay




On Thu, Apr 27, 2017 at 7:37 PM, hanoh haim <hhaim...@gmail.com> wrote:
Hi Alec,
Can you send the output of 
$ top -p <trex-pid> -H 

Hanoh

On Thu, 27 Apr 2017 at 19:23 Alec Hothan <aho...@gmail.com> wrote:

Hi,

Is there a way to quiesce the TRex process when there is no more traffic being generated (other than killing the Trex process)?


Tasks: 669 total,   1 running, 668 sleeping,   0 stopped,   0 zombie
%Cpu(s):  2.7 us,  0.1 sy,  0.0 ni, 96.9 id,  0.3 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 26392556+total, 17957105+free, 14804284 used, 69550216 buff/cache
KiB Swap: 33554428 total, 33554428 free,        0 used. 24791838+avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
25234 root      20   0 8992872  10228   3552 S 104.3  0.0   1:05.42 _t-rex-64
13262 2020      20   0 24.796g 2.904g  40836 S  14.0  1.2 240:45.92 java
13202 2021      20   0 16.544g 911468  16044 S   9.3  0.3 176:47.63 java
12580 root      20   0 2051780  35620   8452 S   4.7  0.0   1:15.81 docker-current
  252 root      20   0       0      0      0 S   0.3  0.0   0:00.20 khungtaskd

Unclear to me if that is a DPDK PMD thread using all the CPU or another TRex specific thread?

We're planning to run TRex in a container on long-lived production servers that are shared to do other important monitoring work so it is better to relinquish the CPU when there is no more traffic being generated.

Thanks

  Alec

--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+unsubscribe@googlegroups.com.

To post to this group, send email to trex...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/b5b4b764-d0ef-4680-9f17-7408beb783b3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
Hanoh
Sent from my iPhone

--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+unsubscribe@googlegroups.com.

To post to this group, send email to trex...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/CA%2BYxBoJdrdfLm%3D_3wst-RSTeP8wZJL43ZE7K%2BoAtx33gcb9uVw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.



--
Itay Marom

Alec Hothan

unread,
Apr 27, 2017, 1:22:53 PM4/27/17
to TRex Traffic Generator, aho...@gmail.com
here you go:

top - 10:22:05 up 22:31,  1 user,  load average: 1.68, 1.51, 1.54
Threads:  13 total,   1 running,  12 sleeping,   0 stopped,   0 zombie
%Cpu(s):  2.7 us,  0.1 sy,  0.0 ni, 97.0 id,  0.3 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 26392556+total, 17876483+free, 14804148 used, 70356576 buff/cache
KiB Swap: 33554428 total, 33554428 free,        0 used. 24791769+avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
25240 root      20   0 8992872  10276   3580 R 99.9  0.0  68:16.64 TRex RX
25234 root      20   0 8992872  10276   3580 S  4.0  0.0   3:22.76 _t-rex-64
25235 root      20   0 8992872  10276   3580 S  0.0  0.0   0:00.00 vfio-sync
25236 root      20   0 8992872  10276   3580 S  0.0  0.0   0:00.00 eal-intr-thread
25237 root      20   0 8992872  10276   3580 S  0.0  0.0   0:00.54 Trex DP core 1
25238 root      20   0 8992872  10276   3580 S  0.0  0.0   0:00.54 Trex DP core 2
25239 root      20   0 8992872  10276   3580 S  0.0  0.0   0:00.55 Trex DP core 3
25244 root      20   0 8992872  10276   3580 S  0.0  0.0   0:00.00 Trex Publisher
25245 root      20   0 8992872  10276   3580 S  0.0  0.0   0:00.00 Trex Publisher
25248 root      20   0 8992872  10276   3580 S  0.0  0.0   0:00.09 Trex ZMQ sync
25249 root      20   0 8992872  10276   3580 S  0.0  0.0   0:00.09 Trex Watchdog
25250 root      20   0 8992872  10276   3580 S  0.0  0.0   0:00.00 Trex ZMQ sync
25251 root      20   0 8992872  10276   3580 S  0.0  0.0   0:00.00 Trex ZMQ sync

Itay Marom

unread,
Apr 27, 2017, 1:24:49 PM4/27/17
to Alec Hothan, TRex Traffic Generator
yep...
the RX core.

you are probably hitting:
https://trex-tgn.cisco.com/youtrack/issue/trex-380

give a 'reset' a try if you can.

Thanks,
Itay



To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+unsubscribe@googlegroups.com.

To post to this group, send email to trex...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/f9668bb6-39ee-4bf8-afca-f364d2545bbf%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
Itay Marom

hanoh haim

unread,
Apr 27, 2017, 1:25:45 PM4/27/17
to Alec Hothan, TRex Traffic Generator
Hi Alec,
As Itay suggested try removing the streams (using reset) 

The issue is that even if there is no traffic RX try to be ready for packets in low latency.


Hanoh

For more options, visit https://groups.google.com/d/optout.

Alec Hothan

unread,
Apr 27, 2017, 6:05:59 PM4/27/17
to TRex Traffic Generator, aho...@gmail.com

We actually call the reset() API at the end of each run.
I also tried to reset from the console and it does not seem to help.


trex>reset

Resetting ports [0, 1]:                                      [SUCCESS]

38.24 [ms]


top - 18:03:19 up 1 day,  3:12,  0 users,  load average: 1.22, 1.21, 1.30
Threads:  19 total,   3 running,  16 sleeping,   0 stopped,   0 zombie
%Cpu(s):  2.9 us,  0.1 sy,  0.0 ni, 96.2 id,  0.8 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 26392556+total, 17511104+free, 14832012 used, 73982504 buff/cache
KiB Swap: 33554428 total, 33554428 free,        0 used. 24788889+avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
  362 root      20   0 8992872  10280   3580 R 99.9  0.0 349:16.32 TRex RX
  356 root      20   0 8992872  10280   3580 R  5.0  0.0  17:01.30 _t-rex-64
    1 root      20   0   11628   1380   1176 S  0.0  0.0   0:00.03 init.sh
   36 root      20   0    4344    348    276 S  0.0  0.0   0:00.00 tail

Thanks

  Alec


hanoh haim

unread,
Apr 28, 2017, 1:07:07 AM4/28/17
to Alec Hothan, TRex Traffic Generator

void CRxCoreStateless::recalculate_next_state() {
    if (m_state == STATE_QUIT) {
        return;
    }

    /* only latency requires the 'hot' state */
    m_state = (is_latency_or_capture_active() ? STATE_HOT : STATE_COLD);
}

Hi Alec,
There are two conditions for HOT state 
1. Latency stream is assigned to one of the ports
2. Capture/service mode is enabled

after a few seconds without these conditions, it should move to cold mode

1. You could attached gdb to your process to check which condition is On
Or 
2. Describe a way to reconstruct this and we will look into this.

Thanks,
Hanoh


--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+u...@googlegroups.com.
To post to this group, send email to trex...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Alec Hothan

unread,
Apr 28, 2017, 12:22:32 PM4/28/17
to TRex Traffic Generator, aho...@gmail.com

We actually just need to start trex manually and it uses 100% CPU right away before we even start to send any traffic.


# ./t-rex-64 -i —no-scapy-server

Error: current path is not readable by user "nobody" (starting at /root).
Two possible solutions:

  1. (Recommended)
     Copy TRex to some public location (/tmp or /scratch, assuming it has proper permissions (chmod 777 etc.))

  2. (Not recommended)
     Change permissions of current path. (Starting from directory /root).
     chmod 777 /root -R

Could not start scapy_daemon_server, which is needed by GUI to create packets.
If you don't need it, use --no-scapy-server flag.
Starting  TRex v2.18 please wait  ...
zmq publisher at: tcp://*:4500
 Number of ports found: 2
 set driver name net_i40e
 wait 1 sec .
port : 0
------------
link         :  link : Link Up - speed 10000 Mbps - full-duplex
promiscuous  : 0
port : 1
------------
link         :  link : Link Up - speed 10000 Mbps - full-duplex
promiscuous  : 0
 -------------------------------
RX core uses TX queue number 1 on all ports
 core, c-port, c-queue, s-port, s-queue, lat-queue
 ------------------------------------------
 1        0      0       1       0      2
 -------------------------------
 number of ports         : 2
 max cores for 2 ports   : 1
 max queue per port      : 3


-Per port stats table
      ports |               0 |               1
 -----------------------------------------------------------------------------------------
   opackets |               0 |               0
     obytes |               0 |               0
   ipackets |               0 |               0
     ibytes |               0 |               0
    ierrors |               0 |               0
    oerrors |               0 |               0
      Tx Bw |       0.00  bps |       0.00  bps

-Global stats enabled
 Cpu Utilization : 0.0  %  0.0 Gb/core
 Platform_factor : 1.0
 Total-Tx        :       0.00  bps
 Total-Rx        :       0.00  bps
 Total-PPS       :       0.00  pps
 Total-CPS       :       0.00  cps

 Expected-PPS    :       0.00  pps
 Expected-CPS    :       0.00  cps
 Expected-BPS    :       0.00  bps

 Active-flows    :        0  Clients :        0   Socket-util : 0.0000 %
 Open-flows      :        0  Servers :        0   Socket :        0 Socket/Clients :  -nan
 drop-rate       :       0.00  bps
 current time    : 3.3 sec
 test duration   : 0.0 sec

Threads:  11 total,   1 running,  10 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.7 us,  0.1 sy,  0.0 ni, 97.8 id,  0.4 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 26392556+total, 17229406+free, 14855220 used, 76776272 buff/cache
KiB Swap: 33554428 total, 33554428 free,        0 used. 24784984+avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
 3828 root      20   0 8854488  10064   3744 R 99.9  0.0   0:23.06 TRex RX
 3823 root      20   0 8854488  10064   3744 S  0.0  0.0   0:05.22 _t-rex-64
 3824 root      20   0 8854488  10064   3744 S  0.0  0.0   0:00.00 vfio-sync
 3826 root      20   0 8854488  10064   3744 S  0.0  0.0   0:00.00 eal-intr-thread
 3827 root      20   0 8854488  10064   3744 S  0.0  0.0   0:00.01 Trex DP core 1
 3830 root      20   0 8854488  10064   3744 S  0.0  0.0   0:00.00 Trex Publisher
 3831 root      20   0 8854488  10064   3744 S  0.0  0.0   0:00.00 Trex Publisher
 3832 root      20   0 8854488  10064   3744 S  0.0  0.0   0:00.00 Trex ZMQ sync
 3833 root      20   0 8854488  10064   3744 S  0.0  0.0   0:00.00 Trex Watchdog
 3834 root      20   0 8854488  10064   3744 S  0.0  0.0   0:00.00 Trex ZMQ sync
 3835 root      20   0 8854488  10064   3744 S  0.0  0.0   0:00.00 Trex ZMQ sync


We're using 2.18
# cat /etc/trex_cfg.yaml
# Config generated by NFVBench tool
        - port_limit : 2
          version    : 2
          interfaces : ['0a:00.0','0a:00.1']
          port_info  :
              - dest_mac : 00:11:94:00:0A:00
                src_mac  : 00:10:94:00:0A:00
                vlan     : 3049
              - dest_mac : 00:10:94:00:0A:00
                src_mac  : 00:11:94:00:0A:00
                vlan     : 3018

# ./dpdk_setup_ports.py -s

Network devices using DPDK-compatible driver
============================================
0000:0a:00.0 'Ethernet Controller X710 for 10GbE SFP+' drv=vfio-pci unused=i40e,igb_uio,uio_pci_generic
0000:0a:00.1 'Ethernet Controller X710 for 10GbE SFP+' drv=vfio-pci unused=i40e,igb_uio,uio_pci_generic


Alec Hothan

unread,
Apr 28, 2017, 12:30:50 PM4/28/17
to TRex Traffic Generator, aho...@gmail.com
Btw, I moved the trex directory under /tmp to get rid of the warning, still same CPU issue.

# ./t-rex-64 -i --no-scapy-server

hanoh haim

unread,
Apr 29, 2017, 1:14:09 PM4/29/17
to Alec Hothan, TRex Traffic Generator
Hi Alec,
Now everything is clear :-)
You are using v2.18 which is very old  in our release process.
From V2.19 there is  fix for this issue (thanks to Itay)

See release notes 
)

Thanks
Hanoh
--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+u...@googlegroups.com.
To post to this group, send email to trex...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Alec Hothan

unread,
May 1, 2017, 11:00:19 AM5/1/17
to TRex Traffic Generator, aho...@gmail.com

Fantastic!
Indeed the release notes does mention it.
We will upgrade to the latest and let you know,

Thanks!

  Alec
Reply all
Reply to author
Forward
0 new messages