Total-tx-bytes and Total-rx-pkt showing huge values

Natarajan M

unread,

Mar 1, 2019, 1:20:14 PM3/1/19

to TRex Traffic Generator

I'm running in ASTF mode across two different VMs where Total-tx-bytes & Total-rx-pkt on server side shows huge values. This is a lentos VM with VMXNET3 driver. When I run with 2 bare metal servers don't see such issue. Any pointers will really help.

Total-tx-bytes : 22327011130238056 bytes

Total-rx-pkt : 6148340669024846825 pkts

[root@localhost v2.53]# lsmod | grep vmxnet3

vmxnet3 53866 0

[root@localhost v2.53]# uname -a

Linux localhost.localdomain 3.10.0-862.el7.x86_64 #1 SMP Fri Apr 20 16:44:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

[root@localhost v2.53]#

[root@localhost v2.53]# nice -n -19 ./t-rex-64 --iom 0 -c 1 -d 100 -f astf/http_simple2.py --astf --astf-server-only -m 0.1 --cfg /etc/trex_cfg.yml --no-key

converting astf profile astf/http_simple2.py to json /tmp/astf.json

The ports are bound/configured.

Starting TRex v2.53 please wait ...

EAL: Started without hugepages support, physical addresses not available

set driver name net_af_packet

driver capability :

set dpdk queues mode to ONE_QUE

Number of ports found: 2 (dummy among them: 1)

zmq publisher at: tcp://*:4500

wait 1 sec .

port : 0

------------

link : link : Link Up - speed 10000 Mbps - full-duplex

promiscuous : 0

number of ports : 2

max cores for 2 ports : 1

tx queues per port : 3

-------------------------------

RX core uses TX queue number 65535 on all ports

core, c-port, c-queue, s-port, s-queue, lat-queue

------------------------------------------

1 0 0 1 0 0

-------------------------------

Using json file /tmp/astf.json

*** TRex is shutting down - cause: 'test has ended'

latency daemon has stopped

All cores stopped !!

==================

interface sum

==================

---------------

port : 0

------------

opackets : 134

obytes : 7772

ipackets : 137

ibytes : 10626

Tx : 382.27 bps

Cpu Utilization : 0.0 %

Platform_factor : 1.0

Total-Tx : 382.27 bps

Total-Rx : 514.09 bps

Total-PPS : 0.82 pps

Total-CPS : 0.00 cps

Expected-PPS : 0.00 pps

Expected-CPS : 0.00 cps

Expected-L7-BPS : 0.00 bps

Active-flows : 0 Clients : 255 Socket-util : 0.0000 %

Open-flows : 0 Servers : 65535 Socket : 0 Socket/Clients : 0.0

drop-rate : 0.00 bps

==================

interface sum

==================

------------------------

per core stats core id : 1

------------------------

per core per if stats id : 1

------------------------

port 0, queue id :0 - client

----------------------------

port 1, queue id :0 - server

----------------------------

==================

generators

==================

normal

-------------

min_delta : 10 usec

cnt : 0

high_cnt : 0

max_d_time : 0 usec

sliding_average : 0 usec

precent : -nan %

histogram

-----------

---------------

port : 0

------------

opackets : 134

obytes : 7772

ipackets : 137

ibytes : 10626

Tx : 382.27 bps

Cpu Utilization : 0.0 %

Platform_factor : 1.0

Total-Tx : 382.27 bps

Total-Rx : 514.09 bps

Total-PPS : 0.82 pps

Total-CPS : 0.00 cps

Expected-PPS : 0.00 pps

Expected-CPS : 0.00 cps

Expected-L7-BPS : 0.00 bps

Active-flows : 0 Clients : 255 Socket-util : 0.0000 %

Open-flows : 0 Servers : 65535 Socket : 0 Socket/Clients : 0.0

drop-rate : 0.00 bps

summary stats

--------------

Total-pkt-drop : 0 pkts

Warning : number of rx packets exceeds 101% of tx packets!

Total-tx-bytes : 22327011130237360 bytes

Total-tx-sw-bytes : 0 bytes

Total-rx-bytes : 10703 byte

Total-tx-pkt : 141 pkts

Total-rx-pkt : 6148340669024846813 pkts

Total-sw-tx-pkt : 0 pkts

Total-sw-err : 0 pkts

Total ARP sent : 2 pkts

Total ARP received : 1 pkts

| client | server |

-----------------------------------------------------------------------------------------

m_active_flows | 0 | 0 | active open flows

m_est_flows | 0 | 0 | active established flows

m_tx_bw_l7_r | 0.00 bps | 0.00 bps | tx L7 bw acked

m_tx_bw_l7_total_r | 0.00 bps | 0.00 bps | tx L7 bw total

m_rx_bw_l7_r | 0.00 bps | 0.00 bps | rx L7 bw acked

m_tx_pps_r | 0.00 pps | 0.00 pps | tx pps

m_rx_pps_r | 0.00 pps | 0.00 pps | rx pps

m_avg_size | 0.00 B | 0.00 B | average pkt size

m_tx_ratio | 0.00 % | 0.00 % | Tx acked/sent ratio

- | --- | --- |

TCP | --- | --- |

- | --- | --- |

UDP | --- | --- |

- | --- | --- |

Flow Table | --- | --- |

- | --- | --- |

err_cwf | 134 | 0 | *client pkt without flow

redirect_rx_ok | 3 | 0 | redirect to rx OK

[root@localhost v2.53]# nice -n -19 ./t-rex-64 --iom 0 -c 1 -d 100 -f astf/http_simple2.py --astf --astf-server-only -m 0.1 --cfg /etc/trex_cfg.yml --no-key --software

converting astf profile astf/http_simple2.py to json /tmp/astf.json

The ports are bound/configured.

Starting TRex v2.53 please wait ...

EAL: Started without hugepages support, physical addresses not available

set driver name net_af_packet

driver capability :

set dpdk queues mode to ONE_QUE

Number of ports found: 2 (dummy among them: 1)

zmq publisher at: tcp://*:4500

wait 1 sec .

port : 0

------------

link : link : Link Up - speed 10000 Mbps - full-duplex

promiscuous : 0

number of ports : 2

max cores for 2 ports : 1

tx queues per port : 3

-------------------------------

RX core uses TX queue number 65535 on all ports

core, c-port, c-queue, s-port, s-queue, lat-queue

------------------------------------------

1 0 0 1 0 0

-------------------------------

Using json file /tmp/astf.json

*** TRex is shutting down - cause: 'test has ended'

latency daemon has stopped

All cores stopped !!

==================

interface sum

==================

---------------

port : 0

------------

opackets : 146

obytes : 8468

ipackets : 149

ibytes : 11562

Tx : 392.35 bps

Cpu Utilization : 0.0 %

Platform_factor : 1.0

Total-Tx : 392.35 bps

Total-Rx : 527.65 bps

Total-PPS : 0.85 pps

Total-CPS : 0.00 cps

Expected-PPS : 0.00 pps

Expected-CPS : 0.00 cps

Expected-L7-BPS : 0.00 bps

Active-flows : 0 Clients : 255 Socket-util : 0.0000 %

Open-flows : 0 Servers : 65535 Socket : 0 Socket/Clients : 0.0

drop-rate : 0.00 bps

==================

interface sum

==================

------------------------

per core stats core id : 1

------------------------

per core per if stats id : 1

------------------------

port 0, queue id :0 - client

----------------------------

port 1, queue id :0 - server

----------------------------

==================

generators

==================

normal

-------------

min_delta : 10 usec

cnt : 0

high_cnt : 0

max_d_time : 0 usec

sliding_average : 0 usec

precent : -nan %

histogram

-----------

---------------

port : 0

------------

opackets : 146

obytes : 8468

ipackets : 149

ibytes : 11562

Tx : 392.35 bps

Cpu Utilization : 0.0 %

Platform_factor : 1.0

Total-Tx : 392.35 bps

Total-Rx : 527.65 bps

Total-PPS : 0.85 pps

Total-CPS : 0.00 cps

Expected-PPS : 0.00 pps

Expected-CPS : 0.00 cps

Expected-L7-BPS : 0.00 bps

Active-flows : 0 Clients : 255 Socket-util : 0.0000 %

Open-flows : 0 Servers : 65535 Socket : 0 Socket/Clients : 0.0

drop-rate : 0.00 bps

summary stats

--------------

Total-pkt-drop : 0 pkts

Warning : number of rx packets exceeds 101% of tx packets!

Total-tx-bytes : 22327011130238056 bytes

Total-tx-sw-bytes : 0 bytes

Total-rx-bytes : 11639 byte

Total-tx-pkt : 153 pkts

Total-rx-pkt : 6148340669024846825 pkts

Total-sw-tx-pkt : 0 pkts

Total-sw-err : 0 pkts

Total ARP sent : 2 pkts

Total ARP received : 1 pkts

| client | server |

-----------------------------------------------------------------------------------------

m_active_flows | 0 | 0 | active open flows

m_est_flows | 0 | 0 | active established flows

m_tx_bw_l7_r | 0.00 bps | 0.00 bps | tx L7 bw acked

m_tx_bw_l7_total_r | 0.00 bps | 0.00 bps | tx L7 bw total

m_rx_bw_l7_r | 0.00 bps | 0.00 bps | rx L7 bw acked

m_tx_pps_r | 0.00 pps | 0.00 pps | tx pps

m_rx_pps_r | 0.00 pps | 0.00 pps | rx pps

m_avg_size | 0.00 B | 0.00 B | average pkt size

m_tx_ratio | 0.00 % | 0.00 % | Tx acked/sent ratio

- | --- | --- |

TCP | --- | --- |

- | --- | --- |

UDP | --- | --- |

- | --- | --- |

Flow Table | --- | --- |

- | --- | --- |

err_cwf | 146 | 0 | *client pkt without flow

redirect_rx_ok | 3 | 0 | redirect to rx OK

[root@localhost v2.53]#

hanoh haim

unread,

Mar 2, 2019, 5:05:57 PM3/2/19

to Natarajan M, TRex Traffic Generator

Thanks for reporting. This huge number is actually a negative number due to bug in the counting of ARP packets

--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+u...@googlegroups.com.
To post to this group, send email to trex...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/f8f28586-4a94-4946-b6f6-d3bb622e6ad7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Hanoh

Sent from my iPhone

Natarajan M

unread,

Mar 2, 2019, 10:02:22 PM3/2/19

to hanoh haim, TRex Traffic Generator

Thanks for prompt response. Is there a patch that I can apply quickly to workaround this?

Thanks

Natarajan

Natarajan M

unread,

Mar 6, 2019, 2:39:57 PM3/6/19

to hanoh haim, TRex Traffic Generator

I moved to MAC instead of ip in /etc/trex_cfg.yaml to work-around ARP issue for now. But 'Total-rx-bytes' still had some huge random value (sounds like negative stuff). Rest of the stats are fine. Any work-arounds or suggestions possible to handle this?

File stats

=================

*** TRex is shutting down - cause: 'test has ended'

latency daemon has stopped

All cores stopped !!

==================

interface sum

==================

---------------

port : 1

------------

opackets : 8073027

obytes : 11060644992

ipackets : 1794006

ibytes : 203320680

Tx : 321.23 Mbps

Cpu Utilization : 19.0 % 3.4 Gb/core

Platform_factor : 1.0

Total-Tx : 321.23 Mbps

Total-Rx : 5.95 Mbps

Total-PPS : 29.42 Kpps

Total-CPS : 0.00 cps

Expected-PPS : 0.00 pps

Expected-CPS : 0.00 cps

Expected-L7-BPS : 0.00 bps

Active-flows : 0 Clients : 1 Socket-util : 0.0000 %

Open-flows : 0 Servers : 1 Socket : 0 Socket/Clients : 0.0

drop-rate : 315.28 Mbps

port : 1

------------

opackets : 8073027

obytes : 11060644992

ipackets : 1794006

ibytes : 203320680

Tx : 321.23 Mbps

Cpu Utilization : 19.0 % 3.4 Gb/core

Platform_factor : 1.0

Total-Tx : 321.23 Mbps

Total-Rx : 5.95 Mbps

Total-PPS : 29.42 Kpps

Total-CPS : 0.00 cps

Expected-PPS : 0.00 pps

Expected-CPS : 0.00 cps

Expected-L7-BPS : 0.00 bps

Active-flows : 0 Clients : 1 Socket-util : 0.0000 %

Open-flows : 0 Servers : 1 Socket : 0 Socket/Clients : 0.0

drop-rate : 315.28 Mbps

summary stats

--------------

Total-pkt-drop : 6296059 pkts

Total-tx-bytes : 11060644997 bytes

Total-tx-sw-bytes : 0 bytes

Total-rx-bytes : 8246760943118609808 byte

Total-tx-pkt : 8090069 pkts

Total-rx-pkt : 1794010 pkts

Total-sw-tx-pkt : 0 pkts

Total-sw-err : 0 pkts

Total ARP sent : 0 pkts

Total ARP received : 0 pkts

| client | server |

-----------------------------------------------------------------------------------------

m_active_flows | 0 | 0 | active open flows

m_est_flows | 0 | 0 | active established flows

m_tx_bw_l7_r | 0.00 bps | 321.02 Mbps | tx L7 bw acked

m_tx_bw_l7_total_r | 0.00 bps | 304.67 Mbps | tx L7 bw total

m_rx_bw_l7_r | 0.00 bps | 2.11 Mbps | rx L7 bw acked

m_tx_pps_r | 0.00 pps | 29.42 Kpps | tx pps

m_rx_pps_r | 0.00 pps | 7.89 Kpps | rx pps

m_avg_size | 0.00 B | 1.08 KB | average pkt size

m_tx_ratio | 0.00 % | 105.37 % | Tx acked/sent ratio

- | --- | --- |

TCP | --- | --- |

- | --- | --- |

tcps_accepts | 0 | 299001 | connections accepted

tcps_connects | 0 | 299001 | connections established

tcps_closed | 0 | 299001 | conn. closed (includes drops)

tcps_segstimed | 0 | 1495005 | segs where we tried to get rtt

tcps_rttupdated | 0 | 1495005 | times we succeeded

tcps_sndtotal | 0 | 8073027 | total packets sent

tcps_sndpack | 0 | 7475025 | data packets sent

tcps_sndbyte | 0 | 10493141094 | data bytes sent by application

tcps_sndbyte_ok | 0 | 10493141094 | data bytes sent by tcp

tcps_sndacks | 0 | 598002 | ack-only packets sent

tcps_rcvpack | 0 | 598002 | packets received in sequence

tcps_rcvbyte | 0 | 75348252 | bytes received in sequence

tcps_rcvackpack | 0 | 1495005 | rcvd ack packets

tcps_rcvackbyte | 0 | 10493141094 | tx bytes acked by rcvd acks

tcps_rcvackbyte_of | 0 | 598002 | tx bytes acked by rcvd acks - overflow acked

- | --- | --- |

UDP | --- | --- |

- | --- | --- |

Flow Table | --- | --- |

- | --- | --- |

err_rx_throttled | 0 | 1 | rx thread was throttled

[root@localhost v2.53]#

Thanks

Natarajan

hanoh haim

unread,

Mar 7, 2019, 6:43:14 AM3/7/19

to Natarajan M, TRex Traffic Generator

We will provide a fix for that.

Please open a github issue. no need to remove the ARP

besi7...@gmail.com

unread,

Mar 7, 2019, 9:27:54 AM3/7/19

to TRex Traffic Generator

Hello,
It seems like we had some uninitialized variables in the code. A quick fix can be found at
https://github.com/bdollma/trex-core/tree/Uninitialized_CPhyEthIFStats
and will probably be pushed in the next release.
Thanks for the report.
Best,
Bes

Natarajan M

unread,

Mar 7, 2019, 12:36:30 PM3/7/19

to besi7...@gmail.com, TRex Traffic Generator

Thanks for quick turnaround.

--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+u...@googlegroups.com.
To post to this group, send email to trex...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/d6c2f842-3118-4f08-9bf8-cc6a0a5893d3%40googlegroups.com.

Reply all

Reply to author

Forward