Increase throughput in t-rex ASTF-mode

Robert Blomberg

unread,

Sep 20, 2017, 1:24:54 PM9/20/17

to TRex Traffic Generator

Hi guys!

I am not getting any good throughput on latest t-rex build(v2.30), any suggestions on how to increase it?

Using Mellanox CX5 with latest firmware on the card.

Using following command:
./t-rex-64 -f astf/http_simple.py -m 10000 -d 1000000 --astf -l 10000

This is the config I am using:

### Config file generated by dpdk_setup_ports.py ###

- port_limit: 2

version: 2

interfaces: ['04:00.0', '04:00.1']

#'82:00.0', '82:00.1']

# c: 14

limit_memory: 1024

# memory:

# mbuf_2048: 65536

# traffic_mbuf_2048: 65536

port_info:

- dest_mac: 7c:fe:90:57:73:31 # MAC OF LOOPBACK TO IT'S DUAL INTERFACE

src_mac: 7c:fe:90:57:73:30

- dest_mac: 7c:fe:90:57:73:30 # MAC OF LOOPBACK TO IT'S DUAL INTERFACE

src_mac: 7c:fe:90:57:73:31

- dest_mac: 7c:fe:90:57:73:33 # MAC OF LOOPBACK TO IT'S DUAL INTERFACE

src_mac: 7c:fe:90:57:73:32

- dest_mac: 7c:fe:90:57:73:32 # MAC OF LOOPBACK TO IT'S DUAL INTERFACE

src_mac: 7c:fe:90:57:73:33

platform:

master_thread_id: 0

latency_thread_id: 2

dual_if:

- socket: 0

threads: [4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70]

- socket: 1

threads: [5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71]

Yaroslav Brustinov

unread,

Sep 21, 2017, 6:40:47 AM9/21/17

to Robert Blomberg, TRex Traffic Generator

Hi, Robert.

Please define "good" :)

Thanks,

Yaroslav.

--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+unsubscribe@googlegroups.com.
To post to this group, send email to trex...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/4d9b2c95-4481-4172-b98d-f0535ce4b62f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Robert Blomberg

unread,

Sep 21, 2017, 7:05:23 AM9/21/17

to TRex Traffic Generator

Hi!

Sorry for cryptic-question.

On previous version I had throughput on at least 5 - 10 Gbps, now I only have 30 - 40 Mbps with http_simple.py.

/Robert

To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+u...@googlegroups.com.

hanoh haim

unread,

Sep 23, 2017, 12:54:04 PM9/23/17

to Robert Blomberg, TRex Traffic Generator

Please send the output of tcp counters and main dashboard.

Hanoh

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/aa3846f1-9878-4f07-b646-d0455dd90e0c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Hanoh

Sent from my iPhone

Robert Blomberg

unread,

Sep 23, 2017, 3:44:07 PM9/23/17

to TRex Traffic Generator

-Per port stats table

ports | 0 | 1

-----------------------------------------------------------------------------------------

opackets | 22985748 | 571979

obytes | 1786024632 | 37750614

ipackets | 571976 | 22985827

ibytes | 37750416 | 1786030758

ierrors | 0 | 0

oerrors | 0 | 0

Tx Bw | 21.54 Mbps | 527.56 Kbps

-Global stats enabled

Cpu Utilization : 100.0 % 0.0 Gb/core

Platform_factor : 1.0

Total-Tx : 22.07 Mbps

Total-Rx : 22.07 Mbps

Total-PPS : 35.67 Kpps

Total-CPS : 11.22 Kcps

Expected-PPS : 0.00 pps

Expected-CPS : 0.00 cps

Expected-L7-BPS : 0.00 bps

Active-flows : 3747599 Clients : 256 Socket-util : 23.2451 %

Open-flows : 8070924 Servers : 65536 Socket : 3747599 Socket/Clients : 14639.1

drop-rate : 0.00 bps

current time : 574.1 sec

test duration : 999425.9 sec

-Latency stats enabled

Cpu Utilization : 0.1 %

if| tx_ok , rx_ok , rx check ,error, latency (usec) , Jitter max window

| , , , , average , max , (usec)

----------------------------------------------------------------------------------------------------------------

0 | 571984, 0, 0, 0, 0 , 0, 0 | 0 0 0 0 0 0 0 0 0 0 0 0 0

1 | 571984, 0, 0, 0, 0 , 0, 0 | 0 0 0 0 0 0 0 0 0 0 0 0 0

*** TRex is shutting down - cause: 'CTRL + C detected'

All cores stopped !!

latency daemon has stopped

AND

| client | server |

-----------------------------------------------------------------------------------------

m_active_flows | 2691610 | 0 | active flows

m_est_flows | 0 | 0 | active est flows

m_tx_bw_l7_r | 0.00 bps | 0.00 bps | tx L7 bw acked

m_tx_bw_l7_total_r | 0.00 bps | 0.00 bps | tx L7 bw total

m_rx_bw_l7_r | 0.00 bps | 0.00 bps | rx L7 bw acked

m_tx_pps_r | 47.10 Kpps | 0.00 pps | tx pps

m_rx_pps_r | 0.00 pps | 0.00 pps | rx pps

m_avg_size | 0.00 B | 0.00 B | average pkt size

m_tx_ratio | 0.00 %% | 0.00 %% | Tx acked/sent ratio

- | --- | --- |

TCP | --- | --- |

- | --- | --- |

tcps_connattempt | 2691610 | 0 | connections initiated

tcps_segstimed | 2691610 | 0 | segs where we tried to get rtt

tcps_sndtotal | 6275981 | 0 | total packets sent

tcps_sndbyte | 670210890 | 0 | data bytes sent by application

tcps_sndctrl | 6275981 | 0 | control (SYN|FIN|RST) packets sent

tcps_rexmttimeo | 3584371 | 0 | *retransmit timeouts

- | --- | --- |

Flow Table | --- | --- |

- | --- | --- |

*** TRex is shutting down - cause: 'CTRL + C detected'

Anything missing?

hanoh haim

unread,

Sep 23, 2017, 3:48:57 PM9/23/17

to Robert Blomberg, TRex Traffic Generator

It the same issue you had in the first email.

The SYN packets are not answered for some reason. This is the reason for low performance.

Is this Mellanox?

Try to work with CentOS 7.3 and latest OFED

Thanks,

Hanoh

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/77d80633-66ac-4963-86f7-d97da1c850e1%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

hanoh haim

unread,

Sep 24, 2017, 4:36:26 AM9/24/17

to Robert Blomberg, TRex Traffic Generator

This is the output from our CX-5 setup.

Core is 2Ghz

processor : 27

vendor_id : GenuineIntel

cpu family : 6

model : 79

model name : Intel(R) Xeon(R) CPU E5-2660 v4 @ 2.00GHz

stepping : 1

microcode : 0xb00001d

cpu MHz : 1200.078

cache size : 35840 KB

physical id : 1

siblings : 14

core id : 14

cpu cores : 14

apicid : 60

initial apicid : 60

fpu : yes

fpu_exception : yes

cpuid level : 20

wp : yes

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch ida arat epb pln pts dtherm intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdseed adx smap xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local

bogomips : 3996.40

clflush size : 64

cache_alignment : 64

address sizes : 46 bits physical, 48 bits virtual

power management:

$cat /etc/redhat-release

CentOS Linux release 7.3.1611 (Core)

$ofed_info

MLNX_OFED_LINUX-4.1-1.0.2.0 (OFED-4.1-1.0.2):

$ibv_devinfo

hca_id: mlx5_1

transport: InfiniBand (0)

fw_ver: 16.20.1010

node_guid: 248a:0703:00a2:f381

sys_image_guid: 248a:0703:00a2:f380

vendor_id: 0x02c9

vendor_part_id: 4121

hw_ver: 0x0

board_id: MT_0000000013

phys_port_cnt: 1

Device ports:

port: 1

state: PORT_ACTIVE (4)

max_mtu: 4096 (5)

active_mtu: 4096 (5)

sm_lid: 0

port_lid: 0

port_lmc: 0x00

link_layer: Ethernet

hca_id: mlx5_0

transport: InfiniBand (0)

fw_ver: 16.20.1010

node_guid: 248a:0703:00a2:f380

sys_image_guid: 248a:0703:00a2:f380

vendor_id: 0x02c9

vendor_part_id: 4121

hw_ver: 0x0

board_id: MT_0000000013

phys_port_cnt: 1

Device ports:

port: 1

state: PORT_ACTIVE (4)

max_mtu: 4096 (5)

active_mtu: 4096 (5)

sm_lid: 0

port_lid: 0

port_lmc: 0x00

link_layer: Ethernet

$more /etc/trex_cfg.yaml

### Config file generated by dpdk_setup_ports.py ###

- port_limit: 2

version: 2

interfaces: ['04:00.0', '04:00.1']

port_info:

- ip: 1.1.1.1

default_gw: 2.2.2.2

- ip: 2.2.2.2

default_gw: 1.1.1.1

platform:

master_thread_id: 0

latency_thread_id: 14

dual_if:

- socket: 0

threads: [1,2,3,4,5,6,7,8,9,10,11,12,13]

[csi-trex-19]> sudo ./t-rex-64 -f astf/http_simple.py -m 20000 -d 1000 -l 1000 --astf -c 1

converting astf profile astf/http_simple.py to json /tmp/astf.json

./astf-sim -f astf/http_simple.py --json > /tmp/astf.json

The ports are bound/configured.

Starting TRex v2.30 please wait ...

because it is only one core with 2 dual ports you can reach

~8.5Gb/sec with 150,000 active flows

thanks,

Hanoh

On Wed, Sep 20, 2017 at 8:24 PM, Robert Blomberg <robert.b...@gmail.com> wrote:

--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.

To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+unsubscribe@googlegroups.com.

To post to this group, send email to trex...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/4d9b2c95-4481-4172-b98d-f0535ce4b62f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Reply all

Reply to author

Forward