Max Tx rate with modifying packets for flows

Andrew Theurer

unread,

Nov 29, 2017, 1:59:49 PM11/29/17

to TRex Traffic Generator

We are using TRex with Intel fortville cards, and we would like to transmit at packet rates beyond our current capability by using more ports.  Right now we use a pair of ports, 1 port per card, and we can transmit at ~36Mpps if we do not make any packet header modifications.  We can do this with a total of 7 cores (5 for Tx).

When we start modifying the headers (src/dst ip/mac), we need a lot more cores to transmit at a similar rate, 14 in total, which I believe is the maximum for TRex,  Eventually we'd like to go to 4 adapters, 1 port each, 2 ports per NUMA node on a 2 socket system.  We would probably need 26 cores to do this (12 Tx per node, plus 1 for Rx and 1 for latency).  I don't think TRex can run beyond 14 cores currently, right?

Is there something we should be doing to modify the packet headers more efficiently, so we can use fewer cores/socket?

I have taken some perf-record samples for both cases.

without packet modification:

    60.60%          1940  Trex DP core 1  _t-rex-64         [.] rte_delay_us_block
    15.87%           508  Trex DP core 1  _t-rex-64         [.] i40e_xmit_pkts
     7.50%           240  Trex DP core 1  _t-rex-64         [.] CNodeGenerator::flush_file_realtime<23, false>
     5.87%           188  Trex DP core 1  _t-rex-64         [.] CCoreEthIFStateless::send_node_common<false>
     4.84%           155  Trex DP core 1  _t-rex-64         [.] CCoreEthIFStateless::send_node_flow_stat
     3.81%           122  Trex DP core 1  _t-rex-64         [.] std::__adjust_heap<__gnu_cxx::__normal_iterator<CGenNode**, std::vector<CGenNode*, std::allocator<CGenNode*> > >, long, CGenNode*, CGenNodeCompare>
     0.66%            21  Trex DP core 1  _t-rex-64         [.] CCoreEthIF::send_burst
     0.38%             1  swapper         [kernel.vmlinux]  [k] vtime_account_idle
     0.22%             7  Trex DP core 1  _t-rex-64         [.] CCoreEthIFStateless::send_node
     0.06%             2  Trex DP core 1  [kernel.vmlinux]  [k] ktime_get
     0.04%             1  Trex DP core 1  [kernel.vmlinux]  [k] fetch_task_cputime
     0.03%             1  Trex DP core 1  [kernel.vmlinux]  [k] trigger_load_balance
     0.03%             1  Trex DP core 1  [kernel.vmlinux]  [k] local_apic_timer_interrupt
     0.03%             1  Trex DP core 1  [kernel.vmlinux]  [k] native_write_msr_safe
     0.03%             1  Trex DP core 1  [kernel.vmlinux]  [k] vtime_account_user
     0.03%             1  Trex DP core 1  [kernel.vmlinux]  [k] _raw_spin_lock
     0.00%             1  swapper         [kernel.vmlinux]  [k] nmi_restore
     0.00%             3  swapper         [kernel.vmlinux]  [k] native_write_msr_safe

with packet modification:

    46.50%          1489  Trex DP core 1  _t-rex-64         [.] StreamDPVmInstructionsRunner::run
    17.24%           552  Trex DP core 1  _t-rex-64         [.] CGenNodeStateless::alloc_node_with_vm
    14.84%           475  Trex DP core 1  _t-rex-64         [.] i40e_xmit_pkts
     4.84%           155  Trex DP core 1  _t-rex-64         [.] CCoreEthIFStateless::send_node_common<false>
     3.12%           100  Trex DP core 1  _t-rex-64         [.] CNodeGenerator::flush_file_realtime<23, false>
     2.97%            95  Trex DP core 1  _t-rex-64         [.] std::__adjust_heap<__gnu_cxx::__normal_iterator<CGenNode**, std::vector<CGenNode*, std::allocator<CGenNode*> > >, long, CGenNode*, CGenNodeCompare>
     2.69%            86  Trex DP core 1  libc-2.17.so      [.] __memcpy_ssse3
     2.69%            86  Trex DP core 1  _t-rex-64         [.] CCoreEthIFStateless::send_node_flow_stat
     2.00%            64  Trex DP core 1  _t-rex-64         [.] common_ring_mp_enqueue
     1.09%            35  Trex DP core 1  _t-rex-64         [.] common_ring_mc_dequeue
     0.56%            18  Trex DP core 1  _t-rex-64         [.] rte_memcpy_generic.constprop.17
     0.41%            13  Trex DP core 1  _t-rex-64         [.] memcpy@plt
     0.31%             1  Trex DP core 1  [kernel.vmlinux]  [k] _raw_spin_lock_irq
     0.25%             8  Trex DP core 1  _t-rex-64         [.] CCoreEthIFStateless::send_node
     0.12%             4  Trex DP core 1  [kernel.vmlinux]  [k] ktime_get
     0.09%             3  Trex DP core 1  _t-rex-64         [.] CCoreEthIF::send_burst
     0.06%             2  Trex DP core 1  [kernel.vmlinux]  [k] trigger_load_balance
     0.03%             1  Trex DP core 1  [kernel.vmlinux]  [k] rcu_bh_qs
     0.03%             1  Trex DP core 1  _t-rex-64         [.] rte_delay_us_block
     0.03%             1  Trex DP core 1  [kernel.vmlinux]  [k] apic_timer_interrupt
     0.03%             1  Trex DP core 1  [kernel.vmlinux]  [k] cpuacct_charge
     0.03%             1  Trex DP core 1  [unknown]         [k] 0x00000000005f080a
     0.03%             1  Trex DP core 1  [kernel.vmlinux]  [k] task_tick_fair
     0.03%             1  Trex DP core 1  [kernel.vmlinux]  [.] irq_return
     0.00%             1  swapper         [kernel.vmlinux]  [k] do_nmi
     0.00%             3  swapper         [kernel.vmlinux]  [k] native_write_msr_safe

Is this what we should expect to see when implementing multiple streams/flows?

Thanks,

-Andrew

Yaroslav Brustinov

unread,

Nov 29, 2017, 2:36:32 PM11/29/17

to Andrew Theurer, TRex Traffic Generator

Hi,

> I don't think TRex can run beyond 14 cores currently, right?

Should be able to...

> pair of ports, 1 port per card

Do the ports share same NUMA? Performance-wise, each pair of ports should belong to same NUMA as CPU core to which they belong.

Please read following sections of docs:

https://trex-tgn.cisco.com/trex/doc/trex_manual.html#_platform_section_configuration

https://trex-tgn.cisco.com/trex/doc/trex_manual.html#_script_for_creating_config_file

Could you share the Python profile you are using to create traffic in this example?

Maybe you can improve the "modify headers" program AKA Field engine by reusing same variable and writing it to different offsets with added/subtracted value.

Did you record with perf only single thread of TRex? There is only 1 core in your screenshots, hope you are using other cores as well...

Thanks,

Yaroslav.

--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+unsubscribe@googlegroups.com.
To post to this group, send email to trex...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/CAD5-U7RLXL1WRiQHL%3DDeN-yUA_OSb6TDoBRHo1FX9We9KgvhYg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Itay Marom

unread,

Nov 29, 2017, 2:53:29 PM11/29/17

to Yaroslav Brustinov, Andrew Theurer, TRex Traffic Generator

Hi Andrew,

Few comments:

1. Perf is problematic with TRex's regular build mode.

this is because delays are not being executed as system calls such as nanosleep.

instead we use pause in a loop - this is way more accurate.

that's why, for example, in your first snapshot, you see rte_delay_us_block 60%

which is probably dead time for delays.

I suggest you do this:

in TRex TUI go to CPU utilization and send the snapshot

see here how:

http://trex-tgn.cisco.com/trex/doc/trex_stateless.html#_performance_tweaking

2. if you really want perf, I've added a compilation flag to allow a more friendly version for PERF:

https://github.com/cisco-system-traffic-generator/trex-core/blob/master/linux_dpdk/ws_main.py#L821

this will switch TRex sleeps to be nanosleep (less accurate but great for debug with perf)

also it will disregard full queue to not confuse perf with retries of TRex.

Regarding the performance issues,

Can you send the CPU util from the TUI and show that all the CPUs are 100% ?

there are alot of tweaking options (cache, pinning, and etc.), but let's verify the problem first...

Itay

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/CAJysGCTwFJaPVmD1kKQ0z8HpPePik%2BbPQ9eHJ65Gv%3DXGoo2_EA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Itay Marom

Andrew Theurer

unread,

Nov 29, 2017, 2:54:49 PM11/29/17

to Yaroslav Brustinov, TRex Traffic Generator

On Wed, Nov 29, 2017 at 1:35 PM, Yaroslav Brustinov <y.bru...@gmail.com> wrote:

Hi,

> I don't think TRex can run beyond 14 cores currently, right?

Should be able to...

$ ./t-rex-64 -i -c 15 --checksum-offload --cfg /var/lib/pbench-agent/tmp/trex_cfg.yaml

Killing Scapy server... Scapy server is killed

Starting Scapy server.... Scapy server is started

The ports are bound/configured.

Starting TRex v2.31 please wait ...

set driver name net_i40e

driver capability : TCP_UDP_OFFLOAD TSO

Warning --checksum-offload is turn on by default, no need to call it

Number of ports found: 2

EAL: Error - exiting with code: 1

Cause: Error: Number of TX queues exceeds 16. Try running with lower -c <val>

> pair of ports, 1 port per card

Do the ports share same NUMA?

​Yes, they do.  Currently I have 2 dual-port PCI cards in the same NUMA node.  Running 1 port per card allows us to get a higher rate per port, since theres 1 port/controller in use vs 2 ports/controller.

When we go to 4 ports, I would use 2 cards in node0 and 2 cards in node1.  The port-pairs would be node0 for one and node1 for the other.

​

Performance-wise, each pair of ports should belong to same NUMA as CPU core to which they belong.
Please read following sections of docs:
https://trex-tgn.cisco.com/trex/doc/trex_manual.html#_platform_section_configuration
https://trex-tgn.cisco.com/trex/doc/trex_manual.html#_script_for_creating_config_file

Could you share the Python profile you are using to create traffic in this example?

https://github.com/atheurer/lua-trafficgen/blob/master/trex-txrx.py#L245

Maybe you can improve the "modify headers" program AKA Field engine by reusing same variable and writing it to different offsets with added/subtracted value.
Did you record with perf only single thread of TRex? There is only 1 core in your screenshots, hope you are using other cores as well...

​Yes, I am using more cores, but I wanted to share the report for 1 cpu for clarity.  The other cpus doing Tx look similar.

BTW, it does not seem to matter if we change only 1 filed in the header or many; the performance impact is the same, which I thought was odd.  Makes me wonder if the impact is not actually changing the header but something else.

Thanks,

-Andrew

hanoh haim

unread,

Nov 29, 2017, 3:02:29 PM11/29/17

to Andrew Theurer, TRex Traffic Generator

Hi Andrew, the numbers does not make sense

See our report.

https://trex-tgn.cisco.com/trex/doc/trex_analytics.html

You should have 15MPPS/core with Field Engine and 35-40MPPS

/core without it

Please send the information Itay asked (TRex CPU%) and the profile you are using

Thanks,

Hanoh

--

You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.

To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+u...@googlegroups.com.

To post to this group, send email to trex...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/CAD5-U7RLXL1WRiQHL%3DDeN-yUA_OSb6TDoBRHo1FX9We9KgvhYg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

Hanoh

Sent from my iPhone

Andrew Theurer

unread,

Nov 29, 2017, 3:08:09 PM11/29/17

to Itay Marom, Yaroslav Brustinov, TRex Traffic Generator

On Wed, Nov 29, 2017 at 1:53 PM, Itay Marom <ita...@gmail.com> wrote:

Hi Andrew,

Few comments:

1. Perf is problematic with TRex's regular build mode.
this is because delays are not being executed as system calls such as nanosleep.
instead we use pause in a loop - this is way more accurate.

that's why, for example, in your first snapshot, you see rte_delay_us_block 60%
which is probably dead time for delays.

I suggest you do this:

in TRex TUI go to CPU utilization and send the snapshot

see here how:
  http://trex-tgn.cisco.com/trex/doc/trex_stateless.html#_performance_tweaking

2. if you really want perf, I've added a compilation flag to allow a more friendly version for PERF:

https://github.com/cisco-system-traffic-generator/trex-core/blob/master/linux_dpdk/ws_main.py#L821

this will switch TRex sleeps to be nanosleep (less accurate but great for debug with perf)
also it will disregard full queue to not confuse perf with retries of TRex.

Regarding the performance issues,
Can you send the CPU util from the TUI and show that all the CPUs are 100% ?

​So far I can only get a single cpu util number:

-Per port stats table

ports | 0 | 1

-----------------------------------------------------------------------------------------

opackets | 2902735117 | 2902678250

obytes | 185775045248 | 185771404544

ipackets | 2379534529 | 2379583028

ibytes | 152290208190 | 152293312254

ierrors | 523124779 | 523115489

oerrors | 0 | 0

Tx Bw | 17.46 Gbps | 17.46 Gbps

-Global stats enabled

Cpu Utilization : 100.0 % 5.0 Gb/core

Platform_factor : 1.0

Total-Tx : 34.93 Gbps

Total-Rx : 28.63 Gbps

Total-PPS : 68.22 Mpps

Total-CPS : 0.00 cps

Expected-PPS : 0.00 pps

Expected-CPS : 0.00 cps

Expected-BPS : 0.00 bps

Active-flows : 0 Clients : 0 Socket-util : 0.0000 %

Open-flows : 0 Servers : 0 Socket : 0 Socket/Clients : -nan

Total_queue_full : 4315

drop-rate : 6.30 Gbps

current time : 93.6 sec

test duration : 0.0 sec

Is there a way to toggle per-cpu stats?

hanoh haim

unread,

Nov 29, 2017, 3:10:35 PM11/29/17

to Andrew Theurer, Itay Marom, TRex Traffic Generator, Yaroslav Brustinov

Use the Console TUI and not the server.

There you can see per core util

To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+u...@googlegroups.com.

To post to this group, send email to trex...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/CAD5-U7RLXL1WRiQHL%3DDeN-yUA_OSb6TDoBRHo1FX9We9KgvhYg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.

To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+u...@googlegroups.com.

To post to this group, send email to trex...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/CAJysGCTwFJaPVmD1kKQ0z8HpPePik%2BbPQ9eHJ65Gv%3DXGoo2_EA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
Itay Marom

--

You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.

To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+u...@googlegroups.com.

To post to this group, send email to trex...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/CAD5-U7Snne%2BOLtUaj%2BS7wuCi_3t9a2TAdSsm-ig%2B-5A-i0CQsQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Itay Marom

unread,

Nov 29, 2017, 3:13:18 PM11/29/17

to hanoh haim, Andrew Theurer, TRex Traffic Generator, Yaroslav Brustinov

./trex-console

trex>tui

press 'u' or 'c' - don't remember...

should look something like this:

Cpu Util(%)

  Thread   | Avg | Latest | -1  | -2  | -3  | -4  | -5  | -6  | -7  | -8

 0   (0)   |  92 |     92 |  92 |  91 |  91 |  92 |  91 |  92 |  93 |  94
 1 (IDLE)  |   0 |      0 |   0 |   0 |   0 |   0 |   0 |   0 |   0 |   0
 2   (1)   |  96 |     95 |  95 |  96 |  96 |  96 |  96 |  95 |  94 |  95
 3 (IDLE)  |   0 |      0 |   0 |   0 |   0 |   0 |   0 |   0 |   0 |   0
 4   (0)   |  92 |     93 |  93 |  91 |  91 |  93 |  93 |  93 |  93 |  93
 5 (IDLE)  |   0 |      0 |   0 |   0 |   0 |   0 |   0 |   0 |   0 |   0
 6   (1)   |  88 |     88 |  88 |  88 |  88 |  88 |  88 |  88 |  87 |  87
 7 (IDLE)  |   0 |      0 |   0 |   0 |   0 |   0 |   0 |   0 |   0 |   0

To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+unsubscribe@googlegroups.com.

To post to this group, send email to trex...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/CAD5-U7RLXL1WRiQHL%3DDeN-yUA_OSb6TDoBRHo1FX9We9KgvhYg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.

To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+unsubscribe@googlegroups.com.

To post to this group, send email to trex...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/CAJysGCTwFJaPVmD1kKQ0z8HpPePik%2BbPQ9eHJ65Gv%3DXGoo2_EA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
Itay Marom

--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.

To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+unsubscribe@googlegroups.com.

To post to this group, send email to trex...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/CAD5-U7Snne%2BOLtUaj%2BS7wuCi_3t9a2TAdSsm-ig%2B-5A-i0CQsQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
Hanoh
Sent from my iPhone

--

Itay Marom

Andrew Theurer

unread,

Nov 29, 2017, 3:23:09 PM11/29/17

to Itay Marom, hanoh haim, TRex Traffic Generator, Yaroslav Brustinov

Thanks for the tips on the console.

Global Statistics

connection : localhost, Port 4501 total_tx_L2 : 34.84 Gb/sec

version : v2.31 total_tx_L1 : 45.72 Gb/sec

cpu_util. : 100.0% @ 14 cores (14 per port) total_rx : 28.54 Gb/sec

rx_cpu_util. : 0.01% / 735.90 pkt/sec total_pps : 68.04 Mpkt/sec

async_util. : 0.1% / 1.36 KB/sec drop_rate : 6.29 Gb/sec

queue_full : 0 pkts

Cpu Util(%)

Thread | Avg | Latest | -1 | -2 | -3 | -4 | -5 | -6 | -7 | -8 | -9 | -10 | -11 | -12 | -13 | -14

-------------------------------------------------------------------------------------------------------------

0 (0,1) | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100

1 (0,1) | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100

2 (0,1) | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100

3 (0,1) | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100

4 (0,1) | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100

5 (0,1) | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100

6 (0,1) | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100

7 (0,1) | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100

8 (0,1) | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100

9 (0,1) | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100

10 (0,1) | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100

11 (0,1) | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100

12 (0,1) | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100

13 (0,1) | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100

Mbuf Util

| 64b | 128b | 256b | 512b | 1024b | 2048b | 4096b | 9kb | RAM(MB)

----------------------------------------------------------------------------------------------------

Total: | 131040 | 65520 | 40950 | 40950 | 40950 | 90220 | 1024 | 2560 | 302

Socket 1: | 31308 | 0 | 0 | 0 | 0 | 8192 | 0 | 0 | 18

Percent: | 23% | 0% | 0% | 0% | 0% | 9% | 0% | 0% |

status: -

browse: 'ESC' - console, 'q' - quit, 'd' - dashboard, 's' - streams, 'l' - latency, 'u' - util,

ustats:

Itay Marom

unread,

Nov 29, 2017, 3:27:01 PM11/29/17

to Andrew Theurer, hanoh haim, TRex Traffic Generator, Yaroslav Brustinov

Try with pinning...

You have 14 cores switching from interface 0 to interface 1 all the time.

http://trex-tgn.cisco.com/trex/doc/trex_stateless.html#_performance_tweaking

you can with the console:

"start ... --pin"

--

Itay Marom

unread,

Nov 29, 2017, 3:28:16 PM11/29/17

to Andrew Theurer, hanoh haim, TRex Traffic Generator, Yaroslav Brustinov

for Python API:

c.start(ports = [port_a, port_b], mult = rate, core_mask=STLClient.CORE_MASK_PIN)

--

Itay Marom

Andrew Theurer

unread,

Nov 29, 2017, 4:38:39 PM11/29/17

to Itay Marom, hanoh haim, TRex Traffic Generator, Yaroslav Brustinov

On Wed, Nov 29, 2017 at 2:28 PM, Itay Marom <ita...@gmail.com> wrote:

for Python API:

c.start(ports = [port_a, port_b], mult = rate, core_mask=STLClient.CORE_MASK_PIN)

​Thanks for this!  We are trying a mask of 0x0F,0xF0, and we are currently seeing ~41Mpps Tx per port, a big improvement.​

There is one issue in debugging, in that our benchmark script loses our connection to the trex server when we connect via 'trex-console' and force the connection to gather the stats like:

Global Statistics

connection : localhost, Port 4501 total_tx_L2 : 42.16 Gb/sec

version : v2.32 total_tx_L1 : 55.33 Gb/sec

cpu_util. : 72.7% @ 11 cores (11 per port) total_rx : 28.49 Gb/sec

rx_cpu_util. : 0.0% / 0.00 pkt/sec total_pps : 82.34 Mpkt/sec

async_util. : 0.07% / 1.50 KB/sec drop_rate : 13.67 Gb/sec

queue_full : 0 pkts

Cpu Util(%)

Thread | Avg | Latest | -1 | -2 | -3 | -4 | -5 | -6 | -7 | -8 | -9 | -10 | -11 | -12 | -13 | -14

-------------------------------------------------------------------------------------------------------------

0 (0) | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100

1 (0) | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100

2 (0) | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100

3 (0) | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100

4 (1) | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100

5 (1) | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100

6 (1) | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100

7 (1) | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100

8 (IDLE) | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0

9 (IDLE) | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0

10 (IDLE) | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0

Mbuf Util

| 64b | 128b | 256b | 512b | 1024b | 2048b | 4096b | 9kb | RAM(MB)

----------------------------------------------------------------------------------------------------

Total: | 131040 | 65520 | 40950 | 40950 | 40950 | 90220 | 1024 | 2560 | 302

Socket 1: | 8329 | 0 | 0 | 0 | 0 | 8192 | 0 | 0 | 17

Percent: | 6% | 0% | 0% | 0% | 0% | 9% | 0% | 0% |

status: \

However, if we could gather stats like this with the API, maybe we could avoid using the trex-console manually to get this info.

BTW, we did see an error when we have latency measurement enabled, so we have disabled for now.  We'll look in to this a little closer, but might have some questions about it later.

Thanks very much for the help.  I will continue to look at the performance tweaking section in the docs to see what else we can do to improve.  We'll be moving to a config where we have 2 ports per socket soon, so hopefully we can push to ~164Mpps/host!

Itay Marom

unread,

Nov 29, 2017, 4:51:00 PM11/29/17

to Andrew Theurer, hanoh haim, TRex Traffic Generator, Yaroslav Brustinov

Hi Andrew,

Not sure I understand..

"our script loses our connection to the trex server when we connect via 'trex-console'"

do you mean the ports are acquired by the console ?

so use TRex conosle in read only mode.

./trex-console -r

Also, from the Python API you can connect multiple clients (or multiple scripts instances) and gather stats...

Simply do not acquire the ports.

get_stats is a read only action and can be performed on non-acquired ports.

search for get_cpu_util in the Python API SDK.

or maybe I misunderstood ?

Itay

--

Itay Marom

Andrew Theurer

unread,

Nov 29, 2017, 5:05:10 PM11/29/17

to Itay Marom, hanoh haim, TRex Traffic Generator, Yaroslav Brustinov

On Wed, Nov 29, 2017 at 3:50 PM, Itay Marom <ita...@gmail.com> wrote:

Hi Andrew,

Not sure I understand..

"our script loses our connection to the trex server when we connect via 'trex-console'"

do you mean the ports are acquired by the console ?

so use TRex conosle in read only mode.

./trex-console -r

​Oops, I was not using the -r.  All should be good with that.​

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/CAPXuVfcJEyeXGfiihMGsfX97getjuTwJ21v7dVaoVYeY8pUeaw%40mail.gmail.com.

hanoh haim

unread,

Nov 30, 2017, 2:44:33 AM11/30/17

to Andrew Theurer, Itay Marom, TRex Traffic Generator, Yaroslav Brustinov

Hi Andrew,

Could you check v2.32?

in this total rate (~70MPPS) you probably hitting the ref-cnt issue that we improved

in v2.33

with latest code (in github) and mlx5 driver TRex can do 150MPPS with 6 cores@64B

see

http://trex-tgn.cisco.com/trex/doc/release_notes.html

another ref doc that we have is this, could you run the same commands and send the results?

http://trex-tgn.cisco.com/trex/doc/trex_stateless_bench.html

thanks,

Hanoh

Yaroslav Brustinov

unread,

Nov 30, 2017, 4:59:44 AM11/30/17

to hanoh haim, Andrew Theurer, Itay Marom, TRex Traffic Generator

> We would probably need 26 cores to do this (12 Tx per node, plus 1 for Rx and 1 for latency). I don't think TRex can run beyond 14 cores currently, right?

Just a note, -c parameter specifies number of TX cores for each pair of interfaces.

Thus, specifying -c 13 would give 26 cores for 4 interfaces total.

Karl Rister

unread,

Dec 5, 2017, 10:07:35 AM12/5/17

to Yaroslav Brustinov, hanoh haim, Andrew Theurer, Itay Marom, TRex Traffic Generator

We have been experimenting with the core_mask functionality and are
seeing impressive performance improvements. However, we are having
one problem. Our traffic profile includes latency packets flowing in
both directions between two ports. When we use
core_mask=STLClient.CORE_MASK_PIN or unique masks the test dies with
this error:

Server has been shutdown - cause: 'assert:
../../src/main_dpdk.cpp:2727 Assertion 'lp_port->m_tx_queue_id_lat !=
INVALID_Q_ID' failed.'

We have only been able to get the test to function by doing one of the
following:

A) Disable the latency streams
B) Specify masks where both ports are allowed to execute on the first
cpu allocated to the pair. It seems that latency packets must be sent
from the first cpu only, is that accurate?

https://github.com/cisco-system-traffic-generator/trex-core/blob/master/src/main_dpdk.cpp#L2704

The problem with approach B is that it results in much higher
utilization on the first cpu since it is bouncing between ports, which
is exactly what we were trying to avoid in the first place. We have
not scaled this scenario up enough yet to know what the final impact
of that imbalance is, but in the meantime is there another approach to
getting latency packets flowing that we haven't uncovered yet that
allows the cpus to be truly dedicated to individual ports?

--
Karl Rister <kri...@redhat.com>

>>>>>>>>>>>>>> it, send an email to trex-tgn+u...@googlegroups.com.

>>>>>>>>>>>>>> To post to this group, send email to
>>>>>>>>>>>>>> trex...@googlegroups.com.
>>>>>>>>>>>>>> To view this discussion on the web visit
>>>>>>>>>>>>>> https://groups.google.com/d/msgid/trex-tgn/CAD5-U7RLXL1WRiQHL%3DDeN-yUA_OSb6TDoBRHo1FX9We9KgvhYg%40mail.gmail.com.
>>>>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> You received this message because you are subscribed to the
>>>>>>>>>>>>> Google Groups "TRex Traffic Generator" group.
>>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from

>>>>>>>>>>>>> it, send an email to trex-tgn+u...@googlegroups.com.

>>>>>>>>>>>>> To post to this group, send email to trex...@googlegroups.com.
>>>>>>>>>>>>> To view this discussion on the web visit
>>>>>>>>>>>>> https://groups.google.com/d/msgid/trex-tgn/CAJysGCTwFJaPVmD1kKQ0z8HpPePik%2BbPQ9eHJ65Gv%3DXGoo2_EA%40mail.gmail.com.
>>>>>>>>>>>>>
>>>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Itay Marom
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> You received this message because you are subscribed to the
>>>>>>>>>>> Google Groups "TRex Traffic Generator" group.
>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from it,

>>>>>>>>>>> send an email to trex-tgn+u...@googlegroups.com.

>>>>>>>>>>> To post to this group, send email to trex...@googlegroups.com.
>>>>>>>>>>> To view this discussion on the web visit
>>>>>>>>>>> https://groups.google.com/d/msgid/trex-tgn/CAD5-U7Snne%2BOLtUaj%2BS7wuCi_3t9a2TAdSsm-ig%2B-5A-i0CQsQ%40mail.gmail.com.
>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Hanoh
>>>>>>>>>> Sent from my iPhone
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Itay Marom
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Itay Marom
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Itay Marom
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Itay Marom
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "TRex Traffic Generator" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send

>>>> an email to trex-tgn+u...@googlegroups.com.

>>>> To post to this group, send email to trex...@googlegroups.com.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/trex-tgn/CAPXuVfcJEyeXGfiihMGsfX97getjuTwJ21v7dVaoVYeY8pUeaw%40mail.gmail.com.
>>>>
>>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>
>>
>>
>> --
>> Hanoh
>> Sent from my iPhone
>
>

> --
> You received this message because you are subscribed to the Google Groups
> "TRex Traffic Generator" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to trex-tgn+u...@googlegroups.com.

> To post to this group, send email to trex...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/trex-tgn/CAJysGCRk4JyjrRC5_w_WGf%2B8ePRy6Kwng3-BusydPao3sVR1Yw%40mail.gmail.com.

Itay Marom

unread,

Dec 5, 2017, 11:25:29 AM12/5/17

to Karl Rister, Yaroslav Brustinov, hanoh haim, Andrew Theurer, TRex Traffic Generator

Hi Karl,

This is a bug.

Current implementation assigns latency streams to the first core of the mask, not the first core for the pair of ports (as you have identified).

Latency streams must be injected by the first core in the pair regardless of the mask.

Latency streams implies first core load implicitly.

The correct fix would be to assign latency streams specifically to the first core of the pair.

I'll try to provide you a fix to test out tomorrow.

Thanks,

Itay

>>>>>>>>>>>>>> it, send an email to trex-tgn+unsubscribe@googlegroups.com.

>>>>>>>>>>>>>> To post to this group, send email to
>>>>>>>>>>>>>> trex...@googlegroups.com.
>>>>>>>>>>>>>> To view this discussion on the web visit
>>>>>>>>>>>>>> https://groups.google.com/d/msgid/trex-tgn/CAD5-U7RLXL1WRiQHL%3DDeN-yUA_OSb6TDoBRHo1FX9We9KgvhYg%40mail.gmail.com.
>>>>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> You received this message because you are subscribed to the
>>>>>>>>>>>>> Google Groups "TRex Traffic Generator" group.
>>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from

>>>>>>>>>>>>> it, send an email to trex-tgn+unsubscribe@googlegroups.com.

>>>>>>>>>>>>> To post to this group, send email to trex...@googlegroups.com.
>>>>>>>>>>>>> To view this discussion on the web visit
>>>>>>>>>>>>> https://groups.google.com/d/msgid/trex-tgn/CAJysGCTwFJaPVmD1kKQ0z8HpPePik%2BbPQ9eHJ65Gv%3DXGoo2_EA%40mail.gmail.com.
>>>>>>>>>>>>>
>>>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Itay Marom
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> You received this message because you are subscribed to the
>>>>>>>>>>> Google Groups "TRex Traffic Generator" group.
>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from it,

>>>>>>>>>>> send an email to trex-tgn+unsubscribe@googlegroups.com.

>>>>>>>>>>> To post to this group, send email to trex...@googlegroups.com.
>>>>>>>>>>> To view this discussion on the web visit
>>>>>>>>>>> https://groups.google.com/d/msgid/trex-tgn/CAD5-U7Snne%2BOLtUaj%2BS7wuCi_3t9a2TAdSsm-ig%2B-5A-i0CQsQ%40mail.gmail.com.
>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Hanoh
>>>>>>>>>> Sent from my iPhone
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Itay Marom
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Itay Marom
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Itay Marom
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Itay Marom
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "TRex Traffic Generator" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send

>>>> an email to trex-tgn+unsubscribe@googlegroups.com.

>>>> To post to this group, send email to trex...@googlegroups.com.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/trex-tgn/CAPXuVfcJEyeXGfiihMGsfX97getjuTwJ21v7dVaoVYeY8pUeaw%40mail.gmail.com.
>>>>
>>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>
>>
>>
>> --
>> Hanoh
>> Sent from my iPhone
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "TRex Traffic Generator" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to trex-tgn+unsubscribe@googlegroups.com.

> To post to this group, send email to trex...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/trex-tgn/CAJysGCRk4JyjrRC5_w_WGf%2B8ePRy6Kwng3-BusydPao3sVR1Yw%40mail.gmail.com.

>
> For more options, visit https://groups.google.com/d/optout.

--

Itay Marom

unread,

Dec 6, 2017, 11:47:49 AM12/6/17

to Karl Rister, Yaroslav Brustinov, hanoh haim, Andrew Theurer, TRex Traffic Generator

Hi,

Please try the attached patch.

It will migrate latency compiled streams to core 0.

I still need to test it thoroughly but it has passed compilation and functional regression.

Please let me know if it solves the problem.

I've managed to reconstruct the same crash and made sure this patch solves it.

P.S: you might need to merge it to the correct place if my base version and yours differ a little, but it should be easy.

Thanks,

Itay

--

Itay Marom

latency_pinning.patch

Karl Rister

unread,

Dec 7, 2017, 4:38:53 PM12/7/17

to Itay Marom, Yaroslav Brustinov, hanoh haim, Andrew Theurer, TRex Traffic Generator

I've been testing the patch today and it seems to be working as
intended. I'm still experimenting with different configurations to
determine what is the best arrangement but so far I have tried quite a
few different scenarios and it seems to be handling everything I've
thrown at it so far.

Thanks.

--
Karl Rister <kri...@redhat.com>

>>> >>>>>>>>>>>>>> trex-tgn+u...@googlegroups.com.

>>> >>>>>>>>>>>>>> To post to this group, send email to
>>> >>>>>>>>>>>>>> trex...@googlegroups.com.
>>> >>>>>>>>>>>>>> To view this discussion on the web visit
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> https://groups.google.com/d/msgid/trex-tgn/CAD5-U7RLXL1WRiQHL%3DDeN-yUA_OSb6TDoBRHo1FX9We9KgvhYg%40mail.gmail.com.
>>> >>>>>>>>>>>>>> For more options, visit
>>> >>>>>>>>>>>>>> https://groups.google.com/d/optout.
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> --
>>> >>>>>>>>>>>>> You received this message because you are subscribed to the
>>> >>>>>>>>>>>>> Google Groups "TRex Traffic Generator" group.
>>> >>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails
>>> >>>>>>>>>>>>> from

>>> >>>>>>>>>>>>> it, send an email to trex-tgn+u...@googlegroups.com.

>>> >>>>>>>>>>>>> To post to this group, send email to
>>> >>>>>>>>>>>>> trex...@googlegroups.com.
>>> >>>>>>>>>>>>> To view this discussion on the web visit
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> https://groups.google.com/d/msgid/trex-tgn/CAJysGCTwFJaPVmD1kKQ0z8HpPePik%2BbPQ9eHJ65Gv%3DXGoo2_EA%40mail.gmail.com.
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> --
>>> >>>>>>>>>>>> Itay Marom
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> --
>>> >>>>>>>>>>> You received this message because you are subscribed to the
>>> >>>>>>>>>>> Google Groups "TRex Traffic Generator" group.
>>> >>>>>>>>>>> To unsubscribe from this group and stop receiving emails from
>>> >>>>>>>>>>> it,

>>> >>>>>>>>>>> send an email to trex-tgn+u...@googlegroups.com.

>>> >>>>>>>>>>> To post to this group, send email to
>>> >>>>>>>>>>> trex...@googlegroups.com.
>>> >>>>>>>>>>> To view this discussion on the web visit
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> https://groups.google.com/d/msgid/trex-tgn/CAD5-U7Snne%2BOLtUaj%2BS7wuCi_3t9a2TAdSsm-ig%2B-5A-i0CQsQ%40mail.gmail.com.
>>> >>>>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>> >>>>>>>>>>
>>> >>>>>>>>>> --
>>> >>>>>>>>>> Hanoh
>>> >>>>>>>>>> Sent from my iPhone
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>> --
>>> >>>>>>>>> Itay Marom
>>> >>>>>>>>
>>> >>>>>>>>
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> --
>>> >>>>>>> Itay Marom
>>> >>>>>>
>>> >>>>>>
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> --
>>> >>>>>> Itay Marom
>>> >>>>>
>>> >>>>>
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> --
>>> >>>> Itay Marom
>>> >>>>
>>> >>>> --
>>> >>>> You received this message because you are subscribed to the Google
>>> >>>> Groups "TRex Traffic Generator" group.
>>> >>>> To unsubscribe from this group and stop receiving emails from it,
>>> >>>> send

>>> >>>> an email to trex-tgn+u...@googlegroups.com.

>>> >>>> To post to this group, send email to trex...@googlegroups.com.
>>> >>>> To view this discussion on the web visit
>>> >>>>
>>> >>>> https://groups.google.com/d/msgid/trex-tgn/CAPXuVfcJEyeXGfiihMGsfX97getjuTwJ21v7dVaoVYeY8pUeaw%40mail.gmail.com.
>>> >>>>
>>> >>>> For more options, visit https://groups.google.com/d/optout.
>>> >>>
>>> >>>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Hanoh
>>> >> Sent from my iPhone
>>> >
>>> >
>>> > --
>>> > You received this message because you are subscribed to the Google
>>> > Groups
>>> > "TRex Traffic Generator" group.
>>> > To unsubscribe from this group and stop receiving emails from it, send
>>> > an

>>> > email to trex-tgn+u...@googlegroups.com.

>>> > To post to this group, send email to trex...@googlegroups.com.
>>> > To view this discussion on the web visit
>>> >
>>> > https://groups.google.com/d/msgid/trex-tgn/CAJysGCRk4JyjrRC5_w_WGf%2B8ePRy6Kwng3-BusydPao3sVR1Yw%40mail.gmail.com.
>>> >
>>> > For more options, visit https://groups.google.com/d/optout.
>>
>>
>>
>>
>> --
>> Itay Marom
>
>
>
>
> --
> Itay Marom
>

> --
> You received this message because you are subscribed to the Google Groups
> "TRex Traffic Generator" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to trex-tgn+u...@googlegroups.com.

> To post to this group, send email to trex...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/trex-tgn/CAPXuVffn2NXmYf9WW5hDa4cVf%3D4egnbP7uf-%3DO55oQSAYX3ZTg%40mail.gmail.com.

Itay Marom

unread,

Dec 10, 2017, 4:36:34 AM12/10/17

to Karl Rister, Yaroslav Brustinov, hanoh haim, Andrew Theurer, TRex Traffic Generator

Integrated to master with commit a14e913f86314f and will be included in the next release.

please see the link below for more details:

https://trex-tgn.cisco.com/youtrack/issue/trex-428

Thanks,

Itay

>>> >>>>>>>>>>>>>> trex-tgn+unsubscribe@googlegroups.com.

>>> >>>>>>>>>>>>>> To post to this group, send email to
>>> >>>>>>>>>>>>>> trex...@googlegroups.com.
>>> >>>>>>>>>>>>>> To view this discussion on the web visit
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> https://groups.google.com/d/msgid/trex-tgn/CAD5-U7RLXL1WRiQHL%3DDeN-yUA_OSb6TDoBRHo1FX9We9KgvhYg%40mail.gmail.com.
>>> >>>>>>>>>>>>>> For more options, visit
>>> >>>>>>>>>>>>>> https://groups.google.com/d/optout.
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> --
>>> >>>>>>>>>>>>> You received this message because you are subscribed to the
>>> >>>>>>>>>>>>> Google Groups "TRex Traffic Generator" group.
>>> >>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails
>>> >>>>>>>>>>>>> from

>>> >>>>>>>>>>>>> it, send an email to trex-tgn+unsubscribe@googlegroups.com.

>>> >>>>>>>>>>>>> To post to this group, send email to
>>> >>>>>>>>>>>>> trex...@googlegroups.com.
>>> >>>>>>>>>>>>> To view this discussion on the web visit
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> https://groups.google.com/d/msgid/trex-tgn/CAJysGCTwFJaPVmD1kKQ0z8HpPePik%2BbPQ9eHJ65Gv%3DXGoo2_EA%40mail.gmail.com.
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> --
>>> >>>>>>>>>>>> Itay Marom
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> --
>>> >>>>>>>>>>> You received this message because you are subscribed to the
>>> >>>>>>>>>>> Google Groups "TRex Traffic Generator" group.
>>> >>>>>>>>>>> To unsubscribe from this group and stop receiving emails from
>>> >>>>>>>>>>> it,

>>> >>>>>>>>>>> send an email to trex-tgn+unsubscribe@googlegroups.com.

>>> >>>>>>>>>>> To post to this group, send email to
>>> >>>>>>>>>>> trex...@googlegroups.com.
>>> >>>>>>>>>>> To view this discussion on the web visit
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> https://groups.google.com/d/msgid/trex-tgn/CAD5-U7Snne%2BOLtUaj%2BS7wuCi_3t9a2TAdSsm-ig%2B-5A-i0CQsQ%40mail.gmail.com.
>>> >>>>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>> >>>>>>>>>>
>>> >>>>>>>>>> --
>>> >>>>>>>>>> Hanoh
>>> >>>>>>>>>> Sent from my iPhone
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>> --
>>> >>>>>>>>> Itay Marom
>>> >>>>>>>>
>>> >>>>>>>>
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> --
>>> >>>>>>> Itay Marom
>>> >>>>>>
>>> >>>>>>
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> --
>>> >>>>>> Itay Marom
>>> >>>>>
>>> >>>>>
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> --
>>> >>>> Itay Marom
>>> >>>>
>>> >>>> --
>>> >>>> You received this message because you are subscribed to the Google
>>> >>>> Groups "TRex Traffic Generator" group.
>>> >>>> To unsubscribe from this group and stop receiving emails from it,
>>> >>>> send

>>> >>>> an email to trex-tgn+unsubscribe@googlegroups.com.

>>> >>>> To post to this group, send email to trex...@googlegroups.com.
>>> >>>> To view this discussion on the web visit
>>> >>>>
>>> >>>> https://groups.google.com/d/msgid/trex-tgn/CAPXuVfcJEyeXGfiihMGsfX97getjuTwJ21v7dVaoVYeY8pUeaw%40mail.gmail.com.
>>> >>>>
>>> >>>> For more options, visit https://groups.google.com/d/optout.
>>> >>>
>>> >>>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Hanoh
>>> >> Sent from my iPhone
>>> >
>>> >
>>> > --
>>> > You received this message because you are subscribed to the Google
>>> > Groups
>>> > "TRex Traffic Generator" group.
>>> > To unsubscribe from this group and stop receiving emails from it, send
>>> > an

>>> > email to trex-tgn+unsubscribe@googlegroups.com.

>>> > To post to this group, send email to trex...@googlegroups.com.
>>> > To view this discussion on the web visit
>>> >
>>> > https://groups.google.com/d/msgid/trex-tgn/CAJysGCRk4JyjrRC5_w_WGf%2B8ePRy6Kwng3-BusydPao3sVR1Yw%40mail.gmail.com.
>>> >
>>> > For more options, visit https://groups.google.com/d/optout.
>>
>>
>>
>>
>> --
>> Itay Marom
>
>
>
>
> --
> Itay Marom
>
> --
> You received this message because you are subscribed to the Google Groups
> "TRex Traffic Generator" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to trex-tgn+unsubscribe@googlegroups.com.

> To post to this group, send email to trex...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/trex-tgn/CAPXuVffn2NXmYf9WW5hDa4cVf%3D4egnbP7uf-%3DO55oQSAYX3ZTg%40mail.gmail.com.

>
> For more options, visit https://groups.google.com/d/optout.

--

Itay Marom

Karl Rister

unread,

Dec 11, 2017, 11:04:12 AM12/11/17

to Itay Marom, Yaroslav Brustinov, hanoh haim, Andrew Theurer, TRex Traffic Generator

Awesome. Thanks!

--
Karl Rister <kri...@redhat.com>

>> >>> >>>>>>>>>>>>>> trex-tgn+u...@googlegroups.com.

>> >>> >>>>>>>>>>>>>> To post to this group, send email to
>> >>> >>>>>>>>>>>>>> trex...@googlegroups.com.
>> >>> >>>>>>>>>>>>>> To view this discussion on the web visit
>> >>> >>>>>>>>>>>>>>
>> >>> >>>>>>>>>>>>>>
>> >>> >>>>>>>>>>>>>> https://groups.google.com/d/msgid/trex-tgn/CAD5-U7RLXL1WRiQHL%3DDeN-yUA_OSb6TDoBRHo1FX9We9KgvhYg%40mail.gmail.com.
>> >>> >>>>>>>>>>>>>> For more options, visit
>> >>> >>>>>>>>>>>>>> https://groups.google.com/d/optout.
>> >>> >>>>>>>>>>>>>
>> >>> >>>>>>>>>>>>>
>> >>> >>>>>>>>>>>>> --
>> >>> >>>>>>>>>>>>> You received this message because you are subscribed to
>> >>> >>>>>>>>>>>>> the
>> >>> >>>>>>>>>>>>> Google Groups "TRex Traffic Generator" group.
>> >>> >>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails
>> >>> >>>>>>>>>>>>> from
>> >>> >>>>>>>>>>>>> it, send an email to

>> >>> >>>>>>>>>>>>> trex-tgn+u...@googlegroups.com.

>> >>> >>>>>>>>>>>>> To post to this group, send email to
>> >>> >>>>>>>>>>>>> trex...@googlegroups.com.
>> >>> >>>>>>>>>>>>> To view this discussion on the web visit
>> >>> >>>>>>>>>>>>>
>> >>> >>>>>>>>>>>>>
>> >>> >>>>>>>>>>>>> https://groups.google.com/d/msgid/trex-tgn/CAJysGCTwFJaPVmD1kKQ0z8HpPePik%2BbPQ9eHJ65Gv%3DXGoo2_EA%40mail.gmail.com.
>> >>> >>>>>>>>>>>>>
>> >>> >>>>>>>>>>>>> For more options, visit
>> >>> >>>>>>>>>>>>> https://groups.google.com/d/optout.
>> >>> >>>>>>>>>>>>
>> >>> >>>>>>>>>>>>
>> >>> >>>>>>>>>>>>
>> >>> >>>>>>>>>>>>
>> >>> >>>>>>>>>>>> --
>> >>> >>>>>>>>>>>> Itay Marom
>> >>> >>>>>>>>>>>
>> >>> >>>>>>>>>>> --
>> >>> >>>>>>>>>>> You received this message because you are subscribed to
>> >>> >>>>>>>>>>> the
>> >>> >>>>>>>>>>> Google Groups "TRex Traffic Generator" group.
>> >>> >>>>>>>>>>> To unsubscribe from this group and stop receiving emails
>> >>> >>>>>>>>>>> from
>> >>> >>>>>>>>>>> it,

>> >>> >>>>>>>>>>> send an email to trex-tgn+u...@googlegroups.com.

>> >>> >>>>>>>>>>> To post to this group, send email to
>> >>> >>>>>>>>>>> trex...@googlegroups.com.
>> >>> >>>>>>>>>>> To view this discussion on the web visit
>> >>> >>>>>>>>>>>
>> >>> >>>>>>>>>>>
>> >>> >>>>>>>>>>> https://groups.google.com/d/msgid/trex-tgn/CAD5-U7Snne%2BOLtUaj%2BS7wuCi_3t9a2TAdSsm-ig%2B-5A-i0CQsQ%40mail.gmail.com.
>> >>> >>>>>>>>>>> For more options, visit
>> >>> >>>>>>>>>>> https://groups.google.com/d/optout.
>> >>> >>>>>>>>>>
>> >>> >>>>>>>>>> --
>> >>> >>>>>>>>>> Hanoh
>> >>> >>>>>>>>>> Sent from my iPhone
>> >>> >>>>>>>>>
>> >>> >>>>>>>>>
>> >>> >>>>>>>>>
>> >>> >>>>>>>>>
>> >>> >>>>>>>>> --
>> >>> >>>>>>>>> Itay Marom
>> >>> >>>>>>>>
>> >>> >>>>>>>>
>> >>> >>>>>>>
>> >>> >>>>>>>
>> >>> >>>>>>>
>> >>> >>>>>>> --
>> >>> >>>>>>> Itay Marom
>> >>> >>>>>>
>> >>> >>>>>>
>> >>> >>>>>>
>> >>> >>>>>>
>> >>> >>>>>> --
>> >>> >>>>>> Itay Marom
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>
>> >>> >>>>
>> >>> >>>>
>> >>> >>>> --
>> >>> >>>> Itay Marom
>> >>> >>>>
>> >>> >>>> --
>> >>> >>>> You received this message because you are subscribed to the
>> >>> >>>> Google
>> >>> >>>> Groups "TRex Traffic Generator" group.
>> >>> >>>> To unsubscribe from this group and stop receiving emails from it,
>> >>> >>>> send

>> >>> >>>> an email to trex-tgn+u...@googlegroups.com.

>> >>> >>>> To post to this group, send email to trex...@googlegroups.com.
>> >>> >>>> To view this discussion on the web visit
>> >>> >>>>
>> >>> >>>>
>> >>> >>>> https://groups.google.com/d/msgid/trex-tgn/CAPXuVfcJEyeXGfiihMGsfX97getjuTwJ21v7dVaoVYeY8pUeaw%40mail.gmail.com.
>> >>> >>>>
>> >>> >>>> For more options, visit https://groups.google.com/d/optout.
>> >>> >>>
>> >>> >>>
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >> --
>> >>> >> Hanoh
>> >>> >> Sent from my iPhone
>> >>> >
>> >>> >
>> >>> > --
>> >>> > You received this message because you are subscribed to the Google
>> >>> > Groups
>> >>> > "TRex Traffic Generator" group.
>> >>> > To unsubscribe from this group and stop receiving emails from it,
>> >>> > send
>> >>> > an

>> >>> > email to trex-tgn+u...@googlegroups.com.

>> >>> > To post to this group, send email to trex...@googlegroups.com.
>> >>> > To view this discussion on the web visit
>> >>> >
>> >>> >
>> >>> > https://groups.google.com/d/msgid/trex-tgn/CAJysGCRk4JyjrRC5_w_WGf%2B8ePRy6Kwng3-BusydPao3sVR1Yw%40mail.gmail.com.
>> >>> >
>> >>> > For more options, visit https://groups.google.com/d/optout.
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >> Itay Marom
>> >
>> >
>> >
>> >
>> > --
>> > Itay Marom
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "TRex Traffic Generator" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an

>> > email to trex-tgn+u...@googlegroups.com.

Reply all

Reply to author

Forward