TRex for benchmarking on AWS EC2

1,157 views
Skip to first unread message

Tino

unread,
Sep 28, 2020, 10:55:53 AM9/28/20
to TRex Traffic Generator
Hello,

I have been able to set up a TRex server using RHEL 7.8 on an m5dn.12xlarge instance type (48 cores, 50 Gbps). I would like to use TRex to benchmark the network performance of the instance to determine it's Bandwidth and maximum PPS. I have not yet gotten to testing my specific application, I just need to know the instance's limitations.

Before I share what I have been trying, is there a guide on how to use TRex on AWS for the use case I have shared above? Is it possible to use TRex for benchmarking the same way we use iperf (i.e. to have two EC2 instances, one client and one server, and then benchmark by sending traffic from one to the other via IP)?

...

I have attached 4 virtual NICs on the instance. All NICs are using the latest version of the ENA driver which I understand to be supported by TRex. All IP addresses are in the same subnet and they use the gateway address 172.31.0.1 (the VPC Router). The config I am using is as follows:

- version: 2
  interfaces: ['00:06.0', '00:07.0', '00:08.0', '00:09.0']
  stack: linux_based
  low_end: false
  services_core: 35
  port_info:
      - ip: 172.31.12.235
        default_gw: 172.31.0.1
      - ip: 172.31.3.50
        default_gw: 172.31.0.1

      - ip: 172.31.14.23
        default_gw: 172.31.0.1
      - ip: 172.31.0.64
        default_gw: 172.31.0.1

  platform:
      master_thread_id: 0
      latency_thread_id: 1
      dual_if:
        - socket: 0
          threads: [2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17]

        - socket: 0
          threads: [18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33]

I have tried the examples given on the TRex manual along with some of the traffic profiles that come with the installation but I get the error "STF mode does not support more than one core by dual interfaces". After changing to "-c 1", the test starts running and I can see "Total-Tx" increasing but "Total-Rx" remains 0 and the drop-rate is equal to the Total-Tx.

I suspect that this is due to misconfigured routing, but I can't figure out how to confirm or resolve the issue. I can't do a packet capture due to DPDK limitations either. Any ideas on how to resolve this?

Regards,
Tino

hanoh haim

unread,
Sep 29, 2020, 3:14:10 AM9/29/20
to Tino, TRex Traffic Generator
Hi Tino, 
see here
https://github.com/cisco-system-traffic-generator/trex-core/wiki/TRex-on-AWS

suggest to move to ASTF but it has limitation too 

thanks
Hanoh


--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/b93c65f4-cf7d-4c3d-821c-a36727baae68n%40googlegroups.com.


--
Hanoh
Sent from my iPhone

John H

unread,
Mar 1, 2021, 10:06:18 AM3/1/21
to TRex Traffic Generator
Hi Hanoch,

I've tried to do some bench-marking in AWS on c5n.2xlarge, c5n.4xlarge, and c5n.9xlarge but I can't seem to scale past 1M total PPS (7-9Gbps total throughput depending on average packet size), no matter how many cores I use.

Used this thread as reference on how to run ASTF (2.87) with ena: https://github.com/cisco-system-traffic-generator/trex-core/issues/509. There's no DUT in my case.

Do you know if this is a DPDK/nic limitation, or is there something else I can try?

Thanks!

hanoh haim

unread,
Mar 1, 2021, 2:56:27 PM3/1/21
to John H, TRex Traffic Generator
TRex supports multi-core with ENA, I assume it is a throttle mechanism in AWS

Thanks
Hanoh 

John H

unread,
Mar 2, 2021, 9:49:02 AM3/2/21
to TRex Traffic Generator
Is it possible we're not utilizing all the queues? One queue performance seems to be around 1Mpps:

I've also tried using 2 dual_ifs (4 ports) but the total pps still maxes out at 1Mpps. This is how I start trex:

/t-rex-64 -i --astf --lro-disable -c 6

/etc/trex_cfg.yaml
### Config file generated by dpdk_setup_ports.py ###

- version: 2
  interfaces: ['00:06.0', '00:07.0']
  port_bandwidth_gb : 50
  rx_desc : 2048
  tx_desc : 2048  
  port_info:
      - dest_mac: 02:6d:e2:16:7b:ef # MAC OF LOOPBACK TO IT'S DUAL INTERFACE
        src_mac:  02:f5:75:ce:45:6b
      - dest_mac: 02:f5:75:ce:45:6b # MAC OF LOOPBACK TO IT'S DUAL INTERFACE
        src_mac:  02:6d:e2:16:7b:ef


  platform:
      master_thread_id: 0
      latency_thread_id: 1
      dual_if:
        - socket: 0
          threads: [2,3,4,5,6,7]

hanoh haim

unread,
Mar 2, 2021, 9:51:58 AM3/2/21
to John H, TRex Traffic Generator
Hi John, 
This is not a TRex issue. it is an ENA infrastructure issue. 
if you could share the TUI counters it could help 

John H

unread,
Mar 2, 2021, 9:58:01 AM3/2/21
to TRex Traffic Generator
Was just about to :)

2 ports

connection   : 127.0.0.1, Port 4501                       total_tx_L2  : 7 Gbps                         
version      : ASTF @ v2.87                               total_tx_L1  : 7.16 Gbps                      
cpu_util.    : 21.62% @ 6 cores (6 per dual port)         total_rx     : 7 Gbps                         
rx_cpu_util. : 0.0% / 0 pps                               total_pps    : 1 Mpps                         
async_util.  : 0.03% / 11.82 Kbps                         drop_rate    : 0 bps                          
total_cps.   : 84.11 Kcps                                 queue_full   : 0 pkts                         

Port Statistics

   port    |         0         |         1         |       total       
-----------+-------------------+-------------------+------------------
owner      |              root |              root |                   
link       |                UP |                UP |                   
state      |      TRANSMITTING |      TRANSMITTING |                   
speed      |           50 Gb/s |           50 Gb/s |                   
CPU util.  |            21.62% |            21.62% |                   
--         |                   |                   |                   
Tx bps L2  |         3.06 Gbps |         3.94 Gbps |            7 Gbps
Tx bps L1  |         3.14 Gbps |         4.02 Gbps |         7.16 Gbps
Tx pps     |       476.61 Kpps |       523.96 Kpps |            1 Mpps
Line Util. |            6.27 % |            8.04 % |                   
---        |                   |                   |                   
Rx bps     |         3.94 Gbps |         3.06 Gbps |            7 Gbps
Rx pps     |       523.97 Kpps |       476.58 Kpps |            1 Mpps
----       |                   |                   |                   
opackets   |           6669807 |           7330221 |          14000028
ipackets   |           7330035 |           6669876 |          13999911
obytes     |        5350176060 |        6881056367 |       12231232427
ibytes     |        6880898747 |        5350205798 |       12231104545
tx-pkts    |        6.67 Mpkts |        7.33 Mpkts |          14 Mpkts
rx-pkts    |        7.33 Mpkts |        6.67 Mpkts |          14 Mpkts
tx-bytes   |           5.35 GB |           6.88 GB |          12.23 GB
rx-bytes   |           6.88 GB |           5.35 GB |          12.23 GB
-----      |                   |                   |                   
oerrors    |                 0 |                 0 |                 0
ierrors    |                 0 |                 0 |                 0

4 ports
connection   : 127.0.0.1, Port 4501                       total_tx_L2  : 7.11 Gbps                      
version      : ASTF @ v2.87                               total_tx_L1  : 7.27 Gbps                      
cpu_util.    : 43.91% @ 8 cores (4 per dual port)         total_rx     : 7.1 Gbps                       
rx_cpu_util. : 0.0% / 0 pps                               total_pps    : 1.02 Mpps                      
async_util.  : 0.03% / 14.93 Kbps                         drop_rate    : 0 bps                          
total_cps.   : 85.58 Kcps                                 queue_full   : 117,595,276 pkts               

Port Statistics

   port    |         0         |         1         |         2         |         3         |       total       
-----------+-------------------+-------------------+-------------------+-------------------+------------------
owner      |              root |              root |              root |              root |                   
link       |                UP |                UP |                UP |                UP |                   
state      |      TRANSMITTING |      TRANSMITTING |      TRANSMITTING |      TRANSMITTING |                   
speed      |           50 Gb/s |           50 Gb/s |           50 Gb/s |           50 Gb/s |                   
CPU util.  |            45.75% |            45.75% |            42.07% |            42.07% |                   
--         |                   |                   |                   |                   |                   
Tx bps L2  |         1.46 Gbps |         1.88 Gbps |         1.65 Gbps |         2.12 Gbps |         7.11 Gbps
Tx bps L1  |         1.49 Gbps |         1.92 Gbps |         1.69 Gbps |         2.17 Gbps |         7.27 Gbps
Tx pps     |       227.14 Kpps |       249.79 Kpps |       257.24 Kpps |       282.51 Kpps |         1.02 Mpps
Line Util. |            2.99 % |            3.83 % |            3.38 % |            4.33 % |                   
---        |                   |                   |                   |                   |                   
Rx bps     |         1.88 Gbps |         1.46 Gbps |         2.12 Gbps |         1.65 Gbps |          7.1 Gbps
Rx pps     |       249.68 Kpps |       227.12 Kpps |       282.46 Kpps |       257.16 Kpps |         1.02 Mpps
----       |                   |                   |                   |                   |                   
opackets   |          49399013 |          54300833 |          49422770 |          54319546 |         207442162
ipackets   |          54300356 |          49399012 |          54318948 |          49422523 |         207440839
obytes     |       39627765864 |       50968240070 |       39645736906 |       50980613006 |      181222355846
ibytes     |       50967804135 |       39627763900 |       50980039468 |       39645543860 |      181221151363
tx-pkts    |        49.4 Mpkts |        54.3 Mpkts |       49.42 Mpkts |       54.32 Mpkts |      207.44 Mpkts
rx-pkts    |        54.3 Mpkts |        49.4 Mpkts |       54.32 Mpkts |       49.42 Mpkts |      207.44 Mpkts
tx-bytes   |          39.63 GB |          50.97 GB |          39.65 GB |          50.98 GB |         181.22 GB
rx-bytes   |          50.97 GB |          39.63 GB |          50.98 GB |          39.65 GB |         181.22 GB
-----      |                   |                   |                   |                   |                   
oerrors    |                 0 |                 0 |                 0 |                 0 |                 0
ierrors    |                 0 |                 0 |                 0 |                 0 |                 0


hanoh haim

unread,
Mar 2, 2021, 11:23:06 AM3/2/21
to John H, TRex Traffic Generator
You have “queue_full” indication, ask AWS infrastructure guys how to disable the throttle 

Back then they did spin to burn CPU in this type of cases 

Thanks
Hanoh

--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+u...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages