LACP bonding and CPU limit

[Tom Kendrick]

Hi, 

I'm using v2.91, with 4 ports in one bond for clients, and 4 ports in a bond for servers. 

  interfaces: ['--vdev=net_bonding0,mode=4,slave=0d:00.0,slave=0e:00.0,slave=11:00.0,slave=12:00.0', '--vdev=net_bonding1,mode=4,slave=0d:00.1,slave=0e:00.1,slave=11:00.1,slave=12:00.1']

This works, but, it seems that as the test loads, although I have 8 ports, I am limited to 2 CPU cores, which seem to run at 100% during the test.

Any way to allow more than 2 cores, as when I try to use -c option, I am advised that only c = 1 is possible, due to two (logical) ports.

port : 0

------------

link         :  link : Link Up - speed 40000 Mbps - full-duplex

promiscuous  : 0

port : 1

------------

link         :  link : Link Up - speed 40000 Mbps - full-duplex

promiscuous  : 0

 number of ports         : 2

 max cores for 2 ports   : 1

 tx queues per port      : 3

Tom

[hanoh haim]

Hi Tom, 

The driver should support multi-queue, could you send the output with -v 7 and the full trex_cfg.yaml

Thanks

Hanoh

[Tom Kendrick]

Hey Hanoch, sure.  I was trying different options in terms of sockets, but below is what I have.

It's probably something simple I'm not aware of, or have hanging over from one of my other installs which I migrated across to an additional installation,  so appreciate you looking.

- version: 2

  interfaces: ['--vdev=net_bonding0,mode=4,slave=0d:00.0,slave=0e:00.0,slave=11:00.0,slave=12:00.0', '--vdev=net_bonding1,mode=4,slave=0d:00.1,slave=0e:00.1,slave=11:00.1,slave=12:00.1']

  zmq_pub_port: 4500

  zmq_rpc_port: 4501

  prefix: instance1

  #rx_desc: 4096

  #tx_desc: 4096

  port_info:

      - ip: 192.168.251.20

        default_gw: 192.168.251.1

      - ip: 192.168.252.20

        default_gw: 192.168.252.1

  platform:

      master_thread_id: 0

      latency_thread_id: 1

      dual_if:

        - socket: 0

          threads: [2,3,4,5]

        - socket: 1

          threads: [6,7,8,9]

With -v 7, hopefully the useful information below:

The ports are bound/configured.

Starting  TRex v2.91 please wait  ...

Using configuration file /etc/trex/trex.yaml

 port limit     :  not configured

 port_bandwidth_gb    :  10

 port_speed           :  0

 port_mtu             :  0

 if_mask        : None

 prefix              : instance1

 is low-end : 0

 stack type :

 thread_per_dual_if      : 1

 if        :  --vdev=net_bonding0,mode=4,slave=0d:00.0,slave=0e:00.0,slave=11:00.0,slave=12:00.0, --vdev=net_bonding1,mode=4,slave=0d:00.1,slave=0e:00.1,slave=11:00.1,slave=12:00.1,

 enable_zmq_pub :  1

 zmq_pub_port   :  4500

 m_zmq_rpc_port    :  4501

 src     : 00:00:00:00:00:00

 dest    : 00:00:00:00:00:00

 src     : 00:00:00:00:00:00

 dest    : 00:00:00:00:00:00

 memory per 2x10G ports

 MBUF_64                                   : 16380

 MBUF_128                                  : 8190

 MBUF_256                                  : 8190

 MBUF_512                                  : 8190

 MBUF_1024                                 : 8190

 MBUF_2048                                 : 4095

 MBUF_4096                                 : 128

 MBUF_9K                                   : 512

 TRAFFIC_MBUF_64                           : 65520

 TRAFFIC_MBUF_128                          : 32760

 TRAFFIC_MBUF_256                          : 8190

 TRAFFIC_MBUF_512                          : 8190

 TRAFFIC_MBUF_1024                         : 8190

 TRAFFIC_MBUF_2048                         : 32760

 TRAFFIC_MBUF_4096                         : 128

 TRAFFIC_MBUF_9K                           : 512

 MBUF_DP_FLOWS                             : 524288

 MBUF_GLOBAL_FLOWS                         : 5120

 master   thread  : 0

 rx  thread  : 1

 dual_if : 0

    socket  : 0

   [   2   3   4   5     ]

 dual_if : 1

    socket  : 1

   [   6   7   8   9     ]

CTimerWheelYamlInfo does not exist

 flags           : 8010a00

 write_file      : 0

 verbose         : 2

 realtime        : 1

 flip            : 0

 cores           : 1

 single core     : 0

 flow-flip       : 0

 no clean close  : 0

 zmq_publish     : 1

 vlan mode       : 0

 client_cfg      : 0

 mbuf_cache_disable  : 0

 cfg file        : /config_yaml/2port_http_1.yaml

 mac file        :

 out file        :

 client cfg file :

 duration        : 10

 factor          : 2880

 mbuf_factor     : 1

 latency         : 0 pkt/sec

 zmq_port        : 4500

 telnet_port     : 4501

 expected_ports  : 2

 tw_bucket_usec  : 20.000000 usec

 tw_buckets      : 1024 usec

 tw_levels       : 3 usec

 port : 0 dst:00:00:00:00:00:00  src:00:00:00:00:00:00

 port : 1 dst:00:00:00:00:00:00  src:00:00:00:00:00:00

 port : 2 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 3 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 4 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 5 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 6 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 7 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 8 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 9 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 10 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 11 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 12 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 13 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 14 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 15 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 16 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 17 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 18 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 19 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 20 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 21 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 22 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 23 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 24 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 25 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 26 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 27 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 28 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 29 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 30 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 31 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 Total Memory :

 MBUF_64                                   : 81900

 MBUF_128                                  : 40950

 MBUF_256                                  : 16380

 MBUF_512                                  : 16380

 MBUF_1024                                 : 16380

 MBUF_2048                                 : 36855

 MBUF_4096                                 : 1024

 MBUF_DP_FLOWS                             : 524288

 MBUF_GLOBAL_FLOWS                         : 5120

 get_each_core_dp_flows                    : 524288

 Total memory                              :     248.40 Mbytes

 core_list : 0,1,2

 sockets : 0

 active sockets : 1

 ports_sockets : 1

0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,

 phy   |   virt

 2      1

DPDK args

 xx  -l  0,1,2  -n  4  --log-level  3  --main-lcore  0  --vdev=net_bonding0,mode=4,slave=0d:00.0,slave=0e:00.0,slave=11:00.0,slave=12:00.0  --vdev=net_bonding1,mode=4,slave=0d:00.1,slave=0e:00.1,slave=11:00.1,slave=12:00.1  -a  0d:00.0  -a  0e:00.0  -a  11:00.0  -a  12:00.0  -a  0d:00.1  -a  0e:00.1  -a  11:00.1  -a  12:00.1  -m  1024  --legacy-mem  --file-prefix  instance1

  size of interfaces_vdevs 2

 ===>>>found net_bonding0 8

 ===>>>found net_bonding1 9

 set driver name net_bonding

 driver capability  : TCP_UDP_OFFLOAD  TSO

 set dpdk queues mode to ONE_QUE

 DPDK devices 10 : 10

-----

 0 : vdev 0d:00.0

 1 : vdev 0d:00.1

 2 : vdev 0e:00.0

 3 : vdev 0e:00.1

 4 : vdev 11:00.0

5 : vdev 11:00.1

 6 : vdev 12:00.0

 7 : vdev 12:00.1

 8 : vdev net_bonding0

 9 : vdev net_bonding1

-----

 Number of ports found: 2

if_index : 0

driver name : net_bonding

min_rx_bufsize : 0

max_rx_pktlen  : 15872

max_rx_queues  : 128

max_tx_queues  : 64

max_mac_addrs  : 16

rx_offload_capa : 0x92e9f

tx_offload_capa : 0xa03f

rss reta_size   : 128

flow_type_rss   : 0x38d34

tx_desc_max     : 4096

tx_desc_min     : 0

rx_desc_max     : 4096

rx_desc_min     : 0

zmq publisher at: tcp://*:4500

 rx_data_q_num : 1

 rx_drop_q_num : 0

 rx_dp_q_num   : 0

 rx_que_total : 1

 --

 rx_desc_num_data_q   : 512

 rx_desc_num_drop_q   : 4096

 rx_desc_num_dp_q     : 512

 total_desc           : 512

 --

 tx_desc_num     : 1024

port 8 desc: Unknown

 rx_qid: 0 (512)

port 9 desc: Unknown

 rx_qid: 0 (512)

 wait 8 sec ........

port : 0

------------

link         :  link : Link Up - speed 40000 Mbps - full-duplex

promiscuous  : 0

port : 1

------------

link         :  link : Link Up - speed 40000 Mbps - full-duplex

promiscuous  : 0

 number of ports         : 2

 max cores for 2 ports   : 1

 tx queues per port      : 3

no client generator pool configured, using default pool

no server generator pool configured, using default pool

 flags           : 8010a00

 write_file      : 0

 verbose         : 2

 realtime        : 1

 flip            : 0

 cores           : 1

 single core     : 0

 flow-flip       : 0

 no clean close  : 0

 zmq_publish     : 1

 vlan mode       : 0

 client_cfg      : 0

 mbuf_cache_disable  : 0

 cfg file        : /config_yaml/2port_http_1.yaml

 mac file        :

 out file        :

 client cfg file :

 duration        : 10

 factor          : 2880

 mbuf_factor     : 1

 latency         : 0 pkt/sec

 zmq_port        : 4500

 telnet_port     : 4501

 expected_ports  : 2

 tw_bucket_usec  : 20.000000 usec

 tw_buckets      : 1024 usec

 tw_levels       : 3 usec

 port : 0 dst:00:00:00:00:00:00  src:00:MAC

 port : 1 dst:00:00:00:00:00:00  src:00:MAC

 port : 2 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 3 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 4 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 5 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 6 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 7 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 8 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 9 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 10 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 11 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 12 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 13 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 14 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 15 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 16 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 17 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 18 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 19 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 20 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 21 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 22 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 23 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 24 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 25 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 26 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 27 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 28 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 29 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 30 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 port : 31 dst:00:00:00:01:00:00  src:00:00:00:00:00:00

 Total Memory :

 MBUF_64                                   : 81900

 MBUF_128                                  : 40950

 MBUF_256                                  : 16380

 MBUF_512                                  : 16380

 MBUF_1024                                 : 16380

 MBUF_2048                                 : 37879

 MBUF_4096                                 : 1024

 MBUF_DP_FLOWS                             : 524288

 MBUF_GLOBAL_FLOWS                         : 5120

 get_each_core_dp_flows                    : 524288

 Total memory                              :     250.59 Mbytes

 -------------------------------

RX core uses TX queue number 65535 on all ports

 core, c-port, c-queue, s-port, s-queue, lat-queue

 ------------------------------------------

 1        0      0       1       0      0

 -------------------------------

no client generator pool configured, using default pool

no server generator pool configured, using default pool

 -- loading cap file avl/delay_http_browsing_0.pcap

 id name                                     tps      cps       f-pkts f-bytes  duration Mb/sec  MB/sec c-flows PPS    errors flows

 00 avl/delay_http_browsing_0.pcap           144000.00 144000.00     39    34701     0.25 39975.55 4996.94   35814 5616000    0    1

 00 sum                                      144000.00 144000.00     39    34701     0.25 39975.55 4996.94   35814 5616000    0    1

 Memory usage

 size_64        : 0

 size_128       : 0

 size_256       : 1

 size_512       : 0

 size_1024      : 0

 size_2048      : 22

 size_4096      : 0

 size_8192      : 0

 size_16384     : 0

 Total    :      44.25 Kbytes  141% util due to buckets

*******Pretest after resolving ********

Pre test info start ===================

Port 0:

  Sources:

    ip: 192.168.251.20 mac: 00:MAC

  Destinations:

    ip: 192.168.251.1 mac: 00:MAC

Port 1:

  Sources:

    ip: 192.168.252.20 mac: 00:MAC

  Destinations:

    ip: 192.168.252.1 mac: 00:MAC

Pre test info end ===================

[hanoh haim]

Do you have the output with -c 2. ?

--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/93f92af3-2ec6-4617-bcd1-8b51768d48e8n%40googlegroups.com.

--

Hanoh

Sent from my iPhone

[Tom Kendrick]

EAL: Detected 12 lcore(s)

EAL: Detected 1 NUMA nodes

EAL: Detected static linkage of DPDK

EAL: Multi-process socket /var/run/dpdk/instance1/mp_socket

EAL: Selected IOVA mode 'PA'

EAL: No available 1048576 kB hugepages reported

[hanoh haim]

I see, only STL and ASTF modes support multi core.

Try ASTF

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/e2fceb74-2036-4ff5-941d-295724e346fbn%40googlegroups.com.

[Tom Kendrick]

Hanoch, thanks for the reply.

So, I switched to advanced stateful mode,  but seemed to have moved the problem (or made more!) Sorry - I'm not a developer, so I'll explain the challanges.

Background My goal, is to be able to launch a script, to allow large volume of traffic generation (~40Gbps to start with but to go much higher in time).  I need a script simple for anyone to pick up and run (none developers) :)

When I changed to -astf mode, and use the guide, I cant launch the console using - ./trex-console -s [server-ip].  I get: 

sudo ./trex-console -v -f

Using 'python3' as Python interpeter

Log:

*** [RPC] - Failed to get server response from tcp://localhost:4501

When using interactive mode, I can launch with -c 4 , but there's no traffic to make anything happen. What did I miss?

sudo ./t-rex-64 --cfg /etc/trex/trex_ASTF.yaml -i -c 4 --astf

If I switch to use a a py file, to create the clients and servers, I see in the notes, that this feature may be removed, and using this mode returns the 1 core limitation (as shown in the documentation).

I changed and copied  astf/http_simple.py's ip_gen_c and ip_gen_s values,  and ran:

sudo ./t-rex-64 --astf --cfg  /etc/trex/trex_ASTF.yaml -f astf/LS.py -m $multiplier -d $duration -w 7 -c 4

The script fails to run, due to -c != 1, so I need to change back to c 1, so I'm back where I started, just in astf mode, I think :)

 STF_BATCH mode does not support more than one core by dual interfaces

try with setting the number of cores to 1 (-c 1)

[hanoh haim]

Hi, 

I'm not sure what is the performance of the bonding DPDK driver. You will need to invest time to understand how ASTF works. It is not like STF with CLI. 

It is an API based, but you would be able to achieve your goal. 

try to ASTF examples 

Thanks

Hanoh