Low CPS rate with 30 core

[Vin Iov]

Hello, 
i'm running the TRex on a VM with 30 core and 2 Mellanox ConnectX-5 adapters to reach  ~200 Gbps (server/client) and i'm perfectly able to reach 80/80 Gbps before seeing errors.

I was testing the CPS rate and i think i'm missing something else because i can't go over 1.4Mcps without having a 100% CPU usage.

I tried adjusting the timeout but the performance is very different if compared to what's written in the documentation.

I'm already using a specific crafted pcap file to the cps.
Do i need to adjust something else?

This is the config i'm testing:

----

Running with the command: start -f custom/tcprate.py -m 3000000

[Vin Iov]

Also, if i try to increase the number of cores, i received this error:
 set driver name net_mlx5
 driver capability  : TCP_UDP_OFFLOAD  TSO  LRO
 set dpdk queues mode to RSS_DROP_QUE_FILTER
 Number of ports found: 2
EAL: Error - exiting with code: 1
  Cause: Error: Number of TX queues exceeds 48. Try running with lower -c <val>
root@trex-b:/opt/trex/v3.04#

[Цветков Андрей]

Hi .
My config :
[{"port_limit": 2, "version": 2, "interfaces": ["0000:3b:00.1", "0000:3b:00.0"], "low_end": false, "enable_zmq_pub": true, "prefix": 1, "zmq_pub_port": 4500, "zmq_rpc_port": 4501
, "trex_zmq_port": 4500, "stack": "linux_based", "c": 20, "platform": {"master_thread_id": 0, "latency_thread_id": 21, "dual_if": [{"socket": 0, "threads": [1, 2, 3, 4, 5, 6, 7,
8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20]}]}}]

where :
low_end: False - it is hardware mode (dpdk) 

low_end: True - it is software mode limit 1 mpps, use 1 cpu and do not use platform section
"c" - cores . where the number of cores is equal to the number of threads

I don't see that the counters (Tx bps / Rx bps) have been increasing.

четверг, 23 мая 2024 г. в 19:31:42 UTC+3, Vin Iov:

[Цветков Андрей]

Hi
you said that you are using a virtual machine - you passthrough 2 Mellanox into VM ?

As I understand it, the connection of the CPU socket and the network card must be consistent.
There are 2 CPUs in the virtual machine , each with 15 cores .
CPU 1 to network card 1, and CPU 2 to network card 2. Then you can add the "socket" section: 1,

Make a virtual machine with 1 CPU with 30 cores.

I have 2 CPUs with 10 cores (20 threads) - I can only use1 CPUs connected to the network card via pcie directly.

суббота, 25 мая 2024 г. в 21:44:32 UTC+3, Цветков Андрей:

[Vin Iov]

Hello Андрей, Thank you so much for your reply.

I have a VM with 32 vCPUs, each with 1 Core assigned. Could this configuration be causing issues?

I've enabled passthrough and i am able to reach around 160 Gbps of ASTF traffic client/server. However, after 160 Gbps, I start to see the queue_full counter increasing, is there something i could do to increase the queue (if that's the problem).

This issue is perplexing me; I'm unsure of what I might be doing wrong. The CPUs don't reach full utilization with 160 Gbps of traffic but is not able to reach 1.5 million packets per second (Mcps). I have attached a screenshot of my ASTF traffic. Perhaps it could help with the diagnosis (i've got some RX Thread throttled), here the CPU usage is around 60/70%

[Цветков Андрей]

trex_book.asciidoc :
       #rx_desc         : 1024 # optional <17>
       #tx_desc         : 1024 # optional <17>
<17> Override the default number of Rx/TX dpdk driver descriptors. For better performance on virtual interfaces better to enlarge these numbers to Rx=4096

Setups with virtual interfaces (SR-IOV/VMXNET3/virtio) has by default, only 512 ex descriptors per queue,
better to enlarge it for multi-rx-queue/high performance setups

trex_faq.asciidoc
===== TPG packets are dropping at low rate

The NIC or software redirecting rings are full.
Are you sending a burst of packets at the same time? Please increase the number of rx descriptors to be bigger than the burst size,
or introduce `isg` to avoid the burst.

суббота, 25 мая 2024 г. в 23:43:48 UTC+3, Vin Iov:

[Vin Iov]

I've already tried tuning those parameters, but I'm using the Mellanox (physical) interfaces with passthrough, not the VMXNET interfaces
This is the result with rx_desc: 4096 and tx_desc: 4096, which completely degraded the performance:


Again, thank you for trying to help me, i'm completely lost on this one. Maybe i am missing some tuning on the Linux Kernel parameters?

[Цветков Андрей]

Connect 2 Mellanox together. Setup L2 connect on Trex and run test without DUT/SUT.

воскресенье, 26 мая 2024 г. в 02:28:21 UTC+3, Vin Iov: