STL not hitting line rate

[tge...@gmail.com]

Hello,

I am running TRex v2.60 with two single-port Mellanox CX-5s (100 G) in loopback (using 20 cores in STL mode). I have never been able to hit line rate on a stateless test, yet TRex should be able to handle 100G traffic. 

In the first test, I am pushing a continuous stream of 1000 Byte packets at 100% linerate, yet it always maxes out at ~82% and the CPU util is relatively high and queue_full goes way up:

What exactly does queue_full mean (I read elsewhere on here that the NIC may be backpressuring), but how can I fix this to get it to run at line rate?

-Per port stats table

      ports |               0 |               1

 -----------------------------------------------------------------------------------------

   opackets |        96926146 |               0

     obytes |     96926154964 |               0

   ipackets |               0 |        96594528

     ibytes |               0 |     96594537960

    ierrors |               0 |               0

    oerrors |               0 |               0

      Tx Bw |      82.76 Gbps |       0.00  bps

 

-Global stats enabled

 Cpu Utilization : 78.1  %  10.6 Gb/core

 Platform_factor : 1.0

 Total-Tx        :      82.76 Gbps

 Total-Rx        :      82.05 Gbps

 Total-PPS       :      10.34 Mpps

 Total-CPS       :       0.00  cps

 

 Expected-PPS    :       0.00  pps

 Expected-CPS    :       0.00  cps

 Expected-BPS    :       0.00  bps

 

 Active-flows    :        0  Clients :        0   Socket-util : 0.0000 %

 Open-flows      :        0  Servers :        0   Socket :        0 Socket/Clients :  -nan

 Total_queue_full : 152397469

 drop-rate       :       0.00  bps

 current time    : 13.8 sec

 test duration   : 0.0 sec

Now, changing the percentage down to 80% is all fine (why is CPU util so low here, but changing it to 100% jumps up so high?):

-Per port stats table

      ports |               0 |               1

 -----------------------------------------------------------------------------------------

   opackets |       120070340 |               0

     obytes |    120070347968 |               0

   ipackets |               0 |       120099234

     ibytes |               0 |    120099243960

    ierrors |               0 |               0

    oerrors |               0 |               0

      Tx Bw |      78.28 Gbps |       0.00  bps

 

-Global stats enabled

 Cpu Utilization : 1.2  %  670.8 Gb/core

 Platform_factor : 1.0

 Total-Tx        :      78.28 Gbps

 Total-Rx        :      78.28 Gbps

 Total-PPS       :       9.78 Mpps

 Total-CPS       :       0.00  cps

 

 Expected-PPS    :       0.00  pps

 Expected-CPS    :       0.00  cps

 Expected-BPS    :       0.00  bps

 

 Active-flows    :        0  Clients :        0   Socket-util : 0.0000 %

 Open-flows      :        0  Servers :        0   Socket :        0 Socket/Clients :  -nan

 Total_queue_full : 9

 drop-rate       :       0.00  bps

 current time    : 16.9 sec

 test duration   : 0.0 sec

Also, I am not sure if this is related or intended behavior, but if I turn on STLFlowStats, the maximum RX rate I can achieve is ~41G and the rest of it is dropped. Could you provide some input? This is at 80% line rate.

-Per port stats table

      ports |               0 |               1

 -----------------------------------------------------------------------------------------

   opackets |       132666891 |               0

     obytes |    132666900960 |               0

   ipackets |               0 |        70381054

     ibytes |               0 |     70381059976

    ierrors |               0 |           10690

    oerrors |               0 |               0

      Tx Bw |      78.29 Gbps |       0.00  bps

 

-Global stats enabled

 Cpu Utilization : 1.3  %  597.3 Gb/core

 Platform_factor : 1.0

 Total-Tx        :      78.29 Gbps

 Total-Rx        :      41.53 Gbps

 Total-PPS       :       9.79 Mpps

 Total-CPS       :       0.00  cps

 

 Expected-PPS    :       0.00  pps

 Expected-CPS    :       0.00  cps

 Expected-BPS    :       0.00  bps

 

 Active-flows    :        0  Clients :        0   Socket-util : 0.0000 %

 Open-flows      :        0  Servers :        0   Socket :        0 Socket/Clients :  -nan

 drop-rate       :      36.76 Gbps

 current time    : 18.0 sec

 test duration   : 0.0 sec

Thank you

[hanoh haim]

Hi,

The recommendations in the appendix is to use the CX-5 version with the two ports.

Try to search why we are recommending it and why your performance is low.

Thanks

Hanoh

--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/e2e9b26f-049d-42bb-aa95-004c6738c377%40googlegroups.com.

--

Hanoh

Sent from my iPhone

[tge...@gmail.com]

MCX516A-CCAT and MCX515A-CCAT will be a bit slower.

Can you explain why these will be slower? What does queue_full indicate exactly?

To unsubscribe from this group and stop receiving emails from it, send an email to trex...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/e2e9b26f-049d-42bb-aa95-004c6738c377%40googlegroups.com.

[tge...@gmail.com]

Solved: issue was not TRex related. The single-port CX-5s are able to get the expected performance as well.