40g performance test problem

[cka...@gmail.com]

Hello friends!

I have DUT with 40g theoretical throughput.
I'm having a problem while running performance test with TRex 40g NIC's.

I set same speed (for example 10g per port) on 2 dual ports and they started to loose/get link, errors occurs... and then i need to set low speed to get trex work stable.

I tried to rise speed by step, trex in this case works better, but on 10-11g speed per port this thing happens again. 

The problem occurs on big packet size (1500-9000), on 64-724 everything is ok.

What it can be with wrong with my test stand?

[cka...@gmail.com]

forgot to mention

on packet size 724 i get 35g throughput...

[Besart Dollma]

Hi, 

Just because your observation on the packet size: Is your DUT maybe fragmenting the packets? What is the MTU?

Thanks,

[cka...@gmail.com]

DUT's MTU is about 9100 approximately.

[cka...@gmail.com]

here I've got some statistics

[hanoh haim]

Each XL710 NIC could do maximum of ~42gbps due to PCIe limit (total on *all* ports)

It mentioned in our FAQ

Thanks

Hanoh

On Mon, 10 Aug 2020 at 18:22 <cka...@gmail.com> wrote:

here I've got some statistics

--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/168dd48e-afae-41da-ab37-48c46d0bc539o%40googlegroups.com.

--

Hanoh

Sent from my iPhone

[cka...@gmail.com]

Yeap, I’ve got it.
But as I mentioned the problem occurs on lower speeds, (about 10g tx) per port.
What do you mean by “total on all ports” ? I should consider tx and rx speeds on both ports, or only tx speeds on both ports?

[hanoh haim]

Total of all Tx (from all the ports) is limited by PCIe

--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/5da58268-1d1f-416f-904e-d86d5bc20ff9o%40googlegroups.com.

[cka...@gmail.com]

Thanks for information.
I suppose this info refers to my statistic screenshot.
I understand that there is a limit.

But I get strange trex behavior (link looses, errors)
om my 2x40G nic. Some cases for Example:
1) 10G speed/packet size 1518 bi direction
2) case 3G /packet size 9000 uni direction

[hanoh haim]

Did you verify in loopback?

--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/c1a27f2c-0c29-4352-a5dd-6a440969c949o%40googlegroups.com.

[cka...@gmail.com]

In loop back TRex works fine.

Here is a screenshot. Here I ran stl/bench.py, got link down and stopped test. We can see here some strange speed numbers after stopping the test...

[hanoh haim]

Is the screenshot is in loopback?

I suggest to try v2.79 with older DPDK version and check firmware if the driver (-v 7 in CLI)

This issue seems to be a NIC/driver issue.

Thanks

Hanoh

--

You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/532ff49d-d11d-4ab1-93cf-fb0b99fcc39co%40googlegroups.com.

[cka...@gmail.com]

I'm sorry for wrong info.
That screenshot is not in loopback. 
It shows whats happen when I test my device.
Here is a screenshot for loopback case.

[cka...@gmail.com]

Here I've got a video of whats happening with my DUT

https://yadi.sk/i/nGGvhlb_5vBVag

[hanoh haim]

Understood, saw the video.

it seems as your DUT issue with interoperability with Intel NIC. The Intel NIC implements the link state machine in the firmware (not software). Suggest to take this issue to Intel and solve the problem in your DUT.

It is not a TRex issue

Thanks

Hanoh

On Tue, 11 Aug 2020 at 15:03 <cka...@gmail.com> wrote:

Here I've got a video of whats happening with my DUT

https://yadi.sk/i/nGGvhlb_5vBVag

--

You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/01ff989e-0267-4c76-87da-98d663dc545eo%40googlegroups.com.

[cka...@gmail.com]

 When you said about interoperability, we decided to connect TRex not directly to DUT, but with mellanox switch. The problem passed away....

Thank you very much!

[Максим Богданов]

Hello, friends!
solve this problem with the settings below:

-latency enabled
-rate bps L1 20g(per each port)

and i get clean TX/RX stats, and no interoperability problems... (trex now connected directly to DUT, without switch)

you can see some stats here: https://yadi.sk/i/JiiunYmCQOiCFg

среда, 12 августа 2020 г. в 17:25:22 UTC+3, cka...@gmail.com:

[hanoh haim]

Could you try adding  —software to CLI and not using Latency Stream?

--


You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.


To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/a760bde5-71ea-4117-8b3d-ec651d07d085n%40googlegroups.com.

[Максим Богданов]

Yes, but the results will be as in video above...
I either get link state down or TX speeds not equal to RX, with losses. 

I can get 39,98 Gbps with p.size=9000 but another question 

why my max line rate is 36,8 Gbps with packet size 1500 bidirectional?

look at the stats: https://yadi.sk/i/sWU3AMl9enyHOA

среда, 19 августа 2020 г. в 12:33:33 UTC+3, Hanoch Haim:

[Максим Богданов]

in case of p.size=700, total line rate will be 17,7 Gbps...

среда, 19 августа 2020 г. в 14:01:15 UTC+3, Максим Богданов:

[Максим Богданов]

what is the idea of lattency in this case? I can't get it.

on smal p.size I get normal results without lattency, but on huge packets I have to use lattency...

среда, 19 августа 2020 г. в 14:04:19 UTC+3, Максим Богданов:

[Максим Богданов]

ok , it reaches max cpu load on small p size (with lattency enabled )...

среда, 19 августа 2020 г. в 14:59:01 UTC+3, Максим Богданов:

[hanoh haim]

You can’t send 40gbps of latency packets, it wasn’t designed for this!

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/7092006b-a340-45f0-b410-1963948bcee1n%40googlegroups.com.

[Максим Богданов]

Do you mean that i can't test perfomance with a latency packets (2x20gbps latency streams) and that it was not designed to test perfomance?
where can i read more info about latency streams?

четверг, 20 августа 2020 г. в 18:24:12 UTC+3, Hanoch Haim:

[hanoh haim]

Read the FAQ and manual 

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/5a3a0640-629d-4494-abea-ff3484e5f590n%40googlegroups.com.

[Максим Богданов]

Ok, I have read about latency idea. thanks for information.

And also thanks for your patience in my questions.

Then I use latency as a constant (100pps) stream along main stream.

here is some of my latency statistics for udp packets(p.size=9000):

bidir 10Gbps: jitter avg=1μs, max lattency= 280 μs (no errors in lattency stats)
bidir 14Gbps: jitter avg=1μs, max lattency= 280 μs (no errors in lattency stats)
bidir 20Gbps: jitter avg=8μs, max lattency= 400 μs (some errors appears in lattency stats)
bidir 30Gbps: jitter avg=11μs, max lattency= 400 μs (errors increases in lattency stats)
bidir 40Gbps: jitter avg=15μs, max lattency= 550 μs (errors highly increasing in lattency stats)  

maybe this is a key to understanding why everything goes bad in the test. or the problem is only in interoperability with intel NIC?
I repeat that in my test rx speeds is not equal to tx, so i get a lot of losses...

пятница, 21 августа 2020 г. в 12:13:31 UTC+3, Hanoch Haim:

[hanoh haim]

Is it in loopback? On our setup max Windows is about 5-10usec

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/cd4a9cb1-1d66-4e5a-9703-1505638fad70n%40googlegroups.com.

[Максим Богданов]

no, it was with DUT.

here is stats in loopback: img

[Максим Богданов]

strange things noticed  

my server has 2x Intel Xeon e5-2650 v3 (10cores 20threads per cpu)

With core number c=20 in cfg

my stats are bad

bidir: total l1 tx 40G, total l1 rx 36G (test with DUT)   

10% total p. losses

latency avg=550 

but I got better speed results, and less latecy/jitter with decreasing cores number to c=1

in test bidirectional: total l1 tx 40G, total l1 rx 39,2G (test with DUT) 

latency avg=350   

2% total p. losses
cpu usage 20%~

пятница, 28 августа 2020 г. в 14:58:03 UTC+3, Максим Богданов: