Need help to get line rate of 100Gbps on ConnectX-5, any limitation ?

[saichandra...@gmail.com]

Hi ,

I am trying to get line rate on 100G card , card used is Mellanox ConnectX-5.

I am not able scale beyond ~35Gbps per port , is there any limitation for X-5 or am i missing some config parameters ?

Stream Details :- Untagged,IPV4,UDP 1500 byte packet with random payload .

Using Trex v 2.43 with stateless GUI 4.4-0-ga199804 & CentOS Linux release 7.4.1708 (Core).

Ports are connected back-to-back 2*100G from same card , using 14CPUs from Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz

Hardware Info :

==========

19:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
19:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
19:00.2 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
19:00.3 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
3b:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]  <<<<<<<<<<<<<<<<<<<<
3b:00.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]  <<<<<<<<<<<<<<<<<<<<
5e:00.0 Ethernet controller: Intel Corporation 82575EB Gigabit Network Connection (rev 02)
5e:00.1 Ethernet controller: Intel Corporation 82575EB Gigabit Network Connection (rev 02)
b1:00.0 Ethernet controller: Intel Corporation Ethernet Switch FM10000 Host Interface
b2:00.0 Ethernet controller: Intel Corporation Ethernet Switch FM10000 Host Interface

Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                40
On-line CPU(s) mask:   0xffffffffff
Thread(s) per core:    1
Core(s) per socket:    20
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 85
Model name:            Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
Stepping:              4
CPU MHz:               2400.000
BogoMIPS:              4800.00
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              1024K
L3 cache:              28160K
NUMA node0 CPU(s):     0x5555555555
NUMA node1 CPU(s):     0xaaaaaaaaaa

Thanks and Regards,

Sai Chandra .

[hanoh haim]

Hi, 

This is the expected results. 

https://trex-tgn.cisco.com/trex/doc/trex_appendix_mellanox.html#_connect_5x_maximum_throughput

I would start from here:

https://trex-tgn.cisco.com/trex/doc/trex_stateless_bench.html

thanks,

Hanoh

--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+unsubscribe@googlegroups.com.
To post to this group, send email to trex...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/08c0f311-0684-4b60-a1fa-d6929ab2c2ec%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Hanoh

Sent from my iPhone

[saichandra...@gmail.com]

Hi Hanoh,

  If you don't mind can you explain a little bit why it's expected , I referred the links provided.

  As per the document i am expecting 100% line rate .

  Please be noted i am using 1500 Byte packet size not 64 .

             4. Connect-5x maximum throughput

Table 1. TRex unidirectional traffic matrix

Packet size	Tx L1 Percent (100Gb)	Rx L1 Percent (100Gb)
64	100%	88%
128	100%	100%
256	100%	100%
512	100%	100%
1024	100%	100%
1500	100%	100% <<<<

To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+u...@googlegroups.com.

To post to this group, send email to trex...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/08c0f311-0684-4b60-a1fa-d6929ab2c2ec%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[hanoh haim]

This are the numbers measured on our low end platform with CX-5EN version.

Have a look on the setup information in the other reports. You should get to those numbers with the benchmark described in the report. Suggest to start from there.

Thanks,

Hanoh

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/76b9f37d-d431-449e-8b59-201c794f430b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[alon....@gmail.com]

I don't think you can run 200G (in+out) on the same NIC.

The PCIe is bottleneck. 

בתאריך יום רביעי, 25 ביולי 2018 בשעה 08:49:23 UTC+3, מאת saichandra...@gmail.com:

[Sai Chandra Reddy Mudireddy]

thanks for comment , i didn't get time to recheck this from my last run , we do have only one NIC in our lab .

Any suggestions on how much throughput i must expect from same NIC (in +out ), at present i am getting ~65G with 2*100G ports on same nic .

Thanks,

Sai Chandra .

--
You received this message because you are subscribed to a topic in the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/trex-tgn/Az7tiLB8AgY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to trex-tgn+u...@googlegroups.com.

To post to this group, send email to trex...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/042bba7c-48dc-4909-b31e-a406b99eedea%40googlegroups.com.

[Maik Pfeil]

Hi,

with flow statistics enabled you will only reach 2x 35 Gbps...  You must disable flow statistics on streams to reach 2x 100 Gbps. Did you tried this?

Instead of:

            self.stream = STLStream(name = self.name,

                         packet = self.pkt,

                         flow_stats = STLFlowStats(pg_id = self.pg_id),    <<< this must be removed

                         mode = STLTXCont(bps_L2 = int(self.bps_L2)))

You should only do:

            self.stream = STLStream(name = self.name,

                         packet = self.pkt,

                         mode = STLTXCont(bps_L2 = int(self.bps_L2)))

Save frame size is 192 byte, everything below and you will see drops.

For eg (with 192 byte frame size):

I don't use GUI, so I don't know if there is any knob for this...

[Sai Chandra Reddy Mudireddy]

Hi Maik pfeil ,

No i didn't try by disabling flow stats , i will check and confirm .

Thanks for the inputs .

Thanks and Regards,

Sai Chandra .

--
You received this message because you are subscribed to a topic in the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/trex-tgn/Az7tiLB8AgY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to trex-tgn+u...@googlegroups.com.
To post to this group, send email to trex...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/4e16de4f-de7e-4dab-82dc-a1595c1a413e%40googlegroups.com.

[hakh...@redhat.com]

Hi Sai,

Were you able to generate traffic more than 50gbps with your conntectX-5 dual port 100 gig nic?

I am facing similar issue. I disabled stats stream still seeing only ~45gbs (21 mpps) with frame size 256 bytes.

I used 14 cores but didnt help. 

Any idea how to increase this throughput number?

Thanks

-Haresh

On Friday, August 31, 2018 at 2:42:57 PM UTC+5:30, Sai Chandra Reddy Mudireddy wrote:

Hi Maik pfeil ,

No i didn't try by disabling flow stats , i will check and confirm .

Thanks for the inputs .

Thanks and Regards,

Sai Chandra .

On Thu, Aug 30, 2018 at 7:31 PM Maik Pfeil <maik...@gmail.com> wrote:

Hi,

with flow statistics enabled you will only reach 2x 35 Gbps...  You must disable flow statistics on streams to reach 2x 100 Gbps. Did you tried this?

Instead of:

            self.stream = STLStream(name = self.name,

                         packet = self.pkt,

                         flow_stats = STLFlowStats(pg_id = self.pg_id),    <<< this must be removed

                         mode = STLTXCont(bps_L2 = int(self.bps_L2)))

You should only do:

            self.stream = STLStream(name = self.name,

                         packet = self.pkt,

                         mode = STLTXCont(bps_L2 = int(self.bps_L2)))

Save frame size is 192 byte, everything below and you will see drops.

For eg (with 192 byte frame size):

I don't use GUI, so I don't know if there is any knob for this...

--
You received this message because you are subscribed to a topic in the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/trex-tgn/Az7tiLB8AgY/unsubscribe.

To unsubscribe from this group and all its topics, send an email to trex...@googlegroups.com.

[陈敬海]

Which file was modified?  " flow_stats = STLFlowStats(pg_id = self.pg_id),     "

What mode can I use for streaming? I limit the use of stl mode to 70Gbps

[陈敬海]

I modified the file and commented it out   " automation/trex_control_plane/interactive/trex/examples/stl/ndr_bench_fs_latency.py  "

However, the speed can reach 100Gbps when the current is diverted, but an error occurs. What is the problem?


[root@localhost v3.00]# python automation/trex_control_plane/interactive/trex/examples/stl/ndr_bench_fs_latency.py --port 0 1 --max-latency 100
Bench run has finished for size imix (1\5).

Argument: 'mult' invalid value: '-26585438868.7bps'

在2018年8月30日星期四 UTC+8 22:01:21<maik....@gmail.com> 写道：

[陈敬海]