ierrors counter Increments for traffic at 60% of line rate on E810-CQDA2 NIC

[vishwanath pai]

Hi,

I am using E810-CQDA2 NIC with Trex to generate traffic. Currently NIC has external loopback and is configured in 25G Mode.  

I am facing 2 issues:

1. If I increase the rate to about 60% using Trex, I see ierror packets at Rx side Incrementing. Is it because we have only 1 core for Rx? 

2. The packets having ip/tcp/udp checksum error are shown as good pkts in the stats. I see that Rx checksum offload is enabled by default by Trex. It seems that NIC is not verifying the checksum. The traffic sent very basic Eth->IPv4->TCP.  What is the option to enable checksum offload at Rx. 

I am using latest  Trex code and drivers . Any assistance about how to work around the Issues will be really helpful. 

Thanks

Vishwanath Pai

[hanoh haim]

1. Yes

2. Forward this issue to dpdk team. The driver is rather new 

--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/CAF4-L%2Bu9GbBTFPf68UnxEK9KF7O9bL2%2Bpyuz%3DgBWQ%3DGE81W%2B3A%40mail.gmail.com.

--

Hanoh

Sent from my iPhone

[vishwanath pai]

Hi,

Thanks for quick reply, What could I do to get rid of ierror packets at Rx side Incrementing. Is there an option to add more core for Rx?  Is it on the roadmap for Trex In future?  

Thanks

Vishwanath Pai

[hanoh haim]

Just add more cores. There is a roadmap (Intel) to add support to the driver to make it work in hardware mode 

[vishwanath pai]

Hi,

After some study with Trex, I don't think one Rx thread (rx_thread_id) is enough to support 4 x25G ports in a Trex Server instance. The queues are getting full and packets are getting Dropped. I guess Trex Only supports  NX10G ports.

2X25G worked without any issue in a trex server instance, But anything beyond that fails with the Rx queue full.  Is there a way to add more Rx Cores?

Thanks

Vishwanath Pai

.    

On Wed, Apr 28, 2021 at 11:59 AM vishwanath pai <vishw...@gmail.com> wrote:

Hi,

I went through the Trex Documentation. I didn't find any attribute which can be given in trex_cfg.yaml to Increase Rx Core Per Port When Trex Server is launched.  -c option is used for DP Cores(Tx). 

Are there any Benchmark values for E810-CQDA2 using Trex? 

Thanks

Vishwanath Pai  

[hanoh haim]

What is the command/mode that you are using? 

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/CAF4-L%2BvAU3YfHQa54KFQ5OGfcC-UuY5fOd2V%2BC8RMj5%3D93F8pA%40mail.gmail.com.

[vishwanath pai]

Hi,

I am using stateless mode 

./t-rex-64 -i --prom -v 7

cfg file:

- version    : 2
  port_limit : 4
  interfaces : ['44:00.0', '44:00.1','44:00.2','44:00.3']
  limit_memory : 2048
  port_info  :
  - dest_mac   : '00:00:00:01:00:00'
    src_mac    : '00:00:00:02:00:00'
  - dest_mac   : '00:00:00:03:00:00'
    src_mac    : '00:00:00:04:00:00'
  - dest_mac   : '00:00:00:05:00:00'
    src_mac    : '00:00:00:06:00:00'
  - dest_mac   : '00:00:00:07:00:00'
    src_mac    : '00:00:00:08:00:00'
  c : 4
  platform:
          master_thread_id: 16
          rx_thread_id: 17
          dual_if:
                  - socket : 1
                    threads: [18, 19, 20 ,21]
                  - socket : 1
                    threads: [22, 23, 24, 25]

All 4 interfaces are in 25G mode.

Thanks

Vishwanath Pai 

[hanoh haim]

What is the profile and the issue?

You might using low number of flows 

[vishwanath pai]

Hi,

I am using 1 stream in profile per port which sends continuous traffic at 100% of line rate. These ports are loopback with external cables, so it receives traffic at the same rate. 

So 4X25G ports sending and receiving traffic at line rate (25Gbps) , this leads to increment of ierrors(rx_missing errors). 

Same works fine for 2X25G ports without any errors.  

Thanks

Vishwanath Pai

[hanoh haim]

Please share the profile 

[vishwanath pai]

Hi,

Attaching my Trex Traffic profile

Thanks

Vishwanath Pai

[hanoh haim]

Don’t you have a python profile for stateless? 

[vishwanath pai]

Hi,

Attaching python profile for stateless. 

[hanoh haim]

The issue is simple as I wrote in the previous email. The distribution to cores is done by the 5 tuple. To use all the cores in the Rx side add more flows.

Try bench.py with size=imix 

[vishwanath pai]

Hi,

Thanks, I will try your suggestion. I had one more observation which I can't understand. 

for 2 X25G Server Instance I see below stats in Trex Server

Cpu Utilization : 29.8  %  46.6 Gb/core

for 4 X25G Server Instance I see below stats in Trex Server  

Cpu Utilization : 7.7  %  180 Gb/core

What does XGb/core signify here?

[vishwanath pai]

Hi,

As suggested I used different traffic flows in profile but still see the same issue. Below is what I observed which could be related to ierror(rx_missing_errors) I am seeing. 

PORTA and PORTB are connected back to back on the server with external cable. 

1. Init, load profile with 4 flows each at 25% line rate

        PORTA(25G)[IDLE]<---------------> PORTB(25G)[IDLE]

2. Start Transmit on PORTA, I see traffic received on PORTB and ierror incrementing. 

        PORTA(25G)[TRANSMITTING]<---------------> PORTB(25G)[IDLE]  

3. Start Transmit on PORTB, I see traffic received on PORTA & PORTB and no ierrors now. 

        PORTA(25G)[TRANSMITTING]<---------------> PORTB(25G)[ TRANSMITTING ]  

I see that ierrors stop when ports are in "TRANSMITTING" state. I went ahead and tried t-rex-64-debug-gdb.

I see that TrexStatelessDpCore::start_schedule does add rx_node. 

if ( get_dpdk_mode()->dp_rx_queues() ){                      
    // add rx node if needed                                  
    CGenNode * node_rx = m_core->create_node() ;              
    node_rx->m_type = CGenNode::STL_RX_FLUSH;                
    node_rx->m_time = now_sec(); /* NOW to warm thing up */  
    m_core->m_node_gen.add_node(node_rx);                    
}                                                            
                                                              

I am new to Trex So not able to make out the exact reason for this in start_Schedule which invokes when we do call start()

Thanks

Vishwanath Pai

[vishwanath pai]

Hi

Adding tui output for reference.  Rx is only able to do 5mpps for a packet size of 64bytes.  

Global Statistics

connection   : 10.190.162.100, Port 4501                  total_tx_L2  : 29.7 Gbps
version      : STL @ v2.89                                total_tx_L1  : 38.98 Gbps
cpu_util.    : 44.68% @ 12 cores (6 per dual port)        total_rx     : 28.58 Gbps
rx_cpu_util. : 0.0% / 0 pps                               total_pps    : 58.01 Mpps
async_util.  : 0% / 0 bps                                 drop_rate    : 0 bps
total_cps.   : 0 cps                                      queue_full   : 0 pkts

Port Statistics

   port    |         0         |         1         |         2         |         3         |       total
-----------+-------------------+-------------------+-------------------+-------------------+------------------
owner      |             vipai |             vipai |             vipai |             vipai |
link       |                UP |                UP |                UP |                UP |
state      |      TRANSMITTING |      TRANSMITTING |      TRANSMITTING |      TRANSMITTING |
speed      |           25 Gb/s |           25 Gb/s |           25 Gb/s |           25 Gb/s |
CPU util.  |            33.55% |            55.58% |            33.89% |            55.69% |
--         |                   |                   |                   |                   |
Tx bps L2  |         8.57 Gbps |          6.3 Gbps |         8.57 Gbps |         6.27 Gbps |         29.7 Gbps
Tx bps L1  |        11.25 Gbps |         8.26 Gbps |        11.25 Gbps |         8.22 Gbps |        38.98 Gbps
Tx pps     |        16.74 Mpps |         12.3 Mpps |        16.74 Mpps |        12.24 Mpps |        58.01 Mpps
Line Util. |           44.99 % |           33.05 % |           44.99 % |            32.9 % |
---        |                   |                   |                   |                   |
Rx bps     |         8.22 Gbps |         6.06 Gbps |         8.21 Gbps |         6.08 Gbps |        28.58 Gbps
Rx pps     |         5.87 Mpps |         5.87 Mpps |         5.57 Mpps |         5.57 Mpps |        22.88 Mpps
----       |                   |                   |                   |                   |
opackets   |        1139015637 |         836747885 |        1139062070 |         832781385 |        3947606977
ipackets   |         399373637 |         399373820 |         379096189 |         379095969 |        1556939615
obytes     |       72897000768 |       53551864384 |       72899972928 |       53298006272 |      252646844352
ibytes     |       69941193928 |       51564364768 |       69857349120 |       51721258844 |      243084166660
tx-pkts    |        1.14 Gpkts |      836.75 Mpkts |        1.14 Gpkts |      832.78 Mpkts |        3.95 Gpkts
rx-pkts    |      399.37 Mpkts |      399.37 Mpkts |       379.1 Mpkts |       379.1 Mpkts |        1.56 Gpkts
tx-bytes   |           72.9 GB |          53.55 GB |           72.9 GB |           53.3 GB |         252.65 GB
rx-bytes   |          69.94 GB |          51.56 GB |          69.86 GB |          51.72 GB |         243.08 GB
-----      |                   |                   |                   |                   |
oerrors    |                 0 |                 0 |                 0 |                 0 |                 0
ierrors    |       739,687,998 |       433,407,368 |       759,919,856 |       457,651,913 |     2,390,667,135

status:  |

Press 'ESC' for navigation panel...
status:

tui(read-only)>

Thanks

Vishwanath Pai

[hanoh haim]

Hi, 

Try to add more rx descriptors (8K) and 1K tx descriptors 

[vishwanath pai]

Hi,

I tried it now, I am not seeing any improvements. Do we have any benchmark numbers for  25G ports (Any NIC) at Line Rate. I just wanted to know what is the highest rate we can achieve without any frame drops or ierror packets. 

Global Statistics

connection   : 10.190.162.100, Port 4501                  total_tx_L2  : 79.52 Gbps
version      : STL @ v2.89                                total_tx_L1  : 82.67 Gbps
cpu_util.    : 98.72% @ 12 cores (6 per dual port)        total_rx     : 73.29 Gbps
rx_cpu_util. : 0.0% / 0 pps                               total_pps    : 19.65 Mpps

async_util.  : 0% / 0 bps                                 drop_rate    : 0 bps

total_cps.   : 0 cps                                      queue_full   : 300,673,361 pkts

Port Statistics

   port    |         0         |         1         |         2         |         3         |       total
-----------+-------------------+-------------------+-------------------+-------------------+------------------
owner      |             vipai |             vipai |             vipai |             vipai |
link       |                UP |                UP |                UP |                UP |
state      |      TRANSMITTING |      TRANSMITTING |      TRANSMITTING |      TRANSMITTING |
speed      |           25 Gb/s |           25 Gb/s |           25 Gb/s |           25 Gb/s |

CPU util.  |            98.01% |            98.01% |            99.28% |            99.57% |
--         |                   |                   |                   |                   |
Tx bps L2  |        19.88 Gbps |        19.88 Gbps |        19.88 Gbps |        19.88 Gbps |        79.52 Gbps
Tx bps L1  |        20.67 Gbps |        20.67 Gbps |        20.67 Gbps |        20.67 Gbps |        82.67 Gbps
Tx pps     |         4.91 Mpps |         4.91 Mpps |         4.91 Mpps |         4.91 Mpps |        19.65 Mpps
Line Util. |           82.67 % |           82.67 % |           82.67 % |           82.67 % |
---        |                   |                   |                   |                   |
Rx bps     |        18.32 Gbps |        18.32 Gbps |        18.32 Gbps |        18.32 Gbps |        73.29 Gbps
Rx pps     |         4.41 Mpps |         4.42 Mpps |         4.42 Mpps |         4.42 Mpps |        17.67 Mpps
----       |                   |                   |                   |                   |
opackets   |         189413065 |         189413057 |         189412857 |         189412849 |         757651828
ipackets   |         170036582 |         170274152 |         170256979 |         170266212 |         680833925
obytes     |       95820867942 |       95820863398 |       95820758904 |       95820755356 |      383283245600
ibytes     |       88281831200 |       88277024474 |       88277351226 |       88277169436 |      353113376336
tx-pkts    |      189.41 Mpkts |      189.41 Mpkts |      189.41 Mpkts |      189.41 Mpkts |      757.65 Mpkts
rx-pkts    |      170.04 Mpkts |      170.27 Mpkts |      170.26 Mpkts |      170.27 Mpkts |      680.83 Mpkts
tx-bytes   |          95.82 GB |          95.82 GB |          95.82 GB |          95.82 GB |         383.28 GB
rx-bytes   |          88.28 GB |          88.28 GB |          88.28 GB |          88.28 GB |         353.11 GB

-----      |                   |                   |                   |                   |
oerrors    |                 0 |                 0 |                 0 |                 0 |                 0

ierrors    |        19,376,467 |        19,138,783 |        19,155,892 |        19,146,643 |        76,817,785

status:  /

Press 'ESC' for navigation panel...
status:

tui(read-only)>

[hanoh haim]

Hi Vishwanath,

have a look here for XL710 performance, the 64B is limited by the chip

https://trex-tgn.cisco.com/trex/doc/trex_stateless_bench.html 

You are using a NIC that has a very basic support and is not part of our regression due to missing parts in the driver. 

Mellanox CX-5/6 can support ~140MPPS @  64B packets for each NIC

Thanks

Hanoh

[vishwanath pai]

Thanks Hanoh, really appreciate your patience in answering my queries.