Unable to get above 60Gbit with Mellanox ConnectX-6 100G in ASTF mode

[Dima Glebov]

Hi, I'm using Mellanox ConnectX-6 Dual Port.

in ASTF mode I'm getting 'queue_full' when trying to push above 60Gbit how can i improve this ? should it be possible to reach 100Gbit in ASTF mode with this NIC ?

thanks.

config:

interfaces: ['03:00.0', '03:00.1']
  stack: linux_based
  prefix: instance1
  zmq_pub_port: 4700
  zmq_rpc_port: 4701
  limit_memory: 4096
  port_info:
      - ip: 192.168.255.1
        default_gw: 192.168.255.254
      - ip: 192.168.255.254
        default_gw: 192.168.255.1

  platform:
      master_thread_id: 0
      latency_thread_id: 31
      dual_if:
        - socket: 0
          threads: [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30]

traffic profile:

from trex.astf.api import *


class Prof1():
    def __init__(self):
        pass

    def get_profile(self, **kwargs):
        # ip generator
        ip_gen_c = ASTFIPGenDist(ip_range=["172.90.0.0", "172.90.255.255"], distribution="seq")
        ip_gen_s = ASTFIPGenDist(ip_range=["172.100.0.0", "172.100.255.255"], distribution="seq")
        ip_gen = ASTFIPGen(glob=ASTFIPGenGlobal(ip_offset="1.0.0.0"),
                           dist_client=ip_gen_c,
                           dist_server=ip_gen_s)
        c_glob_info = ASTFGlobalInfo()
        c_glob_info.tcp.mss = 1300
        c_glob_info.tcp.initwnd = 1

        s_glob_info = ASTFGlobalInfo()
        s_glob_info.tcp.mss = 1300
        s_glob_info.tcp.initwnd = 1

        return ASTFProfile(default_ip_gen=ip_gen,
                           #  Defaults affects all files
                           default_c_glob_info=c_glob_info,
                           default_s_glob_info=s_glob_info,
                            cap_list=[
                                ASTFCapInfo(file="../avl/5MB-exe.cap",cps=1),
                            ])


def register():
    return Prof1()

[hanoh haim]

Hi Dima, 

It seems as a driver issue. I would try to **reduce** the number of cores  

Thanks

Hanoh

[Dima Glebov]

Hi Hanoh,

thank you for your support.

I've made some tests and tried reducing the number of cores as you suggested, I got my best results (71Gbps) with the following configuration:

platform:
      master_thread_id: 0
      latency_thread_id: 15
      dual_if:
        - socket: 0
          threads: [1,2,3,4,5,6,7,8,9,10,11,12,13,14]

seems like going above or below 16 cores in total had a negative impact but with 16 cores I certainly got better result.

any further suggestions how to improve the results beyond 70Gbps ? do you think its Mellanox driver limitation ?

Thanks.

[hanoh haim]

Hi Dima, 

It does not seem as a CPU issue or software issue. 

thanks

Hanoh