interface bonding with Mellanox ConnectX-6 is failing

[Dima Glebov]

Hello,

I'm using Mellanox MT2892 Family ConnectX-6 Dx and my goal is to create 200G LACP bond out of 2x100G interfaces

I'm getting the following error:

Starting Cmds server.... Cmds server is started
The ports are bound/configured.
Starting  TRex v3.03 please wait  ...
bond_ethdev_parse_slave_port_kvarg(125) - Invalid slave port value (47:00.0) specified
bond_ethdev_configure(3754) - Failed to parse slave ports for bonded device net_bonding0
Port0 dev_configure = -1
bond_ethdev_parse_slave_port_kvarg(125) - Invalid slave port value (47:00.1) specified
bond_ethdev_configure(3754) - Failed to parse slave ports for bonded device net_bonding1
Port1 dev_configure = -1
 set driver name net_bonding
 driver capability  : SLRO
 set dpdk queues mode to MULTI_QUE
 Number of ports found: 2
zmq publisher at: tcp://*:4700
Requested TX offload MULTI_SEGS is not supported
Requested RX offload SCATTER is not supported
Killing Cmds server... Cmds server is killed

here is the configuration I'm using:

- version: 2
  interfaces: ['--vdev=net_bonding0,mode=4,slave=47:00.0,slave=03:00.0', '--vdev=net_bonding1,mode=4,slave=47:00.1,slave=03:00.1']
  stack: linux_based
  prefix: instance1
  zmq_pub_port: 4700
  zmq_rpc_port: 4701
  limit_memory: 4096
  port_info:
      - ip: 192.168.255.245
        default_gw: 192.168.255.248
      - ip: 220.0.1.70
        default_gw: 220.0.0.254

  platform:
      master_thread_id: 0
      latency_thread_id: 31
      dual_if:
        - socket: 0
          threads: [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15]
        - socket: 0
          threads: [16,17,18,19,20,21,22,23,24,25,26,27,28,29,30]

just to be clear - without bonding configuration these interfaces work perfect separately. any suggestions ?

[hanoh haim]

Hi Dima, 

We haven't tested this option, only PF and simple VF. Does normal PF works for you? 

I would try same version of DPDK with baseline application on the same machine to understand if DPDK supports it.

Thanks

Hanoh

[Anton Matvienko]

Hello, did you succeed with this? I have the same promblems.

четверг, 15 февраля 2024 г. в 12:03:48 UTC+3, hanoh haim:

[Anton Matvienko]

Hello, Dima, I achieved this without bonding on this NIC (I had the same problems with bonding interface), just with dummy ports:
```

- version: 2
  interfaces: ['98:00.0', 'dummy', '98:00.1', 'dummy']
  port_info:
      - ip: 192.168.253.106 # 98:00.0
        default_gw: 192.168.253.107 # ConnectX-6 Dx on DUT
      - ip: 192.168.253.116 # dummy setup, doesn't matter
        default_gw: 192.168.253.117
      - ip: 192.168.254.106 # 98:00.1
        default_gw: 192.168.254.107 # ConnectX-6 Dx on DUT
      - ip: 192.168.254.116 # dummy setup, doesn't matter
        default_gw: 192.168.254.117

  platform:
      master_thread_id: 0
      latency_thread_id: 2
      dual_if:
        - socket: 1
          threads: [1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55]
        - socket: 1
          threads: [57,59,61,63,65,67,69,71,73,75,77,79,81,83,85,87,89,91,93,95,97,99,101,103,105,107,109,111]

```

I played with number of cores, in my setup 112 cores are available, but Trex only allow to use 23 * 2 = 46, on -c 24 i got crash with assertion.

Also huge page setup should be enough:

```

sudo dpdk-hugepages.py --setup 4G

```

Also see CPU utilization, if it will be the bottleneck, you don't achieve 200 Gpbs, that why amount of cores is matter behind among other things.

четверг, 15 февраля 2024 г. в 12:03:48 UTC+3, hanoh haim:

Hi Dima,