DPDK behind Netronome, multiple cores

[Jonatan Langlet]

Hi,

We are building a setup with DPDK bound to VF ports of a Netronome Agilio CX 2x40 (NFP4000) SmartNIC.
Netronome does some P4 processing on packets, and forwards through SR-IOV to host where dpdk will continue processing.

My problem: in DPDK I can not allocate more than a single RX-queue to the ports.
Multiple dpdk processes can not pull from the same queue, which means that my dpdk setup only works with a single core.

Binding dpdk to PF ports on a simple Intel 2x10G NIC works without a problem, multiple RX-queues (and hence multiple cores) works.


I bind dpdk to Netronome VF ports with the igb_uio driver.
I have seen vfio-pci mentioned, would using this driver allow multiple RX-queues? There were some problems using this driver, which is why it has not yet been tested.


We use T4P4S to compile P4 for dpdk.
For compatibility with Netronome, I have to change the RSS hash function in dpdk_lib_init_hw.c.
To work with Netronome, it has to be: 0xd34
To work with Intel, it has to be: 0x7ef8
Trying to run with an incorrect rss_hf throws as error, explicitly saying which rss_hf should be used.

As I understand it, the RSS hash function is what distributes packets across RX-queues?


Output when attempting to use two RX-queues, one core each, shared by two ports:

EAL: Detected 20 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Probing VFIO support...
EAL: PCI device 0000:05:00.0 on NUMA socket 0
EAL:   probe driver: 8086:1533 net_e1000_igb
EAL: PCI device 0000:06:00.0 on NUMA socket 0
EAL:   probe driver: 8086:1533 net_e1000_igb
EAL: PCI device 0000:65:00.0 on NUMA socket 0
EAL:   probe driver: 8086:1572 net_i40e
EAL: PCI device 0000:65:00.1 on NUMA socket 0
EAL:   probe driver: 8086:1572 net_i40e
EAL: PCI device 0000:67:00.0 on NUMA socket 0
EAL:   probe driver: 8086:1572 net_i40e
EAL: PCI device 0000:67:00.1 on NUMA socket 0
EAL:   probe driver: 8086:1572 net_i40e
EAL: PCI device 0000:b3:00.0 on NUMA socket 0
EAL:   probe driver: 19ee:4000 net_nfp_pf
EAL: PCI device 0000:b3:08.0 on NUMA socket 0
EAL:   probe driver: 19ee:6003 net_nfp_vf
EAL: PCI device 0000:b3:08.1 on NUMA socket 0
EAL:   probe driver: 19ee:6003 net_nfp_vf
EAL: PCI device 0000:b3:08.2 on NUMA socket 0
EAL:   probe driver: 19ee:6003 net_nfp_vf
EAL: PCI device 0000:b3:08.3 on NUMA socket 0
EAL:   probe driver: 19ee:6003 net_nfp_vf
Ethdev port_id=0 nb_rx_queues=2 > 1
EAL: Error - exiting with code: 1
  Cause: Cannot configure device: err=-22, port=0

T4P4S switch exited with error code 1
 - Runtime options were: -c 0x3 -n 4 -- -p 0x3 --config "\"(0,0,0),(0,1,1),(1,0,0),(1,1,1)\""

If more information is required, I will be happy providing it


Thanks,
Jonatan