MLX5 - TREX v2.89 failed to create dpdk flow rule msg

[Haklat Haklat]

Hi,

I have used TREX with mellanox succesfully for some time as docker container in many k8s setups. I have successfully been using v2.85 with OFED 5.1 on Centos 8.2.2004 container.

When trying to build and run new 2.89 version, I hit same thing I can see in issue 675 (https://github.com/cisco-system-traffic-generator/trex-core/pull/675):

Failed to create dpdk flow rule msg: hardware refuses to create flow

Did you manage to understand the root cause of this issue?

I have tried building different containers based on:

Centos7.9 with both OFED5.2 and OFED5.3

&

Centos8.2.2004 with OFED5.3

The all have hit same issue when trying to start up. Below is my Docker file

used (at the time 2.85 image created latest OFED was 5.1) and working fine for trex v2.85 (was modified when trying v2.89):

#Dockerfile base image centos:8.2.2004 with TRex

FROM centos:8.2.2004

RUN yum -y update && yum -y install epel-release bash numactl-libs wget curl

RUN curl https://linux.mellanox.com/public/repo/mlnx_ofed/latest/rhel8.2/mellanox_mlnx_ofed.repo --output /etc/yum.repos.d/mellanox_mlnx_ofed.repo

###RUN yum -y update && yum -y install mlnx-ofed-dpdk-upstream-libs-user-only.noarch

RUN yum -y update && yum -y install mlnx-ofed-dpdk-upstream-libs.noarch

RUN yum install -y sudo pciutils vim kmod strace wget gcc-c++ ethtool python2 tcpdump net-tools screen openssh-clients

RUN cd /usr/bin; ln -s python2 python

RUN mkdir /scratch

WORKDIR /scratch

RUN wget --no-check-certificate https://trex-tgn.cisco.com/trex/release/v2.85.tar.gz

RUN tar -zxvf v2.85.tar.gz && \

    chown root:root v2.85 && \

    rm v2.85.tar.gz

RUN mkdir /scratch/trex-core

WORKDIR /scratch/trex-core

RUN ln -s /scratch/v2.85 scripts

WORKDIR /scratch/v2.85

### Patching TRex to work for Mellanox mlx5 NICs

RUN sed -i 's/--mlx4-so //g' t-rex-64

RUN yum clean all -y

CMD ["/bin/bash"]

BR//Håkan

[hanoh haim]

v2.89 should work with OFED5.3 and CentOs 7.9 

[hanoh haim]

On Thu, 20 May 2021 at 16:36, hanoh haim <hhaim...@gmail.com> wrote:

v2.89 should work with OFED5.3 and CentOs 7.9 

--

Hanoh

Sent from my iPhone

[Engarn Engarn]

Hi,

thanks Hanoch. Cento7.9 with OFED 5.3-1 was first thing I tried with for v2.89, but it failed with "Failed to create dpdk flow rule msg: hardware refuses to create flow".

With docker container and Mellanox DPDK, OFED installation is required on both host system and in container image. In my case host system has OFED version 5.2 (this version I cannot control as it is part of the k8s system).

Thanks//Håkan

[hanoh haim]

Hi Engarn,

I would consult Mellanox DPDK team. It works on our setup with VF and PF mode. 

Thanks

Hanoh

[Haklat Haklat]

Hi,

I will open a support case with Mellanox (Nvida) on the topic. I think general assumption is that the docker DPDK case should work with same OFED major versions (5.x)  on the host and container. But if there is some general issue with DPDK 21.02 and OFED 5.2 (which I think you have also experienced), this might not be the case.

Thanks//Håkan

[Haklat Haklat]

Hi,

just want to share my latest findings on this. Looks like it was maybe firmware version that needed to be latest, at least in my case.

Below is my latest post to NVIDA MLX5 case:

"""""

Hi,

looks like maybe it was the NIC firmware version that was reason for:

"Failed to create dpdk flow rule msg: hardware refuses to create flow"

Unless one time lucky, it looks like maybe e.g. some DPDK flow rule used in TREX DPDK app needed some feature added in latest firmware.

After one server type we have in the system was firmware updated to 16.29.2002 from some 16.28.xxxx release, it looks like it works now.

We however have some DELL platforms sas well in the k8s system. For those we need to take latest DELL firmware package in order to find matching PSID firmware, this firmware version is currently at:

16.28.45.12, 16.28.45.12

Or is it possible to get the 16.29 versions also for NIC with DELL PSID?

[root@trex-mlx-perf v2.89]# ethtool -i net1

driver: mlx5_core

version: 5.2-1.0.4

firmware-version: 16.29.2002 (MT_0000000012)

expansion-rom-version:

bus-info: 0000:18:00.3

supports-statistics: yes

supports-test: yes

supports-eeprom-access: no

supports-register-dump: no

supports-priv-flags: yes

[root@trex-mlx-perf v2.89]# ofed_info -s

MLNX_OFED_LINUX-5.3-1.0.0.1:

""""""

BR//Håkan