Receiving packets using Micro-C

[Raghav Kapoor]

Hello,

I've been struggling to receive packets on an AgilioCX 2x25GbE, which I have connected to a bluefield SmartNIC that is sending ethernet frames to the Netronome using scapy. I've attached more information about the hardware I'm using below in the hardware_info.txt file. 

Here is the configuration of my ports:

I've also attached the code that I'm running below along with other files. I would appreciate any help that anyone can provide. 

Thank you, 

Raghav Kapoor

[Octavio Herrera-Ruiz]

Hello Raghav,

   I believe you are facing a L2 issue here. The SmartNICs are not able to link to each other. 

  Here is what I suggest:

1) Confirm that both cards can link up to themselves with the cable you have. That is, put a loop between P0 and P1 on the card(s) and observe whether link (LED) comes UP. Assuming that this test will work w/o mayor issues for both cards, you could move to the following step. Otherwise, follow up with us or the manufacturer of the other SmartNIC. 

2) Find settings to link UP across your cards

 2.1) It is up to you to confirm what options the bluefield card supports and how to set them up

 2.2) For the Netronome case (as root or with sudo), the command "nfp-media" will show you your current settings. Check the output of "nfp-media --help-25g" for a list of the options you can use. I suggest forcing the mode to 25G (e.g., for port 0: "nfp-media --set-aneg=phy0=F", then reboot for the change to take effect). 

Once the LEDs (or output of nfp-phymod) indicate that you have a link UP between your SmartNICs, you can move on with your testing on upper layers (i.e., your SW). 

Have a nice day!

Octavio H.

[Raghav Kapoor]

Hello Octavio,

Thank you for getting back to me. When I plug in my cable between p0 and p1 of the Netronome, I notice that something odd happens. While the LEDs on the Netronome do turn on indicating a link, I get no output when I run nfp-phymod. Is this supposed to happen?

Thank you, 

Raghav Kapoor

[Octavio Herrera-Ruiz]

Hello Raghav,

   I have never seen such behavior. What I would suggest is unloading any firmware you may have on the SmartNIC (i.e., nfp-nffw unload) before you do your link test(s).

[Raghav Kapoor]

Hello Octavio, 

I did as you suggested, and now I'm able to see the link LED light on and nfp-phymod produces the following output:

Thank you, 

Raghav Kapoor

[Raghav Kapoor]

Hello, 

After being able to successfully validate that a link has been established between both ports of my Netronome SmartNIC, I tried seeing if I could actually send ethernet frames from one port to another using scapy. When sending an ethernet frame from one interface to another and sniffing the receiving interface, I see that no packets can be sniffed even though I receive confirmation after calling "sendp". I also tried sniffing the sending port and I see that no packets can be sniffed either. Are there some steps that I need to follow prior to using scapy to send ethernet frames between the ports?

Thank you, 

Raghav Kapoor

[Octavio Herrera-Ruiz]

Hello,

   You did not specify what firmware you are using in the SmartNIC. To simplify your test, I would suggest trying with Netronome's CoreNIC first. Then use namespaces to assign one of the interfaces to a new TCP stack, then assign IPs on each interface and you should be able to run "normal" traffic from one port to the other. 

   As a reference/sample, I include the following script that I used before to do some testing. Review it at your convenience and adapt it to your testing needs
--------

#!/bin/sh

if [ $# -ne 3 ]; then
        if [ $# -eq 1 ]; then

                # set up client side (for iperf3 test, e.g. "iperf3 -s 10.0.10.2 -i15 -t65 -p 5201" )

                ip a a dev $1 10.0.10.1/24; ip l set dev $1 mtu 9200; ip l set dev $1 up
                exit
        else
                echo " Please use:
                 Set-NS <NS_name> <interface> <IP>

                e.g.
               # ./Set-NS.sh SFP-Test eth1 10.0.10.2

                  which would move interface eth1 into name space SFP-Test"
                exit
        fi

fi

SPACE="$1"
INT="$2"
IP="$3"
# Create namespace

ip netns add $SPACE
# Move interface into the new namespace

ip link set eth1 netns $SPACE
# Assign IP address

ip netns exec $SPACE ip a a dev $INT ${IP}/24
# Change mtu

ip netns exec $SPACE ip link set $INT mtu 9200
# Bring interface UP

ip netns exec $SPACE ip link set $INT up
# Start iperf3 server side: one or more will be used depending of performance goal(s)

ip netns exec $SPACE iperf3 -s -D -p 5201
ip netns exec $SPACE iperf3 -s -D -p 5202
ip netns exec $SPACE iperf3 -s -D -p 5203

[Raghav Kapoor]

Hello Octavio, 

I followed the guidelines you provided, but, when I try to test the link between the two interfaces using iperf3, I get the following error:

Raghav Kapoor

[Octavio Herrera-Ruiz]

Hello,

   We are  missing some configuration here...

   To guide you further, I'll need you to track/confirm any setup that you have done.

1) System does boot OK? Output of 'dmesg | grep nfp' should be enough to determine this.

1.1) Are you loading the Basic firmware? Again, the output of 'dmesg | grep nfp' should provide clues on this, but we may want to confirm that you actually have that (and only that one) package installed in your system. The output of 'apt list | grep "agilio\|nfp" ' and/or 'dpkg -l | grep "agilio\|nfp" ' should tell you what Netronome packages you have installed and whether any of these are currently installed.

2) You already have link UP, I assume. The output of 'nfp-phymod' should confirm this. If any of the ports is shown as down, try bringing them explicitly UP with 'ip link set <dev> up'

2) Are you using namespaces? If so, can you check that each stack is working "fine":
#List name space(s)

ip netns list

# Assuming you have set IPs like P0=10.0.10.1/24 and P1=10.0.10.2/24 and P1 is in name space = "TEST"

# Check that each interface is "present"

ip link

ip netns exec ip link

# ping each interface's IP

ping -c3 10.0.10.1

ip netns exec TEST ping -c3 10.0.10.2

3) If the steps above are OK, now you can ping the interface at the other end:

ping -c3 10.0.10.2

ip netns exec TEST ping -c3 10.0.10.1

# if you still get the unable to connect message, check disabling the firewall (i.e., iptables) in your system. Then, test again.

   Have a nice day!

Octavio H.

[Raghav Kapoor]

Hello Octavio, 

I tried following the steps you mentioned above, but I keep getting stuck at the issue of having the link UP. After several reboots and following the same process I had followed prior to set up a link (i.e. making sure the firmware was unloaded), the link LEDs and phymod will confirm the link is down or the link LEDS will be on briefly then phymod will indicate the link is down and the link LEDs will turn off. It seems like there might be something with my Netronome card. 

Raghav Kapoor

[Octavio Herrera-Ruiz]

Hello Raghav,

   To speed up the process, maybe we should do a remote session. How is your availability this coming Wednesday (Sep 21st)? I am on EDT...

[Raghav Kapoor]

Hello Octavio,
I'll be available on Wednesday at noon PST.

Raghav Kapoor