NFP Physical Ports Shut Down Unexpectedly

[shaoke xi]

Hi everyone, 

My agilio 1x40GbE cannot forward packets through physical media now. The 40GbE port is split into 4x10GbE ports, I can see them with the ifconfig command (they are named nfp_p0/1/2/3, respectively). However, when I compile the basic wire P4 lab program (Lab p4wire, wire.p4) and start hardware debug, those 4 ports seem to be shut down. Using ifconfig nfp_p0 up command only gets a timeout error. I provide more details in the following. Has anyone encountered this problem before? Any help is very much appreciated.

With regards,

Shaoke

Sequentially execute commands

# rmmod nfp

# modprobe nfp

# ifconfig nfp_p0 up; ifconfig nfp_p1 up; ifconfig nfp_p2 up; ifconfig nfp_p3 up

dmesg shows

[1880827.008743] nfp src version: rev- (o-o-t)

nfp src path: /var/lib/dkms/nfp/1.1.r431/build/

nfp build user id: root

nfp build user: root

nfp build host: p4.tooyum.com

nfp build path: /var/lib/dkms/nfp/1.1.r431/build

[1880827.008745] Netronome NFP CPP API

[1880827.008768] nfp-net-vnic: NFP vNIC driver, Copyright (C) 2010-2015 Netronome Systems

[1880827.008967] nfp 0000:01:00.0: Netronome Flow Processor NFP4000/NFP6000 PCIe Card Probe

[1880827.009056] nfp 0000:01:00.0: RESERVED BARs: 0.0: General/MSI-X SRAM, 0.1: PCIe XPB/MSI-X PBA, 0.4: Explicit0, 0.5: Explicit1, free: 20/24

[1880827.009098] nfp 0000:01:00.0: Model: 0x40010010, SN: 00:15:4d:12:1f:83, Ifc: 0x10ff

[1880827.017074] nfp 0000:01:00.0: Assembly: SMAAMDA0081-000116110042-15 CPLD: 0x1030000

[1880827.304473] nfp 0000:01:00.0: BSP: 020028.020028.02007f

[1880827.304478] nfp 0000:01:00.0: nfp: Looking for firmware file in order of priority:

[1880827.304499] nfp 0000:01:00.0: Direct firmware load failed with error -2

[1880827.304500] nfp 0000:01:00.0: Falling back to user helper

[1880887.448053] nfp 0000:01:00.0: nfp:   netronome/serial-00-15-4d-12-1f-83-10-ff.nffw: not found

[1880887.448482] nfp 0000:01:00.0: Direct firmware load failed with error -2

[1880887.448484] nfp 0000:01:00.0: Falling back to user helper

[1880947.612561] nfp 0000:01:00.0: nfp:   netronome/pci-0000:01:00.0.nffw: not found

[1880947.613803] nfp 0000:01:00.0: nfp:   netronome/nic_AMDA0081-0001_4x10.nffw: found, loading...

[1880947.613806] nfp 0000:01:00.0: Soft-reset, loading FW image

[1880950.408998] i40e 0000:02:00.1 enp2s0f1: NIC Link is Down

[1880950.409064] net2: port 1(enp2s0f1) entered disabled state

[1880950.417069] i40e 0000:02:00.2 enp2s0f2: NIC Link is Down

[1880950.425139] i40e 0000:02:00.3 enp2s0f3: NIC Link is Down

[1880950.988558] i40e 0000:02:00.1 enp2s0f1: NIC Link is Up 10 Gbps Full Duplex, Flow Control: None

[1880950.988599] net2: port 1(enp2s0f1) entered forwarding state

[1880950.988609] net2: port 1(enp2s0f1) entered forwarding state

[1880950.996609] i40e 0000:02:00.3 enp2s0f3: NIC Link is Up 10 Gbps Full Duplex, Flow Control: None

[1880951.084380] i40e 0000:02:00.2 enp2s0f2: NIC Link is Up 10 Gbps Full Duplex, Flow Control: None

[1880954.681047] nfp 0000:01:00.0: Finished loading FW image

[1880954.688516] nfp 0000:01:00.0: irq 31 for MSI/MSI-X

[1880954.688524] nfp 0000:01:00.0: irq 167 for MSI/MSI-X

[1880954.688530] nfp 0000:01:00.0: irq 168 for MSI/MSI-X

[1880954.688536] nfp 0000:01:00.0: irq 169 for MSI/MSI-X

[1880954.688543] nfp 0000:01:00.0: irq 170 for MSI/MSI-X

[1880954.688549] nfp 0000:01:00.0: irq 171 for MSI/MSI-X

[1880954.688554] nfp 0000:01:00.0: irq 172 for MSI/MSI-X

[1880954.688560] nfp 0000:01:00.0: irq 173 for MSI/MSI-X

[1880954.688568] nfp 0000:01:00.0: irq 174 for MSI/MSI-X

[1880954.688574] nfp 0000:01:00.0: irq 175 for MSI/MSI-X

[1880954.688579] nfp 0000:01:00.0: irq 176 for MSI/MSI-X

[1880954.688585] nfp 0000:01:00.0: irq 177 for MSI/MSI-X

[1880954.688592] nfp 0000:01:00.0: irq 178 for MSI/MSI-X

[1880954.688598] nfp 0000:01:00.0: irq 179 for MSI/MSI-X

[1880954.688604] nfp 0000:01:00.0: irq 180 for MSI/MSI-X

[1880954.688609] nfp 0000:01:00.0: irq 181 for MSI/MSI-X

[1880954.688615] nfp 0000:01:00.0: irq 182 for MSI/MSI-X

[1880954.688620] nfp 0000:01:00.0: irq 183 for MSI/MSI-X

[1880954.688626] nfp 0000:01:00.0: irq 184 for MSI/MSI-X

[1880954.688632] nfp 0000:01:00.0: irq 185 for MSI/MSI-X

[1880954.688638] nfp 0000:01:00.0: irq 186 for MSI/MSI-X

[1880954.688644] nfp 0000:01:00.0: irq 187 for MSI/MSI-X

[1880954.688650] nfp 0000:01:00.0: irq 188 for MSI/MSI-X

[1880954.688656] nfp 0000:01:00.0: irq 189 for MSI/MSI-X

[1880954.688661] nfp 0000:01:00.0: irq 190 for MSI/MSI-X

[1880954.688667] nfp 0000:01:00.0: irq 191 for MSI/MSI-X

[1880954.688674] nfp 0000:01:00.0: irq 192 for MSI/MSI-X

[1880954.688680] nfp 0000:01:00.0: irq 193 for MSI/MSI-X

[1880954.688687] nfp 0000:01:00.0: irq 194 for MSI/MSI-X

[1880954.688693] nfp 0000:01:00.0: irq 195 for MSI/MSI-X

[1880954.688698] nfp 0000:01:00.0: irq 196 for MSI/MSI-X

[1880954.688705] nfp 0000:01:00.0: irq 197 for MSI/MSI-X

[1880954.688710] nfp 0000:01:00.0: irq 198 for MSI/MSI-X

[1880954.688716] nfp 0000:01:00.0: irq 199 for MSI/MSI-X

[1880954.688722] nfp 0000:01:00.0: irq 200 for MSI/MSI-X

[1880954.688729] nfp 0000:01:00.0: irq 201 for MSI/MSI-X

[1880954.688736] nfp 0000:01:00.0: irq 202 for MSI/MSI-X

[1880954.688743] nfp 0000:01:00.0: irq 203 for MSI/MSI-X

[1880954.688749] nfp 0000:01:00.0: irq 204 for MSI/MSI-X

[1880954.688755] nfp 0000:01:00.0: irq 205 for MSI/MSI-X

[1880954.690315] nfp 0000:01:00.0 eth0: Netronome NFP-6xxx Netdev: TxQs=8/8 RxQs=8/8

[1880954.690317] nfp 0000:01:00.0 eth0: VER: 0.0.4.5, Maximum supported MTU: 9216

[1880954.690320] nfp 0000:01:00.0 eth0: CAP: 0xa3160eff PROMISC L2BCFILT L2MCFILT RXCSUM TXCSUM RXVLAN TXVLAN GATHER TSO1 RSS1 RSS2 CTAG_FILTER AUTOMASK IRQMOD VXLAN NVGRE LIVE_ADDR 

[1880954.692252] nfp 0000:01:00.0 eth1: Netronome NFP-6xxx Netdev: TxQs=8/8 RxQs=8/8

[1880954.692254] nfp 0000:01:00.0 eth1: VER: 0.0.4.5, Maximum supported MTU: 9216

[1880954.692257] nfp 0000:01:00.0 eth1: CAP: 0xa3160eff PROMISC L2BCFILT L2MCFILT RXCSUM TXCSUM RXVLAN TXVLAN GATHER TSO1 RSS1 RSS2 CTAG_FILTER AUTOMASK IRQMOD VXLAN NVGRE LIVE_ADDR 

[1880954.694260] nfp 0000:01:00.0 eth2: Netronome NFP-6xxx Netdev: TxQs=8/8 RxQs=8/8

[1880954.694262] nfp 0000:01:00.0 eth2: VER: 0.0.4.5, Maximum supported MTU: 9216

[1880954.694265] nfp 0000:01:00.0 eth2: CAP: 0xa3160eff PROMISC L2BCFILT L2MCFILT RXCSUM TXCSUM RXVLAN TXVLAN GATHER TSO1 RSS1 RSS2 CTAG_FILTER AUTOMASK IRQMOD VXLAN NVGRE LIVE_ADDR 

[1880954.696274] nfp 0000:01:00.0 eth3: Netronome NFP-6xxx Netdev: TxQs=8/8 RxQs=8/8

[1880954.696277] nfp 0000:01:00.0 eth3: VER: 0.0.4.5, Maximum supported MTU: 9216

[1880954.696279] nfp 0000:01:00.0 eth3: CAP: 0xa3160eff PROMISC L2BCFILT L2MCFILT RXCSUM TXCSUM RXVLAN TXVLAN GATHER TSO1 RSS1 RSS2 CTAG_FILTER AUTOMASK IRQMOD VXLAN NVGRE LIVE_ADDR 

[1880988.823190] nfp 0000:01:00.0 nfp_p0: RV00: irq=168/002

[1880988.823201] nfp 0000:01:00.0 nfp_p0: RV01: irq=169/003

[1880988.823209] nfp 0000:01:00.0 nfp_p0: RV02: irq=170/004

[1880988.823217] nfp 0000:01:00.0 nfp_p0: RV03: irq=171/005

[1880988.823224] nfp 0000:01:00.0 nfp_p0: RV04: irq=172/006

[1880988.823231] nfp 0000:01:00.0 nfp_p0: RV05: irq=173/007

[1880988.823239] nfp 0000:01:00.0 nfp_p0: RV06: irq=174/008

[1880988.823247] nfp 0000:01:00.0 nfp_p0: RV07: irq=175/009

[1880988.958110] IPv6: ADDRCONF(NETDEV_UP): nfp_p0: link is not ready

[1880988.958115] 8021q: adding VLAN 0 to HW filter on device nfp_p0

[1880992.876819] nfp 0000:01:00.0 nfp_p1: RV00: irq=178/012

[1880992.876830] nfp 0000:01:00.0 nfp_p1: RV01: irq=179/013

[1880992.876838] nfp 0000:01:00.0 nfp_p1: RV02: irq=180/014

[1880992.876845] nfp 0000:01:00.0 nfp_p1: RV03: irq=181/015

[1880992.876853] nfp 0000:01:00.0 nfp_p1: RV04: irq=182/016

[1880992.876861] nfp 0000:01:00.0 nfp_p1: RV05: irq=183/017

[1880992.876869] nfp 0000:01:00.0 nfp_p1: RV06: irq=184/018

[1880992.876876] nfp 0000:01:00.0 nfp_p1: RV07: irq=185/019

[1880992.959453] nfp 0000:01:00.0 nfp_p1: NIC Link is Up

[1880992.959526] 8021q: adding VLAN 0 to HW filter on device nfp_p1

[1880996.220505] nfp 0000:01:00.0 nfp_p2: RV00: irq=188/022

[1880996.220515] nfp 0000:01:00.0 nfp_p2: RV01: irq=189/023

[1880996.220522] nfp 0000:01:00.0 nfp_p2: RV02: irq=190/024

[1880996.220529] nfp 0000:01:00.0 nfp_p2: RV03: irq=191/025

[1880996.220536] nfp 0000:01:00.0 nfp_p2: RV04: irq=192/026

[1880996.220542] nfp 0000:01:00.0 nfp_p2: RV05: irq=193/027

[1880996.220549] nfp 0000:01:00.0 nfp_p2: RV06: irq=194/028

[1880996.220555] nfp 0000:01:00.0 nfp_p2: RV07: irq=195/029

[1880996.355524] nfp 0000:01:00.0 nfp_p2: NIC Link is Up

[1880996.355591] 8021q: adding VLAN 0 to HW filter on device nfp_p2

[1881000.701125] nfp 0000:01:00.0 nfp_p3: RV00: irq=198/032

[1881000.701136] nfp 0000:01:00.0 nfp_p3: RV01: irq=199/033

[1881000.701144] nfp 0000:01:00.0 nfp_p3: RV02: irq=200/034

[1881000.701152] nfp 0000:01:00.0 nfp_p3: RV03: irq=201/035

[1881000.701159] nfp 0000:01:00.0 nfp_p3: RV04: irq=202/036

[1881000.701167] nfp 0000:01:00.0 nfp_p3: RV05: irq=203/037

[1881000.701175] nfp 0000:01:00.0 nfp_p3: RV06: irq=204/038

[1881000.701182] nfp 0000:01:00.0 nfp_p3: RV07: irq=205/039

[1881000.835879] nfp 0000:01:00.0 nfp_p3: NIC Link is Up

[1881000.835992] 8021q: adding VLAN 0 to HW filter on device nfp_p3

Then I load the firmware compiled from the basic wire P4 program (Lab p4wire, wire.p4), dmesg shows:

[1882276.038698] nfp 0000:01:00.0 nfp_p1: NIC Link is Down

[1882276.038970] nfp 0000:01:00.0 nfp_p2: NIC Link is Down

[1882276.039244] nfp 0000:01:00.0 nfp_p3: NIC Link is Down

[1882276.049446] i40e 0000:02:00.2 enp2s0f2: NIC Link is Down

[1882276.057445] i40e 0000:02:00.1 enp2s0f1: NIC Link is Down

[1882276.065447] i40e 0000:02:00.3 enp2s0f3: NIC Link is Down

[1882277.040730] net3: port 1(enp2s0f2) entered disabled state

[1882277.040788] net2: port 1(enp2s0f1) entered disabled state

[1882277.040836] net4: port 1(enp2s0f3) entered disabled state

[1882293.132462] i40e 0000:02:00.2 enp2s0f2: NIC Link is Up 10 Gbps Full Duplex, Flow Control: None

[1882293.132506] net3: port 1(enp2s0f2) entered forwarding state

[1882293.132515] net3: port 1(enp2s0f2) entered forwarding state

[1882293.244332] i40e 0000:02:00.1 enp2s0f1: NIC Link is Up 10 Gbps Full Duplex, Flow Control: None

[1882293.244361] net2: port 1(enp2s0f1) entered forwarding state

[1882293.244366] net2: port 1(enp2s0f1) entered forwarding state

[1882293.252378] i40e 0000:02:00.3 enp2s0f3: NIC Link is Up 10 Gbps Full Duplex, Flow Control: None

[1882293.252404] net4: port 1(enp2s0f3) entered forwarding state

[1882293.252410] net4: port 1(enp2s0f3) entered forwarding state

# ifconfig nfp_p1 down

[1882496.281047] nfp 0000:01:00.0 nfp_p1: Reconfig timeout: 0x00000023

[1882496.281053] nfp 0000:01:00.0 nfp_p1: Could not disable device: -5

[1882496.465085] nfp 0000:01:00.0 nfp_p1: nfp_p1 down

[1882496.471518] i40e 0000:02:00.2 enp2s0f2: NIC Link is Down

[1882496.471584] net3: port 1(enp2s0f2) entered disabled state

# ifconfig nfp_p1 up

SIOCSIFFLAGS: input/output error

dmesg shows:

[1882580.595214] nfp 0000:01:00.0 nfp_p1: RV00: irq=178/012

[1882580.595222] nfp 0000:01:00.0 nfp_p1: RV01: irq=179/013

[1882580.595227] nfp 0000:01:00.0 nfp_p1: RV02: irq=180/014

[1882580.595232] nfp 0000:01:00.0 nfp_p1: RV03: irq=181/015

[1882580.595236] nfp 0000:01:00.0 nfp_p1: RV04: irq=182/016

[1882580.595241] nfp 0000:01:00.0 nfp_p1: RV05: irq=183/017

[1882580.595245] nfp 0000:01:00.0 nfp_p1: RV06: irq=184/018

[1882580.595249] nfp 0000:01:00.0 nfp_p1: RV07: irq=185/019

[1882585.779787] nfp 0000:01:00.0 nfp_p1: Reconfig timeout: 0x00000127

[1882590.781171] nfp 0000:01:00.0 nfp_p1: Reconfig timeout: 0x00000023

[1882590.781180] nfp 0000:01:00.0 nfp_p1: Could not disable device: -5

Finally, the program does not work.

[shaoke xi]

Sorry to bother you all, I know why my packets are being dropped now. Ports are working normally. It turns out that I set the port number wrong. I changed the SFP wire sequence days ago, but I haven't thought that the sequence actually matters! I find the problem when watching the Debug Windows > P4 Inspectors in the programmer studio, it shows that the src mac is unmatched with the ingress port. This information is very useful.

Wish you all good lucks.

Shaoke