SmartNIC 2x40G with Breakout Cables

[matthi...@gmail.com]

Hello everyone,

I am having some problems programming a SmartNIC 6000 2x40G with breakout cables to 8x10G.

The following card is installed:

$ sudo /opt/netronome/bin/nfp-hwinfo -n 0

nfp.interface=pci.0.0

nfp.model=0x60120010

nfp.serial=00:15:4d:12:11:05

board.exec=bootloader.bin

uart.baud=115200

preinit.setup.version=nfp-bsp-6000-b0 (59bbe8b39e4d)

pcie0.type=ep

board.setup.version=nfp-bsp-6000-b0 (59bbe8b39e4d)

vpd=ee2:0:0x50:0x0

assembly.model=starfighter1

assembly.partno=AMDA0058-0012

assembly.revision=05

assembly.serial=15350050

assembly.vendor=SMA

chip.serial=xxxxxxxxxxxxxx

ddr0.spd=ee2:0:0x50:0xf00

ddr1.spd=ee2:0:0x50:0xf00

ddr2.spd=ee2:0:0x50:0xf00

ddr3.spd=ee2:0:0x50:0xf00

ddr4.spd=none

ddr5.spd=none

emu2.type=cache

ethm.mac=00:15:4d:12:11:05

eth.mac=00:15:4d:12:11:06

eth.macs=8

ddr.speed=1866

chip.model=NFP6012

chip.revision=B0

core.speed=1000

me.speed=1200

arm.speed=600

chip.model.device=0x6200aa20

chip.identifier=0x4679d005aa

chip.model.hard=0x0

chip.model.soft=0x60120058

chip.route=0xe51dffc8

chip.island=0x3007f37003332

mem.setup.version=nfp-bsp-6000-b0 (59bbe8b39e4d)

The Ports are reconfigured to 10G via nfp-media:

$ sudo /opt/netronome/bin/nfp-phymod -n 0

phy0: NBI0.0(4) "0" QSFP conn:No separable [40GBASE-CR4]

 "Fiberstore" "Q-4SPC03" "A7841600002"  oui:0x93a type:0x28 len:3m active:0 cc:0

  eth0: NBI0.0(1) "0.0" 00:15:4d:12:11:06 10G Down Bootable

Link Fault: RX_BASE RX_BLOCK

  eth2: NBI0.1(1) "0.1" 00:15:4d:12:11:08 10G Down

Link Fault: RX_BASE RX_BLOCK RX_DISABLE

  eth3: NBI0.2(1) "0.2" 00:15:4d:12:11:09 10G Up

  eth4: NBI0.3(1) "0.3" 00:15:4d:12:11:0a 10G Up

phy1: NBI0.4(4) "1" QSFP conn:No separable [40GBASE-CR4]

 "Fiberstore" "Q-4SPC03" "A7841600003"  oui:0x93a type:0x28 len:3m active:0 cc:0

  eth1: NBI0.4(1) "1.0" 00:15:4d:12:11:07 10G Down

Link Fault: RX_BASE RX_BLOCK

  eth5: NBI0.5(1) "1.1" 00:15:4d:12:11:0b 10G Down

Link Fault: RX_BASE RX_BLOCK RX_DISABLE

  eth6: NBI0.6(1) "1.2" 00:15:4d:12:11:0c 10G Up

  eth7: NBI0.7(1) "1.3" 00:15:4d:12:11:0d 10G Up

To test I am loading the wire application form lab5.

I am able to execute code on all microengines and write something to the mailboxes.

But no packets seem to arrive. Is there more documentation on the json-config-files?

I was not able to find anything and probably the selection of the jsons is bad.

(They are attached as well as the modified loading script)

This is the microc Code I am running:

int main(void) {

    __gpr struct pkt_ms_info msi;

    __addr40 char *pbuf;

    __xread struct nbi_meta_catamaran nbi_meta;

    __xread struct nbi_meta_pkt_info *pi = &nbi_meta.pkt_info;

    __gpr enum PKT_CTM_SIZE ctm_buf_size;

    __xread pkt_status_t pkt_status;

    __gpr int in_port, out_port, pkt_off;

    __mem struct pkt_hdr *pkt_hdr;

    // test code execution on MEs

    WRITE_MAILBOX_2(0xa);

    for (;;) {

        /* Receive a packet */

        pkt_nbi_recv(&nbi_meta, sizeof(nbi_meta));

        in_port = MAC_TO_PORT(nbi_meta.port);

        pbuf = pkt_ctm_ptr40(pi->isl, pi->pnum, 0);

        pkt_hdr = pkt_ctm_ptr40(pi->isl, pi->pnum, PKT_NBI_OFFSET);

        WRITE_MAILBOX_0(READ_MAILBOX_0()+1); // packet-counter

        WRITE_MAILBOX_1(0xf0 | in_port);            // write receiving port

        /* Send the packet */

        /* Write the MAC egress CMD and adjust offset and len accordingly */

        pkt_off = PKT_NBI_OFFSET + MAC_PREPEND_BYTES;

        pkt_mac_egress_cmd_write(pbuf, pkt_off, 1, 1);

        pkt_off -= 4;

        msi = pkt_msd_write(pbuf, pkt_off);

        pkt_nbi_send(pi->isl, pi->pnum, &msi, pi->len - MAC_PREPEND_BYTES + 4,

            NBI, PORT_TO_TMQ(out_port),

            nbi_meta.seqr, nbi_meta.seq, PKT_CTM_SIZE_256);

    }

    return 0;

}

Will the ports be outnumbered from 0-7 if all two breakout cables function properly?

When running nfp-macinit I get the following error:

Non-qualified Direct Attach Copper - Fiberstore Q-4SPC03 Using: 3m_GENERIC 3m

What does this mean?

Let me know, if you need more information! Thanks.

Regards,

Matthias

[matthi...@gmail.com]

Gentle reminder for help :)

[David George]

Hi Matthais

In your wire.sh you have the following:

        nfp -n $CARD -m mac -e set port rx 0 0 enable &> $OUTPUT || exit 1

        nfp -n $CARD -m mac -e set port rx 0 4 enable &> $OUTPUT || exit 1

        nfp -n $CARD -m mac set port ifup 0 0 &> $OUTPUT || exit 1

        nfp -n $CARD -m mac set port ifup 0 4 &> $OUTPUT || exit 1

I'm not sure which ports you are using, but perhaps you should bring up all the ports:
PORTS=0 1 2 3 4 5 6 7

for p in $PORTS:

   nfp -n $CARD -m mac -e set port rx 0 $p enable &> $OUTPUT || exit 1

   ...

David

[matthi...@gmail.com]

Hi David,

thank you for the answer!

I am now able to get a received packet count in the mailboxes.

Are these Macros correct?, I saw with them only mappings to port 1/3 in the mailboxes, but when I read out nbi_meta.port directly I was able to distinguish between 0x0f/0x0b  0x1f/0x1b. for connector 3/4 of the breakout cable on both ports.

#define MAC_CHAN_PER_PORT   8

#define TMQ_PER_PORT        (MAC_CHAN_PER_PORT * 8)

#define MAC_TO_PORT(x)      (x / MAC_CHAN_PER_PORT)

#define PORT_TO_TMQ(x)      (x * TMQ_PER_PORT)

How I can calculate the TMQ via PORT_TO_TMQ?

Thanks!

-Matthias

[David George]

Hi Matthias

Let me try to give a quick overview of the MAC and TM port mapping on the NFP.

When traffic ingresses on a port, one of the first component it enters is the MAC. The MAC converts the port number into and ingress channel number. Typically a single port will map to a range of channel numbers, VLAN ID (or lack thereof) is used to determine the placement within this range. In the MAC config you will see these lines,

  "channel_base": 0,

  "num_channels": 4,

  "untagged_chan": 3,

which control this behaviour. This effectively means the MicroEngines "see" a MAC channel number on ingress and not the port number.

The macro MAC_TO_PORT converts the channel back to a port number in your application. Note that the num_channels configuration field is 4 and your macro is 8, which is likely why you are losing the port information. They should be the same.

The MAC channel assignments in the above configuration are symmetrical for ingress and egress; on egress the channel number will be converted back into a port. However, to complicate things a little more, the egress port value within your application is neither a port nor a MAC channel, but a Traffic Manager (TM) queue.

The TM, among other things, is a block which contains configurable cascading schedulers and 1024 input queues. The MAC channel number is determined by the TM queue divide by 8 due to the scheduler layout.

David

[Matthias Jasny]

Hi David,

I changed the num_channels value for each of the eight ports and also
adapted the channel_base value. The modified file is attached.

Unfortunately the mapping written to the mailboxes from micro-c didn't
change at all and stayed the same to port 1/3.

Can you please take a look? Thanks.

-Matthias

[Matthias Jasny]

correction: I didn't adapt the untagged_chan value. See the attached file.
Now I get mappings of 3,2,7,6 for 4 connected cables. I'll do more tests
later...

What is the purpose of the "untagged_chan" config value ??


-Matthias

[Pallav Gupta]

Hello Mattihas, 

I saw you got similar error:

Non-qualified Direct Attach Copper - OEM SFP-H10GB-CU3M Using: 3m_GENERIC 3m

Did you find solution for that?

Regards
Pallav

[Matthias Jasny]

Hi Pallav,

No this error is still there.
Were you able to succefully send packets to a specified port? I can only
receive as stated above.

-Matthias

[David George]

correction: I didn't adapt the untagged_chan value. See the attached file.
Now I get mappings of 3,2,7,6 for 4 connected cables. I'll do more tests later...

The untagged_chan field specifies which MAC channel to map the untagged (no vlan) traffic to. This is an absolute value, not relative to the mac channel group. So that value should be port_no * 8 in your case.

[David George]

I saw you got similar error:

Non-qualified Direct Attach Copper - OEM SFP-H10GB-CU3M Using: 3m_GENERIC 3m

This is describing the cable you have installed and is not really an error.

[Matthias Jasny]

> The untagged_chan field specifies which MAC channel to map the untagged
> (no vlan) traffic to. This is an absolute value, not relative to the mac
> channel group. So that value should be port_no * 8 in your case.
>

I adapted untagged_chan like you said. The mapping displayed in the
mailboxes stayed the same. (2,3,6,7).
Although currently the other four cables are connected to a switch.

In my wire_main.c I wrote a simple switch-statement to do basic
port-routing.
I'm running tcpdump on all 4 connected ports.


switch(in_port) {
case 2: out_port = 3; break;
case 3: out_port = 2; break;
case 6: out_port = 7; break;
case 7: out_port = 6; break;
default: out_port = in_port; break;
}

The packet sent is an arp-request, every tcpdump instance is silent
except the one on port 2 (and the sending port with the outgoing arp).

The broken received packet is:


14:22:29.789064 ff:ff:ff:ff:00:1b (oui Unknown) > c0:00:00:00:ff:ff (oui
Unknown), ethertype Unknown (0x21a5), length 64:
0x0000: 9cbc 0806 0001 0800 0604 0001 001b 21a5 ..............!.
0x0010: 9cbc c0a8 2c0a 0000 0000 0000 c0a8 2c58 ....,.........,X
0x0020: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0030: 0000


I attached my micro_c code and the modified json. Can you please take a
look what may be a cause for the packet-loss.

Thanks! -Matthias

[David George]

It appears as though you are adding a MAC egress prepend (to do the checksums in hardware), but you don't have egress command prepend enabled in you mac json config file.

Modify the egress_cmd_prepend_mask* values from 0x0 to 0xffffffff

[Matthias Jasny]

The packet sent on port 2 is not broken anymore. It is the same one as
sent. Thanks!

Remaining is that only sending out to out_port=2 works but not to the
other three ports.
Am I missing something else?

-Matthias

[David George]

Your TM configuration has lots of gaps in it, so your egress port will hit queues that are disabled in the TM.

Try the attached configuration file.

--

David George
Principal Software Engineer

Netronome | 1st Floor, Southdowns Ridge Office Park, Cnr John Vorster &

                      Nellmapius Drive, Irene, Centurion 0157, South Africa

Phone: +27 (012) 665-4427| www.netronome.com

[Matthias Jasny]

Thank you. I tried the file. Now sending out to Port 3 and 2 works. Port
6 and 7 are still silent.

Additional question to the file, I compared it with my old one:

"cluster0": {
"start_q":0,
"end_q":31,
"q_enable":true,
"q_drop_enable":true,
"q_size":16, <--- why 16 and not 32 ???
"q_red_enable" :false,
"q_dma_thresh":"0x0",
"q_drop_range":false
},

Is 12x10GE correct? The breakout cable provides only 8x10GE ports or is
it just the old filename?

On 29.05.2018 09:25, David George wrote:
> Your TM configuration has lots of gaps in it, so your egress port will
> hit queues that are disabled in the TM.
>
> Try the attached configuration file.
>
>
> --

> *David George*
> /Principal Software Engineer/
>
> *Netronome* | 1st Floor, Southdowns Ridge Office Park, Cnr John Vorster &

>                       Nellmapius Drive, Irene, Centurion 0157, South Africa

> Phone: + <tel:%28408%29%20802-2775>27 (012) 665-4427
> <tel:%28012%29%665-4427>| www.netronome.com <http://www.netronome.com/>
>
>

[David George]

"cluster0": {
    "start_q":0,
    "end_q":31,
    "q_enable":true,
    "q_drop_enable":true,
    "q_size":16,        <--- why 16 and not 32 ???

that is the depth of the individual queues in the group

 

Is 12x10GE correct? The breakout cable provides only 8x10GE ports or is it just the old filename?

I thought it would work, but after looking closely noticed gaps in the queues too (sorry). I tweaked the queues and renamed it (see attached), hopefully this will work.

[Matthias Jasny]

Ports 2,3 and 7 work, Port 6 is still missing.

[David George]

doh - I left it out (I edited these files by hand without testing)

port = 6

TMQ_per_port = 8 * 8

port  * TMQ_per_port = 384

in that file insert a new cluster entry between cluster11 and cluster12 from 384 to 415 and it should work

--

David George
Principal Software Engineer

Netronome | 1st Floor, Southdowns Ridge Office Park, Cnr John Vorster &

                      Nellmapius Drive, Irene, Centurion 0157, South Africa

[Matthias Jasny]

I didn't check it either afterwards. But now all 4 Ports work.
Thank you David!

When loading and starting the firmware I get the following output:

nbi0.mac0.0: ***Timeout waiting for link up
nbi0.mac0.1: ***Timeout waiting for link up
nbi0.mac0.2: 10GE Link Up! (delay: 1)
nbi0.mac0.3: 10GE Link Up! (delay: 1)
nbi0.mac0.4: ***Timeout waiting for link up
nbi0.mac0.5: ***Timeout waiting for link up
nbi0.mac0.6: 10GE Link Up! (delay: 1)
nbi0.mac0.7: 10GE Link Up! (delay: 1)

The timeouted Ports 0,1,4,5 are connected to a switch whereas 2,3,6,7
are connected directly to a server.
Can this be a configuration problem on my side or shall I contact my
system admin?

On 29.05.2018 10:30, David George wrote:
> doh - I left it out (I edited these files by hand without testing)
>
> port = 6
> TMQ_per_port = 8 * 8
> port  * TMQ_per_port = 384
>
> in that file insert a new cluster entry between cluster11 and cluster12
> from 384 to 415 and it should work
>
> On 29 May 2018 at 10:26, Matthias Jasny <matthi...@gmail.com

> <mailto:matthi...@gmail.com>> wrote:
>
> Ports 2,3 and 7 work, Port 6 is still missing.
>
>
>
> On 29.05.2018 10:05, David George wrote:
>
>
>
>     "cluster0": {
>          "start_q":0,
>          "end_q":31,
>          "q_enable":true,
>          "q_drop_enable":true,
>          "q_size":16,        <--- why 16 and not 32 ???
>
>
> that is the depth of the individual queues in the group
>
>     Is 12x10GE correct? The breakout cable provides only 8x10GE
> ports or
>     is it just the old filename?
>
>
> I thought it would work, but after looking closely noticed gaps
> in the queues too (sorry). I tweaked the queues and renamed it
> (see attached), hopefully this will work.
>
>
>
>
> --

> *David George*
> /Principal Software Engineer/
>

> *Netronome* | 1st Floor, Southdowns Ridge Office Park, Cnr John Vorster &

>                       Nellmapius Drive, Irene, Centurion 0157, South Africa

[David George]

Well - check if your server ports link with the switch ports, if not then the problem is with the switch.

[Matthias Jasny]

I want to implement packet broadcasting, e.g. ARP Requests. P4 provided
instructions for cloning. I could not find micro-c functions that handle
cloning. Am I required to generate a packet for each output port
separately and copy the packet-data to each one or is there a more
convenient solution?

Second question: What is the exact purpose of PKT_NBI_OFFSET = 64?

Thanks!

-Matthias

[David George]

I want to implement packet broadcasting, e.g. ARP Requests. P4 provided instructions for cloning. I could not find micro-c functions that handle cloning. Am I required to generate a packet for each output port separately and copy the packet-data to each one or is there a more convenient solution?

See the attached for how to create packets from scratch. You will obviously copy your packet data, rather than generate your own.

 

Second question: What is the exact purpose of PKT_NBI_OFFSET = 64?

This the offset within the CTM buffer that the NBI will use when copying packets. The offset is needed to for packet metadata, both system and potentially user. You can use 128B as well.

David

--

David George
Principal Software Engineer

Netronome | 1st Floor, Southdowns Ridge Office Park, Cnr John Vorster &

                      Nellmapius Drive, Irene, Centurion 0157, South Africa

[Matthias Jasny]

Hi David,

thanks for the code, works fine. Is there a way to set the size of the MU buffer or what's the limit?

-Matthias

[David George]

thanks for the code, works fine.

No problem - glad to hear it works.

 

Is there a way to set the size of the MU buffer or what's the limit?

It is set to a fixed size at firmware compile time, you likely have 10kB max.

Look for NBI[89]_BLQ_EMU_[01234]_PKTBUF_SIZE in blm_custom.h.

The CTM buffer size is tunable. For all the possibilities look PKT_CTM_SIZE_* for in your microc libraries.

David