NFP-400 traffic fowarding issues
Sean Pennefather
I am currently attempting to implement a basic variant of the 'bump-in-the-wire' simulation discussed in the 'The Joy of Micro-C Filter Annex'. In this variant i am using a NFP-400 2x10GbE card and I'm trying to use actual network traffic operating at 1Gbs instead of running through a simulator. The goal is to try get a working example of forwarding traffic between the two interfaces on this card so if anyone has a working example and is willing to share, that would be a huge help. For testing i am simply sending generic UDP traffic (no VLAN) directly to the card.
Currently i am using lab5 from the c_packetprocessing as my basis and at the moment can receive traffic from the two physical interfaces on the card but cannot retransmit. I am also unsure of how the 'enet_ports' in the json file for configuring the MAC should be set up for this car. It only has two physical interfaces, so should only two port be configured? If so, which ones and if not, how are the ports associated with the interfaces?
I have been following a similar discussion initiated by Matthias Jasny (SmartNIC 2x40G with Breakout Cables) which has been very informative but i haven't quite been able to adapt the progress of that thread to my simple application.
I have attached the relevant files below, if anyone can spot some obvious issues i would be very grateful.
Many thanks
Sean
Joy of Micro-C Annex: (https://open-nfp.org/m/documents/the-joy-of-micro-c-annex_E8Wpvht.pdf)
c_packetprocessing lab5: (https://github.com/open-nfpsw/c_packetprocessing/tree/master/apps/lab5)
David George
Sean Pennefather
Thank you for the response. After including your suggestions and reviewing the previous thread again, i have been able to re-implement the basic reflect application for the NFP 400. I have put the basic application on github (https://github.com/g10p0016/nfp400_traffic_forward) in case anyone else has a similar problem and would like to use it for a bit of context.
I am however still unsure about how to forward traffic between the two NBIs.
the function pkt_nbi_send takes in a NBI indicator as one of the parameters which i assume indicates which of the (in this case two) NBIs the packet should be submitted over. I added the following flag to the makefile for application running on islands 32 and 33:
$(eval $(call micro_c.add_flags,wire0_obj, -DSRC_NBI=0)),
And for islands 35 and 36, I added the flag:
$(eval $(call micro_c.add_flags,wire0_obj, -DSRC_NBI=1)),
The hope was that these flags could be used to identify the source NBI and thus could be used to select the opposing NBI through a simple operation such as:
pkt_nbi_send(pi->isl, pi->pnum, &msi, pi->len - MAC_PREPEND_BYTES + 4, ((SRC_NBI == 0) ? 1 : 0), PORT_TO_TMQ(out_port), nbi_meta.seqr, nbi_meta.seq, PKT_CTM_SIZE_256);
This unfortunately does not seem to work. I don't know if it is because i am misunderstanding how to send data to a different physical interface or because the out_port parameter should also be changed. Please let me know if you have any insight into this, ill keep exploring and see if can come up with a workable solution in the meantime.
Many thanks
Sean
Pallav Gupta
I am having similar kind of problem. Could you tell me what was your actual physical set up for this experiment ?
Pallav
Sean Pennefather
Hi Pallav,
Gladly. For this experiment, my setup is very simple (see image attached):
- The NFP card is in a host PC running Ubuntu 18, kernel version 4.15.0-22-generic.
- The two SFP interfaces on the NFP are each connected directly to a host device running a generic Linux based OS.
- In my case these connections include a media translator to convert the fiber medium to a copper medium.
This configuration allows me to have a direct connection to the NFP card for both client devices which means i don't have to worry about the type of traffic i send as no routing is required.
The application is then built on the host and flashed to the NFP card. In this experiment, the host has no involvement in the network traffic and is effectively only responsible for flashing and powering the NFP card.
Once the application is running, i then run sudo tshark -x -i eth0 on both Client A and B to monitor for traffic.
- I would also recommend to enable promiscuous mode on these interfaces before running tshark. This be done on Linux by:
ip link set eth0 promisc on (though this command will depend on your OS). - You can check if promiscuous is enabled using: netstat -i and checking if the 'P' flag is set for the interface (assuming you are using a Linux based OS).
sudo tcpreplay --intf1=eth0 single_pkt.pcap on either Client A or B to direct arbitrary network traffic to the NFP card. - I have attached a pacp with contains a single udp packet for this example.
Capturing on 'eth0'
0000 10 10 10 10 10 10 18 18 18 18 18 18 08 00 45 00 ..............E.
0010 00 2e 00 01 00 00 40 11 3a bd 10 00 00 01 30 00 ......@.:.....0.
0020 00 01 04 01 00 0c 00 1a 7f 6f 78 78 78 78 78 78 .........oxxxxxx
0030 78 78 78 78 78 78 78 78 78 78 78 78 xxxxxxxxxxxx
0000 10 10 10 10 10 10 18 18 18 18 18 18 08 00 45 00 ..............E.
0010 00 2e 00 01 00 00 40 11 3a bd 10 00 00 01 30 00 ......@.:.....0.
0020 00 01 04 01 00 0c 00 1a 7f 6f 78 78 78 78 78 78 .........oxxxxxx
0030 78 78 78 78 78 78 78 78 78 78 78 78 xxxxxxxxxxxx
The recording depicts two instances of the packet; one for the egress and one which was reflected back by the NFP.
I hope this helps, please shout if you need me to clarify anything.
Regards
Sean
Sean Pennefather
As an update, i have been able to forward traffic between the two
physical interfaces on the device and have updated the repository accordingly.
After some general testing however, i do have two questions which i
would be very grateful if anyone could answer.
Firstly, i noted that for the NFP-400 2x10GE, all traffic received would
have an port number of either 0 or 4 depending on which physical
interface it was received over. Why is there and offset of 4 between the
physical interfaces and what are ports 1-3 reserved for?
Secondly, as best i can tell, traffic is only sent or received from one
NBI (all traffic is being handled by engines on island 32). After
reading the The Joy of MicroC Annex
(https://open-nfp.org/m/documents/the-joy-of-micro-c-annex_E8Wpvht.pdf),
i was under the impression that there would a separate NBI for each
physical interface. Do some architectures contain only a single NBI, and
how are the NBIs associated with physical interfaces? Is there
documentation indicating which products contain multiple NBIs and how
they are associated with NFP cards supporting multiple interfaces?
Kind regards,
Sean
David George
As an update, i have been able to forward traffic between the two physical interfaces on the device and have updated the repository accordingly. After some general testing however, i do have two questions which i would be very grateful if anyone could answer.
Firstly, i noted that for the NFP-400 2x10GE, all traffic received would have an port number of either 0 or 4 depending on which physical interface it was received over. Why is there and offset of 4 between the physical interfaces and what are ports 1-3 reserved for?
Secondly, as best i can tell, traffic is only sent or received from one NBI (all traffic is being handled by engines on island 32). After reading the The Joy of MicroC Annex (https://open-nfp.org/m/documents/the-joy-of-micro-c-annex_E8Wpvht.pdf), i was under the impression that there would a separate NBI for each physical interface. Do some architectures contain only a single NBI, and how are the NBIs associated with physical interfaces? Is there documentation indicating which products contain multiple NBIs and how they are associated with NFP cards supporting multiple interfaces?