Understanding multiple interface support on ZCU106

[John Merrill]

Hi Alex,

I have a question related to how support for multiple interfaces maps to physical hardware. We're booting Corundum on a ZCU106 with one physical ethernet connection (on the RJ45 port). We're using a Corundum configuration with 2 interfaces. Loading the driver provides us with 2 network interfaces, which we are able to bring up simultaneously and use interchangeably to access the single physical ethernet connection. This brings into question what an "interface" corresponds to physically on the board. Could you help clarify this? Is there code or documentation we can look at to better understand this mapping?

Thank you,

John

[Alex Forencich]

That's strange; one interface can have multiple ports associated, but each port should only be attached to one interface.  The default design should connect one interface to each SFP+ port, with nothing connected to the RJ45 port (that port goes directly to the zynq PL).  I can test on my ZCU106 to see if I can replicate what you're seeing, although I might need a few more details on your setup. 

Alex Forencich

--
You received this message because you are subscribed to the Google Groups "corundum-nic" group.
To unsubscribe from this group and stop receiving emails from it, send an email to corundum-nic...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/corundum-nic/061df180-d67d-4bde-966b-47c2d2763dd7n%40googlegroups.com.

[John Merrill]

Thanks for the clarification; it makes much more sense taking the SFP ports into consideration. As for our ZCU106 setup, here are some more details of how we tested:

- We have the ZCU106 connected via PCI-e to a host (Ubuntu 20.04) workstation.

- We have the RJ45 on the ZCU106 looping back to an onboard RJ45 port on the workstation.

- We tested the two interfaces by assigning them static IPs in ifconfig then SSH'ing to each of them. SSH'ing to either interface gave us a shell on the workstation.

As for setting up SFP, we don't have any RJ45-to-SFP adapters in-house to test with. A search for the right adapter reveals that there are a few different brands and transceiver types with varying levels of device compatibility. Do you know of a particular adapter that works well with the ZCU106?

I appreciate your help.

Thanks,

John

[Alex Forencich]

My suggestion is to buy a 10G SFP+ PCIe NIC and an SFP+ DAC.  Currently, the SFP+ modules only run at 10 Gbps and don't (currently) support negotiating down, so you would need 10GBASE-T adapters and a 10G BASE-T NIC.  However, that is probably far more expensive than picking up a used 10G SFP+ NIC and DAC off of ebay.  Although it depends on what your final application would be.  If you ultimately want to run at 1 Gbps, then it is possible to swap out the 10GBASE-R PHYs for 1000BASE-X PHYs on the FPGA, and then you should be able to use gigabit RJ45 to SFP adapters.  I have not tested the corundum datapath at 8 bits, I may need to tweak some parameters to get it to work properly.  But you can just use AXI stream width converters for the time being. 

Also, bear in mind that linux tries to be smart and will route the packets internally if it can.  Most likely none of your test packets left the host.  There are a few solutions for this.  Most reliable is to use two machines.  Another option is to use network namespaces.  I have a script for that: https://github.com/alexforencich/verilog-ethernet/blob/master/scripts/dev-netns-shell.sh .  You run that script with the interface name as an argument, and then it will create a namespace, add the interface to the namespace, and then drop you into a shell running in that namespace.  It's also smart enough to manage the namespace nicely so you can run multiple instances of the script attached to the same interface.  From there, you cannot see any of the other NICs, so traffic is forced to use the associated NIC instead of getting routed internally.  So you can try running your ssh client from there as a test case.  My guess is that if you repeat the experiment, you won't be able to connect at all because the gigabit port on the ZCU106 is not used. 

Alex Forencich

To view this discussion on the web visit https://groups.google.com/d/msgid/corundum-nic/b4ea05e7-ad54-497b-8ede-d78b86ef7d81n%40googlegroups.com.