Re: DPDK example - "No Ethernet ports" error

Message has been deleted

Waldek Kozaczuk

unread,

Sep 9, 2021, 10:38:57 PM9/9/21

to OSv Development

Hi,

I have just found some time to look at it and play with this DPDK app. And I have some interesting news to share.

On Sunday, August 29, 2021 at 8:29:45 PM UTC-4 Sergio Inzunza wrote:

Hello,

I hope everyone is doing great. I was wondering if anyone had any experience with running the dpdk-example image https://github.com/cloudius-systems/osv-apps/tree/184beda56c0bc01f8991ca2abcabacc6181ff57c/dpdk-example

I have tried running the image by exporting it to Lib-Virt as stated in the example and then running the l2fwd application. However, I get the following error:

EAL: Error - exiting with code: 1
Cause: No Ethernet ports - bye

First of all, I assume you have managed to setup two bridges - default and net2 - as l2fwd requires two NICs. Setting up the net2 two is not really described in the app readme and I am planning to update it.

For now, you can follow method 2 in https://computingforgeeks.com/how-to-create-and-configure-bridge-networking-for-kvm-in-linux/. In essence, you can export the default network using virsh, change what is needed and use virsh to create a new bridge using new xml file. Roughly like so:

virsh net-dumpxml default

cat net2.xml

<nat>

</nat>

</forward>

<dhcp>

</dhcp>

</ip>

</network>

virsh net-create --file ./net2.xml

I had trouble seeing any output from OSv guest when using virsh to actually run OSv as it is documented in README (it would be nice to get to the bottom of this). But instead, you can use "./script/run.py -n --dry-output" to create a shell script and manually add a second device for virbr1 bridge like so:

./scripts/run.py -n --dry-run

/home/wkozaczuk/projects/osv-master/scripts/../scripts/imgedit.py setargs /home/wkozaczuk/projects/osv-master/build/last/usr.img "--rootfs=zfs --console=serial --verbose --maxnic=0 /l2fwd --no-shconf -c f -n 2 --log-level 8 -m 768 -- -p 3"

qemu-system-x86_64 \

-m 2G \

-smp 4 \

-vnc :1 \

-gdb tcp::1234,server,nowait \

-device virtio-blk-pci,id=blk0,drive=hd0,scsi=off,bootindex=0 \

-drive file=/home/wkozaczuk/projects/osv-master/build/last/usr.img,if=none,id=hd0,cache=none,aio=native \

-netdev bridge,id=hn0,br=virbr0,helper=/usr/lib/qemu/qemu-bridge-helper \

-device virtio-net-pci,netdev=hn0,id=nic0 \

-device virtio-rng-pci \

-enable-kvm \

-cpu host,+x2apic \

-chardev stdio,mux=on,id=stdio,signal=off \

-mon chardev=stdio,mode=readline \

-device isa-serial,chardev=stdio

add 2 lines:

-netdev bridge,id=hn1,br=virbr1,helper=/usr/lib/qemu/qemu-bridge-helper \

-device virtio-net-pci,netdev=hn1,id=nic1 \

I was wondering if maybe there is some additional steps to be done that weren't included. Possibly on binding the unikernel nic to the dpdk driver.

But even if you did all this, l2fwd would still not detect any NICs just like in our example.

After some digging, I have discovered the problem lies in lib/librte_eal/osvapp/eal/eal_pci.cc, where we have logic to discover PCI devices that are not attached. In short, the logic there does not work due to some changes I made when refactoring virtio devices and drivers code layer in OSv (see the commit https://github.com/cloudius-systems/osv/commit/43777010faf8e3fe3abbfa65c915ce05b2715581) 2 years ago when doing work to support Firecracker. For more insight, read the section Virtio of the blog article - http://blog.osv.io/blog/2019/04/19/making-OSv-run-on-firecraker/. In essence, the device objects registered in DeviceManager are not necessarily instances of pci::device class (they may be instances of virtio::virtio_device instead). So this code in eal_pci.cc becomes problematic:

/* Scan one pci entry, and fill the devices list from it. */

static int

pci_scan_one(hw::hw_device* dev)

{

u8 bus, device, func;

auto pci_dev = static_cast<pci::device*>(dev);

auto rte_dev = new rte_pci_device();

/* get bus id, device id, function no */

pci_dev->get_bdf(bus, device, func);

****

}

/*

* Scan the content of the PCI bus, and add the devices in the devices

* list. Call pci_scan_one() for each pci entry found.

*/

static int

pci_scan(void)

{

unsigned dev_count = 0;

int err = 0;

auto dm = hw::device_manager::instance();

dm->for_each_device([&dev_count, &err] (hw::hw_device* dev) {

if (dev->is_attached())

return;

int ret = pci_scan_one(dev);

if (ret < 0) {

err++;

} else {

dev_count += ret;

}

});

if (err)

return -1;

RTE_LOG(ERR, EAL, "PCI scan found %u devices\n", dev_count);

return 0;

}

As you can see we call for_each_device() on device_manager and then in pci_scan_one() cast each dev to pci::device* which does NOT always work (it actually never works for devices we are interested - virtio ones).

The virtio ones registered in device_manager are instances of virtio::virtio_device.

I do not have a correct fix for that (most likely it will require some changes to OSv interfaces and the DPDK OSv app as well) but be on the lookout for some patches. If somebody has good suggestions for some clean design solution here, I would appreciate it (how to expose PCI devices to DPDK app).

Meanwhile, I hacked things and the app starts correctly:

sudo ./run_with_2_nics.sh

OSv v0.56.0-3-g985d4d76

4 CPUs detected

Firmware vendor: SeaBIOS

console=serial

bsd: initializing - done

VFS: mounting ramfs at /

VFS: mounting devfs at /dev

net: initializing - done

-> Device Manager: registered device with ID:305627270

-> Device Manager: registered device with ID:1879081094

-> Device Manager: registered device with ID:1880129670

-> Device Manager: registered device with ID:1897103494

-> Device Manager: registered device with ID:286331444

-> Device Manager: registered device with ID:137972

-> Device Manager: registered device with ID:72436

-> Device Manager: registered device with ID:269044

virtio-blk: Add blk device instances 0 as vblk0, devsize=268435456

random: virtio-rng registered as a source.

random: intel drng, rdrand registered as a source.

random: <Software, Yarrow> initialized

Config space of: 00:00.0

Config space of: 00:01.0

Config space of: 00:01.1

Config space of: 00:01.3

Config space of: 00:02.0

Config space of: 00:03.0

Config space of: 00:04.0

Config space of: 00:05.0

Config space of: 00:06.0

random: device unblocked.

VFS: unmounting /dev

VFS: mounting zfs at /zfs

zfs: mounting osv/zfs from device /dev/vblk0.1

VFS: mounting devfs at /dev

VFS: mounting procfs at /proc

VFS: mounting sysfs at /sys

BSD shrinker: event handler list found: 0xffffa00000d9d080

BSD shrinker found: 1

BSD shrinker: unlocked, running

Booted up in 607.68 ms

Cmdline: /l2fwd --no-shconf -c f -n 2 --log-level 8 -m 768 -- -p 3

EAL: Detected lcore 0 as core 0 on socket 0

EAL: Detected lcore 1 as core 1 on socket 0

EAL: Detected lcore 2 as core 2 on socket 0

EAL: Detected lcore 3 as core 3 on socket 0

EAL: Support maximum 128 logical core(s) by configuration.

EAL: Detected 4 lcore(s)

EAL: pci_scan_one() read BDF: [0:2.0]!

EAL: bar0 mmio at 0xffff8000fd000000

EAL: bar1 not available

pci_dev:0xffffa00000d8d780

EAL: pci_scan_one() read BDF: [0:4.0]!

EAL: bar0 NON-mmio at 0xc080

EAL: bar1 mmio at 0xffff8000feb92000

EAL: bar2 not available

pci_dev:0xffffa00000d8d880

EAL: pci_scan_one() read BDF: [0:5.0]!

EAL: bar0 NON-mmio at 0xc0a0

EAL: bar1 mmio at 0xffff8000feb93000

EAL: bar2 not available

pci_dev:0x60002

EAL: pci_scan_one() read BDF: [0:1.0]!

EAL: bar0 not available

EAL: pci_scan_one() inserted BEFORE

EAL: pci_scan_one() read BDF: [0:1.1]!

EAL: bar0 not available

EAL: pci_scan_one() inserted BEFORE

EAL: pci_scan_one() read BDF: [0:1.3]!

EAL: bar0 not available

EAL: pci_scan_one() inserted BEFORE

EAL: PCI scan found 6 devices

EAL: Setting up memory...

EAL: Mapped memory segment 0 @ 0xffff80013d800000: physaddr:0x13d800000, len 33554432

EAL: Mapped memory segment 0 @ 0xffff80013b600000: physaddr:0x13b600000, len 33554432

EAL: Mapped memory segment 0 @ 0xffff800139400000: physaddr:0x139400000, len 33554432

EAL: Mapped memory segment 0 @ 0xffff800137200000: physaddr:0x137200000, len 33554432

EAL: Mapped memory segment 0 @ 0xffff800135000000: physaddr:0x135000000, len 33554432

EAL: Mapped memory segment 0 @ 0xffff800132e00000: physaddr:0x132e00000, len 33554432

EAL: Mapped memory segment 0 @ 0xffff800130c00000: physaddr:0x130c00000, len 33554432

EAL: Mapped memory segment 0 @ 0xffff80012ea00000: physaddr:0x12ea00000, len 33554432

EAL: Mapped memory segment 0 @ 0xffff80012c800000: physaddr:0x12c800000, len 33554432

EAL: Mapped memory segment 0 @ 0xffff80012a600000: physaddr:0x12a600000, len 33554432

EAL: Mapped memory segment 0 @ 0xffff800128400000: physaddr:0x128400000, len 33554432

EAL: Mapped memory segment 0 @ 0xffff800126200000: physaddr:0x126200000, len 33554432

EAL: Mapped memory segment 0 @ 0xffff800124000000: physaddr:0x124000000, len 33554432

EAL: Mapped memory segment 0 @ 0xffff800121e00000: physaddr:0x121e00000, len 33554432

EAL: Mapped memory segment 0 @ 0xffff80011fc00000: physaddr:0x11fc00000, len 33554432

EAL: Mapped memory segment 0 @ 0xffff80011da00000: physaddr:0x11da00000, len 33554432

EAL: Mapped memory segment 0 @ 0xffff80011b800000: physaddr:0x11b800000, len 33554432

EAL: Mapped memory segment 0 @ 0xffff800119600000: physaddr:0x119600000, len 33554432

EAL: Mapped memory segment 0 @ 0xffff800117400000: physaddr:0x117400000, len 33554432

EAL: Mapped memory segment 0 @ 0xffff800115200000: physaddr:0x115200000, len 33554432

EAL: Mapped memory segment 0 @ 0xffff800113000000: physaddr:0x113000000, len 33554432

EAL: Mapped memory segment 0 @ 0xffff800110e00000: physaddr:0x110e00000, len 33554432

EAL: Mapped memory segment 0 @ 0xffff80010ec00000: physaddr:0x10ec00000, len 33554432

EAL: Mapped memory segment 0 @ 0xffff8000bde00000: physaddr:0xbde00000, len 33554432

EAL: TSC frequency is ~0 KHz

EAL: Master lcore 0 is ready (tid=d8e400;cpuset=[0])

PMD: ENICPMD trace: rte_enic_pmd_init

EAL: lcore 1 is ready (tid=d8e430;cpuset=[1])

EAL: lcore 2 is ready (tid=d8e440;cpuset=[2])

EAL: lcore 3 is ready (tid=d8e420;cpuset=[3])

EAL: PCI device 0000:00:04.0 on NUMA socket -1

EAL: probe driver: 1af4:1000 rte_virtio_pmd

EAL: PCI device 0000:00:05.0 on NUMA socket -1

EAL: probe driver: 1af4:1000 rte_virtio_pmd

Lcore 0: RX port 0

Lcore 1: RX port 1

Initializing port 0... done:

Port 0, MAC address: 52:54:00:12:34:56

Initializing port 1... done:

Port 1, MAC address: 52:54:00:12:34:57

Checking link statusdone

Port 0 Link Up - speed 10000 Mbps - full-duplex

Port 1 Link Up - speed 10000 Mbps - full-duplex

L2FWD: entering main loop on lcore 1

L2FWD: entering main loop on lcore 0

L2FWD: lcore 3 has nothing to do

L2FWD: lcore 2 has nothing to do

L2FWD: -- lcoreid=1 portid=1

L2FWD: -- lcoreid=0 portid=0

Port statistics ====================================

Statistics for port 0 ------------------------------

Packets sent: 22

Packets received: 14

Packets dropped: 0

Statistics for port 1 ------------------------------

Packets sent: 14

Packets received: 22

Packets dropped: 0

Statistics for port 2 ------------------------------

...

Waldek

PS. Have you had a chance to update the app https://github.com/syuu1228/osv-dpdk/tree/osv-head to newest DPDK?

Thanks in advance,
Sergio Inzunza

Waldek Kozaczuk

unread,

Sep 13, 2021, 9:43:33 AM9/13/21

to OSv Development

Hi,

I have pushed some commits to address this issue.

Waldek

Message has been deleted

Waldek Kozaczuk

unread,

Sep 13, 2021, 6:57:20 PM9/13/21

to Sergio Inzunza, OSv Development

I have pushed a commit to a dpdk-example that changes pci_scan_one() to correctly identify PCI devices. It also relies on some changes I made to OSv itself to allow for these changes.

Waldek

On Mon, Sep 13, 2021 at 18:24 Sergio Inzunza <sinz...@ucr.edu> wrote:

Hi Waldek,

wow this some interesting stuff!

I did try setting up the two network bridges as you described a while back, however I still was not able to run l2fwd.

I have been working on a modern DPDK implementation for OSv, specifically for DPDK v20.05. However, I have not been able to test if it fully works yet. I was testing it on the l2fwd application, but without it being able to run I had no results.

The PCI driver was where I probably had the majority of trouble. As the modern DPDK versions have restructured the driver code rather significantly. But the functionality for pci_scan_one() and pci_scan() remained relatively similar. I still however, was casting each dev to pci::device. I am going to try messing with your new findings over the next couple of days.

Thank you so much!

Sincerely,
Sergio Inzunza

--
You received this message because you are subscribed to the Google Groups "OSv Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osv-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/07ed352e-315a-4379-ae20-3072e5a1cc47n%40googlegroups.com.

Waldek Kozaczuk

unread,

Sep 13, 2021, 7:08:53 PM9/13/21

to Sergio Inzunza, OSv Development

https://github.com/cloudius-systems/osv-apps/commit/63a4abde3ad71d3ef160831b9ad0a5efda99a5ec

Message has been deleted

Waldek Kozaczuk

unread,

Sep 14, 2021, 10:10:53 AM9/14/21

to OSv Development

Also, make sure to use the latest OSv with this commit - https://github.com/cloudius-systems/osv/commit/b4b4829e33400d9c22285f16c42a0119d79851cf.

On Monday, September 13, 2021 at 7:20:41 PM UTC-4 Sergio Inzunza wrote:

Thank you so much Waldek. I will look over these changes. If I have success with the modern DPDK implementation I will update the group.

Reply all

Reply to author

Forward