Problem with using SRIOV in openshift

[Pravein Govindan Kannan]

Hi All,

 I am working on moving some VMs to Openshift Virtualization with SRIOV. We faced a problem when launching the VM with SRIOV network interface on a bare-metal worker node. We observe the below logs (snippets) from the VM launcher, and eventually the VM start fails:

1) Initial logs when launch of VM (from launcher pod)

{"component":"virt-launcher","level":"info","msg":"Connecting to libvirt daemon: qemu:///system","pos":"libvirt.go:380","timestamp":"2021-03-05T07:28:29.655999Z"}
{"component":"virt-launcher","level":"info","msg":"Connecting to libvirt daemon failed: virError(Code=38, Domain=7, Message='Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory')","pos":"libvirt.go:388","timestamp":"2021-03-05T07:28:29.656241Z"}
{"component":"virt-launcher","level":"info","msg":"libvirt version: 6.0.0, package: 25.5.module+el8.2.1+8680+ea98947b (Red Hat, Inc. \u003chttp://bugzilla.redhat.com/bugzilla\u003e, 2020-11-06-13:17:30, )","subcomponent":"libvirt","thread":"71","timestamp":"2021-03-05T07:28:29.686000Z"}
{"component":"virt-launcher","level":"info","msg":"hostname: testvm","subcomponent":"libvirt","thread":"71","timestamp":"2021-03-05T07:28:29.686000Z"}
{"component":"virt-launcher","level":"error","msg":"internal error: Child process (/usr/sbin/dmidecode -q -t 0,1,2,3,4,17) unexpected exit status 1: /dev/mem: No such file or directory","pos":"virCommandWait:2713","subcomponent":"libvirt","thread":"71","timestamp":"2021-03-05T07:28:29.686000Z"}
{"component":"virt-launcher","level":"info","msg":"Connected to libvirt daemon","pos":"libvirt.go:396","timestamp":"2021-03-05T07:28:30.158963Z"}
{"component":"virt-launcher","level":"info","msg":"Registered libvirt event notify callback","pos":"client.go:394","timestamp":"2021-03-05T07:28:30.163827Z"}
{"component":"virt-launcher","level":"info","msg":"Marked as ready","pos":"virt-launcher.go:71","timestamp":"2021-03-05T07:28:30.164249Z"}
{"component":"virt-launcher","level":"info","msg":"SR-IOV PCI device allocated: 0000:19:02.5","pos":"converter.go:1305","timestamp":"2021-03-05T07:28:33.452457Z"}

2) Finally, the launch fails with the below errors:

{"component":"virt-launcher","level":"warning","msg":"Failed to read from '/sys/bus/pci/devices/0000:19:02.5/config' : No such file or directory","pos":"virPCIDeviceRead:346","subcomponent":"libvirt","thread":"65","timestamp":"2021-03-05T07:28:33.871000Z"}

{"component":"virt-launcher","level":"error","msg":"unsupported configuration: pci backend driver 'default' is not supported","pos":"virHostdevGetPCIHostDevice:254","subcomponent":"libvirt","thread":"57","timestamp":"2021-03-05T07:28:33.908000Z"}
{"component":"virt-launcher","level":"error","msg":"Failed to allocate PCI device list: unsupported configuration: pci backend driver 'default' is not supported","pos":"virHostdevReAttachPCIDevices:1133","subcomponent":"libvirt","thread":"57","timestamp":"2021-03-05T07:28:33.908000Z"}
{"component":"virt-launcher","kind":"","level":"error","msg":"Starting the VirtualMachineInstance failed.","name":"testvm","namespace":"lab-vms","pos":"manager.go:1297","reason":"virError(Code=67, Domain=10, Message='unsupported configuration: host doesn't support passthrough of host PCI devices')","timestamp":"2021-03-05T07:28:33.909616Z","uid":"1b7ec2fd-c993-489b-be8d-e132bd93aacb"}

Kindly note that, IOMMU is on in the kernel settings, and VFs are allocated in the worker node. I logged in to the launcher pod, and was able to see the PCI device and libvirt-sock.

Kindly suggest what I would need to do?

Thanks,

Pravein

[Petr Horacek]

st 10. 3. 2021 v 11:37 odesílatel Pravein Govindan Kannan <prav...@gmail.com> napsal:

Hello,

It seems that you are using the wrong driver for the SR-IOV VF. KubeVirt's SR-IOV passthrough requires the VF to be set up with the vfio-pci driver.

Could you verify you use them? See "deviceType" in https://kubevirt.io/user-guide/virtual_machines/interfaces_and_networks/#how-to-expose-sr-iov-vfs-to-kubevirt.

Petr

 

Thanks,

Pravein

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/d0033cd0-8233-452c-87f9-78fdf931853en%40googlegroups.com.

[Pravein GK]

Hi Petr,

Thanks for your response. I am using the below yaml, where the deviceType is netdevice :

apiVersion: sriovnetwork.openshift.io/v1 

kind: SriovNetworkNodePolicy 

metadata:

  name: bcm-sriov1-policy

  namespace: openshift-sriov-network-operator

spec:

  resourceName: bcmsriov1

  nodeSelector:

    node-role.kubernetes.io/sriov: ""

  numVfs: 8

  nicSelector:

    pfNames: ['eno1np0'] 

  deviceType: netdevice

Thanks,

Pravein

[Pravein GK]

Hi Petr,

When I change the "deviceType" to "vfio-pci", The SRIOV device plugin, does not add the VF to the resource pool : 

I0310 10:53:15.394232 25 netDeviceProvider.go:78] netdevice AddTargetDevices(): device found: 0000:19:03.5 02 Broadcom Inc. and... NetXtreme-E Ethernet Virtual Function 

I0310 10:53:15.394362 25 netDeviceProvider.go:78] netdevice AddTargetDevices(): device found: 0000:19:03.6 02 Broadcom Inc. and... NetXtreme-E Ethernet Virtual Function 

I0310 10:53:15.394468 25 netDeviceProvider.go:78] netdevice AddTargetDevices(): device found: 0000:19:03.7 02 Broadcom Inc. and...NetXtreme-E Ethernet Virtual Function 

I0310 10:53:15.394572 25 main.go:66] Initializing resource servers 

I0310 10:53:15.394579 25 manager.go:112] number of config: 1 

I0310 10:53:15.394582 25 manager.go:115] 

I0310 10:53:15.394585 25 manager.go:116] Creating new ResourcePool: bcmsriov1 

I0310 10:53:15.394587 25 manager.go:117] DeviceType: netDevice 

I0310 10:53:15.406104 25 manager.go:130] 

no devices in device pool, skipping creating resource server for bcmsriov1 

I0310 10:53:15.406132 25 main.go:72] Starting all servers... 

I0310 10:53:15.406144 25 main.go:77] All servers started. 

I0310 10:53:15.406155 25 main.go:78] Listening for term signals 

Thanks,

Pravein

[Petr Horacek]

Unfortunately, I don't know how to help you with this. It is not specific to KubeVirt. If I understand correctly, you are using OpenShift, so you may need to go through different channels - either through OpenShift or the SR-IOV project.

st 10. 3. 2021 v 12:04 odesílatel Pravein GK <prav...@gmail.com> napsal:

[Pravein GK]

Thanks for the help Petr.

If anyone else knows the problem, please suggest.

Thanks,

Pravein

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/CAGkpguVSU9UWoNDQuBPHtneL%2BEw%2Br%2BRwxhiUrf9qGpUCYt62dg%40mail.gmail.com.

[Sebastian Scheinkman]

Hi Pravein,

Can you please share the `sriovnetworknodestates.sriovnetwork.openshift.io` from the `openshift-sriov-network-operator` namespace.

Thanks!

Sebastian

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/CAE4%3DsScVnFOr_xk3zuih3fHFvbaTKUjSFz3WACKTC%2BfjMJF_6A%40mail.gmail.com.

--

Sebastian Scheinkman

Red Hat

[Pravein GK]

Hi Sebastian,

Please find the sriovnetworknodestates.sriovnetwork.openshift.io of the worker node attached.

Thanks,

Pravein

[Or Shoval]

On Wed, Mar 10, 2021 at 3:54 PM Sebastian Scheinkman <ssch...@redhat.com> wrote:

Hi Pravein,

Can you please share the `sriovnetworknodestates.sriovnetwork.openshift.io` from the `openshift-sriov-network-operator` namespace.

Thanks!

Sebastian

Hi,

Assuming you are using sriov-operator 4.3 / 4.4  ?

sriov - operator 4.4 has a bug that sometimes you need to delete the sriov-network-config-daemon

and then delete the sriov-network-device-plugin (in this order), 

in case it didn't create VFs (which can result in no VF in the device pool),

we can see if its the case from the NodeState that Sebastian asked above,

sriov-operator 4.8 is much more stable in this aspect btw.

Thanks

 

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/CAD7DGuKQUBOeMZGuWmpSDTuk%2BJL%2B3XoCDOeL6FJyPjOCa6GiLg%40mail.gmail.com.

[Pravein GK]

I use 4.6 version. I tried to restart the pods. However, I get the same problem. Please find the sriov-network-device-plugin logs.

Thanks,
Pravein

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/CAMF_ihCfP4REbLsBq%3DpGfB9X0F1r8iF08Wn%2BXt%2BM91%2B4a%3DpjCQ%40mail.gmail.com.

[Or Shoval]

On Wed, Mar 10, 2021 at 4:15 PM Pravein GK <prav...@gmail.com> wrote:

I use 4.6 version. I tried to restart the pods. However, I get the same problem. Please find the sriov-network-device-plugin logs.

Thanks,
Pravein

I0310 14:10:38.747579      25 manager.go:130] no devices in device pool, skipping creating resource server for bcmsriov1

Do you have VFs in ip link command ? 

if no, can you create please VF manually and then restart the pods ?

see please 

https://bugzilla.redhat.com/show_bug.cgi?id=1878566

[Pravein GK]

Yes, "ip link" lists the VFs : 

# ip link

2: eno1np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000

    link/ether bc:97:e1:a4:9b:ea brd ff:ff:ff:ff:ff:ff

    vf 0     link/ether e6:b7:9e:d9:63:bd brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off

    vf 1     link/ether 3e:21:f7:1e:99:df brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off

    vf 2     link/ether b6:b9:14:e8:79:ad brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off

    vf 3     link/ether a6:7a:8a:eb:89:84 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off

    vf 4     link/ether 26:e3:fa:a9:39:9d brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off

    vf 5     link/ether be:eb:e5:49:8e:10 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off

    vf 6     link/ether da:20:3a:01:96:52 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off

    vf 7     link/ether b2:1b:5c:44:87:91 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off

3: eno3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000

    link/ether bc:97:e1:a4:9b:e8 brd ff:ff:ff:ff:ff:ff

4: eno4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000

    link/ether bc:97:e1:a4:9b:e9 brd ff:ff:ff:ff:ff:ff

5: eno2np1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000

    link/ether bc:97:e1:a4:9b:eb brd ff:ff:ff:ff:ff:ff

    vf 0     link/ether 82:9d:11:29:66:0c brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off

    vf 1     link/ether 4e:c7:98:39:4e:ad brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off

    vf 2     link/ether aa:3b:83:f4:c8:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off

    vf 3     link/ether 42:38:0a:eb:2b:50 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off

    vf 4     link/ether ee:47:38:31:71:8d brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off

    vf 5     link/ether f6:cd:08:90:d9:13 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off

    vf 6     link/ether 8e:25:43:26:49:1f brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off

    vf 7     link/ether ea:65:ce:dc:97:43 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off

[Or Shoval]

Only ideas left from my side (based on the same that i had):

1. Delete create the SriovPolicy, as it will retrigger reconciliation

(might need to add after it the delete of the 2 pods mentioned earlier, where order is important)

2. undeploy and redeploy the sriov operator

3. Bump to sriov operator 4.8 to see if its solves

But it might be something else of course, let's hear please other opinions

[Sebastian Scheinkman]

Hi Pravein,

Can you please also share the sriov-config-daemon logs

btw in the spec you have only 1 policy but in the status I see it configure 2 devices.

Last comment the nic that you are using is not a support nic by the sriov-network-operator from the openshift documentation so I am not sure if it will work.

Thanks!

Sebastian