Integrating KubeVirt into StarlingX -- SRIOV issue

[Greg Waines]

Hey,

I work in the StarlingX open-source group; https://www.starlingx.io/    https://docs.starlingx.io/ .

StarlingX provides infrastructure management for setting up a Kubernetes cluster on bare metal ( and optionally a containerized openstack cloud on top of that).

We/StarlingX are looking at supporting KubeVirt on the StarlingX Kubernetes cluster as well.

I have been doing some basic tests of KubeVirt/virtctl/CDI on StarlingX Kubernetes deployment ... e.g. simple ubuntu server and simple windows server, just using basic CNI networking.  And all works well.

I am starting to look at SRIOV now.

I notice that KubeVirt docs refer to OpenShift/OKD -specific CRDs for SRIOV interfaces and SRIOV networks  (i.e. SriovNetworkNodePolicy and SriovNetwork).

Startlingx has its own infrastructure management mechanisms for configuring SRIOV interfaces and configuring the sriov device plugin (see https://docs.starlingx.io/node_management/kubernetes/node_interfaces/provisioning-sr-iov-interfaces-using-the-cli.html )

AND

Starlingx uses NetworkAttachmentDefinition to do the sriov network defintion.

In my StarlingX enviornment, I can configure a vanilla pod/container with an SRIOV interface successfully.

However if try to launch a KubeVirt VM with SRIOV (with the multus:networkName pointing to my NetworkAttachmentDefinition), then the KubeVirt VM schedules properly, but has the following error:

  - lastProbeTime: null

    lastTransitionTime: "2022-07-10T12:53:56Z"

    message: 'server error. command SyncVMI failed: "LibvirtError(Code=67, Domain=10,

      Message=''unsupported configuration: host doesn''t support passthrough of host

      PCI devices'')"'

    reason: Synchronizing with the Domain failed.

    status: "False"

    type: Synchronized

Any ideas what KubeVirt is looking for here ?  ... to determine that the host supports PCI devices ?  ... that is somehow missing in the StarlingX Infrastructure setup ?

Thanks in advance,

Greg.

[Petr Horacek]

Hello Greg,

This looks like your VFs were configured with netdevice instead of vfio-pci driver. In the SR-IOV operator this is controlled using `deviceType` attribute: https://github.com/k8snetworkplumbingwg/sriov-network-operator#sriovnetworknodepolicy

You may need to reconfigure VFs you want to use in VMs on your hosts to use the vfio-pci driver.

HTH,

Petr

po 11. 7. 2022 v 15:21 odesílatel Greg Waines <greg....@gmail.com> napsal:

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/8e53cf0a-4f61-4747-a8cd-d4347ae6a00an%40googlegroups.com.

[Greg Waines]

Thanks Petr ... that was the issue.

[Greg Waines]

Thanks Petr !!!

You were absolutely correct ... I had configured the VFs with netdevice.

FYI, in StarlingX's infrastructure management you configure the sriov interface's vf driver as follows:

[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-list compute-1
+--------------------------------------+--------+-----------+----------+---------+-----------------+----------+-------------+------------+
| uuid                                 | name   | class     | type     | vlan id | ports           | uses i/f | used by i/f | attributes |
+--------------------------------------+--------+-----------+----------+---------+-----------------+----------+-------------+------------+
| a1075cd6-8d90-424d-bfe8-138e815b8828 | sriov0 | pci-sriov | ethernet | None    | [u'enp24s0f0']  | []       | []          | MTU=9216   |
| a732ff98-060b-434d-9f92-8dbfb9b8b551 | mgmt0  | platform  | ethernet | None    | [u'enp175s0f0'] | []       | []          | MTU=9216   |
+--------------------------------------+--------+-----------+----------+---------+-----------------+----------+-------------+------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify  compute-0 srov0  -N 64  --vf-driver=vfio

My KubeVirt VM with the CNI interface and the SRIOV interface scheduled and run ... and was able to configure IP Address on SRIOV interface and connect to IP Networks attached to SRIOV interface.

:-)

thanks again,

Greg.

On Monday, July 11, 2022 at 10:54:45 AM UTC-4 phor...@redhat.com wrote:

[Petr Horacek]

Heck yeah! Happy that helped o/

čt 14. 7. 2022 v 8:56 odesílatel Greg Waines <greg....@gmail.com> napsal:

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/ed8f4590-f9bc-42a0-85cb-84e51c75cf54n%40googlegroups.com.