USB passthrough - fvio and iommu groups

[Tyler Brandon]

Hi,

KubeVirt's vfio implementation is pretty amazing. 

In the case where a USB controller, identified by the PCI bus and pciVendorSelector, is on an iommu group of its own, it can be passed into a Windows VM using the pciVendorSelector.  However, in the situation where a machine has the USB controller in a shared iommu group, it is not possible to pass in the USB controller, unless all the other PCI devices in the iommu group are unbound and rebound to a vfio driver. Which is not an option when those PCI devices are integral to the operation of the host system.

The use case is on the edge where containers co-exist with VMs and need to access external peripherals (hot plug in some cases).

How should this type of situation be handled in KubeVirt?  Maybe vfio is not the way to go in this situation?  Should we be looking at a different method? 

Thank you.

Tyler

[Victor Toso]

Hi Tyler,

I've been working on USB redirection for virtctl [0] and I was
wondering if that would fit some of your use cases or you need
USB devices plugged in the actual nodes for some reason?

The way that [0] works for virtctl can likely be extended as
virt-handler and virtctl are acting basically as proxies between
QEMU and usbredirect from usbredir library [1].

Note that this kind of USB redirection works seamless to Windows
and Linux guests which I have tested so far.

[0] https://github.com/kubevirt/kubevirt/pull/4089#
[1] https://gitlab.freedesktop.org/spice/usbredir


> Thank you.
> Tyler

Cheers,
Victor

[Tyler Brandon]

Hi Victor,

Thank you for the suggestion.  It appears usbredir supports individual device redirection.  In our use case there will be multiple devices and the USB device vendor and product IDs are not known ahead of time.  We can control what USB port(s) on the host machine devices are plugged into.

Thank you.

Tyler

[Victor Toso]

Hi Tyler,

Just to clarify, my end goal is to see if there is an opportunity
to extend what I have been working on to cover more interesting
use cases.

On Fri, Mar 12, 2021 at 06:17:30AM -0800, 'Tyler Brandon' via kubevirt-dev wrote:
> Hi Victor,
>
> Thank you for the suggestion. It appears usbredir supports
> individual device redirection.

Not really, it depends on how many usb devices configured for usb
redirection you have in your VM. In the proposal, I've limited to
4 but if we consider host passthrough, this can be increased too.

> In our use case there will be multiple devices and the USB
> device vendor and product IDs are not known ahead of time.

Not knowing vendor-product is fine. We can add a redirection on
hotplug, for instance.

> We can control what USB port(s) on the host machine devices are
> plugged into.

If using the host's port is a must, I wonder if someone could
bring some light in a possible design. I'd be happy to work on
it.

(1) I know that QEMU can already be configured to do that itself. Not
sure that adding this kind of configuration is interesting to
KubeVirt or even possible considering limitations in Pod's
permissions. I'll check.

(2) It can also be possible to e.g: launch usbredirect in the node,
configured to listen to a specific usb port and redirect what is
connected there to the VMI. I'm not sure where such configuration
should be placed and if this is better (design wise) than (1).

Cheers,
Victor

> --
> You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/f4fc7eb4-4041-4978-be44-d1fb7ac6e509n%40googlegroups.com.

[Tyler Brandon]

Hi Victor,

Thank you for the response.  

In regards to usbredir, there is some concern about the reliability of Windows and thus would like to first pursue options that handle the passthrough as close to the hardware as possible.  Your point that QEMU can already passthrough a USB controller will be explored.  A PCIe USB card is also being looked at to confirm the vfio approach will work as expected. 

Thank you.

Tyler