Best way to handle USB Controller passthrough?

[CoRfr]

Hi,

I'm looking at Kubevirt to switch from a homegrown solution to manage QEmu VM instances on few bare metal servers.

The idea behind the infra is that each VM Host has a number of USB PCI controllers.

It's in the form of the 1 or 2 USB controllers that generally come with a motherboard, + extra controllers using extension cards (see https://www.startech.com/Cards-Adapters/USB-3.0/Cards/PCI-Express-USB-3-Card-4-Dedicated-Channels-4-Port~PEXUSB3S44V for instance).

From what I can see from the various device plugins (kubevirt vfio, kubevirt kvm, new GPU feature ...), it seems like there is some clear abstraction in the declaration for a VM or VMI seems to simply state 'I want 1 GPU', which is perfectly understandable since there isn't really any functional differences between 2 GPUs.

In this case however, the USB Controller is the tip of the iceberg. Behind there are few USB devices and, because of that physical mapping, I need to be able to target a specific USB controller on a specific host when creating a VMI.

Here is the output on one of the host:

core@carmd-er-lxtb0cc47a4c27b6 ~ $ lspci -Dnn | grep USB
0000:00:1a.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 [8086:8c2d] (rev 05)
0000:00:1d.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 [8086:8c26] (rev 05)
0000:03:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142]
0000:04:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142]
0000:05:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142]
0000:06:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142]

Using something like `1b21_1142` would not offer much precision. and at this point I'm not sure if this is handled by kubevirt anyway (only SR-IOV and the GPU PR seems to be doing some PCI passthrough).

I'm thinking that the VMI could be declared to have:

spec:
  domain:
    devices:
      disks:
      - disk:
          bus: virtio
        name: containerdisk
      interfaces:
      - masquerade: {}
        name: default
      rng: {}
      pcis:
      - slot: "0000:03:00.0"

It seems like the change would be quite similar to the GPU one ( https://github.com/kubevirt/kubevirt/pull/2597 ), but what I'm also not sure is if I need to develop a new device plugin (or extend the vfio one) or not.

My understanding is that a device plugin tells the cluster 'Device X is available on host Y', so that when we request a device X, the VMI is created on host Y (or some other host Z that also has a Device X).

Here "0000:03:00.0" might be valid on multiple hosts, but in the declaration of the VMI I need to somehow mention that it's only for one particular host.

Is there a way to do that already in the spec? Or should I have something like:
      pcis:
      - slot: "0000:03:00.0"

        node: "carmd-er-lxtb0cc47a4c27b6"

Reusing the kubevirt vfio device plugin would probably help to also allow declarations such as:
      pcis:
      - id: "1b21:1142"

Please let me know what you thing about this. I would like to provide something that is not too business-specific as to be able to upstream the solution.

Thanks,

CoRfr

[Petr Kotas]

Hi CoRfr,

thanks for the proposal. Indeed the idea was discussed before, but was never demanded.

Currently there is no support for this.

Would you mind drafting and issue with proposal? https://github.com/kubevirt/kubevirt/issues

We can than discuss it there.

Thanks a lot for your input!

Kind regards,

Petr

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/3c132789-7f57-4f98-a105-6aeaf557d2f8%40googlegroups.com.

[CoRfr]

Thanks Petr, I opened https://github.com/kubevirt/kubevirt/issues/2686

I'll try to work on it a bit this afternoon so as to come up with something a bit more precise.

Thanks,

Bertrand

On Monday, 9 September 2019 08:27:49 UTC-7, Petr Kotas wrote:

Hi CoRfr,

thanks for the proposal. Indeed the idea was discussed before, but was never demanded.

Currently there is no support for this.

Would you mind drafting and issue with proposal? https://github.com/kubevirt/kubevirt/issues

We can than discuss it there.

Thanks a lot for your input!

Kind regards,

Petr

To unsubscribe from this group and stop receiving emails from it, send an email to kubevi...@googlegroups.com.

[Petr Kotas]

Hi CoRfr,

thanks a lot!

I will review it as soon as possible.

Thanks,

Petr

To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/e3dcd972-ecbe-4293-b639-0fe09b359f4f%40googlegroups.com.