MSI-x support in domU

[perm...@gmail.com]

I have a Mellanox 10G adapter in my Qubes box.
When I spin up a domU with the device passed-thru, I am unable to load the device driver due to what appears to be a lack of support for MSI-x interrupt mapping.

There was some chatter a while back regarding MSI support:

https://www.google.com/url?q=https%3A%2F%2Fwww.qubes-os.org%2Fnews%2F2017%2F10%2F18%2Fmsi-support%2F&sa=D&sntz=1&usg=AFQjCNGXTUp6QSX9Fb0v5Q6hyVc0i6NwfQ

One might hope that this would enable MSI-x support as well, but apparently, not. My guess is that someone added a kluge to permit writing to the MSI enable regions in PCI config space for a device, neglecting to add the kluges for MSI-x as well.

I find that I need to flag the mellanox as 'permissive' or else I get errors flagged on the dom0 console log regarding attempts to write to PCI config space. Once I do so, I am still unable to load the device driver for the ConnectX 4-LX device (mlx5_core). It fails when it attempts to allocate the set of MSIX vectors sized based on the # of CPUs online.

The driver assumes MSIx are available. No fall-back to MSI. No fall-back to INT A/ INT B.

Are there some other magic knobs I need to tweak?

MSIx and Xen does raise some interesting issues. I would like to have to option of spinning up a domU with, say, 20 VCPU and, knowing that the Mellanox will assign queues to MSIx and I can assign MSIx to CPUs, I would like to have dom0 bind the vCPU to real CPU so that the interrupt mapping works correctly. This would be for some network performance work I have to do occasionally.

I am also keen to enable the VF devices in the adapter (using some domU instance to enable) so that these VF instances can be passed to other domU instances. Also want to see if I can get hardware offload working and OVS working in qubes. Just for fun.

Q: if a domU kernel enables VF devices in a mapped PF device instance, will the dom0 kernel discover the VF devices? IE: what is the mechanism whereby a kernel discovers the need for a bus-walk?
This has to work correctly for Xen, no?

[awokd]

I had somewhat similar weirdness with an old Atheros AR9280- see
https://github.com/QubesOS/qubes-issues/issues/3609. It initialized
properly in a stock Xen HVM, but the driver would crash on a Qubes HVM.
Never did manage to figure it out. Mine could have been memory mapping
related, though.

[Tai...@gmx.com]

On 08/07/2018 06:41 PM, perm...@gmail.com wrote:

> Q: if a domU kernel enables VF devices in a mapped PF device instance, will the dom0 kernel discover the VF devices? IE: what is the mechanism whereby a kernel discovers the need for a bus-walk?
> This has to work correctly for Xen, no?

What do you mean by bus walk?

SR-IOV must be enabled in dom0 (but doesn't require networking packages
just enabling on the device to create the VF's)

I would of course look at a xen guide and tweak for your pleasure but
SR-IOV with more than one VM assigned to the same port will decrease
security as would using a multi port networking asic as opposed to one
with a separate chip per port.

Of course every device must have separate IOMMU groups including each VF.

I think SR-IOV is one of the coolest technologies out there and would
love to find out how to get it working for my LSI HBA's as well but I
would like to note there is only one way that it can improve qubes
security and that is by the restricted VF device parameters and also
thus not being able to perform hostile firmware updates from the guest etc.

Do you already have the nics? FYI you definitely don't want to buy a
first gen SR-IOV NIC as they have problems.