Failing to passthrough an USB3 PCIE card

[David Hobach]

Dear users,

for some reason I cannot get my USB3 PCIE card to passthrough to any
AppVM - even though the card works fine in dom0 and other Linux OSes.

Apparently the xHCI drivers do not load in the AppVM:
dmesg|grep xhci:
[ 2.084058] xhci_hcd 0000:00:00.0: Xen PCI mapped GSI19 to IRQ59
[ 2.084438] xhci_hcd 0000:00:00.0: xHCI Host Controller
[ 2.084618] xhci_hcd 0000:00:00.0: new USB bus registered, assigned
bus number 2
[ 24.713368] xhci_hcd 0000:00:00.0: can't setup: -110
[ 24.713402] xhci_hcd 0000:00:00.0: USB bus 2 deregistered
[ 24.713560] xhci_hcd 0000:00:00.0: init 0000:00:00.0 fail, -110
[ 24.713568] xhci_hcd: probe of 0000:00:00.0 failed with error -110

Plugging in devices works in dom0, but not in any AppVM (nothing
happens, no kernel messages, nothing).

lspci -nn
00:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720201 USB
3.0 Host Controller [1912:0014] (rev 03)

So far I tested this device, another one with the same NEC chipset
(which didn't work in any Linux) and a VLI chipset which did work for
passthrough, but appears to be very instable (a common issue with VLI
chipsets & Linux according to the Internet). So I already produced quite
a pile of junk so far...

Did anyone else have the issue mentioned above and found out how to fix
it? Or does anyone have a recommendation for a PCIE card which he could
pass through to an AppVM?

Actually I also read that USB2 might be a better choice for passthrough,
but getting USB2 PCIE cards is kind of tough nowadays.

Kind Regards
David

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Fri, Jan 29, 2016 at 05:52:20PM +0100, David Hobach wrote:
> Dear users,
>
> for some reason I cannot get my USB3 PCIE card to passthrough to any AppVM -
> even though the card works fine in dom0 and other Linux OSes.
>
> Apparently the xHCI drivers do not load in the AppVM:
> dmesg|grep xhci:
> [ 2.084058] xhci_hcd 0000:00:00.0: Xen PCI mapped GSI19 to IRQ59
> [ 2.084438] xhci_hcd 0000:00:00.0: xHCI Host Controller
> [ 2.084618] xhci_hcd 0000:00:00.0: new USB bus registered, assigned bus
> number 2
> [ 24.713368] xhci_hcd 0000:00:00.0: can't setup: -110
> [ 24.713402] xhci_hcd 0000:00:00.0: USB bus 2 deregistered
> [ 24.713560] xhci_hcd 0000:00:00.0: init 0000:00:00.0 fail, -110
> [ 24.713568] xhci_hcd: probe of 0000:00:00.0 failed with error -110
>
> Plugging in devices works in dom0, but not in any AppVM (nothing happens, no
> kernel messages, nothing).
>
> lspci -nn
> 00:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720201 USB 3.0
> Host Controller [1912:0014] (rev 03)

I have very similar one:
Renesas Technology Corp. uPD720202 USB 3.0 Host Controller [1912:0015]

Initializing xhci driver looks like this:
[ 2.172267] pcifront pci-0: Installing PCI frontend
[ 2.172832] pcifront pci-0: Creating PCI Frontend Bus 0000:00
[ 2.172905] pcifront pci-0: PCI host bridge to bus 0000:00
[ 2.172914] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]
[ 2.172922] pci_bus 0000:00: root bus resource [mem 0x00000000-0xfffffffff]
[ 2.172931] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 2.173458] pci 0000:00:00.0: [1912:0015] type 00 class 0x0c0330
[ 2.174210] pci 0000:00:00.0: reg 0x10: [mem 0xe2200000-0xe2201fff 64bit]
[ 2.205378] pcifront pci-0: claiming resource 0000:00:00.0/0
[ 2.205771] pci 0000:00:00.0: Xen PCI mapped GSI18 to IRQ51
[ 2.317124] xen:events: xen_bind_pirq_gsi_to_irq: returning irq 51 for gsi 18
[ 2.317141] xhci_hcd 0000:00:00.0: Xen PCI mapped GSI18 to IRQ51
[ 2.317285] xhci_hcd 0000:00:00.0: xHCI Host Controller
[ 2.317704] xhci_hcd 0000:00:00.0: new USB bus registered, assigned bus number 2
[ 2.323395] xhci_hcd 0000:00:00.0: hcc params 0x014051cf hci version 0x100 quirks 0x00000090
[ 2.324979] xen:events: Failed to obtain physical IRQ 56
[ 2.325003] xen:events: Failed to obtain physical IRQ 57
[ 2.325029] xen:events: Failed to obtain physical IRQ 58
[ 2.325042] xen:events: Failed to obtain physical IRQ 59
[ 2.325053] xen:events: Failed to obtain physical IRQ 60
[ 2.325207] usb usb2: New USB device found, idVendor=1d6b, idProduct=0002
[ 2.325218] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 2.325228] usb usb2: Product: xHCI Host Controller
[ 2.325234] usb usb2: Manufacturer: Linux 4.1.13-8.pvops.qubes.x86_64 xhci-hcd
[ 2.325243] usb usb2: SerialNumber: 0000:00:00.0
[ 2.325472] hub 2-0:1.0: USB hub found
[ 2.325544] hub 2-0:1.0: 2 ports detected
[ 2.325702] xhci_hcd 0000:00:00.0: xHCI Host Controller
[ 2.325846] xhci_hcd 0000:00:00.0: new USB bus registered, assigned bus number 3
[ 2.326603] usb usb3: We don't know the algorithms for LPM for this host, disabling LPM.
[ 2.326655] usb usb3: New USB device found, idVendor=1d6b, idProduct=0003
[ 2.326663] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 2.326671] usb usb3: Product: xHCI Host Controller
[ 2.326677] usb usb3: Manufacturer: Linux 4.1.13-8.pvops.qubes.x86_64 xhci-hcd
[ 2.326685] usb usb3: SerialNumber: 0000:00:00.0
[ 2.326943] hub 3-0:1.0: USB hub found
[ 2.327033] hub 3-0:1.0: 2 ports detected


Then it looks exactly as you've described - plugging in USB devices have
no effect. And the same device works just fine in dom0.
It may have something to do with those "Failed to obtain physical IRQ"
messages. But unfortunately there is no more details about it. According
to kernel sources, the actual error code is not logged anywhere...

> So far I tested this device, another one with the same NEC chipset (which
> didn't work in any Linux) and a VLI chipset which did work for passthrough,
> but appears to be very instable (a common issue with VLI chipsets & Linux
> according to the Internet). So I already produced quite a pile of junk so
> far...

Same here...

One of them is:
00:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042
SuperSpeed USB Host Controller [1b21:1042]

And AFAIR it was working with passthrough in R3.0 (Xen 4.4). It doesn't
work in R3.1 though (Xen 4.6). With some other error:
[ 2.349043] BUG: unable to handle kernel paging request at
ffffc90000591028
[ 2.349058] IP: [<ffffffffa00abcb9>] xhci_mem_init+0x5e9/0xd10
[xhci_hcd]

It may be some regression in Xen...

On the other hand, I can confirm that Intel integrated USB 3.0
controller works just fine with passthrough. Of course if you have one...

> Did anyone else have the issue mentioned above and found out how to fix it?
> Or does anyone have a recommendation for a PCIE card which he could pass
> through to an AppVM?
>
> Actually I also read that USB2 might be a better choice for passthrough, but
> getting USB2 PCIE cards is kind of tough nowadays.
>
> Kind Regards
> David

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJWq+0uAAoJENuP0xzK19csPc4H/jKZq7SjFe67pSBsH6I+9FgJ
nOIVUNUO9QX3leb5yRCDTwAT+qe2VLvaoqGQN/Kr+xYpPtW8B2X7UiT9Qb4J0pOI
UDIxpNV4VFqV/hILyP0quSAiesQ5jj5RZ2sMr0vvp0eT2NraQ0xDgC9EVbEtGGY+
ISDmGZv0YJj2uv0QUaPBjonhMFkj6Di58jT7O8j1hbeG2qBgb6yftUQBMElOwkGZ
7XiadCgxXKC/tLC+51qpg9O4VJGLci/Ktq1lxHF0ac15pz1m4M84nxoO0wZl+zMJ
6s3Oodmx956kL46UEnCbPIxLZBCUygOfXs5zGDBOmncOz8JzvHZaRdjBSYPjfYI=
=slY/
-----END PGP SIGNATURE-----

[Eric Shelton]

Not only the Xen version, but also the kernel version may be a factor in this.

FWIW, this bears some similarity (superficially) to the problems I experienced with doing passthrough of network adapters to sys-net when trying to run Linux 4.4.

[    2.780563] BUG: unable to handle kernel paging request at 0000000056704b47

[    2.780581] IP: [<ffffffff811fa270>] get_partial_node.isra.62+0x30/0x220

(full dmesg output attached)

Perhaps Xen's kernel code has a regression triggered by PCI passthrough that is growing worse over time?  Two different network adapters ran into the same problem doing passthrough on 4.4 kernels.  I don't know how well their regression tests cover PCI passthrough.

The "xen:events: Failed to obtain physical IRQ" message came up here on xen-devel (but with 3.16 kernel and Xen 4.4.1):

http://lists.xen.org/archives/html/xen-devel/2016-01/msg03553.html

Another possible thread of interest (failed passthrough of xhci to HVM domain):

http://lists.xen.org/archives/html/xen-devel/2015-06/msg03923.html

Eric

[David Hobach]

Just tried again after a fresh reboot and didn't get the error messages
described by me above, but something quite similar to your logs:

[ 1.807595] systemd[1]: Started Journal Service.
[ 1.808617] parport_pc parport_pc.956: Unable to set coherent dma
mask: disabling DMA
[ 1.808658] parport_pc parport_pc.888: Unable to set coherent dma
mask: disabling DMA
[ 1.808690] parport_pc parport_pc.632: Unable to set coherent dma
mask: disabling DMA
[ 1.810546] fuse init (API version 7.23)
[ 1.816370] dummy_hcd dummy_hcd.0: USB Host+Gadget Emulator, driver
02 May 2005
[ 1.816379] dummy_hcd dummy_hcd.0: Dummy host controller
[ 1.816416] dummy_hcd dummy_hcd.0: new USB bus registered, assigned
bus number 1
[ 1.816499] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[ 1.816504] usb usb1: New USB device strings: Mfr=3, Product=2,
SerialNumber=1
[ 1.816508] usb usb1: Product: Dummy host controller
[ 1.816511] usb usb1: Manufacturer: Linux 4.1.13-8.pvops.qubes.x86_64
dummy_hcd
[ 1.816515] usb usb1: SerialNumber: dummy_hcd.0
[ 1.816611] hub 1-0:1.0: USB hub found
[ 1.816638] hub 1-0:1.0: 1 port detected
[ 1.827589] systemd-udevd[186]: starting version 215
[ 1.859392] input: PC Speaker as /devices/platform/pcspkr/input/input0
[ 1.862857] pcifront pci-0: Installing PCI frontend
[ 1.863373] pcifront pci-0: Creating PCI Frontend Bus 0000:00
[ 1.863401] pcifront pci-0: PCI host bridge to bus 0000:00
[ 1.863405] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]
[ 1.863409] pci_bus 0000:00: root bus resource [mem
0x00000000-0x7fffffffff]
[ 1.863412] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 1.863544] pci 0000:00:00.0: [1912:0014] type 00 class 0x0c0330
[ 1.863748] pci 0000:00:00.0: reg 0x10: [mem 0xeb800000-0xeb801fff 64bit]
[ 1.947408] pcifront pci-0: claiming resource 0000:00:00.0/0
[ 1.947481] pci 0000:00:00.0: Xen PCI mapped GSI19 to IRQ51
[ 1.952597] EXT4-fs (dm-0): re-mounted. Opts: (null)
[ 1.961625] xen_netfront: Initialising Xen virtual ethernet driver
[ 1.989800] alg: No test for crc32 (crc32-pclmul)
[ 1.995190] intel_rapl: Found RAPL domain package
[ 1.995212] intel_rapl: Found RAPL domain core
[ 1.995222] intel_rapl: Found RAPL domain uncore
[ 1.995231] intel_rapl: Found RAPL domain dram
[ 2.005269] Error: Driver 'pcspkr' is already registered, aborting...
[ 2.006800] xen:events: xen_bind_pirq_gsi_to_irq: returning irq 51
for gsi 19
[ 2.006807] xhci_hcd 0000:00:00.0: Xen PCI mapped GSI19 to IRQ51
[ 2.006899] xhci_hcd 0000:00:00.0: xHCI Host Controller
[ 2.006949] xhci_hcd 0000:00:00.0: new USB bus registered, assigned
bus number 2
[ 2.012355] xhci_hcd 0000:00:00.0: hcc params 0x014051cf hci version
0x100 quirks 0x00000010
[ 2.012721] xen:events: Failed to obtain physical IRQ 60
[ 2.012729] xen:events: Failed to obtain physical IRQ 61
[ 2.012734] xen:events: Failed to obtain physical IRQ 62
[ 2.012739] xen:events: Failed to obtain physical IRQ 63
[ 2.012743] xen:events: Failed to obtain physical IRQ 64
[ 2.012790] usb usb2: New USB device found, idVendor=1d6b, idProduct=0002
[ 2.012794] usb usb2: New USB device strings: Mfr=3, Product=2,
SerialNumber=1
[ 2.012798] usb usb2: Product: xHCI Host Controller
[ 2.012800] usb usb2: Manufacturer: Linux 4.1.13-8.pvops.qubes.x86_64
xhci-hcd
[ 2.012804] usb usb2: SerialNumber: 0000:00:00.0
[ 2.012883] hub 2-0:1.0: USB hub found
[ 2.012895] hub 2-0:1.0: 4 ports detected
[ 2.012973] xhci_hcd 0000:00:00.0: xHCI Host Controller
[ 2.013018] xhci_hcd 0000:00:00.0: new USB bus registered, assigned
bus number 3
[ 2.015926] usb usb3: We don't know the algorithms for LPM for this
host, disabling LPM.
[ 2.015943] usb usb3: New USB device found, idVendor=1d6b, idProduct=0003
[ 2.015955] usb usb3: New USB device strings: Mfr=3, Product=2,
SerialNumber=1
[ 2.015959] usb usb3: Product: xHCI Host Controller
[ 2.015961] usb usb3: Manufacturer: Linux 4.1.13-8.pvops.qubes.x86_64
xhci-hcd
[ 2.015965] usb usb3: SerialNumber: 0000:00:00.0
[ 2.016046] hub 3-0:1.0: USB hub found
[ 2.016058] hub 3-0:1.0: 4 ports detected

As before, plugging in devices has absolutely no effect.

I will try your hint wrt the Intel USB controller; I just tried to avoid
passing it through so far because I require more ports for dom0 than for
domX and the PCIE card only has two external ports... I guess I'll go
for an external USB hub if it works...

This appears to be a Xen issue as the card works in dom0 and I have
4.1.13-8 both in domX as well as in dom0.

I agree however that the kernel may be an issue as well (for the VLI
chipset which started to become shaky after the latest Qubes upgrade I
guess it was that as it did work in domX somehow, but just failed
randomly during operation until the next restart; admittedly I never
tested it in dom0 though).

[David Hobach]

> I will try your hint wrt the Intel USB controller; I just tried to avoid
> passing it through so far because I require more ports for dom0 than for
> domX and the PCIE card only has two external ports... I guess I'll go
> for an external USB hub if it works...

Tried it and didn't work unfortunately as the PCIE card is only
recognized after the kernel is loaded (+ I like to be able to use my
keyboard in the BIOS or GRUB...). Moreover I cannot unbind the
controller; I guess that could be fixed somehow, but the first reason
already made me stop invest further time.

Considering USB 2 now and hoping for better support with ehci... If
anyone has experience with PCIE USB 2 card passthrough that would be
nice to hear!

[Eric Shelton]

On Saturday, January 30, 2016 at 6:37:07 AM UTC-5, David Hobach wrote:

> I will try your hint wrt the Intel USB controller; I just tried to avoid
> passing it through so far because I require more ports for dom0 than for
> domX and the PCIE card only has two external ports... I guess I'll go
> for an external USB hub if it works...

Tried it and didn't work unfortunately as the PCIE card is only
recognized after the kernel is loaded (+ I like to be able to use my
keyboard in the BIOS or GRUB...). Moreover I cannot unbind the
controller; I guess that could be fixed somehow, but the first reason
already made me stop invest further time.

 

Usually this type of problem can be solved by making sure the appropriate driver(s) are in the initramfs.  Qubes 3.1 rc1 had an issue with not all of the xhci drivers being in the initramfs - I don't know if this issue persists in rc2.  That particular issue is addressed by running "dracut -f -d xhci-pci".

See:

https://groups.google.com/d/msg/qubes-users/rzSPpEBSZTI/7mNAXPl7CQAJ

https://groups.google.com/d/msg/qubes-users/s3lUsfM-ha0/tFUjAphljQMJ

If there are other kernel modules involved in getting your USB3 adapter running, you can identify them by seeing what drivers dom0 loads when using the adapter, and using a similar "dracu -f -d" command.

Hope that helps,

Eric

[Eric Shelton]

The analysis is not quite that straightforward, because running a device through PCI passthrough is a significantly different environment from the typical non-passthrough environment of dom0.  There are certain Linux device drivers that may not be robust enough to work in the passthrough environment.  See:

https://groups.google.com/d/msg/qubes-devel/YqdhlmYtMY0/1XhSKLrFBgAJ

https://groups.google.com/d/msg/qubes-devel/YqdhlmYtMY0/TleqJXJRCAAJ

I mentioned the kernel as a possibility, because I have seen passthrough break simply by changing from a pre-4.4 kernel to the 4.4 kernel.  However, it is quite likely that it is changes made to both the hypervisor and the kernel that are working in concert to cause these issues.

It seems like a lot of changes to PCI passthrough - both hypervisor and kernel side - made in the last year were in connection with issues like XSAs 120 and 157.  This may be connected to why Xen 4.4 may work better than 4.6.

 

Eric

[Vít Šesták]

Hello,

On Saturday, January 30, 2016 at 4:16:11 PM UTC+1, Eric Shelton wrote:

The analysis is not quite that straightforward, because running a device through PCI passthrough is a significantly different environment from the typical non-passthrough environment of dom0.  There are certain Linux device drivers that may not be robust enough to work in the passthrough environment.

Good point. Considering how PV works, this sounds plausible. This also brings some ideas:
* For a VT-d (IOMMU) devices, these problems should not be present on HVMs.
* In long term, switching to PVH might be a solution provided that VT-d will be on virtually all relevant devices.

Anyone with similar issues:
* Do you have VT-d/IOMMU?
* If yes, does it work on HVM?

I have similar issues, but I don't have VT-d/IOMMU support. USB 3.0 used to work, but does not work anymore (Qubes 3.0). USB 2.0 on the same system works.

I mentioned the kernel as a possibility, because I have seen passthrough break simply by changing from a pre-4.4 kernel to the 4.4 kernel.

Hmm, do you mean the dom0 kernel, or domU kernel? I've tries downgrading domU kernel to anything installed, but without any positive result. I haven't tried downgrading the dom0 kernel, as there might be more security considerations.

It seems like a lot of changes to PCI passthrough - both hypervisor and kernel side - made in the last year were in connection with issues like XSAs 120 and 157.  This may be connected to why Xen 4.4 may work better than 4.6.

So, 3.1 might work worse than 3.0…

Regards,
Vít Šesták 'v6ak'

[Eric Shelton]

On Saturday, January 30, 2016 at 12:31:08 PM UTC-5, Vít Šesták wrote:

Hello,

On Saturday, January 30, 2016 at 4:16:11 PM UTC+1, Eric Shelton wrote:

The analysis is not quite that straightforward, because running a device through PCI passthrough is a significantly different environment from the typical non-passthrough environment of dom0.  There are certain Linux device drivers that may not be robust enough to work in the passthrough environment.

Good point. Considering how PV works, this sounds plausible. This also brings some ideas:
* For a VT-d (IOMMU) devices, these problems should not be present on HVMs.

The main issue for drivers and passthrough devices is that Xen restricts access to certain PCI config registers, and some drivers may expect or even require access to those registers.  I don't think that issue is any different for HVM domains.

 

* In long term, switching to PVH might be a solution provided that VT-d will be on virtually all relevant devices.

Joanna suggested a switch to PVH for other reasons - reducing hypervisor complexity by taking advantage of EPT (https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qsb-022-2015.txt):

This bug might also be considered an argument for the view of ditching

of para-virtualized (PV) VMs, and switch to HVMs, or better yet: PVH

VMs for better isolation. This seems to be a valid argument indeed,

but only if the underlying processor also supports SLAT (e.g. EPT).

Otherwise the complexity of the hypervisor code needed to implement

Shadow Paging offsets any potential benefits of using CPU-assisted

virtualization, such as VT-x. Luckily it seems majority of the recent

modern laptops does support SLAT, so this might be the direction we go

for Qubes 4.

 

Anyone with similar issues:
* Do you have VT-d/IOMMU?
* If yes, does it work on HVM?

PCI passthrough is working with HVM domains since Qubes 3.0.

 

I mentioned the kernel as a possibility, because I have seen passthrough break simply by changing from a pre-4.4 kernel to the 4.4 kernel.

Hmm, do you mean the dom0 kernel, or domU kernel? I've tries downgrading domU kernel to anything installed, but without any positive result. I haven't tried downgrading the dom0 kernel, as there might be more security considerations.

I changed both, but the problem specifically manifests in the sys-net domU - it is unable to successfully load kernel modules for network adapters.

 

It seems like a lot of changes to PCI passthrough - both hypervisor and kernel side - made in the last year were in connection with issues like XSAs 120 and 157.  This may be connected to why Xen 4.4 may work better than 4.6.

So, 3.1 might work worse than 3.0…

Some reports suggest this, but we don't really seem to have enough information to understand just what the problems with PCI passthrough are, when they started, or what the likely causes are.  For example, do these same things show up in a vanilla Xen setup, or is it limited to Qubes for some reason?  Is it possible to substitute Xen 4.4 for 4.6 on a Qubes 3.1 RC2 install to see if there is an improvement?

Eric

[Eric Shelton]

On Saturday, January 30, 2016 at 4:30:00 PM UTC-5, Eric Shelton wrote:

On Saturday, January 30, 2016 at 12:31:08 PM UTC-5, Vít Šesták wrote:

Hello,

On Saturday, January 30, 2016 at 4:16:11 PM UTC+1, Eric Shelton wrote:

The analysis is not quite that straightforward, because running a device through PCI passthrough is a significantly different environment from the typical non-passthrough environment of dom0.  There are certain Linux device drivers that may not be robust enough to work in the passthrough environment.

Good point. Considering how PV works, this sounds plausible. This also brings some ideas:
* For a VT-d (IOMMU) devices, these problems should not be present on HVMs.

The main issue for drivers and passthrough devices is that Xen restricts access to certain PCI config registers, and some drivers may expect or even require access to those registers.  I don't think that issue is any different for HVM domains.

 

* In long term, switching to PVH might be a solution provided that VT-d will be on virtually all relevant devices.

Joanna suggested a switch to PVH for other reasons - reducing hypervisor complexity by taking advantage of EPT (https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qsb-022-2015.txt):

This bug might also be considered an argument for the view of ditching

of para-virtualized (PV) VMs, and switch to HVMs, or better yet: PVH

VMs for better isolation. This seems to be a valid argument indeed,

but only if the underlying processor also supports SLAT (e.g. EPT).

Otherwise the complexity of the hypervisor code needed to implement

Shadow Paging offsets any potential benefits of using CPU-assisted

virtualization, such as VT-x. Luckily it seems majority of the recent

modern laptops does support SLAT, so this might be the direction we go

for Qubes 4.

 

Anyone with similar issues:
* Do you have VT-d/IOMMU?
* If yes, does it work on HVM?

PCI passthrough is working with HVM domains since Qubes 3.0.

That should be "has _not_ been working".

Eric

[Vít Šesták]

On Saturday, January 30, 2016 at 10:30:00 PM UTC+1, Eric Shelton wrote:

The main issue for drivers and passthrough devices is that Xen restricts access to certain PCI config registers, and some drivers may expect or even require access to those registers.  I don't think that issue is any different for HVM domains.

Hmm, I expected HVMs to be as close to physical HW as possible. But there might be some exception.

Joanna suggested a switch to PVH for other reasons - reducing hypervisor complexity by taking advantage of EPT (https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qsb-022-2015.txt):

Yes, I remember this :)
 

Anyone with similar issues:
* Do you have VT-d/IOMMU?
* If yes, does it work on HVM?

PCI passthrough is working with HVM domains since Qubes 3.0.

I am not sure if we understand each other there:
* PCI passthrough on HVM is working, but requires VT-d/IOMMU.
* My question was if the bug with PCI passthrough of some USB controllers is also present in HVM on one's system. (I can't test is, because I don't have VT-d.)

Regards,
Vít Šesták 'v6ak'

[Eric Shelton]

No, I just mistyped my initial response (and corrected it with a followup post a minute later).

HVM PCI passthrough has been broken since Qubes 3.0, whether you have an IOMMU or not.  It is a known issue:

https://github.com/QubesOS/qubes-issues/issues/1659

Eric

[Vít Šesták]

Aha, got it. :(

[David Hobach]

> Considering USB 2 now and hoping for better support with ehci... If
> anyone has experience with PCIE USB 2 card passthrough that would be
> nice to hear!

Just tested an USB2 PCIE card and can happily report that it appears to
be working perfectly for passthrough.

For anyone interested: It's an Exsys EX-11064 with a NEC D72010GJ chipset.
lspci -nn
00:00.0 USB controller [0c03]: NEC Corporation OHCI USB Controller
[1033:0035] (rev 43)
00:00.1 USB controller [0c03]: NEC Corporation OHCI USB Controller
[1033:0035] (rev 43)
00:00.2 USB controller [0c03]: NEC Corporation uPD72010x USB 2.0
Controller [1033:00e0] (rev 04)

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

I've uploaded kernel-4.1.13-8.3 to qubes-dom0-unstable repo (for R3.1).
It have applied patches sent by Konrad on xen-devel in this thread:
http://markmail.org/message/dxiyuob24e7mp3lt
devel-4.1 branch in my qubes-linux-kernel repo on github.

Can you test it out? It should fix USB 3.0 passthrough issue
("xen:events: Failed to obtain physical IRQ" error, not sure about the
other cases).

BTW Qubes ticket for reference:
https://github.com/QubesOS/qubes-issues/issues/1734

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJWug+BAAoJENuP0xzK19csWmsH/ijbPk9eg/CKKimO+DvnTp2r
wQkPvvj4Qh4OldN7TVvGtEQW4fUK+/MoTbKoPDkDu1ShKt3X1uLyE2QL3ki0dy63
hN1i4uBe3nc3L0OxZy7YZGRUEjmmFz3R7i1UtWixjNIq5WJ2nhQVsxGtZo27NCC5
D1fUYTGtrC9UtNPMQp/Zut1rC9UozhDteknxsUIWO+gXUnYZXAQnc8KKDDqI4H3u
3r8rNgqqnm05q2hUzwuREW3IFigAeAyf6p0IxgaPo/y3KD057cdTSVvl0PUbBVQD
pVGLPppWbH3RBQAu4cRphrl1ahHadWvP7PUHJLf02rycgDKs9ZseeXWjUqRNf/M=
=Aps2
-----END PGP SIGNATURE-----

[David Hobach]

> I've uploaded kernel-4.1.13-8.3 to qubes-dom0-unstable repo (for R3.1).
> It have applied patches sent by Konrad on xen-devel in this thread:
> http://markmail.org/message/dxiyuob24e7mp3lt
> devel-4.1 branch in my qubes-linux-kernel repo on github.
>
> Can you test it out? It should fix USB 3.0 passthrough issue
> ("xen:events: Failed to obtain physical IRQ" error, not sure about the
> other cases).
>
> BTW Qubes ticket for reference:
> https://github.com/QubesOS/qubes-issues/issues/1734

Hi Marek,

thanks for your efforts!

Unfortunately I sent the unusable hardware back to get a refund and went
for the USB 2 PCIE card solution, i.e. I cannot test it anymore, sorry.
I only have one USB 3 PCIE card left, which didn't work half the time
for passthrough due to apparently different issues.

Maybe someone else read this thread, has similar issues and is willing
to provide feedback?

Kind Regards
David

[zhong...@gmail.com]

Disabling MSI (Message Signaled Interrupts) for the usb 3.0 card specifically may solve your issue.