Downgrade Xen / switch to KVM? (for GPU passthrough experimentation)

[Marcus at WetwareLabs]

Hello everyone!

What would be the steps for installing and trying out different Xen versions in Qubes 3.1? Or even switching to KVM? Shouldn't HAL make this possible on Qubes 3.0+ ?

I'm mainly interested in testing Xen 4.3 branch, since there's anecdotal evidence that something might have broken with GPU passthrough between Xen versions 4.4 and 4.3 and I have not seen any success stories of passthrough after 4.3. 

http://www.gossamer-threads.com/lists/xen/users/349649

https://lime-technology.com/forum/index.php?topic=36101.0

https://www.linuxserver.io/index.php/2013/09/12/xen-4-3-windows-8-with-vga-passthrough-on-arch-linux/

Only exception is here:

https://groups.google.com/forum/#!topic/qubes-users/cmPRMOkxkdA

with Qubes 3.0RC2 but he seems to be using AMD GPU & CPU whereas I'm with Intel and Nvidia.

Personally I've been trying to get the GPU passthrough (as secondary GPU) working for the past two weeks now, without luck. It's always the same result: Windows BSODs during the first boot after driver installation. I've tried Windows  7 Pro SP1 and Windows 8.1 and both act the same way. I know it's not a hardware problem, since GPU passthrough using KVM on Arch Linux works without hiccup. Also the same BSOD happens with Xen on Arch Linux, so I also know that it's not restricted to just Qubes. Also it's not about the well-known problem of "BSOD after 2nd boot", since with KVM I could boot DomU many times flawlessly without any requirement to boot Dom0 (to reset the GPU as well).

I've tried out these OS's with stock Xen versions:

Arch Linux, Xen 4.6.1: BSOD on DomU boot

Qubes 3.1, Xen 4.6.0: BSOD on DomU boot

Qubes 3.0 RC 2, Xen 4.4.2: BSOD on DomU boot

Qubes 2.0, Xen 4.1.6:  Sadly BSOD on DomU boot here as well..  

My current HW is:

Intel I7-5820K

Asrock X99 WS

EVGA GTX 980 (passthrough GPU)

Asus Radeon R5 230 (dom0 GPU)

I've tried also Radeon as passthrough GPU on Xen 4.6.0 with many driver versions (win 7 pro), but with same results.

I would be very interested hearing what kind of results others have achieved!

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Thu, Jun 09, 2016 at 06:20:51AM -0700, Marcus at WetwareLabs wrote:
> Hello everyone!
>
> What would be the steps for installing and trying out different Xen
> versions in Qubes 3.1? Or even switching to KVM? Shouldn't HAL make this
> possible on Qubes 3.0+ ?

As for KVM - it isn't that simple - requires writing few components -
namely vchan library and a little modification to gui-daemon (currently
uses Xen-specific feature to map memory pages from VM).

> I'm mainly interested in testing Xen 4.3 branch,

Take a look at my github account, there is xen-4.4 branch, it should be
quite simple to get xen 4.3 from there. When you build such package in
Qubes 3.1 environment it should just work, at least in theory. But see
below.

> since there's anecdotal
> evidence that something might have broken with GPU passthrough between Xen
> versions 4.4 and 4.3 and I have not seen any success stories of passthrough
> after 4.3.
> http://www.gossamer-threads.com/lists/xen/users/349649
> https://lime-technology.com/forum/index.php?topic=36101.0
> https://www.linuxserver.io/index.php/2013/09/12/xen-4-3-windows-8-with-vga-passthrough-on-arch-linux/
> Only exception is here:
> https://groups.google.com/forum/#!topic/qubes-users/cmPRMOkxkdA
> with Qubes 3.0RC2 but he seems to be using AMD GPU & CPU whereas I'm with
> Intel and Nvidia.
>
> Personally I've been trying to get the GPU passthrough (as secondary GPU)
> working for the past two weeks now, without luck. It's always the same
> result: Windows BSODs during the first boot after driver installation. I've
> tried Windows 7 Pro SP1 and Windows 8.1 and both act the same way. I know

> it's not a hardware problem, since GPU passthrough using *KVM on Arch Linux* *works
> without hiccup*. Also the same BSOD happens with Xen on Arch Linux, so I
> also know that *it's not restricted to just Qubes. *Also it's not about the

> well-known problem of "BSOD after 2nd boot", since with KVM I could boot
> DomU many times flawlessly without any requirement to boot Dom0 (to reset
> the GPU as well).

Do you see PCI device in the VM at all? There is a problem with this in
Qubes 3.0+ :
https://github.com/QubesOS/qubes-issues/issues/1659

And as you can see, there is some progress recently, but it isn't solved
yet.

> I've tried out these OS's with stock Xen versions:
> Arch Linux, Xen 4.6.1: BSOD on DomU boot
> Qubes 3.1, Xen 4.6.0: BSOD on DomU boot
> Qubes 3.0 RC 2, Xen 4.4.2: BSOD on DomU boot
> Qubes 2.0, Xen 4.1.6: Sadly BSOD on DomU boot here as well..

Ok, this is some hint that solving #1659 would not be enough...

> My current HW is:
> Intel I7-5820K
> Asrock X99 WS
> EVGA GTX 980 (passthrough GPU)
> Asus Radeon R5 230 (dom0 GPU)
>
> I've tried also Radeon as passthrough GPU on Xen 4.6.0 with many driver
> versions (win 7 pro), but with same results.
>
> I would be very interested hearing what kind of results others have
> achieved!
>

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJXWXppAAoJENuP0xzK19csjJkIAI479ZpDcVLSE35zrEslIevD
Hsj65Lj15yPfEbU797vciZgGql04yUQwBjaZkgCMyWpPrizr1GSZHuMerRo4dJk7
4DY1DFAjykBxucPdQlL539JDWgO5DdL4bFb4o+zD+rPSNzDwQqOt1LDX36AewRW1
DCfVTsoZbdl+PBxpqByd2QnjehBAaceWg1LC57+4BXTJM8IViZQWOxu+IMnBVHLa
bhGsj+nKJIJxrYcaVMymbQbBMxCGjrsayBBRjVl9txf5q5QwJLKbQ8zA24FOC8HA
TnVYcQNK4NCMUEJKd21PMRFsSb5r7cgU8JPFpwbAjctHfzisrqy65D/xZnVJlIM=
=rFfY
-----END PGP SIGNATURE-----

[marko....@gmail.com]

Marek,

thank you for your prompt reply.

On Thursday, June 9, 2016 at 5:17:22 PM UTC+3, Marek Marczykowski-Górecki wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> On Thu, Jun 09, 2016 at 06:20:51AM -0700, Marcus at WetwareLabs wrote:
> > Hello everyone!
> >
> > What would be the steps for installing and trying out different Xen
> > versions in Qubes 3.1? Or even switching to KVM? Shouldn't HAL make this
> > possible on Qubes 3.0+ ?
>
> As for KVM - it isn't that simple - requires writing few components -
> namely vchan library and a little modification to gui-daemon (currently
> uses Xen-specific feature to map memory pages from VM).
>
> > I'm mainly interested in testing Xen 4.3 branch,
>
> Take a look at my github account, there is xen-4.4 branch, it should be
> quite simple to get xen 4.3 from there. When you build such package in
> Qubes 3.1 environment it should just work, at least in theory. But see
> below.

I haven't done any Qubes development/building before, so this should be interesting :)
You mean setup the dev environment as here
https://www.qubes-os.org/doc/qubes-builder/
and then 3-way merge between your branch and vanilla xen-4.3 & 4.4 branches?

Should also experiment with these FLR related patches in case they are not yet merged:
http://lists.xen.org/archives/html/xen-devel/2014-06/msg03107.html
http://lists.xenproject.org/archives/html/xen-devel/2014-07/msg01111.html
as explained by Radoslaw here:
https://groups.google.com/d/msg/qubes-devel/MfHy2jmXhXM/W30ZbbLoyI8J

Also this old patch from 2010 should be checked:
http://old-list-archives.xen.org/archives/html/xen-devel/2010-05/msg00441.html

I apologize, I should have been more specific. I followed the footsteps from XO..@riseup (link above), so I used only xl tools directly (with qemu-xen-traditional) and not libvirt.
The Windows installation can see the device and Nvidia driver installer can detect the GPU, but after driver installation it either BSODs or refuses to acknowledge the installed driver (Yellow mark and Code 43 in Device Manager).
I've now tested following Nvidia drivers (64bit Win7) under Qubes 2.0, Xen 4.1.6 (xl tools):
From 353.30 to 362.00: Code 43
from 364.72 to newest 368.39: BSOD

So there's a difference here with the drivers as well. According to discussion here
https://lime-technology.com/forum/index.php?topic=38664.0
the Code 43 could mean that Nvidia drivers detect that they are running inside VM and refuse to work (since that functionality is reserved to higher-end Quadro cards). That could be the reason why KVM works but Xen doesn't (on newer KVM versions hypervisor can be hidden more easily, and that's one thing Xen lacks at the moment, as far as I know). I did try to disable "viridian" but that did not have any effect.

Also, I think KVM handles the device reset better than Xen. In addition, the UEFI boot support might also have something to with the better GPU passthrough support on KVM, but my knowledge on that is very vague at the moment. Please correct me if I'm wrong.

Then I tried GPU passthrough to Linux Mint 17.3 64bit VM, but that wouldn't work either (defaulted to emulated Cirrus via VNC). Granted, it doesn't have the most recent kernel (3.19) nor nouveau drivers, so I should install a distro with newest patches and dig more into this. Debugging Linux is a bit more fun than figuring out why a blue screen appears.. :) Oddly, shutting down the Linux Mint VM (with passed through GPU) crashes the dom0 as well, so I should start testing at least with Xen 4.3 if this reappears.

One thing that works at least on Qubes 2.0 and Xen 4.1.6 (xl tools) is normal PCI passthrough. I can pass a USB controller and Win7Pro detects it. USB sticks, a mouse and USB audio device (ODAC) work just fine. No crashes during shutdown, and rebooting VM does not cause any problems.
I remember trying to pass through PCI devices in Qubes 3.1 to Win7 HVM (via libvirt) but that did not seem to work out of the box (devices were not found, I think) and I didn't look further into it at that time. I guess that was related to the #1659 then.

So,

my task list for near future:
- build and install Xen 4.3 for Qubes 3.1
- quick tests if GPU passthrough on Win7 works
- if not, try GPU passthrough with most recent Linux VMs
- try miscellaneous Xen patches

Thank you for your advice!

Best regards,
Marcus

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Thu, Jun 09, 2016 at 03:37:13PM -0700, marko....@gmail.com wrote:
> On Thursday, June 9, 2016 at 5:17:22 PM UTC+3, Marek Marczykowski-Górecki wrote:
> > On Thu, Jun 09, 2016 at 06:20:51AM -0700, Marcus at WetwareLabs wrote:
> > > I'm mainly interested in testing Xen 4.3 branch,
> >
> > Take a look at my github account, there is xen-4.4 branch, it should be
> > quite simple to get xen 4.3 from there. When you build such package in
> > Qubes 3.1 environment it should just work, at least in theory. But see
> > below.
>
> I haven't done any Qubes development/building before, so this should be interesting :)
> You mean setup the dev environment as here
> https://www.qubes-os.org/doc/qubes-builder/

Yes.

> and then 3-way merge between your branch and vanilla xen-4.3 & 4.4 branches?

The repository contains only build scripts and Qubes specific patches,
so shouldn't be that hard. Simply change "version" file and try to build
it. You'll probably need to adjust some of those patches, but it
shouldn't be that hard.

Also, you'll probably need to rebuild components linked with Xen:
- core-libvirt
- core-vchan-xen
- gui-daemon

> Should also experiment with these FLR related patches in case they are not yet merged:
> http://lists.xen.org/archives/html/xen-devel/2014-06/msg03107.html
> http://lists.xenproject.org/archives/html/xen-devel/2014-07/msg01111.html
> as explained by Radoslaw here:
> https://groups.google.com/d/msg/qubes-devel/MfHy2jmXhXM/W30ZbbLoyI8J
>
> Also this old patch from 2010 should be checked:
> http://old-list-archives.xen.org/archives/html/xen-devel/2010-05/msg00441.html

I guess all of them are either merged (so are present in 4.6), or were
rejected for some reason (so you should also think twice before using
them).

> I apologize, I should have been more specific. I followed the footsteps from XO..@riseup (link above), so I used only xl tools directly (with qemu-xen-traditional) and not libvirt.

Ah, I see. That also include running qemu in dom0, right? Keep in mind
that such setup greatly reduce security of the system, as qemu is full
of security bugs (take a look at Xen Security Advisories - most of them,
especially recent ones and those with greatest impact, are about qemu).

> The Windows installation can see the device and Nvidia driver installer can detect the GPU, but after driver installation it either BSODs or refuses to acknowledge the installed driver (Yellow mark and Code 43 in Device Manager).
> I've now tested following Nvidia drivers (64bit Win7) under Qubes 2.0, Xen 4.1.6 (xl tools):
> From 353.30 to 362.00: Code 43
> from 364.72 to newest 368.39: BSOD
>
> So there's a difference here with the drivers as well. According to discussion here
> https://lime-technology.com/forum/index.php?topic=38664.0
> the Code 43 could mean that Nvidia drivers detect that they are running inside VM and refuse to work (since that functionality is reserved to higher-end Quadro cards). That could be the reason why KVM works but Xen doesn't (on newer KVM versions hypervisor can be hidden more easily, and that's one thing Xen lacks at the moment, as far as I know). I did try to disable "viridian" but that did not have any effect.

While "code 43" might mean such a problem, it looks like to be just a
generic error for not working driver:
"Windows has stopped this device because it has reported problems. (Code
43)"

> Also, I think KVM handles the device reset better than Xen. In addition, the UEFI boot support might also have something to with the better GPU passthrough support on KVM, but my knowledge on that is very vague at the moment. Please correct me if I'm wrong.
>
> Then I tried GPU passthrough to Linux Mint 17.3 64bit VM, but that wouldn't work either (defaulted to emulated Cirrus via VNC). Granted, it doesn't have the most recent kernel (3.19) nor nouveau drivers, so I should install a distro with newest patches and dig more into this. Debugging Linux is a bit more fun than figuring out why a blue screen appears.. :) Oddly, shutting down the Linux Mint VM (with passed through GPU) crashes the dom0 as well, so I should start testing at least with Xen 4.3 if this reappears.

Try to collect crash messages, especially in case of dom0 crashes. The
most reliable way would be to attach serial console (on some laptop
systems, it is available on docking station, if you have one).
A situation when VM can induce dom0 crash is a serious issue.

> One thing that works at least on Qubes 2.0 and Xen 4.1.6 (xl tools) is normal PCI passthrough. I can pass a USB controller and Win7Pro detects it. USB sticks, a mouse and USB audio device (ODAC) work just fine. No crashes during shutdown, and rebooting VM does not cause any problems.
> I remember trying to pass through PCI devices in Qubes 3.1 to Win7 HVM (via libvirt) but that did not seem to work out of the box (devices were not found, I think) and I didn't look further into it at that time. I guess that was related to the #1659 then.

Yes, exactly.

> my task list for near future:
> - build and install Xen 4.3 for Qubes 3.1
> - quick tests if GPU passthrough on Win7 works
> - if not, try GPU passthrough with most recent Linux VMs
> - try miscellaneous Xen patches

Looks like a good plan.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJXWfl5AAoJENuP0xzK19cssOgH/ilQzTN4DwkM+668HTKmC5AF
MFPovYb+ZQa3sMWWnbtofyvulU3+0cDbyVn8p0WilpjghLMXxyDRj5fsVHEAg0az
WgDkTmFYfBQrjgSNRo65pXj+DgK0lbw1ibq8qToTXMOdlqBc6VrdIdCpoKzJyGVu
N2vVv/16Womn3zY7t9ZttGTm0RjkF5MhGWEIpNHCMsC1+T1BTvjIufm1ihytOrdZ
2dGFRZgQxiSWIu0FLq4dG4XjJ4lFbOQj/IGzK9rkPoy7I/PQlfWK1Y65SQrwzsZt
iwSluE97EJQ+261uPM5y+Ie4cBG8b+Iuoq/IHyQx/JT5smY0Lye3Vmx2IC71nFg=
=cZBd
-----END PGP SIGNATURE-----

[Marcus at WetwareLabs]

Ok,  so here are the results of the time-consuming process of patching, building, fixing, rebuilding, installing and testing (ad nauseam) of various builds of Qubes to get GPU passthrough working :)

In a nutshell, I have now tried both Win7 Pro and Win 8.1 with these Xen versions:

- 4.6.1

- 4.6.0

- 4.4.4

- 4.4.2

- 4.3.4

- 4.3.2

I started with Nvidia GTX 980, but I had no luck with ANY of the Xen hypervisors or Qubes versions. All hang on Nvidia Catalyst installation (or BSOD during boot, if the driver is first extracted and then manually installed via Device Manager).  Then I tried Arch Linux VM: The text mode frame buffer could find and initialize GTX 980 and survive for few DomU boots, but then it would not work again until Dom0 was booted again. Then I tried Xorg, but the Nvidia open source driver still doesn't seem to support newest GPUs and frankly, at this point I was so tired that I didn't even try the binary blob driver from Nvidia.

Then, I switched the GPU to my old Radeon 6950 (Cayman) and had much better success.

- Xen 4.6.1 & 4.6.0 (Qubes 3.1): 

Win 8.1: Passthrough works by manual driver installation. Catalyst installation hangs when the devices are being detected.  DomU survives multiple boots.

Win 7: Passthrough works, but the whole system is very sluggish. Mouse moves quickly but there is about 1 second lag before windows are redrawn. Not very usable.

Arch Linux: Open Source ATI driver works fine. DomU survives multiple boots.

Curiously, Win 8.1 can be booted multiple times without problem with passthrough, but after Win 7 or Arch Linux is started even once (after Dom0 boot), BSOD (or code 43) occurs when booting Win 8.1. So Win7 and ATI Xorg drivers leave the GPU in such state that Win 8.1 driver cannot overcome this. Dom0 reboot then fixes this so that Win 8.1 can be started again.

Surprisingly, the same occurs with Arch Linux: DomU survives multiple boots, but GPU state is garbled if Win 8.1 (or Win 7) VM has been started even once. Xorg starts, but half of the screen is messed.

- Xen 4.3.4 (Qubes 3.0 patched), 4.4.2, 4.4.4 (Qubes 3.0 vanilla):  Win 8.1 works fine as above. Win 7 works without the sluggishness. Win7 DomU also survives few boots, but hopping between Win 7 and 8.1 causes always BSODs or Code 43 (device could not be initialised), which won’t go away until Dom0 is restarted.

For Win 7 HVMs, the 4.3.4 version seemed to work quite nicely, so if someone wants to experiment, you can find my adaptation for 4.3 -branch for Qubes here: https://github.com/WetwareLabs/qubes-vmm-xen  . Note that this version can be built only on top of Qubes 3.0, since 3.1 has already progressed a lot and quite many Qubes-related patches would have to adapted from Xen 4.6 to 4.3. I’m not saying it isn’t possible, just very time consuming :)

For Win 8.1 HVMs, I suggest using 4.6.0/4.6.1 since they have all the newest security updates. But like Marek said above, there's much more attack surface available for malware and whatnot, since Qemu is run in dom0 when using the qemu-emu-traditional mode. But again, it's up to the user to evaluate the risks and benefits.

I'll write another post later that includes basic tutorial and HCL.

Best regards,

Marcus

[Marcus at WetwareLabs]

Update on the matter of sluggishness of Win 7 on Xen 4.6.1: Disabling MSI translation by setting "pci_msitranslate = 0" in VM config file resolves this. So both Win 7 and 8.1 seem to work fine on newers Qubes OS, and no need thus to mess with Xen 4.3 :)