Gpu passthrough nvidia 4.0

[Mataku]

Hello Qubes users,
I just starting my voyage in thr qubes world and i think it is going to be funny :D. Cannot wait to be become an expert. But for now i am a nooby. So please be kind :).
I get my qubes os on this system :
Motherboard : Asus Rog maximus ix
Cpu : i7 7700k
Ram : 32g
Nvme pci : optane
Gpu : nvidia pascal titan X
Qubes : 4.0 (from yesterday)

My goal is to get my nvidia card passing into my win7new i have create from the tutorial on the qubes website.
When i'm adding the nvidia card on the VM and start it...
Bang! Black screen and reboot.
The devices in Qubes Manager is VGA
compatible controller: NVIDIA Corporation GP102 [TITAN X] (rev a1)

ps : i got the same issue if my main screen come from the gpu of the cpu or the gpu nvidia.

Thank you in advance for your time.

Best regards

[Mataku]

I found this... but...

[Mataku]

Le jeudi 10 janvier 2019 18:55:54 UTC+1, Mataku a écrit :

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

If you're going to assign nvidia GPU to some VM, you can't use it at the
same time in dom0 (to your main X server). The best way to ensure it,
is to unload "nouveau" kernel module in dom0, but if it's already
grabbed by the X server, you may need to add "nouveau.modeset=0" option
to the kernel command line (/boot/efi/EFI/qubes/xen.cfg on EFI system,
/etc/default/grub + grub2-mkconfig -o /boot/grub2/grub.cfg on legacy).

If that doesn't help, you'll need to collect the crash message. It's
either dom0 panic, or Xen panic. For dom0 panic, it may be enough to
switch to text console and start VM from there (qvm-start ...), with a
camera handy. If you're not fast enough, add "noreboot" option to Xen
command line (just after "smt=off").
If that's Xen panic, serial console may be necessary to extract it.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAlw3pC0ACgkQ24/THMrX
1yyAbAf+Jx1jmDYYPiWrUl9Wda0d8O4DrgXJb1N2Cww7770qGVbn5NMydSgbD8NL
C1iyVqcU3Ea2TAuiBZEyay3XKiBgaqM7yFoxDN1QmALmPsNRlX23Id/Gu1Y+s+Nt
KiWiCR61O3dQxOjRVdNPDxYVTwgGQAKe+t6AZjSayj50VR3HzRrlH5luO0P4Az8Y
9cQfS/A1d0nakaptnG3kS1MB+qwB/6+BuscijrupkJQ0fn7CFh1OBehchQO0wPXT
qbiU/Nj5RS7SaV1xg2Y5Ve5hEBNORfi6bXllw/VTE4lS0cYgecwmIUbMV6ZGCK/u
BIUsgIeu+w3+l2CPI4VaKb3Y6sEQZQ==
=g4Ff
-----END PGP SIGNATURE-----

[Mataku]

Thank you!!!!
Seams to be a kernel panic

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

> Thank you!!!!
> Seams to be a kernel panic

There is clearly nouveau module in this call trace. Please try the first
part of my response.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAlw3zVMACgkQ24/THMrX
1yy9WQf9HK5W18EXFhatK2P5cxhs8sHEgV0bpWjXorkntSxHhIzjw7CnDlFtK18T
H057AQl6ibrwnjauFnE/Hu1hYGFS9zmplu2xTPHnznToy8bwk1KqgOom1JMcJcrx
ffE42ib1Z+HX5/nN1CU+vltZp9I2KCb5FBYBxPKQL3Q4mzglk1S34fSAo+Ej4Bwe
6NMw5VQdVHpVPWvZi4X4bI9npMdZ9Tb+wANaElaCJJOz4MJmBaLeUKfPlrKXxmtN
crXJeOAtBYmgFqyR/COI6ZdaQqjMCliaOSJ7rMD5VsQhK5XB9CW+Ksyb2LQT939I
oa+tkOPpYDmlcPlas8431DQq9Nyw3A==
=WITW
-----END PGP SIGNATURE-----

[Mataku]

OK, i think i didn't write it like it is supposed to be.
My xen.cfg file was empty, so i just add nouveau.modset.... but without the "".
And i add it just after the smt part in grub.
I have miss something i guess.

[Mataku]

Better but he is unable to reset pci device

[Mataku]

Ok, i force the reset inside the qubes manager but i get another error

[Mataku]

That is for the win7new vm (i get a bluescreen on it with debug mode)  but for my win10new vm so without tools, blackscreen on the debug console and the vm stop

[Mataku]

I try by adding after "nouveau.modeset=0" this : rd.driver.blacklist=nouveau video=vesa:off
regarding this documentation : https://www.qubes-os.org/doc/nvidia-troubleshooting/

On Win10 i guet the windows10 bluescreen
On Win7 i get the little bar horizontal for pre-loading, and nothing, timeout300 for no respond of the qubes tools.

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

What bluescreen exactly? Try also with Linux, maybe some live distro to
rule out interference from qubes tools. See https://www.qubes-os.org/doc/hvm/

Check also /var/log/xen/console/guest-(VMNAME)-dm.log and
/var/log/xen/console/hypervisor.log, maybe you'll find some
device-related errors.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAlw4+uEACgkQ24/THMrX
1yz7qQf/TFb9Z9eGisEwACzXiWdD7WSFI2dniM1orJG7LFw2o6e+o3RWqXj1+ETp
w4PcdMkLWcf83jPu6fK5Xg2tEZUURGVuPqLRLxkySm9LxujckJdeKB/JOPCJIdTI
Kce7Uewms85/++ZndDy6m0e5sCHJorkol9ijSLXgD2GPZ+b6N0Ar8KtPlRB48IfH
53TalNyT2U6HHtfb9yHZ+Jt1gwnyQUQle1VF7QKeJ6+5c/6EZKZaMz8GdS+tkcGk
gkLZgvfQOQ9/8yrdXu7WDyHPnO7KdRZ/pPqpx32vOvgv7fHfh5qKOkKeUb+xV67Z
gEVxsHTDMfqNm8NLpodWdWbruhGzig==
=hjYl
-----END PGP SIGNATURE-----

[Mataku]

My last test was on Windows 7 where i get a loafing screen with the "yellow green bar" and after blackscreen.
I guess i have to wait the 4.1 with a new version of xen.

[Mataku]

For

"Check also /var/log/xen/console/guest-(VMNAME)-dm.log and
/var/log/xen/console/hypervisor.log, maybe you'll find some
device-related errors. "

I get this following code for the windows 10 nothin (bluescreen)
And same with the windows 7 with bluescreen (memory_management)

You can find the file attach.

[Eric Shelton]

Just a word of warning that passthrough of Nvidia cards to Windows is somewhere between difficult and impossible on Xen, let alone Qubes:

https://groups.google.com/d/msg/qubes-devel/J0rWz3Pnmtc/jJGbBGHgAgAJ

Basically, it sounds like Nvidia's windows drivers are deliberately designed NOT to work in a VM. It used to be a result of weird mappings of PCI config space into MMIO space (there were one to one memory mapping tricks that once would work around that), but it sounds like drivers now simply detect if you are in a VM (there are a number of VM detection techniques that, as I understand it, cannot be defeated).

The exception to this seems to be explicit support for VM passthrough for certain Quadro and Tesla cards (which I have seen first hand work under Xen; it just worked, and was probably the easiest way to do passthrough at one point -- never tried it under Qubes though).

https://www.nvidia.com/en-us/design-visualization/technologies/virtual-gpu/

If you value your time, you are likely much better off using an AMD card over a consumer Nvidia card - preferably one already demonstrated to work with Xen PCI passthrough. At one point, you had to be careful about how AMD drivers got installed and updated, but I believe it works much better these days.

If you still want to work through it, KVM has put a LOT more effort into GPU passthrough than Xen. It might help to see what the state of things is over in KVM to give you an idea of what has to be addressed for Nvidia passthrough to work under Xen.

Finally, if you are really enterprising, you might try recoding qemu to report a Quadro or Tesla PCI device ID for a consumer Nvidia GPU. At one point, people were resoldering surface mount resistors on GTX cards to report the PCI device IDs for their Quadro/Tesla counterparts.

https://news.ycombinator.com/item?id=5398555

The stock Nvidia drivers would work just fine. _Maybe_ you can accomplish the same trick in software instead.

Eric

[John Mitchell]

KVM is much more pass through friendly and has workarounds for some of the problems.

A KVM VM can currently hide itself from the Nvidia drivers. This may not work in future Nvidia products.

AMD consumer based GPU cards have a reset problem in a VM. After using the VM when you shutdown the VM the card doesn't reset properly. It is reported that this problem does not exist in the RX 5xx cards and older.

I am currently building a system for GPU pass through using Nvidia for the default GPU and AMD RX 590 for the pass through GPU.

Also note the two cards should not be identical or KVM may have trouble telling which card is which.

Let me know if anyone needs links to get started. Plan on reading a lot and watching many videos. This has only been working for about a year or so.

Performance is about 95% compared to bare metal.

[Mataku]

I am looking for this yes.

[Radoslaw Szkodzinski]

On Tue, Jan 22, 2019 at 12:47 PM John Mitchell <sonw...@gmail.com> wrote:
> AMD consumer based GPU cards have a reset problem in a VM. After using the VM when you shutdown the VM the card doesn't reset properly. It is reported that this problem does not exist in the RX 5xx cards and older.

AMD messed up time and again (in all cards) by not connecting its PCIe
bridge reset to full card reset, instead relying on some command or a
series of commands beforehand.

The reset workaround magic is not yet implemented for Vega and up.
I've been working on that for some time and am very slowly
implementing required MMIO logging to grab only command accesses to
see how Windows driver installer hard resets the card, since that
actually works.
But the "recovery" soft reset after GPU crash does not, so there is
something missing.

As a nasty but secure workaround, you can enter S3 (suspend to ram)
which cuts power to the card, then issue resets to card side (GPU and
audio) of the PCIe bridge, and then rescan the card's bridge.
I'm pretty sure S3 will fail with Xen in many interesting ways on many
mainboards.

R.

[John Mitchell]

On Tuesday, January 22, 2019 at 1:51:52 PM UTC+1, Radosław Szkodziński wrote:

<snip>

Thank you for your work on this issue and I hope you find success.

In an ideal world Nvidia would allow running a commercial grade card in a single VM and AMD would fix their reset issue.

I read in a forum about AMD that there was a comment in the reset code section of their driver code (I think it was driver code?) that it was a mess and another post said AMD went dark on this issue.

With that said AMD does provide better Linux support than Nvidia. And both have their issues.

John

[StefanUrkel]

Anyone in this thread successfully get GPU passthrough working on Qubes 4.x with an AMD Radeon 64 FE GPU? Are you using a separate GPU for dom-0? I'm interested in this and wondering if it compromises the Qubes security architecture at all, which would kind of somewhat defeat the purpose of choosing Qubes over a Linux distribution such as Debian. My other option is Debian and KVM.