How risky is GPU pass-through?

[Demi M. Obenour]

Someone I know is interested in using QubesOS.  However, they are also a
gamer: if they could not have a Windows VM with access to a dedicated
graphics card for use by games, then QubesOS is not an option for them.

How risky is GPU pass-through?  My understanding is that on most
laptops, the primary (internal) display is connected to the integrated
GPU.  Therefore, it appears to me that the risks are no more than
pass-through of the USB, Ethernet, or wireless controllers, all of which
QubesOS does by default.

[Zrubi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 12/23/18 9:34 PM, Demi M. Obenour wrote:
> Someone I know is interested in using QubesOS. However, they are
> also a gamer: if they could not have a Windows VM with access to a
> dedicated graphics card for use by games, then QubesOS is not an
> option for them.

Short answer:
Qubes OS is not an option for them.


The risk part would come only after this feature exists in practice ;)
Search back for the details.

- --
Zrubi
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEmAe1Y2qfQjTIsHwdVjGlenYHFQ0FAlwgIk0ACgkQVjGlenYH
FQ2S2g//cIr3vnUjFbhTmWFwSu7SHxpGiT3RORuFOWKb5xilsksyFCBzCwNKphMl
IyUG1czc66fiR+9ZcdwtX0iTyAjWsnQYLysfbaxOFGjF980pDc5stunWYSJN8NXE
jkMiUwbTDXCLW0d/VrG2KMVf3GYt1vS/G9vuS3N5JAolGucTEJPOo8TuXYs44kM1
U/XlkqDdJyJnhDzw0YeJCGMGrHRtwlwfsyErOIMyt9hYyF/qAaegNlBg2hxbsB4h
CR/UCHCDHKCw7Z7bO6SKYKdZfDn01c7HEba5zxIEGQj9fdGe9dAOUgCEi8UG0MWZ
Auj5IdWV0uXtcvo7boWddu6hBiBYog96xv+ypzlhW035yLoht47z+qcyX1cw/nLx
QLTp2fMivwwWGkxmJY9GaixGbMvMBsNRlvmhiisy9sc1pk0FCxNGhVtVwZBFf+C8
bUc7AfSBmfdu2iNZGCqg+uCu0gTjgmSlJWpBhzvS5eio+0O7oO04aJxawBFBINKi
goILZ/7ezulvNlhhPgwnEyfy4C/2sCu9FQrXiy1dCIMngZ3pKa/BO9QqRItTQoGD
GDQjEr1S6Cmlki5yf0dz+T8k8wiqAG0q9IRYHp3pxa4VTdjCE0pTOBvfqI+seRkG
pv/n4O1zhzlrekaRcYNdNm6lIIA+fDl5/QNY9/py/iNyHDN4Sr8=
=CU8M
-----END PGP SIGNATURE-----

[awokd]

Zrubi:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> On 12/23/18 9:34 PM, Demi M. Obenour wrote:
>> Someone I know is interested in using QubesOS. However, they are
>> also a gamer: if they could not have a Windows VM with access to a
>> dedicated graphics card for use by games, then QubesOS is not an
>> option for them.
>
> Short answer:
> Qubes OS is not an option for them.
>
>
> The risk part would come only after this feature exists in practice ;)
> Search back for the details.

What Zrubi said, really. Not a practical solution at this point. If at
some point it works better, it seems to me too if the secondary GPU can
be blocked from ever seeing dom0 and vice versa, it could be passed
through without too large an attack surface increase. Assuming here
Qubes/Xen/IOMMU can restrict overly large BARs, but that's any PCIe device.

[John Mitchell]

Laptops are not going to work well with PCI-passthrough. My laptop has almost every device in it's own IOMMU group. However, sadly it only has one GPU.

Solutions that have worked for others, look here.

https://forum.level1techs.com/t/play-games-in-windows-on-linux-pci-passthrough-quick-guide/108981

Also watch the youtube videos and read the forums from this group. There is much information to glean.

unRAID.net may also be a consideration with lower security though. Again many videos that have information that can be gleaned. Search youtube for spaceinvader one.

I am considering a Proxmox server for my personal needs since qubes doesn't have support for GPU pass-through.

With all of that said I really wish qubes would allow the user to determine how much security they want in their system build and just support GPU pass-through so qubes would be an option.

[Hugo Costa]

Best option would be to dual boot. Unless that person is always switching from game to desktop, this solution could probably be an acceptable compromise.

[John Mitchell]

In my opinion, dual booting is just as risky as GPU pass-through, maybe more so. Having a GPU sandboxed in a VM that will never see production VMs would be ideal. Although if you only have one GPU then there is no better option at the moment.

[qubenix]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Zrubi:

> On 12/23/18 9:34 PM, Demi M. Obenour wrote:
>> Someone I know is interested in using QubesOS. However, they
>> are also a gamer: if they could not have a Windows VM with access
>> to a dedicated graphics card for use by games, then QubesOS is
>> not an option for them.
>
> Short answer: Qubes OS is not an option for them.
>

Why do you say that? If you search this list there are people that
successfully game on Win vm with gpu passthrough.

>
> The risk part would come only after this feature exists in practice
> ;) Search back for the details.
>
>

I can't speak to the security risk from personal experience or
knowledge, but I found this:
https://security.stackexchange.com/questions/162122/gpu-passthrough-security/162175.

- --
qubenix

CODE PGP: FE7454228594B4DDD034CE73A95D4D197E922B20
EMAIL PGP: 96096E4CA0870F1C5BAF7DD909D159E1241F9C54
IRC OTR: DFD1DA35 D74E775B 3E3DADB1 226282EE FB711765
-----BEGIN PGP SIGNATURE-----

iQKTBAEBCgB9FiEElgluTKCHDxxbr33ZCdFZ4SQfnFQFAlwgMjdfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDk2
MDk2RTRDQTA4NzBGMUM1QkFGN0REOTA5RDE1OUUxMjQxRjlDNTQACgkQCdFZ4SQf
nFR7IA//YE7rVNDmYFiXmIU9v7d7j9Bg3bPNSQ6wFnWNclylA3NSvzJ2k/uurcXW
HSz/7r3jDSnJgD6trVan8SMOLlVhU48Hz9FCOxrVagwU69Ch+70vEZplauDcbEC7
UKu3vTFaC5Gawu8EHSqeT97eYCjSqvc/K82g6Wlij9uYOp7juTpQXX9ekIYH4i94
2TI+ZEYCJ/IaoL12aNQbDz6TzR6lsQDnsUiEppd1hnCX/yQphVymRlFG4qBQsXUA
40cAiqSUvpoAchxiWuTS7o4wCblSgrYkHHNzBvX0i+8JhSVmiknloPb+rBZmUVrs
0AoS2cqW3ojKIDXdfQ5Yn27p9TSR9AkoGbNDN9hZSl0CQTjXDGKV/Lcdj9qSSy+X
+xOEJL63nYp94hofsDmZhg7EfcARA5C5JbLF0TzA2fyXlO7hgoX/SsCAv+KaDWhE
8B3Sq+sWH7MAfiJOK/UZN52Bi+I5hUsYsdXPTDSxqkhc6aOnYL8i9wi89gPZ4iVi
JTQ6Tzn87Y5fWeBnz10viMWyfj71rD1AktA9GM20zsw60jx+GcDwtxOHxQLWRTNb
vR1KuET9E+XaS4oEmTcNDACNj0ui+H7OgCRt64plfOttrc9FDtUXgTLMHypMx0bV
zNsV02DucRNWaFSpG6ZrXJMarqvC4NLihAFzhpo2QsGQSpTgiME=
=suwp
-----END PGP SIGNATURE-----

[John Smiley]

If your friend is just poking around with Qubes and doesn't have anything on the gaming box that needs protecting, I say go with dual boot. That's what I did. Running games from within a Xen VM is going to suck performance-wise compared to running naively from Win10.

If he *does* have things that need real protection, he should move them off of the Win10 box immediately.

[John Smiley]

On Tuesday, December 25, 2018 at 1:02:05 PM UTC-8, qubenix wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
>
> Zrubi:
> > On 12/23/18 9:34 PM, Demi M. Obenour wrote:
> >> Someone I know is interested in using QubesOS. However, they
> >> are also a gamer: if they could not have a Windows VM with access
> >> to a dedicated graphics card for use by games, then QubesOS is
> >> not an option for them.
> >
> > Short answer: Qubes OS is not an option for them.
> >
>
> Why do you say that? If you search this list there are people that
> successfully game on Win vm with gpu passthrough.

While it is certainly possible to play games with modest hardware requirements under a virt and still have acceptable performance, games with high hardware requirements running at high frame rates, at high resolutions, and maxed out display settings are going to run much more slowly under a virt than they will on Win10 running natively on the same hardware. Most people who spend the kind of money needed to buy such a system will not be satisfied with the performance provided by a virtual machine.

If the reasons for this are not obvious to you, take it as an opportunity to learn about how virtualization works.

[John Mitchell]

Poor gaming performance in a VM is a myth. This may have been true several years ago however it is no longer true with KVM. XEN needs to step up their game. Here are two videos that will show you what I am referring too.

https://www.youtube.com/watch?v=FvcxPufSRNo

https://www.youtube.com/watch?v=Ww2xpxkhitk&t=229s

[John Smiley]

Sorry, you are woefully misinformed. I have been on the design teams for two well known clouds. You can disprove your assertion with a simple test.

[John Mitchell]

On Friday, December 28, 2018 at 11:57:00 PM UTC+1, John Smiley wrote:
> Sorry, you are woefully misinformed. I have been on the design teams for two well known clouds. You can disprove your assertion with a simple test.

Hi John,

I believe we are talking about two different things. I was referring to gaming in a VM not through the cloud. It doesn't seem like you actually watched either of the videos?

Here is another video showing 7 users on 1 CPU (ok dual CPUs but a single motherboard) gaming in VMs at near bare-metal performance (not through the cloud).

https://www.youtube.com/watch?v=opX-AsJ5Uy8

Cloud based solutions may also be possible one day with caveats. Here is a video about that,

https://www.youtube.com/watch?v=0BQ4bXNdEQI

Like I wrote poor gaming performance in a VM is no longer true. You just need the correct hardware and setup to make it all work.

If anyone is interest in learning more the Level1tech forums are a great place to start. Search for IOMMU and GPU PCI passthrough. Be prepared to do lots of reading.

KVM is leading the pack for gaming in a VM running Windows 10.

Peace,

John

[John Smiley]

No. I knew exactly what you were talking about. That’s okay. You just keep on with your mind in neutral. I won’t waste time n a closed mind.

[John Mitchell]

On Sunday, December 30, 2018 at 9:34:58 AM UTC+1, John Smiley wrote:
> No. I knew exactly what you were talking about. That’s okay. You just keep on with your mind in neutral. I won’t waste time n a closed mind.

John,

You never commented on the videos that show gaming working in a VM so I am not sure who has the closed mind?

Anyway, no problems, we can agree we disagree and part friends.

Blessings,

John

[John Smiley]

I don't need a core sample of the moon to know that it isn't made of green cheese. Doesn't matter what the videos showed. There are lots of videos that "prove" and impossible claim. If you want to believe that, it's completely up to you.

VMs have longer code paths than native. That alone would cause a perf hit. Then there is the noisy neighbor problem and the fact that dom0 has to cycle steal. Anyone with a lick of common sense would see the impossibility of such a claim.

[John Mitchell]

<snip>

> I don't need a core sample of the moon to know that it isn't made of green cheese. Doesn't matter what the videos showed. There are lots of videos that "prove" and impossible claim. If you want to believe that, it's completely up to you.
>
> VMs have longer code paths than native. That alone would cause a perf hit. Then there is the noisy neighbor problem and the fact that dom0 has to cycle steal. Anyone with a lick of common sense would see the impossibility of such a claim.

You are correct their are some wacky videos out on the Internet. I wouldn't trust these videos if they didn't come from reliable sources, Level1techs are legit.

I will assume by longer code paths you are referring to the execution times. This is true however the path is roughly only a 5% longer to execute time penalty yielding 95% of the bare metal performance. Red Hat provides most of the speed boost with their virtio drivers. You can learn more here,

https://docs.fedoraproject.org/en-US/quick-docs/creating-windows-virtual-machines-using-virtio-drivers/index.html

When we believe strongly about something we tend to have tunnel vision and can't see outside the box. I can only present that it works, I can not open your mind up to see outside the box nor do I want too. I respect your opinion and will hope you can do the same.

Anyway, this will be my last response unless I have something new to share. I freely give you the last word if you need it.

I hope you and family and everyone on this group have a very Happy New Year!

Peace!

[dimi]

Used to run Gentoo with qemu / kvm and was passing through 3 GPU's each running win10 for over a year. It took some time to setup and was a fun project to figure out things. There was no lag or crashes and Games were running smooth thanks to vfio! Of course bare Metal would be faster but Benchmarks tend to go from 3% to 5% lower ratting, nothing that worried me.

[Tai...@gmx.com]

One of the reasons I hate the qubes mailinglist is because of the large
amount of people here who claim to be experts while being absolutely
clueless.

I max out new games in a VM on my libre firmware piledriver opteron
IOMMU-GFX setup.

I would say the performance is almost native and that I don't have any
complaints in regards to FPS.

I can also run other VM's on another NUMA node or on another CPU without
noticing.


BUT WAIT! Because some new guy with an annoying and weird name hasn't
seen it done himself I must be lying and so is red hat - we are part of
the the vm gaming conspiracy trying to entice mere mortals in to buying
expensive enterprise grade hardware for no reason!

[John Mitchell]

If I may ask what OS do you use for the host?

[cooloutac]

On Monday, February 25, 2019 at 4:02:38 PM UTC-5, John Mitchell wrote:
> If I may ask what OS do you use for the host?

Guest the latest QSA answers this question somehwat lol.

[Tai...@gmx.com]

On 02/25/2019 04:02 PM, John Mitchell wrote:
> If I may ask what OS do you use for the host?
>

Devuan, it is debian without systemd.

I compile most of the related packages though like libvirtd, qemu etc
cause the ones from the distro are way too outdated to support what I need.

You should get a new non-gmail email btw.

[John Mitchell]

Thank you for the reply.

I know Google (facebook, etc.) owns me. :( And most of the rest of us.

Anyway I moved on to Xubuntu. It provides enough security for my needs and the GPU pass through is working. Also there is a patch coming for QEMU that should bump the performance so I am satisfied with my setup. I'll continue to keep an eye on qubes hoping one day the PCI pass through catches up. I realize Qubes is way ahead on the security side though.

[unman]

Do you run Qubes? On what hardware?

[John Mitchell]

On Tuesday, April 9, 2019 at 2:53:25 PM UTC+2, unman wrote:

<snip>

> Do you run Qubes? On what hardware?

I wanted to use Qubes however I didn't feel that my usage case would be supported here so I opted for Xubuntu running QEMU and Virtual Machine Manager. I have it working, responding here from a VM. I've been following Qubes since version 1, just not using because of the many security features.

AMD Ryzen 2700X, 8 cores, 16 threads
32 GB ram
GeForce GT 1030 (desktop GPU)
Radeon RX 590 (gaming GPU, pass through, also working)

The gaming GPU is blocked in the kernel from the host OS (Xubuntu) with virtio. I suppose virtio could be a security risk. The host OS is restricted to 4 GB (hugepages) and one core (two threads). I have RAID 10 running on the host CPU. KVM shares the host memory however it has one core for itself for iothreads, etc. The rest is available for VMs. Neither of the two CPUs for the host and KVM have ever maxed usage for longer than half a second.

I was planning to use bcache to speed up the RAID although I may skip that since I am not feeling a need for speed. RAID 10 is plenty fast when the drives are not spun down. I have SMART monitoring setup too along with temp and fan monitoring. The host runs from an SSD. Next month I will add a backup solution.

I have some bloat in the host that I need to clean up. Overall it is a solid setup, certainly not as secure as Qubes. However I don't believe I would have this working with Qubes.

[unman]

Thanks John: I hope you'll come back to Qubes in the future.

However, my question was addressed to Taiidan.

[throwaway...@gmail.com]

Just for information:
I have a gaming VM inside Qubes OS
It is a windows 7 HVM, with a dedicated GPU.
Performance are very good.
I referenced some useful links here https://neowutran.ovh/qubeos.pdf

[799]

Hello throwaway42,

<throwaway...@gmail.com> schrieb am Di., 9. Apr. 2019, 21:17:

(...)

Just for information:
I have a gaming VM inside Qubes OS
It is a windows 7 HVM, with a dedicated GPU.
Performance are very good.
I referenced some useful links here https://neowutran.ovh/qubeos.pdf

Nice write-up ... Thanks.

Why don't you add this information to the Qubes Community Docs, so that it can be rea(che)d by a broader audience?

Hypertext is such a great invention compared to PDFs ;-)

- O

[John Mitchell]

Hey throwaway42,

Thank you for the information! I wish I had this 6 months ago when I began planning my personal VM server.

[awokd]

799 wrote on 4/9/19 7:31 PM:

> Hello throwaway42,
>
> <throwaway...@gmail.com> schrieb am Di., 9. Apr. 2019, 21:17:
>
>> (...)
>> Just for information:
>> I have a gaming VM inside Qubes OS
>> It is a windows 7 HVM, with a dedicated GPU.
>> Performance are very good.
>> I referenced some useful links here https://neowutran.ovh/qubeos.pdf
>
>
> Nice write-up ... Thanks.

Seconded! This is the first report I've seen of successful GPU
pass-through under 4.0.

[awokd]

From Throwaway42's document:

> GRUB\_CMDLINE\_LINUX="....
> rd.qubes.hide\_pci=0a:00.0,0a:00.1
> modprobe=xen-pciback.passthrough=1
> xen-pciback.permissive"

Instead of xen-pciback.permissive on the Linux options line, could you
set the GPU's two PCI devices to permissive
https://www.qubes-os.org/doc/pci-devices/#permissive ? Seems it would
make it a little more restrictive. Also, is that modprobe required? I'd
think Qubes would load that module by default. Hiding it here makes sense.

[throwaway...@gmail.com]

I updated the docs.
In fact, the permissive flag wasn't necessary ( at least, for the RX580)

[throwaway...@gmail.com]

Full doc, with latest patchs

https://github.com/Qubes-Community/Contents/blob/master/docs/customization/windows-gaming-hvm.md