GVT-g Discussion

[Dylanger Daly]

Will the Qubes team consider enabling the use of GVT-g so we can enjoy hardware accelerated graphics?

After upgrading to DDR4 Memory I've noticed a significant increase in performance, I assume this is because of the many, many memory copies taking place, GVT-g should lighten the load.

[pixel fairy]

This comes up often, and i suspect qubes-os is the inspiration for the xen version of it, but it also opens up a big can of attack surface. maybe after splitting the guivm off from dom0?

[David Schissler]

I've been lurking on Qubes channels for a long time.  For one particular minor version some years ago 25% of the TODOs were small quality of life improvements that I had suggested.  My story is that I've tried to use Qubes but have ended up not due to many quality of life issues since I don't really need that level of protection.  I also live out of a bag so carrying two laptops isn't very practical.  Also I've found more benefit in upgrading notebooks than staying back to use Qubes.  That said, if I could get GPU acceleration for just my green zones for local development work then it would be amazing and probably worth the jump for me.  In my case I'd likely put some proprietary games in their own environment and browse the web unaccelerated.  I think that if this feature were available even if it needed to be turned on manually from the CLI (to make sure people are aware of the ramifications) then it would bring a lot more mind share to Qubes.  The less than ideal degraded GPU security would be better than what I'm running now.  I think that other than the labor the question is if Qubes wants to offer the most protection to many different use cases and threat levels or if its only interested in being the most ideal security environment period.

[Qubes123]

As far as I know in Qubes R4.1 will have split admin and gui domains, so neither of these functions will run in dom0.

For that I assume there's already intensive work and testing are being done to test GPU passthrough to guivm. Considering the significantly larger Intel install base - I guess the integrated Intel GPU passthrough this will come first. Then probably the AMD integrated GPU passthrough support, then the rest of the other PCI cards. However, full HW acceleration (especially 3D) might not be trivial as the Qubes gui agent implementation is code developed by the Qubes dev team, so not all features and possibilities of Xen- HW gui acceleration capabilities is implemented within. But I could be mistaken here, and it could be, that as soon as the guivm separation is done and tested (eg. also security wise), then it'll be much more easy to have HW acceleration (2D->3D) in an AppVM.

[Kardykov Ivan]

bump thread

[Kardykov Ivan]

Sorry for that bumping, but looks like my posts from work domain (tabit.pro) stucked at moderate or
spam checks.. So my message was about GVT-g.

We have some success with adaptation Intel gtv-g technology to Qubes OS.
At now it is rather proof of concept state, but it probably could help
to test guivm features and discuss of applicability in practice.

In this [1] repository we placed specs and patches to build core Qubes
components with ability to run and work with GPU accelerated VM. We
oriented to Fedora 29 dom0 and Xen 4.12 versions with plans of 4.1
compatibility. Now it is up to Fedora 31 and Xen 4.13 and we will get to
this versions in some time.

We made and adapted patches to xen, kernel, libvirt, qemu to dom0 and
xorg dummy driver to guest environment. Our work mostly based on Intel
[2][3] and XenServer [4] public repositories.

The most controversial question is probably running qemu without any
restrictions, but there are several ways to improve it.

At now it works with some issues and limitations, most of them described
in repository readme. We have tested it on Lenovo X1 gen5/gen6 and T480
laptops with KabyLake/KabyLake R chipsets.

[1] https://github.com/tabit-pro/tabit-qubes-repo
[2] https://github.com/intel/gvt-linux
[3] https://github.com/intel/igvtg-qemu
[4] https://github.com/xenserver/xen.pg.git


Regards,
Ivan Kardykov
tabit.pro

 

[Demi M. Obenour]

On 2020-01-02 15:33, Kardykov Ivan wrote:
> Sorry for that bumping, but looks like my posts from work domain (tabit.pro) stucked at moderate or
> spam checks.. So my message was about GVT-g.
>
> We have some success with adaptation Intel gtv-g technology to Qubes OS.
> At now it is rather proof of concept state, but it probably could help
> to test guivm features and discuss of applicability in practice.
>
> In this [1] repository we placed specs and patches to build core Qubes
> components with ability to run and work with GPU accelerated VM. We
> oriented to Fedora 29 dom0 and Xen 4.12 versions with plans of 4.1
> compatibility. Now it is up to Fedora 31 and Xen 4.13 and we will get to
> this versions in some time.
>
> We made and adapted patches to xen, kernel, libvirt, qemu to dom0 and
> xorg dummy driver to guest environment. Our work mostly based on Intel
> [2][3] and XenServer [4] public repositories.
>
> The most controversial question is probably running qemu without any
> restrictions, but there are several ways to improve it.

As Marek has stated before, this is a non-starter. QEMU is not
considered trusted. Xen’s attack surface is already too high.

The approach I would like to see is a paravirtualized approach
similar to Genode’s: a small (<10K SLOC), auditable (ideally
formally verified) driver running in dom0 and/or its own domain that
communicates with a paravirtualized driver in the VMs. Any needed
emulation is done in the stubdomain. This is likely to be difficult
for many reasons, but is the only way that I see can hardware
accelerated graphics being secure enough for Qubes with commodity
hardware. If you are willing to pay literally thousands of dollars
for server-grade GPUs, then SR-IOV and friends become options, which
make secure hardware acceleration much easier. However, those are
far too expensive for virtually all Qubes users. In particular,
they are far more expensive than just buying another computer for
graphics-intensive activities.

The main activities I know of that need hardware-accelerated graphics,
and which a Qubes user is likely to want to partake in, are gaming
and GPGPU. Video games, in particular, should be considered completely
untrusted, and so need to be walled off as tightly as possible.

Sincerely,

Demi

[Dylanger Daly]

 My bad, I accidentally replied directly to Demi

Hey Ivan,

This is fantastic work! I'll test this out today, I think this will help with Firefox's WebRender engine, meaning much less CPU spikes, and reduced wattage when I have like 2 Firefox Tabs open.

Not to mention watching YouTube.

Demi you make some valid Security points, however one major GPU activity is WebRender (Web Content being rendered within the iGPU)

It's also my understanding Marek mentioned R4.1 would support GVT-g.

I would love to see an approach similar to Genode as well, I just don't think it will happen, I'm almost certain GVT-g will be used.

Cheers!

Demi Replied with

> Hey Ivan,
>
> This is fantastic work! I'll test this out today, I think this will help
> with Firefox's WebRender engine, meaning much less CPU spikes, and reduced
> wattage when I have like 2 Firefox Tabs open.
> Not to mention watching YouTube.
>
> Demi you make some valid Security points, however one major GPU activity is
> WebRender (Web Content being rendered within the iGPU)
> It's also my understanding Marek mentioned R4.1 would support GVT-g.
>
> I would love to see an approach similar to Genode as well, I just don't
> think it will happen, I'm almost certain GVT-g will be used.
>
> Cheers!

Regarding WebRender: My personal approach (and recommendation) is to
just accept the higher power consumption.  If you can’t, a separate
computer is probably your best option.  The trend in QubesOS is to
emulate more devices (such as audio hardware), and rely on QEMU being
sandboxed for security.

GVT-g may be supported in QubesOS at some point, but not until it
doesn’t require unsandboxed QEMU.  Honestly, I consider QEMU
being in stubdoms to be one of QubesOS’s main selling points.
Especially since dom0 is so out of date that there is almost certain
to be a trivially-exploitable vulnerability there.

Sincerely,

Demi

[Ivan Kardykov]

> --
> You received this message because you are subscribed to the Google
> Groups "qubes-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to qubes-devel...@googlegroups.com
> <mailto:qubes-devel...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/qubes-devel/f608a18f-15e2-4ad1-94b3-da7ae8b1c553%40googlegroups.com
> <https://groups.google.com/d/msgid/qubes-devel/f608a18f-15e2-4ad1-94b3-da7ae8b1c553%40googlegroups.com?utm_medium=email&utm_source=footer>.

Sorry for that bumping, but looks like my posts stuck at moderate or

spam checks.. So my message was about GVT-g.

We have some success with adaptation Intel gtv-g technology to Qubes OS.
At now it is rather proof of concept state, but it probably could help
to test guivm features and discuss of applicability in practice.

In this [1] repository we placed specs and patches to build core Qubes
components with ability to run and work with GPU accelerated VM. We
oriented to Fedora 29 dom0 and Xen 4.12 versions with plans of 4.1
compatibility. Now it is up to Fedora 31 and Xen 4.13 and we will get to
this versions in some time.

We made and adapted patches to xen, kernel, libvirt, qemu to dom0 and
xorg dummy driver to guest environment. Our work mostly based on Intel
[2][3] and XenServer [4] public repositories.

The most controversial question is probably running qemu without any
restrictions, but there are several ways to improve it.

[Ivan Kardykov]

On 2020-01-02 18:34, Demi M. Obenour wrote:
> On 2020-01-02 15:33, Kardykov Ivan wrote:
>> Sorry for that bumping, but looks like my posts from work domain (tabit.pro) stucked at moderate or
>> spam checks.. So my message was about GVT-g.
>>
>> We have some success with adaptation Intel gtv-g technology to Qubes OS.
>> At now it is rather proof of concept state, but it probably could help
>> to test guivm features and discuss of applicability in practice.
>>
>> In this [1] repository we placed specs and patches to build core Qubes
>> components with ability to run and work with GPU accelerated VM. We
>> oriented to Fedora 29 dom0 and Xen 4.12 versions with plans of 4.1
>> compatibility. Now it is up to Fedora 31 and Xen 4.13 and we will get to
>> this versions in some time.
>>
>> We made and adapted patches to xen, kernel, libvirt, qemu to dom0 and
>> xorg dummy driver to guest environment. Our work mostly based on Intel
>> [2][3] and XenServer [4] public repositories.
>>
>> The most controversial question is probably running qemu without any
>> restrictions, but there are several ways to improve it.
>
> As Marek has stated before, this is a non-starter. QEMU is not
> considered trusted. Xen’s attack surface is already too high.
>

As I see it, useful results could be with testing it to get more about
advantages and finding ways to minimize risks. May be smth with this [1]
movements are compatible with gvt-g.

[1] https://wiki.xenproject.org/wiki/Linux_stub_domains

[Jean-Philippe Ouellet]

Hello,

In case anyone needed a reminder that exposing hardware graphics
acceleration to untrusted workloads is dangerous, watch CVE-2019-14615
for details in the coming days.

Already we see:
- https://www.intel.com/content/www/us/en/security-center/advisory/intel-sa-00314.html
- https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=63d264fe0868cb755a3a887ee7b7b6e3da786804
- https://patchwork.freedesktop.org/patch/348646/
- https://ftp.openbsd.org/pub/OpenBSD/patches/6.5/common/028_inteldrmctx.patch.sig

And previously:
- https://ftp.openbsd.org/pub/OpenBSD/patches/6.5/common/018_inteldrm.patch.sig

Some news and preliminary benchmarks:
- https://www.phoronix.com/scan.php?page=news_item&px=Intel-CVE-2019-14615
- https://www.phoronix.com/scan.php?page=article&item=intel-gen7-hit&num=1

Nothing here yet, but probably (hopefully?) soon:
- https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-14615

If something can't manage to clear state across context switches, it
begs the question of what else has been overlooked. To me, seeing this
relatively fundamental requirement being messed up indicates that this
area has not received much scrutiny with an attacker mindset (at least
from someone who publishes instead of exploits and stays quiet), and I
strongly suspect that this advisory is merely the tip of an iceberg.

Once again, I think we can safely say that another aspect of Qubes'
architecture is clearly being validated as having been a good
decision.

Anyone considering hardware acceleration to untrusted guests should
take a close look at their threat model, and think twice about the
impacts that a totally broken isolation primitive would have in that
context.

Best regards,
Jean-Philippe

[Ivan Kardykov]

Greetings,

there is an update of our experimental GVT-g integration with Qubes.

Code was slightly improved to get rid of qemu. Vgpu init routine was
moved to libxl and it works with stubdomain.


https://github.com/tabit-pro/tabit-qubes-repo

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

This is really interesting development!
Can you say something more about it? Specifically:

1. What a VM have access to really? Full GPU, some context? If a
context, how is it enforced?

2. What components are involved in GPU commands processing and how are
they isolated?

3. Is it possible to enable it only for some VMs - in a way outside of VM
control?

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAl5Cv4gACgkQ24/THMrX
1ywSQAf/bZhoYV4kd++KmF/o38mZkBUblyZZFfRlXRYuvDYqAFWdmCMVmLyLzLBn
fRDTjph/4XuY3UC0hADBXpVYDSAoSVlCw4E2dza5g+gFipvXEC8l0dCAP3hS3nob
N03t4UGmqBS+Vq6dekVpvgCyLnUGVc1SUDtgM04LNdtJbt+NfeRdsL9QFa5tBaCI
MFmg1HTvdz5/Qg4gmxvLIuwMRwXi6tSPWAcd84a3+3c+kGR8paZsJhztdNbyhrY+
QPPWqNxwNbL5aUEv/HqljM9Qcxt/iXRrVkv0y3ZXHnZMN3ff2ZVru6r3+Ie+tCWc
ZjgSKxHf6YnEsEkxyPd8DOrkdwacZA==
=TCpb
-----END PGP SIGNATURE-----

[Ivan Kardykov]

On 2020-02-11 17:51, Marek Marczykowski-Górecki wrote:
> On Thu, Feb 06, 2020 at 02:53:09PM +0300, Ivan Kardykov wrote:
>> Greetings,
>
>> there is an update of our experimental GVT-g integration with Qubes.
>
>> Code was slightly improved to get rid of qemu. Vgpu init routine was
>> moved to libxl and it works with stubdomain.
>
>
>> https://github.com/tabit-pro/tabit-qubes-repo
>
> This is really interesting development!
> Can you say something more about it? Specifically:
> > 1. What a VM have access to really? Full GPU, some context? If a
> context, how is it enforced?
>
> 2. What components are involved in GPU commands processing and how are
> they isolated?
>

In theory [1] I see that each VM has access to frame and command buffers
in system memory. The command buffer reused across VMs with smart
shadowing and GVT framework validates graphics memory addresses before
they used by the GPU.
Access to the privileged resources (GPU Page Table Entries and I/O
registers) is handled by xengt module, based on trap-and-emulation.
Render context switching is used per vGPU.

In practice, security audit is the matter of our further experiments.

[1]
https://www.usenix.org/system/files/conference/atc14/atc14-paper-tian.pdf

> 3. Is it possible to enable it only for some VMs - in a way outside of VM
> control?
>

We made several changes to the existing implementation to use it with
qubes via qvm-features (video-model):
- Patch to libxl creates vGPU instance via sysfs, during hvm domain
execution.
- Stubdomain part extends video model to prevent unnecessary
initialization of emulated VGA.
- Libvirt configuration adapted to parse xengt parameters.


---
Regads,
IK
tabit.pro

[thorsten...@gmail.com]

Hello everyone,

it has been a while since the last post here, is there any update on this topic?

Any gpu virtualization on the horizon in the (near) future?

Thorsten Schierer