Enabling virgl on KubeVirt VMs

[Frederik Carlier]

Hello,

I'm exploring KubeVirt, and trying to figure out whether it's a good choice to easily deploy Android emulators at scale using Android-x86.

I have a basic RegistryDisk [1] which contains a copy of Android-x86. I can create VMs and connect to them over VNC and console.

As a next step, I would like to enable OpenGL acceleration inside the VM.
From what I understand [2], I would essentially need to pass the "-vga virtio -display sdl,gl=on" flags to qemu, but I haven't found a way to configure my VirtualMachineInstance so that it launches a VM with these flags.

Is this currently possible?

Thanks,

Frederik.

[1]: https://github.com/qmfrederik/kube-virt-droid-images
[2]: https://groups.google.com/forum/#!searchin/android-x86/virgl%7Csort:date/android-x86/enPcst6oQ_w/3Kgb9PXABgAJ

[Fabian Deutsch]

On Wed, Nov 21, 2018 at 5:11 PM Frederik Carlier
<frederik...@quamotion.mobi> wrote:
>
> Hello,
>
> I'm exploring KubeVirt, and trying to figure out whether it's a good choice to easily deploy Android emulators at scale using Android-x86.

Nice!

> I have a basic RegistryDisk [1] which contains a copy of Android-x86. I can create VMs and connect to them over VNC and console.
> As a next step, I would like to enable OpenGL acceleration inside the VM.
> From what I understand [2], I would essentially need to pass the "-vga virtio -display sdl,gl=on" flags to qemu, but I haven't found a way to configure my VirtualMachineInstance so that it launches a VM with these flags.
>
> Is this currently possible?

Heya Frederik,

wrt qemu flags - you'll need a custom hook to configure the VM if you
can not configure it with a custom hook to modify the domxml as
needed.
One example is here:
https://github.com/kubevirt/kubevirt/tree/master/cmd/example-hook-sidecar

You would effectively write a hook which is modifying the domxml to
include the gl support.

The issue I am seeing is that "-display sdl" means a local (to the
node) display will be spawned using SDL. You will not be able to
access this using VNC I suppose.

Martin, Richard, do you know if virgl is available for SPICE or VNC?

Cool to see this work, Frederik.

Greetings
- fabian

>
> Thanks,
>
> Frederik.
>
> [1]: https://github.com/qmfrederik/kube-virt-droid-images
> [2]: https://groups.google.com/forum/#!searchin/android-x86/virgl%7Csort:date/android-x86/enPcst6oQ_w/3Kgb9PXABgAJ
>

> --
> You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.
> To post to this group, send email to kubevi...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/AM4PR0501MB22751F88C3FE1DD1441333D2EADA0%40AM4PR0501MB2275.eurprd05.prod.outlook.com.
> For more options, visit https://groups.google.com/d/optout.

[Richard W.M. Jones]

[Adding Gerd]

--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top

[Frederik Carlier]

Hi Fabien,

Thanks for the quick reply!

> wrt qemu flags - you'll need a custom hook to configure the VM if you can not
> configure it with a custom hook to modify the domxml as needed.
> One example is here:
> https://github.com/kubevirt/kubevirt/tree/master/cmd/example-hook-sidecar
>
> You would effectively write a hook which is modifying the domxml to include
> the gl support.

I tried to create a very simple hook [1], but encountered a couple of issues along the way and though it I'd share them here:

- I had to configure a couple of overrides in Gopkg.toml, some of which were surprising.
* kubevirt.io/kubevirt doesn't seem to work properly and needs to be redirected to GitHub
* github.com/go-kit/kit needs to be pinned at a very specific commit
You can find my Gopkg.toml on GitHub [2]. It's my first foray into Go, so perhaps I missed something obvious.
Can this be simplified in the future, or would I always need to specify these overrides?

- Some libvirt options are not present in the KubeVirt API. For example, I don't think there's a way to express something like this:
<video>
<model type='virtio' heads='1' primary='yes'>
<acceleration accel3d='yes'/>
</model>
</video>
because the acceleration node doesn't exist in the kube-virt API.

If I'm not mistaken, I can modify the XML in my custom hook and add any attribute I want, but since the XML is marshalled back to Go structs, any elements/attributes which are not defined in the KubeVirt API would be ignored?

- The version of libvirt which ships with KubeVirt appears to be 4.2.0 and doesn't support the egl-headless graphics yet.
Is there an easy way to upgrade the version of libvirt?

- I tried to work around this by adding the arguments directly to qemu:commandline. I've written a unit test which shows that the hook adds the commandline arguments to the XML, but when unmarshalling the XML, these command line args seem to be missing. The source code for the unit test is here: [3].
Did I miss something there?

Thanks, and apologies for all the questions 😉.

Frederik.

[1] https://github.com/qmfrederik/kube-virt-droid-images/tree/master/android-x86-hook
[2] https://github.com/qmfrederik/kube-virt-droid-images/blob/master/android-x86-hook/Gopkg.toml
[3] https://github.com/qmfrederik/kube-virt-droid-images/blob/master/android-x86-hook/android-x86-hook_test.go#L170

[Roman Mohr]

On Thu, Nov 22, 2018 at 2:33 PM Frederik Carlier <frederik...@quamotion.mobi> wrote:

Hi Fabien,

Thanks for the quick reply!

> wrt qemu flags - you'll need a custom hook to configure the VM if you can not
> configure it with a custom hook to modify the domxml as needed.
> One example is here:
> https://github.com/kubevirt/kubevirt/tree/master/cmd/example-hook-sidecar
>
> You would effectively write a hook which is modifying the domxml to include
> the gl support.

I tried to create a very simple hook [1], but encountered a couple of issues along the way and though it I'd share them here:

- I had to configure a couple of overrides in Gopkg.toml, some of which were surprising.
  * kubevirt.io/kubevirt doesn't seem to work properly and needs to be redirected to GitHub
  * github.com/go-kit/kit needs to be pinned at a very specific commit
  You can find my Gopkg.toml on GitHub [2]. It's my first foray into Go, so perhaps I missed something obvious.
  Can this be simplified in the future, or would I always need to specify these overrides?

- Some libvirt options are not present in the KubeVirt API. For example, I don't think there's a way to express something like this:
  <video>
    <model type='virtio' heads='1' primary='yes'>
      <acceleration accel3d='yes'/>
    </model>
  </video>
  because the acceleration node doesn't exist in the kube-virt API.

  If I'm not mistaken, I can modify the XML in my custom hook and add any attribute I want, but since the XML is marshalled back to Go structs, any elements/attributes which are not defined in the KubeVirt API would be ignored?

Yes, that's something we should change. We should not convert it back to a struct. Instead we should "blindly" pass it through to libvirt. I want to keep our "limited" go struct, to explicitly express in our code what we understand and care about when interacting with libvirt, but hooks should obviously then do in addition whatever they want. Let me create a PR.

 

- The version of libvirt which ships with KubeVirt appears to be 4.2.0 and doesn't support the egl-headless graphics yet.
  Is there an easy way to upgrade the version of libvirt?

We update it in [4] from time to time. Which version will you need? We can do an update and merge it in kubevirt master.

Best Regards,

Roman

 

- I tried to work around this by adding the arguments directly to qemu:commandline. I've written a unit test which shows that the hook adds the commandline arguments to the XML, but when unmarshalling the XML, these command line args seem to be missing. The source code for the unit test is here: [3].
  Did I miss something there?

Thanks, and apologies for all the questions 😉.

Frederik.

[1] https://github.com/qmfrederik/kube-virt-droid-images/tree/master/android-x86-hook
[2] https://github.com/qmfrederik/kube-virt-droid-images/blob/master/android-x86-hook/Gopkg.toml
[3] https://github.com/qmfrederik/kube-virt-droid-images/blob/master/android-x86-hook/android-x86-hook_test.go#L170

[4] https://github.com/kubevirt/libvirt 

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.
To post to this group, send email to kubevi...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/VI1PR0501MB2286B0EA2CA3742A0C8C9825EADB0%40VI1PR0501MB2286.eurprd05.prod.outlook.com.

[frederik...@quamotion.mobi]

On Thursday, 22 November 2018 14:47:00 UTC+1, Roman Mohr wrote:

On Thu, Nov 22, 2018 at 2:33 PM Frederik Carlier <frederik...@quamotion.mobi> wrote:

- The version of libvirt which ships with KubeVirt appears to be 4.2.0 and doesn't support the egl-headless graphics yet.

  Is there an easy way to upgrade the version of libvirt?

We update it in [4] from time to time. Which version will you need? We can do an update and merge it in kubevirt master.

It looks like the commit I'm after went into v4.6.0 [5]. I've opened a PR [6] to bump the version in kubevirt/libvirt to the latest version, 4.9.0.

I can amend the PR to target another version, if required, and give it a spin as soon as that hits kubevirt master.

Thanks!

Frederik.

 

[1] https://github.com/qmfrederik/kube-virt-droid-images/tree/master/android-x86-hook
[2] https://github.com/qmfrederik/kube-virt-droid-images/blob/master/android-x86-hook/Gopkg.toml
[3] https://github.com/qmfrederik/kube-virt-droid-images/blob/master/android-x86-hook/android-x86-hook_test.go#L170

[4] https://github.com/kubevirt/libvirt 

[5] https://github.com/libvirt/libvirt/commit/d8266ebe1615c4b043db6b8d486465722cdd0ef8  

[6] https://github.com/kubevirt/libvirt/pull/18

[Martin Kletzander]

For egl-headless you need three things:

1) libvirt 4.6.0 or newer

2) qemu 2.10 or newer

3) do a little workaround by (it will be supported in libvirt, but it is not
yet, there's a BZ for it)

- setting permissions (setfacl is enough) on /dev/dri/renderD* so that qemu
can access it

- adding the DRI render node files in the cgroup_device_acl in qemu.conf

For info how to use it see egl-headless in our documentation:

https://libvirt.org/formatdomain.html#elementsGraphics

I thought we talked about using the new way of enabling the copr repo for
virt-preview, but as I see now we don't. I see Frederik sent a PR for that as I
was writing it. Thanks.

[Martin Kletzander]

On Thu, Nov 22, 2018 at 06:21:23AM -0800, frederik...@quamotion.mobi wrote:
>
>
>On Thursday, 22 November 2018 14:47:00 UTC+1, Roman Mohr wrote:
>>
>> On Thu, Nov 22, 2018 at 2:33 PM Frederik Carlier <
>> frederik...@quamotion.mobi <javascript:>> wrote:
>>
>>> - The version of libvirt which ships with KubeVirt appears to be 4.2.0
>>> and doesn't support the egl-headless graphics yet.
>>>
>> Is there an easy way to upgrade the version of libvirt?
>>>
>>
>> We update it in [4] from time to time. Which version will you need? We can
>> do an update and merge it in kubevirt master.
>>
>
>It looks like the commit I'm after went into v4.6.0 [5]. I've opened a PR
>[6] to bump the version in kubevirt/libvirt to the latest version, 4.9.0.
>I can amend the PR to target another version, if required, and give it a
>spin as soon as that hits kubevirt master.
>

The patches for cgroups and permissions will be posted in a while, so if all
goes well and swiftly, then they will land in v4.10.0 (cca. two weeks to
release), otherwise it will have to wait till the next release.

>Thanks!
>
>Frederik.
>
>
>>
>>> [1]
>>> https://github.com/qmfrederik/kube-virt-droid-images/tree/master/android-x86-hook
>>> [2]
>>> https://github.com/qmfrederik/kube-virt-droid-images/blob/master/android-x86-hook/Gopkg.toml
>>> [3]
>>> https://github.com/qmfrederik/kube-virt-droid-images/blob/master/android-x86-hook/android-x86-hook_test.go#L170
>>>
>> [4] https://github.com/kubevirt/libvirt
>>
>[5] https://github.com/libvirt/libvirt/commit/d8266ebe1615c4b043db6b8d486465722cdd0ef8
>[6] https://github.com/kubevirt/libvirt/pull/18
>

>--
>You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
>To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.
>To post to this group, send email to kubevi...@googlegroups.com.

>To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/74179983-c424-4c3c-8ceb-aad4c9a0f2c9%40googlegroups.com.

[Frederik Carlier]

> -----Original Message-----
> From: kubevi...@googlegroups.com <kubevi...@googlegroups.com> On
> Behalf Of Martin Kletzander
> Sent: Thursday, November 22, 2018 3:39 PM
>
> For egl-headless you need three things:
>

[...]

>
> 3) do a little workaround by (it will be supported in libvirt, but it is not
> yet, there's a BZ for it)
>
> - setting permissions (setfacl is enough) on /dev/dri/renderD* so that qemu
> can access it
>
> - adding the DRI render node files in the cgroup_device_acl in qemu.conf
>
> For info how to use it see egl-headless in our documentation:
>
> https://libvirt.org/formatdomain.html#elementsGraphics
>

If I understand this correctly, this requires:
- /dev/dri/rendrD* to be available in the compute container that is created for my VMI
- patching of the qemu.conf file inside that container.

Is there an extensibility hook in KubeVirt I can use to achieve this?

Frederik.

[Roman Mohr]

On Thu, Nov 22, 2018 at 4:38 PM Frederik Carlier <frederik...@quamotion.mobi> wrote:

> -----Original Message-----
> From: kubevi...@googlegroups.com <kubevi...@googlegroups.com> On
> Behalf Of Martin Kletzander
> Sent: Thursday, November 22, 2018 3:39 PM
>
> For egl-headless you need three things:
>
[...]
>
>  3) do a little workaround by (it will be supported in libvirt, but it is not
>     yet, there's a BZ for it)
>
>     - setting permissions (setfacl is enough) on /dev/dri/renderD* so that qemu
>       can access it

Hm, we will have to pass through all devices which are needed into the container. There are different options:

 1) a k8s device plugin, or

 2) an OCI hooks

 

>
>     - adding the DRI render node files in the cgroup_device_acl in qemu.conf

Hm, maybe that is not needed, we would have to try it out. We at least tried to disable extra cgroup and namespace protection mechanisms by qemu, to not double-manage them. If a change is needed, maybe we can add a general change.

[Roman Mohr]

Here the PR: https://github.com/kubevirt/kubevirt/pull/1744. 

[frederik...@quamotion.mobi]

On Thursday, 22 November 2018 17:32:07 UTC+1, Roman Mohr wrote:

On Thu, Nov 22, 2018 at 4:38 PM Frederik Carlier <frederik...@quamotion.mobi> wrote:

> -----Original Message-----
> From: kubevi...@googlegroups.com <kubevi...@googlegroups.com> On
> Behalf Of Martin Kletzander
> Sent: Thursday, November 22, 2018 3:39 PM
>
> For egl-headless you need three things:
>
[...]
>
>  3) do a little workaround by (it will be supported in libvirt, but it is not
>     yet, there's a BZ for it)
>
>     - setting permissions (setfacl is enough) on /dev/dri/renderD* so that qemu
>       can access it

Hm, we will have to pass through all devices which are needed into the container. There are different options:

 1) a k8s device plugin, or

 2) an OCI hooks

I tried the Intel GPU plugin [1], and at the very least it does get you the /dev/dri/renderD* devices in the compute pod.

These are the permissions:

[root@android-x86 /]# ls -l /dev/dri/

total 0

crw-rw---- 1 root 44 226,   1 Nov 22 21:08 card1

crw-rw---- 1 root 44 226, 128 Nov 22 21:08 renderD128

So I wanted to check whether this setup would be sufficient:

- Are these devices sufficient for qemu to perform headless OpenGL, or did I miss anything?

- What permissions should these files have? Should they be owned by qemu? What mechanism could I use to make sure the ownership is set correctly once the container starts? 

Thanks!

Frederik.

[1] https://github.com/intel/intel-device-plugins-for-kubernetes/blob/master/cmd/gpu_plugin/README.md 

[Martin Kletzander]

Yeah, I replied generally, but in kubevirt you don't need to do this.

[frederik...@quamotion.mobi]

I gave this another shot today and came up with a 'patched' virt-launcher image [1].

So here's what kind of worked for me, starting with the latest kubevirt/virt-launcher:v0.10.0 image:

1. Update qemu to 3.0.0 and libvrtd to 4.9.0, respectively

2. Install the mesa-dri-drivers

3. Replace /usr/bin/upstream-virt-launcher with a stub which calls setfacl and grants qemu permissions on /dev/dri/renderD128 

I did not change the cgroup_device_acl.

After this, the VM is capable of booting the kernel, which seems progress. (It's followed by a crash which is the next on my todo list).

A fix for 1 is already in kubevirt master, 3 is being worked on in libvirt if I understood it correctly.

That leaves 2. Would it be acceptable to add the mesa-dri-drivers to the virt-launcher image by default?

Frederik.

[1] https://github.com/qmfrederik/kube-virt-droid-images/tree/master/android-x86-launcher-image 

[Frederik Carlier]

Hi Gerd,

> From: Gerd Hoffmann <kra...@redhat.com>
> "-spice gl=on,..." works only locally (qemu and spice client must run on the
> same machine).
>
> "-display egl-headless -spice gl=off,..." works remotely. Not very efficient, it'll
> effectively do ReadPixels on the rendered framebuffer and send them over
> spice like non-gl display updates. But it might still be better than using
> llvmpipe in the android guest.
>
> "-display egl-headless -vnc ..." works too.

I could only get this to work on a headless system when I applied the following patch to qemu (3.0.0 and master).
qemu would crash (SIGSEV) otherwise.
I've seen this behavior this with Ubuntu 18.04 and Android-x86 as a guest; the qemu command line was qemu-system-x86_64 -enable-kvm -vga virtio -display egl-headless [...].

Is this the only way to get this working on a headless system (as a kubevirt server likely is)?

diff --git a/hw/display/virtio-gpu-3d.c b/hw/display/virtio-gpu-3d.c
index 55d76405a9..7482e37063 100644
--- a/hw/display/virtio-gpu-3d.c
+++ b/hw/display/virtio-gpu-3d.c
@@ -624,7 +624,7 @@ int virtio_gpu_virgl_init(VirtIOGPU *g)
{
int ret;

- ret = virgl_renderer_init(g, 0, &virtio_gpu_3d_cbs);
+ ret = virgl_renderer_init(g, VIRGL_RENDERER_USE_SURFACELESS | VIRGL_RENDERER_USE_EGL, &virtio_gpu_3d_cbs);
if (ret != 0) {
return ret;
}

Thanks,

Frederik.

[Frederik Carlier]

> > I could only get this to work on a headless system when I applied the
> following patch to qemu (3.0.0 and master).
> > qemu would crash (SIGSEV) otherwise.
>

> Do you have a stack trace?

Yes, here it is:

#0 0x00007ffff5cb7703 in __rawmemchr_avx2 () at ../sysdeps/x86_64/multiarch/memchr-avx2.S:61
#1 0x00007ffff5bbd352 in _IO_str_init_static_internal (address@hidden, address@hidden, address@hidden, address@hidden) at strops.c:41
#2 0x00007ffff5bae78d in _IO_vsscanf (string=0x0, format=0x7ffff7babf61 "%i.%i", address@hidden) at iovsscanf.c:40
#3 0x00007ffff5ba81a4 in __sscanf (s=<optimized out>, format=<optimized out>) at sscanf.c:32
#4 0x00007ffff7b69063 in vrender_get_glsl_version (glsl_version=0x555557c07f28) at vrend_renderer.c:7969
#5 0x00007ffff7b615f7 in vrend_create_context (id=0, nlen=4, debug_name=0x7ffff7baba40 "HOST") at vrend_renderer.c:5481
#6 0x00007ffff7b8a797 in vrend_renderer_context_create_internal (handle=0, nlen=4, debug_name=0x7ffff7baba40 "HOST") at vrend_decode.c:1307
#7 0x00007ffff7b60e6c in vrend_renderer_init (cbs=0x7ffff7dcc2c0 <virgl_cbs>, flags=0) at vrend_renderer.c:5311
#8 0x00007ffff7b5060c in virgl_renderer_init (cookie=0x555557263990, flags=0, cbs=0x55555655a700 <virtio_gpu_3d_cbs>) at virglrenderer.c:342
#9 0x00005555558c0e74 in virtio_gpu_virgl_init (g=0x555557263990) at /root/qemu-3.0.0/hw/display/virtio-gpu-3d.c:627
#10 0x00005555558bcc99 in virtio_gpu_handle_ctrl (vdev=0x555557263990, vq=0x7ffff025a010) at /root/qemu-3.0.0/hw/display/virtio-gpu.c:926
#11 0x00005555558bcda3 in virtio_gpu_ctrl_bh (opaque=0x555557263990) at /root/qemu-3.0.0/hw/display/virtio-gpu.c:953
[...]

In frame 4, I can see version_str is NULL, which would explain the exception.

Hope it helps,

Frederik.

[Frederik Carlier]

> From: Gerd Hoffmann <kra...@redhat.com>
> Sent: Thursday, November 29, 2018 12:47 PM

> Hmm, strange why that behaves differently in egl-headless vs. "-spice gl=on".
>
> What do you get with the attached patch in both cases?

Here's the output with -display egl-headess:
egl info:
mode : core
version : 3.0 Mesa 18.0.5
vendor : Intel Open Source Technology Center
renderer : Mesa DRI Intel(R) HD Graphics 620 (Kaby Lake GT2)
shader : 1.30
[ guest console output ]
gl_version 45 - core profile enabled
Segmentation fault

And with -spice gl=on
mode : core
version : 3.0 Mesa 18.0.5
vendor : Intel Open Source Technology Center
renderer : Mesa DRI Intel(R) HD Graphics 620 (Kaby Lake GT2)
shader : 1.30
[ guest console output ]
gl_version 45 - core profile enabled
[ guest boots correctly]

Hope it helps,

Frederik.

[Frederik Carlier]

> > > What do you get with the attached patch in both cases?
> >
> > Here's the output with -display egl-headess:
> > egl info:
> > mode : core
> > version : 3.0 Mesa 18.0.5
> > vendor : Intel Open Source Technology Center
> > renderer : Mesa DRI Intel(R) HD Graphics 620 (Kaby Lake GT2)
> > shader : 1.30
> > [ guest console output ]
> > gl_version 45 - core profile enabled
> > Segmentation fault
> >
> > And with -spice gl=on
> > mode : core
> > version : 3.0 Mesa 18.0.5
> > vendor : Intel Open Source Technology Center
> > renderer : Mesa DRI Intel(R) HD Graphics 620 (Kaby Lake GT2)
> > shader : 1.30
> > [ guest console output ]
> > gl_version 45 - core profile enabled
> > [ guest boots correctly]
>

> Think I've found it, see patch.
>

Yes, I can confirm that patch works. Thanks!

Frederik.

[Fabian Deutsch]

Hey Frederik,

I just wondered - were you in the end able to run a VM on KubeVirt
with virgl enabled?

- fabian

[Frederik Carlier]

Hey Fabian,

In terms of an "initial light up", yes, I got this working.

It required:
- A custom hook to set the virgl configuration [1]
- A custom launcher image [2], with a patched version of qemu [3] (looks like this patch didn't make it into 3.1-rc4) and a wrapper around virt-launcher [4] which sets the permissions on /dev/dri/renderD128
- A device plugin to assign a GPU to the compute pod [5]
- Lots of help and patience from Gerd :)

So for Android 7.1-r2, I ended up with a VMI definition which "kind of" worked [6].

I hit a couple of snags with virgl - display would be messed up after a reboot, and starting the screen capture client [7] which we use would cause the VM to crash and reboot.
I had a quick stab at trying to troubleshoot those issues, but it looks like there are a lot of moving parts involved and I'm not very familiar with the graphics stack.

Hence, I’m currently experimenting with GVT-g. The results with "bare" qemu look very promising, as well as running qemu inside a Kubernetes pod.
At the KubeVirt level, it looks like I'm missing two things:
- Making arbitrary changes to the libvirt domain (waiting for PR 1744)
- When the device plugin attaches the vGPU device to the compute container, I'll need to fetch the UUID of the vGPU and update the libvirt domain. I'm not sure yet how that would work. I can probably hardcode that value for now.

Out of curiosity, is the GPU-acceleration using virgl/GVT-g an important workload for kubevirt?

I'll keep you guys posted, let me know if there's anything else you want me to share.

Frederik.

[1]: https://github.com/kubedroid/android-x86-hook/blob/master/android-x86-hook.go
[2]: https://github.com/kubedroid/android-x86-launcher
[3]: https://lists.gnu.org/archive/html/qemu-devel/2018-11/msg06180.html
[4]: https://github.com/kubedroid/android-x86-launcher/blob/master/virt-launcher-wrapper.go
[5]: https://github.com/intel/intel-device-plugins-for-kubernetes/tree/master/cmd/gpu_plugin
[6]: https://github.com/kubedroid/kubedroid/blob/master/android-x86.yml
[7]: https://github.com/openstf/minicap

[Fabian Deutsch]

On Wed, Dec 5, 2018 at 11:20 AM Frederik Carlier
<frederik...@quamotion.mobi> wrote:
>
> Hey Fabian,
>
> In terms of an "initial light up", yes, I got this working.

Hey Frederik,

well - at least the lights are on :) Nice actually.

> It required:
> - A custom hook to set the virgl configuration [1]
> - A custom launcher image [2], with a patched version of qemu [3] (looks like this patch didn't make it into 3.1-rc4) and a wrapper around virt-launcher [4] which sets the permissions on /dev/dri/renderD128
> - A device plugin to assign a GPU to the compute pod [5]
> - Lots of help and patience from Gerd :)

+1

> So for Android 7.1-r2, I ended up with a VMI definition which "kind of" worked [6].
>
> I hit a couple of snags with virgl - display would be messed up after a reboot, and starting the screen capture client [7] which we use would cause the VM to crash and reboot.

Hm.

> I had a quick stab at trying to troubleshoot those issues, but it looks like there are a lot of moving parts involved and I'm not very familiar with the graphics stack.
>
> Hence, I’m currently experimenting with GVT-g. The results with "bare" qemu look very promising, as well as running qemu inside a Kubernetes pod.
> At the KubeVirt level, it looks like I'm missing two things:
> - Making arbitrary changes to the libvirt domain (waiting for PR 1744)

Yes, that PR should help making arbitrary changes ...

> - When the device plugin attaches the vGPU device to the compute container, I'll need to fetch the UUID of the vGPU and update the libvirt domain. I'm not sure yet how that would work. I can probably hardcode that value for now.

Will the UUID be predictabkle, or will the UUID also be exposed in an
env variable? If so, then it can probably be autodetected and tkae
into account when generating the domxml.

> Out of curiosity, is the GPU-acceleration using virgl/GVT-g an important workload for kubevirt?

Well, there was no high demand for vGPU in particular yet.
But there is interest at least from our side to enable device
passthrough, which would also include vGPU passthrough.

> I'll keep you guys posted, let me know if there's anything else you want me to share.

+1 Thanks for sharing this already.
How do you think about bringing basics into shape i.e. in the device
passthrough area in order to also enable your use case?

Will you be at KubeCOn, might be a good opportunity to sync up.

- fabian

[Frederik Carlier]

Hi Fabian,

> > - When the device plugin attaches the vGPU device to the compute container,
> I'll need to fetch the UUID of the vGPU and update the libvirt domain. I'm not
> sure yet how that would work. I can probably hardcode that value for now.
>
> Will the UUID be predictabkle, or will the UUID also be exposed in an env
> variable? If so, then it can probably be autodetected and tkae into account
> when generating the domxml.

My current understanding is that the UUID is known to the device plugin, and can be exposed as an env variable.

When you say it can be autodetected - is this something that can be done today (an existing feature or via a hook), or would it require a change to KubeVirt?

> How do you think about bringing basics into shape i.e. in the device passthrough
> area in order to also enable your use case?

That would be helpful. The Intel folks redirected me to their SRIOV device plugin [1] (although it seems geared towards network devices), and it looks like the kubevirt/kubernetes-device-plugins may also be an option.
This all is pretty much uncharted territory for me; so I'm looking for a (well-defined) path to pass through a (virtualized) GPU to a VMI/pod.
As soon as I have that, I can probably take things from there.

> Will you be at KubeCOn, might be a good opportunity to sync up.

Unfortunately not, but do let me know if there will be a delegation at FOSSDEM - we're based in Brussels and FOSSDEM is a couple of blocks away from the office.

Frederik.

[1] https://github.com/intel/sriov-network-device-plugin

[Roman Mohr]

On Wed, Dec 5, 2018 at 11:20 AM Frederik Carlier <frederik...@quamotion.mobi> wrote:

Hey Fabian,

In terms of an "initial light up", yes, I got this working.

It required:
- A custom hook to set the virgl configuration [1]
- A custom launcher image [2], with a patched version of qemu [3] (looks like this patch didn't make it into 3.1-rc4) and a wrapper around virt-launcher [4] which sets the permissions on /dev/dri/renderD128
- A device plugin to assign a GPU to the compute pod [5]
- Lots of help and patience from Gerd :)

So for Android 7.1-r2, I ended up with a VMI definition which "kind of" worked [6].

I hit a couple of snags with virgl - display would be messed up after a reboot, and starting the screen capture client [7] which we use would cause the VM to crash and reboot.
I had a quick stab at trying to troubleshoot those issues, but it looks like there are a lot of moving parts involved and I'm not very familiar with the graphics stack.

Hence, I’m currently experimenting with GVT-g. The results with "bare" qemu look very promising, as well as running qemu inside a Kubernetes pod.
At the KubeVirt level, it looks like I'm missing two things:
- Making arbitrary changes to the libvirt domain (waiting for PR 1744)

Thar PR is merged and should be available in https://github.com/kubevirt/kubevirt/releases/tag/v0.12.0-alpha.0.

It should help you a bit.

Roman

  

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.
To post to this group, send email to kubevi...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/AM4PR0501MB2275CE8C4E39D2AFEFB900ABEAA80%40AM4PR0501MB2275.eurprd05.prod.outlook.com.