How to make Kubernetes Cluster on GCP with Nested Virtualization Enabled instances?

[Hussain Parsaiyan]

Following the instructions here I was able to get a Nested Virtualization Enabled instance up but how can I have such an instance in a Kubernetes cluster? It only gives me option of COS and Ubuntu.

I probably could spawn a K8s cluster, make image of one of its instances, then follow the instructions to enable NV and then change the clusters instance template definition with a Rolling update. But it that the only way?

Background: I am trying to make a Gitlab CI runner which would do UI testing on Android emulator which needs an NV enabled instance.

[cbo...@google.com]

Hi Hussain,

I am not able to provide at the moment further insight. This is feature is still quite new.

I was able to verify a request from our internal teams to work on it. I would like to add your feedback to this request. Can you provide some additional details on your use case and the reason GKE would be ideal for the implementation?

[Nelson Chen]

So as of right now, there still doesn't seem to be some native way to teach GKE to enable NV. Between the time of your post and now, Nested Virtualization went GA sometime in April but they still mandate a image/license requirement to enable by design. I was hoping that they would enable it by default and that the managed Kubernetes images would pick it up as a side-effect but that did not happen. 

That said, Hussain, your workaround is working great for my initial tests. I may not have 100% management, but I still have a lot of it and a Kubernetes+Containers API that CI systems are cognizant of. I don't think I've seen workaround steps like what you've described before during my search.

For my use case, I am also using it for nested virtualization testing, but also for testing IO inputs and outputs of a popular, larger, and common desktop OS with qemu. It's simply easier to manage auto-scaling clusters with Kubernetes and CI systems are generally aware of it.

[Nelson Chen]

I accidentally posted in gce-discussion, but a helpful Googler there linked me this issue which is about this topic:

https://issuetracker.google.com/issues/110507927