Cloud Hypervisor integration

[Boeuf, Sebastien]

Hi everyone,

I'm reaching out because I'm looking for feedback about the best
approach for integrating Cloud Hypervisor (VMM written in Rust) with
KubeVirt.

Currently, KubeVirt supports QEMU because QEMU is integrated with
libvirt and KubeVirt relies on libvirt for managing VMs. But since
libvirt is very much QEMU oriented, and since libvirt adds yet another
layer of indirection, I was wondering if there would be a way to
integrate Cloud Hypervisor directly with KubeVirt, based on an existing
interface (or by creating a new one if needed).

I've got some advices from Fabian Deutsch, recommending I should look
into the VMI spec, which I'm currently evaluating in order to verify
Cloud Hypervisor supports the bare minimum that is required from a
KubeVirt perspective.

I'm really looking for guidance about what could be done, and what is
acceptable from a community standpoint.

Thank you for your time,
Sebastien
---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number: 302 456 199 R.C.S. NANTERRE
Capital: 5 208 026.16 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

[Boeuf, Sebastien]

After looking into the code, I think replacing the virt-launcher binary
would be the best option. As long as it can serve the same API (the
CmdServer interface that's served through the gRPC server), I think the
integration would be mostly seamless. There might be some QEMUisms and
libvirtisms left in the virt-handler but that's something we could
easily handle with a bit of refactoring.

What do you all think? Would this proposal be acceptable?

I'm planning on getting started with a prototype to validate it's
doable.

Thanks,
Sebastien

[Roman Mohr]

On Tue, Jun 14, 2022 at 9:45 AM Boeuf, Sebastien <sebasti...@intel.com> wrote:

After looking into the code, I think replacing the virt-launcher binary
would be the best option. As long as it can serve the same API (the
CmdServer interface that's served through the gRPC server), I think the
integration would be mostly seamless. There might be some QEMUisms and
libvirtisms left in the virt-handler but that's something we could
easily handle with a bit of refactoring.

What do you all think? Would this proposal be acceptable?

I'm planning on getting started with a prototype to validate it's
doable.

I think from a technical perspective, for as long as the replacement-hypervisor is kvm-based, and all "vm-processes" stay therefore within

the cgroups and namespaces of the pod (in contrast to xen where the vm lives kind of  in parallel to the worker node), that sounds like the an approach which should work.

I am at least not aware of any technical limitations apart from the process needs outlined above.

Let us know how it goes.

Best regards,

Roman

 

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/58b2beadac4476f0e26ef226cbc93c0dd0a974da.camel%40intel.com.

[Alice Frosi]

Hi Sebastien,

below is how I'd start

On Mon, Jun 13, 2022 at 1:00 PM Boeuf, Sebastien <sebasti...@intel.com> wrote:

Hi everyone,

I'm reaching out because I'm looking for feedback about the best
approach for integrating Cloud Hypervisor (VMM written in Rust) with
KubeVirt.

Currently, KubeVirt supports QEMU because QEMU is integrated with
libvirt and KubeVirt relies on libvirt for managing VMs. But since
libvirt is very much QEMU oriented, and since libvirt adds yet another
layer of indirection, I was wondering if there would be a way to
integrate Cloud Hypervisor directly with KubeVirt, based on an existing
interface (or by creating a new one if needed).

It is true that libvirt adds another layer of indirection but it makes it easier to interact with the VM. Virt-launcher uses the libvirt go binding and strongly relies on this to manage the VM. Does cloud hypervisor support an interface that can be easily integrated with go? 

I'd suggest you concentrate on the virt-launcher part, especially in the virtwrap part [1] and a good interface to start with is the DomainManager [2].

What are your bare-minimal working example and features?

IMO, another important part of virt-launcher is the monitoring of the VM events and errors. Do you have something similar for cloud hypervisors? This part is covered in the send domain event [3] and the interfaces between virt-handler and virt-launcher is defined in the proto files in [4].

The last thing, we use bazel to build the container images, but I don't think it is easy to quickly modify the image for experimenting. So. you need a way to overwrite the virt-launcher image with your custom one. You could overwrite the sha of the virt-launcher image in [5] to point to your custom virt-launcher image. If you use the kubevirt development k8s cluster (with make cluster-up), you can push your custom image in the local registry by finding the port with:

  registry_port=$(docker port k8s-1.21-dnsmasq 5000 |grep :: | sed 's/://g')

and the tagging and push your custom image with:

  docker tag virt-launcher-custom localhost:$registry_port/virt-launcher:devel

This is just one way of doing it.

 

I've got some advices from Fabian Deutsch, recommending I should look
into the VMI spec, which I'm currently evaluating in order to verify
Cloud Hypervisor supports the bare minimum that is required from a
KubeVirt perspective.

I'm really looking for guidance about what could be done, and what is
acceptable from a community standpoint.

Thank you for your time,
Sebastien
---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 5 208 026.16 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/0740656c8ff0ad3b5ad3b73dacfa5581ef499dd3.camel%40intel.com.

[1] https://github.com/kubevirt/kubevirt/tree/main/pkg/virt-launcher/virtwrap

[2] https://github.com/kubevirt/kubevirt/blob/main/pkg/virt-launcher/virtwrap/manager.go#L100

[3] https://github.com/kubevirt/kubevirt/blob/main/pkg/virt-launcher/notify-client/client.go#L171

[4] https://github.com/kubevirt/kubevirt/tree/main/pkg/handler-launcher-com

[5] https://github.com/kubevirt/kubevirt/blob/main/hack/parse-shasums.sh#L40

I hope this can help you :)

Regards,

Alice

[Boeuf, Sebastien]

On Tue, 2022-06-14 at 09:49 +0200, Roman Mohr wrote:
>
>
> On Tue, Jun 14, 2022 at 9:45 AM Boeuf, Sebastien
> <sebasti...@intel.com> wrote:
> > After looking into the code, I think replacing the virt-launcher
> > binary
> > would be the best option. As long as it can serve the same API (the
> > CmdServer interface that's served through the gRPC server), I think
> > the
> > integration would be mostly seamless. There might be some QEMUisms
> > and
> > libvirtisms left in the virt-handler but that's something we could
> > easily handle with a bit of refactoring.
> >
> > What do you all think? Would this proposal be acceptable?
> >
> > I'm planning on getting started with a prototype to validate it's
> > doable.
> >
>
>
> I think from a technical perspective, for as long as the replacement-
> hypervisor is kvm-based, and all "vm-processes" stay therefore within
> the cgroups and namespaces of the pod (in contrast to xen where the
> vm lives kind of  in parallel to the worker node), that sounds like
> the an approach which should work.

Yes that's the case, Cloud Hypervisor is a VMM relying on KVM,
therefore you can easily constraints the processes and threads related
to each VM.

>
> I am at least not aware of any technical limitations apart from the
> process needs outlined above.

Cool!

>
> Let us know how it goes.

I will.

[Boeuf, Sebastien]

Hi Alice,

On Tue, 2022-06-14 at 10:25 +0200, Alice Frosi wrote:
> Hi Sebastien,
>
> below is how I'd start
>
> On Mon, Jun 13, 2022 at 1:00 PM Boeuf, Sebastien
> <sebasti...@intel.com> wrote:
> > Hi everyone,
> >
> > I'm reaching out because I'm looking for feedback about the best
> > approach for integrating Cloud Hypervisor (VMM written in Rust)
> > with
> > KubeVirt.
> >
> > Currently, KubeVirt supports QEMU because QEMU is integrated with
> > libvirt and KubeVirt relies on libvirt for managing VMs. But since
> > libvirt is very much QEMU oriented, and since libvirt adds yet
> > another
> > layer of indirection, I was wondering if there would be a way to
> > integrate Cloud Hypervisor directly with KubeVirt, based on an
> > existing
> > interface (or by creating a new one if needed).
> >
>
>
> It is true that libvirt adds another layer of indirection but it
> makes it easier to interact with the VM. Virt-launcher uses the
> libvirt go binding and strongly relies on this to manage the VM. Does
> cloud hypervisor support an interface that can be easily integrated
> with go? 

Cloud Hypervisor can be reached through a UNIX socket, sending HTTP
requests to manage the lifecycle of the VM. It shouldn't be too hard to
implement a small package to build HTTP request and send them over
(Kata Containers already does that for instance).

>
> I'd suggest you concentrate on the virt-launcher part, especially in
> the virtwrap part [1] and a good interface to start with is
> the DomainManager [2].

Looking at the code I reached the same conclusion, except that I feel
it might be easier to replace the virt-launcher entirely. The
DomainManager interface that you're pointing at is a mirror of the
CmdServer interface being served by the gRPC server. And by replacing
the binary entirely, I will be able to prototype faster since I won't
have to deal with libvirt specific things that might get in the way.

>
> What are your bare-minimal working example and features?

I was thinking it'd be nice if I could run a VM and access it with
console access (pty). Networking is always more complicated so if  I
can avoid it at first that'd be good.

>
> IMO, another important part of virt-launcher is the monitoring of the
> VM events and errors. Do you have something similar for
> cloud hypervisors? This part is covered in the send domain event [3]
> and the interfaces between virt-handler and virt-launcher is defined
> in the proto files in [4].

Cloud Hypervisor has the ability to report certain type of events,
reporting these through a file. I could have virt-launcher relying on
inotify to know when the file is being updated (therefore when new
events are received), so that we could provide a similar mechanism to
the one that exist with libvirt.

>
> The last thing, we use bazel to build the container images, but I
> don't think it is easy to quickly modify the image for experimenting.
> So. you need a way to overwrite the virt-launcher image with your
> custom one. You could overwrite the sha of the virt-launcher image in
> [5] to point to your custom virt-launcher image. If you use the
> kubevirt development k8s cluster (with make cluster-up), you can push
> your custom image in the local registry by finding the port with:
>   registry_port=$(docker port k8s-1.21-dnsmasq 5000 |grep :: | sed
> 's/://g')
> and the tagging and push your custom image with:
>   docker tag virt-launcher-custom localhost:$registry_port/virt-
> launcher:devel
>
> This is just one way of doing it.

Thanks for the explanation :)

Sebastien

[Fabian Deutsch]

On Tue, Jun 14, 2022 at 9:45 AM Boeuf, Sebastien <sebasti...@intel.com> wrote:

After looking into the code, I think replacing the virt-launcher binary
would be the best option. As long as it can serve the same API (the
CmdServer interface that's served through the gRPC server), I think the
integration would be mostly seamless. There might be some QEMUisms and
libvirtisms left in the virt-handler but that's something we could
easily handle with a bit of refactoring.

Sebastien, thanks for reaching out again on this matter.

 

What do you all think? Would this proposal be acceptable?

There were already some good remarks about the integration points.

But for a proposal  (assuming a proposal to integrate with kubeVirt and have this released), we will need to look beyond the integration.

A proposal of this size requires a formal design in the kubevirt/community repo, and it should also cover the points of how testing will be done for this new VMM.

After all we are (simply spoken) speaking about doubling the testing matrix - and there is currently not enough capacity (of human time or machines) to support this.

A proposal should also dive into tradeoffs, and differences between the two VMMs.

 

Thus while this is not blocking any research (and this research is welcome!) I still wanted to mention it to set expectations right.

Greetings

- fabian

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/58b2beadac4476f0e26ef226cbc93c0dd0a974da.camel%40intel.com.

[Boeuf, Sebastien]

Hi Fabian,

Oh of course this totally makes sense. There's a lot of work ahead but
right now I really want to validate this can work before going further
into the process of integrating the code in KubeVirt.

>  
> Thus while this is not blocking any research (and this research is
> welcome!) I still wanted to mention it to set expectations right.

I appreciate it :)

Thanks,
Sebastien

[Luboslav Pivarc]

Hi Sebastien,

I think we would need some work on virt-handler <-> virt-launcher API.

I see these points where problems can arise:

1. Metadata - Currently we store metadata in libvirt domain (Libvirtisms - I very much like how you named it) and we send it to virt-handler where we make a decision based on it. I think we would need to define API for this. The same goes for the domain(we could probably port anything needed to metadata API).

2. We are expecting a few UNIX sockets on some concrete paths. For example for the console. I think the same would be needed from Cloud Hypervisor or it would need to be covered by API.

3. Process manipulation. Currently, I can only remember that we set memory limits on qemu if VFIO is used.

4. Migration. I am not sure how Cloud Hypervisor implementation works so it would be good to see if what we do in virt-handler is enough.

I believe for a minimal working solution you would need to address the 1st point(or fake it for the start).

With a good enough API(and with the removal of assumptions), the Cloud Hypervisor could be drop-in replacement for virt-lancher and we might not even require a full test matrix(rather tests of the API).

Looking forward to your progress,

Lubo

On Tue, Jun 14, 2022 at 9:45 AM Boeuf, Sebastien <sebasti...@intel.com> wrote:

[Andrea Bolognani]

On Mon, Jun 13, 2022 at 11:00:10AM +0000, Boeuf, Sebastien wrote:
> Hi everyone,
>
> I'm reaching out because I'm looking for feedback about the best
> approach for integrating Cloud Hypervisor (VMM written in Rust) with
> KubeVirt.
>
> Currently, KubeVirt supports QEMU because QEMU is integrated with
> libvirt and KubeVirt relies on libvirt for managing VMs. But since
> libvirt is very much QEMU oriented, and since libvirt adds yet another
> layer of indirection, I was wondering if there would be a way to
> integrate Cloud Hypervisor directly with KubeVirt, based on an existing
> interface (or by creating a new one if needed).

While the QEMU driver is almost certainly the one that receives the
most attention overall, libvirt is explicitly designed to be an
abstract interface that can be used to interact with various
hypervisors. As proof of that, just consider the fact that for a long
time the only backend it supported was Xen :)

In fact, among the many driver libvirt comes with, there's even one
for Cloud Hypervisor:

https://libvirt.org/drvch.html

The driver is fairly basic at the moment, but as far as I know the
only thing preventing it from being as full-featured as the QEMU
driver is lack of development effort towards that goal.

Based on the above, I would recommend you try using this driver while
leaving libvirt in place instead of reinventing the entire thing from
scratch. There are probably a number of places in KubeVirt where it's
implicitly assumed that the QEMU driver is in use, but you'd have to
take care of those anyway. Architecturally, I think it makes a lot
more sense to leave most things unchanged and only alter the stack
from the libvirt driver down.

Good luck!

--
Andrea Bolognani / Red Hat / Virtualization

[dvo...@redhat.com]

On Tuesday, June 14, 2022 at 3:45:35 AM UTC-4 Sebastien Boeuf wrote:

After looking into the code, I think replacing the virt-launcher binary
would be the best option. As long as it can serve the same API (the
CmdServer interface that's served through the gRPC server), I think the
integration would be mostly seamless. There might be some QEMUisms and
libvirtisms left in the virt-handler but that's something we could
easily handle with a bit of refactoring.

I want to make sure this work isn't trivialized. Virt-handler is going to be a challenge. Look at pkg/virt-handler/vm.go and the usage of the api.Domain objects. Everywhere you see one of those objects being accessed is a bit of code that is directly tied to the libvirt domainxml.

If you decide you want to dedicate resources to this project, my advice is to start investigating how the virt-handler <-> virt-launcher communication both works and is processed within virt-handler. There are two connections, the cmd socket which you're familiar with, and the event socket which virt-launcher uses to "inform" virt-handler when libvirt domains change for any reason. Both of these communication channels are tied to the libvirt domain xml

[Vladik Romanovsky]

Hi Sebastien

On Tue, Jun 14, 2022 at 3:45 AM Boeuf, Sebastien <sebasti...@intel.com> wrote:

After looking into the code, I think replacing the virt-launcher binary
would be the best option. As long as it can serve the same API (the
CmdServer interface that's served through the gRPC server), I think the
integration would be mostly seamless. There might be some QEMUisms and
libvirtisms left in the virt-handler but that's something we could
easily handle with a bit of refactoring.

There were great points mentioned already.
I would also add that kubevirt relies on some tooling that libvirt provides, such as :

 - domain capabilities that are being used by the node labeler[1] to coordinate live migration and VMI scheduling

 - domain stats to monitor the live migration process and to provide stats for Prometheus 

Libvirt also provides a PCI layout and guest ABI stability, in general. 

It allows us to define a domain and see the devices/controllers relationship before running the domain - KubeVirt has features that use this knowledge.

If I'm not mistaken, plugging and unplugging devices also rely on this mechanism in libvirt.

I'm sure there are other aspects that I'm forgetting about, such as the need to replace the libvirt lifecycle events.

Libvirt provides us with some convenience and it's well integrated into KubeVirt. . I don't see how we can use it for one emulator, not the other.

As a first step, I would second Andrea and suggest keeping libvirt in place and using the existing libvirt's Cloud-hypervisor driver.

I'm also wondering if you had any thoughts on how KubeVirt could handle features that are not supported by the could-hypervisor? 

Thanks,

Vladik

[1] https://github.com/kubevirt/kubevirt/tree/main/pkg/virt-handler/node-labeller

 

[Boeuf, Sebastien]

Hi Vladik and Andrea,

I understand your point about libvirt being well integrated at this
point inside KubeVirt,
and I also understand that a lot of things in KubeVirt are based on
libvirt since this is the
only virtualization interface that's supported at the moment.

The main reason I don't really want to integrate Cloud Hypervisor in
libvirt (by developing
the existing driver further) is because I'd like the whole stack (k8s
all the way down to the
VMM) to be as simple as possible. And adding libvirt into the picture
defeats this whole idea.

Given the VMM (Cloud Hypervisor or any other one) provides the knobs
and hooks to be
managed and monitored, it shouldn't be forced to go through yet another
layer of indirection.
Removing a layer could provide a few benefits such as better/faster
debuggability and a smaller
memory footprint per VM.

Prototyping the support of Cloud Hypervisor inside KubeVirt is also the
opportunity for identifying
where a generic virtualization interface could be defined in KubeVirt,
which could benefit other
projects in the future as well.

Thanks,
Sebastien

[Boeuf, Sebastien]

Hi everyone,

I've made some progress to a point where I think I can say there's no
technical limitation integrating Cloud Hypervisor into KubeVirt. You
can have a look at the code here:
https://github.com/kubevirt/kubevirt/pull/8056

Of course I don't expect this PR to be merged, it's only there as a
reference for review.

I've listed on the PR description the list of features that I support
through this POC, and the workarounds/fixes I had to use.

I can see a few possible factorizations with the existing code, which I
didn't implement there since it was only a POC. The goal was to prove I
could use KubeVirt to spawn Cloud Hypervisor VMs, and show it didn't
require a lot of new code.

IMO the POC shows where the duplication happens and therefore where we
can improve the code creating some new Go interfaces to support more
than just libvirt.

I'm looking for feedback from the community about general thoughts and
possible agreement on moving forward with a proper plan to integrate
Cloud Hypervisor with KubeVirt.

Thanks,
Sebastien


On Mon, 2022-06-13 at 13:00 +0200, Sebastien Boeuf wrote: