Use KubeVirt libvirtd in combination with other tools

[Thomas Buchinger]

Hello,

I just installed KubeVirt in my lab and managed to install the first VM. But the thing I cannot get my head around is that:

* KubeVirt uses a containerized libvirtd, instead of the system daemon. (I guess there is a good reason for that)

* Existing tools in traditional virtualization (e.g. Cockpit, Foreman) usually expect to connect to libvirt through either "qemu:///system" or "ssh+qemu://<user>@<host>/system" URIs.

Therefore traditional tools don't know anything about VMs managed by KubeVirt, which to me seems counter intuitive, given the projects mission to bring virtualization based workloads

closer to containers. I wasn't able to find anything in the docs regarding the topic (neither stating that it is possible, nor that it is not) and from my own testing I don't see an obvious 

way to do it.  

The high level question is: How is KubeVirt supposed to work in combination with other virtualization tools? And is it even supposed to do that?

Some smaller question:

* Is there a preferred/official way to connect to KubeVirt's libvirt-daemon?

* Is it even a good idea to modify guests from somewhere else, or is the reconciliation loop going to undo all the changes anyway? Both in terms of VMs existing only in libvirt and VMs

existing in KubeVirt, but have some advanced configuration not covered by KubeVirt

* Are you guys planning on exposing every last knob of libvirt on the KubeVirt API? Otherwise there isn't even the possibility to manually configure libvirt in case someone is running some exotic 

configuration. On the other hand, I have no idea if that is an issue in reality

* I think it would be valuable to have at least read-only access to libvirt, in case someone is using existing tools to react on changes in the environment. (I am thinking something like ManageIQ, 

although ManageIQ needs oVirt as a provider). Or someone could use Foreman/oVirt as dynamic inventory source, because "thats where all the VMs are registered"

* Is it possible to use the libvirt-daemon on the host with KubeVirt? Is there an obvious game stopper I am missing?

Since I am new to KubeVirt I might have completely wrong expectations on what KubeVirt should and should not do. However I feel like a paragraph somewhere in the documentation about this 

topic would help people figure out how KubeVirt fits in this space between virtualization and containers.

-Thomas

[Itamar Heim]

On 11/4/18 11:48 AM, Thomas Buchinger wrote:
> Hello,
> I just installed KubeVirt in my lab and managed to install the first VM.
> But the thing I cannot get my head around is that:
> * KubeVirt uses a containerized libvirtd, instead of the system daemon.
> (I guess there is a good reason for that)

yes. in the kubernetes ecosystem you expected the nodes to not have to
be specifically configured and deploy everything via kubernetes, hence
podified libvirt.
there are other aspects like using kubelet to launch the pod/container
rather than libvirt due to namespace issues and others.

> * Existing tools in traditional virtualization (e.g. Cockpit, Foreman)
> usually expect to connect to libvirt through either "qemu:///system" or
> "ssh+qemu://<user>@<host>/system" URIs.

That is true when using a single host approach, not when using a
virtualization management system (OpenStack, oVirt, etc.).
In these cases, you'd connect to that higher level api to have a broad
view, rather than to hosts directly (for a lot of good reasons).

The current mode is not necessarily a final one - there is work on the
libvirt side to allow using a single podified libvirt having the picture
on the hosts for multiple VMs running in their own pods. But it requires
changes to libvirt. I would still not advise to manipulate VMs directly
in a managed environment though.

The other aspect is ecosystem enablement. In that regard there is work
to add KubeVirt API level support to tools used to manage VMs (already
going on for Foreman, Vagrant, etc.).

>
> Therefore traditional tools don't know anything about VMs managed by
> KubeVirt, which to me seems counter intuitive, given the projects
> mission to bring virtualization based workloads
> closer to containers. I wasn't able to find anything in the docs
> regarding the topic (neither stating that it is possible, nor that it is
> not) and from my own testing I don't see an obvious
> way to do it.
>
> The high level question is: How is KubeVirt supposed to work in
> combination with other virtualization tools? And is it even supposed to
> do that?

We are looking for input to prioritize which "traditional tools" need
this enablement, so more feedback is welcome!
for Foreman, I think this is the relevant one (Ohad/Piotr/Moti can keep
me honest)
https://community.theforeman.org/t/plug-in-kubevirt-request-for-feedback/11484

>
> Some smaller question:
> * Is there a preferred/official way to connect to KubeVirt's libvirt-daemon?

not sure, though can you provide more details on the use case?

> * Is it even a good idea to modify guests from somewhere else, or is the
> reconciliation loop going to undo all the changes anyway? Both in terms
> of VMs existing only in libvirt and VMs
> existing in KubeVirt, but have some advanced configuration not covered
> by KubeVirt

There are ways to tackle this in KubeVirt like using hooks.
In general, no managed system would like you to do something
undercutting it's behavior.

> * Are you guys planning on exposing every last knob of libvirt on the
> KubeVirt API? Otherwise there isn't even the possibility to manually
> configure libvirt in case someone is running some exotic
> configuration. On the other hand, I have no idea if that is an issue in
> reality

I expect a mix of "it makes sense and keeps broad adoption/usage simple"
and "you can do that by using a hook if you really need to".
Again, mostly we are looking for feedback on needs/usage patterns.

> * I think it would be valuable to have at least read-only access to
> libvirt, in case someone is using existing tools to react on changes in
> the environment. (I am thinking something like ManageIQ,
> although ManageIQ needs oVirt as a provider). Or someone could use
> Foreman/oVirt as dynamic inventory source, because "thats where all the
> VMs are registered"

Exactly - for more than single nodes it may be easier to integrate at
kubevrit api. When we get back to libvirt per host architecture, may be
relevant to revisit aspects there.
providing more info on what you are trying to achieve will allow to
provide more guidance.

> * Is it possible to use the libvirt-daemon on the host with KubeVirt? Is
> there an obvious game stopper I am missing?

Not sure.

>
> Since I am new to KubeVirt I might have completely wrong expectations on
> what KubeVirt should and should not do. However I feel like a paragraph
> somewhere in the documentation about this
> topic would help people figure out how KubeVirt fits in this space
> between virtualization and containers.

That is great feedback, thank you for pointing this out.

Thanks,
Itamar

> -Thomas
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "kubevirt-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to kubevirt-dev...@googlegroups.com
> <mailto:kubevirt-dev...@googlegroups.com>.
> To post to this group, send email to kubevi...@googlegroups.com
> <mailto:kubevi...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/kubevirt-dev/e2196f4e-7faf-4f39-b254-cf193894b65e%40googlegroups.com
> <https://groups.google.com/d/msgid/kubevirt-dev/e2196f4e-7faf-4f39-b254-cf193894b65e%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

[Moti Asayag]

That's correct, we're asking for a feedback on foreman-kubevirt plugin on the link above.

I've added there a link to short demo of the KubeVirt compute resource on Foreman.

  

> <mailto:kubevirt-dev+unsub...@googlegroups.com>.

[Fabian Deutsch]

On Sun, Nov 4, 2018 at 6:05 PM Itamar Heim <ih...@redhat.com> wrote:

On 11/4/18 11:48 AM, Thomas Buchinger wrote:
> Hello,
> I just installed KubeVirt in my lab and managed to install the first VM.
> But the thing I cannot get my head around is that:
> * KubeVirt uses a containerized libvirtd, instead of the system daemon.
> (I guess there is a good reason for that)

yes. in the kubernetes ecosystem you expected the nodes to not have to
be specifically configured and deploy everything via kubernetes, hence
podified libvirt.
there are other aspects like using kubelet to launch the pod/container
rather than libvirt due to namespace issues and others.

Let me just add, decoupling the life-cycle management of the platform and kubevirt was one reason.

 

> * Existing tools in traditional virtualization (e.g. Cockpit, Foreman)
> usually expect to connect to libvirt through either "qemu:///system" or
> "ssh+qemu://<user>@<host>/system" URIs.

That is true when using a single host approach, not when using a
virtualization management system (OpenStack, oVirt, etc.).
In these cases, you'd connect to that higher level api to have a broad
view, rather than to hosts directly (for a lot of good reasons).

The current mode is not necessarily a final one - there is work on the
libvirt side to allow using a single podified libvirt having the picture
on the hosts for multiple VMs running in their own pods. But it requires
changes to libvirt. I would still not advise to manipulate VMs directly
in a managed environment though.

Yes, some more details here: Beyond kubevirt, there are other cases where people would like to use libvirt's features rather on a per-process model, instead of the centralized deamon. However, due to the ecosystem there is the vision to still have the daemon which provides a unified access to all libvirtm anaged guests, regardless of how they were launched (i.e. through systemd or kubevrit).

If it makes sense to modify those guests outside of their management platform is a different uestion. I tend to say that it will probably not work in the vast majority of cases.

The other aspect is ecosystem enablement. In that regard there is work
to add KubeVirt API level support to tools used to manage VMs (already
going on for Foreman, Vagrant, etc.).

>
> Therefore traditional tools don't know anything about VMs managed by
> KubeVirt, which to me seems counter intuitive, given the projects
> mission to bring virtualization based workloads
> closer to containers. I wasn't able to find anything in the docs
> regarding the topic (neither stating that it is possible, nor that it is
> not) and from my own testing I don't see an obvious
> way to do it.
>
> The high level question is: How is KubeVirt supposed to work in
> combination with other virtualization tools? And is it even supposed to
> do that?

We are looking for input to prioritize which "traditional tools" need
this enablement, so more feedback is welcome!
for Foreman, I think this is the relevant one (Ohad/Piotr/Moti can keep
me honest)
https://community.theforeman.org/t/plug-in-kubevirt-request-for-feedback/11484

>
> Some smaller question:
> * Is there a preferred/official way to connect to KubeVirt's libvirt-daemon?

not sure, though can you provide more details on the use case?

libvirt is an implementation detail. Aka not a publci API we ask users to use.

There is a hooks API where you can apply a filter the domxml:

https://github.com/kubevirt/kubevirt/tree/master/cmd/example-hook-sidecar

 

> * Is it even a good idea to modify guests from somewhere else, or is the
> reconciliation loop going to undo all the changes anyway? Both in terms
> of VMs existing only in libvirt and VMs
> existing in KubeVirt, but have some advanced configuration not covered
> by KubeVirt

There are ways to tackle this in KubeVirt like using hooks.
In general, no managed system would like you to do something
undercutting it's behavior.

+1

But what would you like to do if you could do this?

Aka what's your use-case? :)

> * Are you guys planning on exposing every last knob of libvirt on the
> KubeVirt API? Otherwise there isn't even the possibility to manually
> configure libvirt in case someone is running some exotic
> configuration. On the other hand, I have no idea if that is an issue in
> reality

I expect a mix of "it makes sense and keeps broad adoption/usage simple"
and "you can do that by using a hook if you really need to".
Again, mostly we are looking for feedback on needs/usage patterns.

+1

We want to go beyond just supporting a strict set of guests but otoh we don't - we can't (UX) wise - expose every knob iof the libvirt API.

However, as Itamar mentioned, that is where hooks can help - the ability to control every aspect of a domxml.

 

> * I think it would be valuable to have at least read-only access to
> libvirt, in case someone is using existing tools to react on changes in
> the environment. (I am thinking something like ManageIQ,
> although ManageIQ needs oVirt as a provider). Or someone could use
> Foreman/oVirt as dynamic inventory source, because "thats where all the
> VMs are registered"

Exactly - for more than single nodes it may be easier to integrate at
kubevrit api. When we get back to libvirt per host architecture, may be
relevant to revisit aspects there.
providing more info on what you are trying to achieve will allow to
provide more guidance.

> * Is it possible to use the libvirt-daemon on the host with KubeVirt? Is
> there an obvious game stopper I am missing?

Not sure.

It is not possible.

The key change that took place with moving from a centralized libvirt to a decentralized one, was teh fact that we gave the node control to the kubelet, in favor of benefiting from the node management.

IOW a range of features is based on K8s features, and these features areonly "accessible" in pods, as pods are K8s atomic computing unit. Providing storage, networking, guaranteed CPU, memoy/cpu qos - All this is inherited from K8s and only possible to achieve in an architecture where libvirt and the vm are living inside a pod.

It sounds liek you'd liek to integrate existnig tools with kubevirt, is this corretc, and do you mind giving some examples?

Greetings

- fabian

 

>
> Since I am new to KubeVirt I might have completely wrong expectations on
> what KubeVirt should and should not do. However I feel like a paragraph
> somewhere in the documentation about this
> topic would help people figure out how KubeVirt fits in this space
> between virtualization and containers.

That is great feedback, thank you for pointing this out.

Thanks,
    Itamar

> -Thomas
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "kubevirt-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to kubevirt-dev...@googlegroups.com
> <mailto:kubevirt-dev...@googlegroups.com>.
> To post to this group, send email to kubevi...@googlegroups.com
> <mailto:kubevi...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/kubevirt-dev/e2196f4e-7faf-4f39-b254-cf193894b65e%40googlegroups.com
> <https://groups.google.com/d/msgid/kubevirt-dev/e2196f4e-7faf-4f39-b254-cf193894b65e%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.
To post to this group, send email to kubevi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/fd1d25a9-0d2b-06f3-dbfb-418fb8fb5d92%40redhat.com.