Refactor proposal: replace schema.go with libvirtxml types

[Alice Frosi]

Hi everyone,

I'd like to refresh an old topic which is still in our backlog [1] which, in my opinion, is important since from time to time it bites us. I would like to know if we truly want to switch from our scheme to libvirtxml structures.

Many reasons have been provided in the upstream issue, but let me summarize and refresh them:

1. The golang parser for the XML loses values if they aren't defined in the schema structures. We used this mechanism as a "filter" for the fields we are interested in. This causes the problem then after we wonder where we lost them. Some recent example [2]
2. It is a reliable API for the information that we get from libvirt and we already use it in multiple places
• When we need to change a field in the migration XML [3]
• For sidecars [4]
• For the node labeler [5]

The fact that we transfer the domain from the launcher to the handler and that many fields are unnecessary were among the arguments made against the use of libvirtxml. We could take advantage of the opportunity to improve the way that the launcher/libvirt and the launcher/handler communicate. A new, lighterway API might enhance the launcher-handler communication, while libvirtxml might serve as the interface with libvirt.

However, libvirtxml has a problem that needs to be fixed before KubeVirt can use it. Libvirtxml structures do not define DeepCopy and DeepEqual. To support this, some effort would be needed. This is the rationale behind bringing up the subject once more. Would we be open to replacing the schema with libvirtxml? If so, we might be able to defend extending libvirtxml to include those methods.

[1] https://github.com/kubevirt/kubevirt/issues/10844

[2] https://github.com/kubevirt/kubevirt/pull/14025

[3] https://github.com/kubevirt/kubevirt/pull/10847

[4] https://github.com/kubevirt/kubevirt/issues/11817

[5] https://github.com/kubevirt/kubevirt/pull/12652

Many thanks,

Alice

[Orel Misan]

Hi Alice,

Thank you for raising this issue again. I've raised it in the KubeVirt v1.6 unconference several weeks ago.
I think that a first step for this task is re-evaluating our communication channel between virt-handler and virt-launcher - our custom implementation of the domain is passed through it.
The custom domain informer in the virt-handler will probably also be affected.
This of course will probably be challenging and will require a VEP with an upgrade path.

Several claims were raised against using libvirtxml, among them were:
1. Increased memory consumption.
2. Logic around the PCI controllers.

I am personally in favor of using it in virt-launcher to communicate with libvirt.

Best regards,

Orel

[Vladik Romanovsky]

Hi Alice,

Thank you for this summary.
As you know, I have reservations about replacing our domain schema entirely with the libvirtxml structures.

KubeVirt keeps a minimalistic domain schema mainly because of two main reasons:

1. Memory consumption: KubeVirt periodically gets the domain XML from libvirt caches and passes it to virt-handler. The large number of additional fields would affect the virt-launcher's memory footprint (which we are trying to keep minimalistic), and would affect the virt-handler as it processes domain objects from all running VMs.
2. Feature support clarity: A full switch to libvirtxml might mislead advanced users into assuming KubeVirt supports all libvirt configurations. For example, adding a field like IO throttling would be enough to make it "just work".

However, it makes sense to use the libvirt-go-xml selectively in the places we are currently using.

That's my take on it.

Thanks,
Vladik

[Roman Mohr]

On Tue, May 6, 2025 at 8:56 AM 'Vladik Romanovsky' via kubevirt-dev <kubevi...@googlegroups.com> wrote:

Hi Alice,

Thank you for this summary.
As you know, I have reservations about replacing our domain schema entirely with the libvirtxml structures.

KubeVirt keeps a minimalistic domain schema mainly because of two main reasons:

1. Memory consumption: KubeVirt periodically gets the domain XML from libvirt caches and passes it to virt-handler. The large number of additional fields would affect the virt-launcher's memory footprint (which we are trying to keep minimalistic), and would affect the virt-handler as it processes domain objects from all running VMs.
2. Feature support clarity: A full switch to libvirtxml might mislead advanced users into assuming KubeVirt supports all libvirt configurations. For example, adding a field like IO throttling would be enough to make it "just work".

(2) was also always my take on the other recurring requests to replace it. If I add new features, adding a few fields is really not the issue, but having a very clear understanding of what we use and what we don't is really valuable imo :)

Best regards,

Roman

 

However, it makes sense to use the libvirt-go-xml selectively in the places we are currently using.

That's my take on it.

Thanks,
Vladik

On Tue, May 6, 2025 at 11:05 AM Alice Frosi <afr...@redhat.com> wrote:

Hi everyone,

I'd like to refresh an old topic which is still in our backlog [1] which, in my opinion, is important since from time to time it bites us. I would like to know if we truly want to switch from our scheme to libvirtxml structures.

Many reasons have been provided in the upstream issue, but let me summarize and refresh them:

1. The golang parser for the XML loses values if they aren't defined in the schema structures. We used this mechanism as a "filter" for the fields we are interested in. This causes the problem then after we wonder where we lost them. Some recent example [2]
2. It is a reliable API for the information that we get from libvirt and we already use it in multiple places
• When we need to change a field in the migration XML [3]
• For sidecars [4]
• For the node labeler [5]

The fact that we transfer the domain from the launcher to the handler and that many fields are unnecessary were among the arguments made against the use of libvirtxml. We could take advantage of the opportunity to improve the way that the launcher/libvirt and the launcher/handler communicate. A new, lighterway API might enhance the launcher-handler communication, while libvirtxml might serve as the interface with libvirt.

However, libvirtxml has a problem that needs to be fixed before KubeVirt can use it. Libvirtxml structures do not define DeepCopy and DeepEqual. To support this, some effort would be needed. This is the rationale behind bringing up the subject once more. Would we be open to replacing the schema with libvirtxml? If so, we might be able to defend extending libvirtxml to include those methods.

[1] https://github.com/kubevirt/kubevirt/issues/10844

[2] https://github.com/kubevirt/kubevirt/pull/14025

[3] https://github.com/kubevirt/kubevirt/pull/10847

[4] https://github.com/kubevirt/kubevirt/issues/11817

[5] https://github.com/kubevirt/kubevirt/pull/12652

Many thanks,

Alice

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/kubevirt-dev/CAOTosK%2BT%2Bj15KrcmgQWWXGGqH79TJG0KWY_WivtOmTGjfgcqzw%40mail.gmail.com.

[Roman Mohr]

On Tue, May 6, 2025 at 8:58 AM Roman Mohr <rm...@google.com> wrote:

On Tue, May 6, 2025 at 8:56 AM 'Vladik Romanovsky' via kubevirt-dev <kubevi...@googlegroups.com> wrote:

Hi Alice,

Thank you for this summary.
As you know, I have reservations about replacing our domain schema entirely with the libvirtxml structures.

KubeVirt keeps a minimalistic domain schema mainly because of two main reasons:

1. Memory consumption: KubeVirt periodically gets the domain XML from libvirt caches and passes it to virt-handler. The large number of additional fields would affect the virt-launcher's memory footprint (which we are trying to keep minimalistic), and would affect the virt-handler as it processes domain objects from all running VMs.
2. Feature support clarity: A full switch to libvirtxml might mislead advanced users into assuming KubeVirt supports all libvirt configurations. For example, adding a field like IO throttling would be enough to make it "just work".

(2) was also always my take on the other recurring requests to replace it. If I add new features, adding a few fields is really not the issue, but having a very clear understanding of what we use and what we don't is really valuable imo :)

However, after checking the attached issues, there may indeed be some benefit in replacing it. How big does the xml get?

[Vladik Romanovsky]

On Tue, May 6, 2025 at 6:03 PM Roman Mohr <rm...@google.com> wrote:

On Tue, May 6, 2025 at 8:58 AM Roman Mohr <rm...@google.com> wrote:

On Tue, May 6, 2025 at 8:56 AM 'Vladik Romanovsky' via kubevirt-dev <kubevi...@googlegroups.com> wrote:

Hi Alice,

Thank you for this summary.
As you know, I have reservations about replacing our domain schema entirely with the libvirtxml structures.

KubeVirt keeps a minimalistic domain schema mainly because of two main reasons:

1. Memory consumption: KubeVirt periodically gets the domain XML from libvirt caches and passes it to virt-handler. The large number of additional fields would affect the virt-launcher's memory footprint (which we are trying to keep minimalistic), and would affect the virt-handler as it processes domain objects from all running VMs.
2. Feature support clarity: A full switch to libvirtxml might mislead advanced users into assuming KubeVirt supports all libvirt configurations. For example, adding a field like IO throttling would be enough to make it "just work".

(2) was also always my take on the other recurring requests to replace it. If I add new features, adding a few fields is really not the issue, but having a very clear understanding of what we use and what we don't is really valuable imo :)

However, after checking the attached issues, there may indeed be some benefit in replacing it. How big does the xml get?

For example, all controllers are expanded. It can get pretty large, depending on the configuration.

One of the issues is that the expanded size is not in the hands of KubeVirt.

Before we move forward, I would like to understand what is the memory impact on the virt-launcher and virt-handler. 

[Alice Frosi]

Hi Vladik, hi Roman,

 

On Tue, May 6, 2025 at 8:56 AM 'Vladik Romanovsky' via kubevirt-dev <kubevi...@googlegroups.com> wrote:

Hi Alice,

Thank you for this summary.
As you know, I have reservations about replacing our domain schema entirely with the libvirtxml structures.

KubeVirt keeps a minimalistic domain schema mainly because of two main reasons:

1. Memory consumption: KubeVirt periodically gets the domain XML from libvirt caches and passes it to virt-handler. The large number of additional fields would affect the virt-launcher's memory footprint (which we are trying to keep minimalistic), and would affect the virt-handler as it processes domain objects from all running VMs.
2. Feature support clarity: A full switch to libvirtxml might mislead advanced users into assuming KubeVirt supports all libvirt configurations. For example, adding a field like IO throttling would be enough to make it "just work".

(2) was also always my take on the other recurring requests to replace it. If I add new features, adding a few fields is really not the issue, but having a very clear understanding of what we use and what we don't is really valuable imo :)

However, after checking the attached issues, there may indeed be some benefit in replacing it. How big does the xml get?

I don't completely understand the second point. Isn't the KubeVirt API that dictates which feature we support? If VM spec isn't translated into Libvirt field, then we don't support that option. 

Additionally, libvirt does expand and set certain fields, e.g the controllers that Vladik mentioned for example. So, the schema isn't really a complete representation of the libvirt features we support.

[Alice Frosi]

On Wed, May 7, 2025 at 12:19 AM Vladik Romanovsky <vrom...@redhat.com> wrote:

On Tue, May 6, 2025 at 6:03 PM Roman Mohr <rm...@google.com> wrote:

On Tue, May 6, 2025 at 8:58 AM Roman Mohr <rm...@google.com> wrote:

On Tue, May 6, 2025 at 8:56 AM 'Vladik Romanovsky' via kubevirt-dev <kubevi...@googlegroups.com> wrote:

Hi Alice,

Thank you for this summary.
As you know, I have reservations about replacing our domain schema entirely with the libvirtxml structures.

KubeVirt keeps a minimalistic domain schema mainly because of two main reasons:

1. Memory consumption: KubeVirt periodically gets the domain XML from libvirt caches and passes it to virt-handler. The large number of additional fields would affect the virt-launcher's memory footprint (which we are trying to keep minimalistic), and would affect the virt-handler as it processes domain objects from all running VMs.
2. Feature support clarity: A full switch to libvirtxml might mislead advanced users into assuming KubeVirt supports all libvirt configurations. For example, adding a field like IO throttling would be enough to make it "just work".

(2) was also always my take on the other recurring requests to replace it. If I add new features, adding a few fields is really not the issue, but having a very clear understanding of what we use and what we don't is really valuable imo :)

However, after checking the attached issues, there may indeed be some benefit in replacing it. How big does the xml get?

For example, all controllers are expanded. It can get pretty large, depending on the configuration.

One of the issues is that the expanded size is not in the hands of KubeVirt.

Before we move forward, I would like to understand what is the memory impact on the virt-launcher and virt-handler. 

I think here we need to distinguish between the handler and the launcher. 

As far as it regards the handler, yes, I do understand the concerns. However, we can filter out the data that are strictly necessary on the launcher side before sending them. This should be part of the evaluation of the launcher and handler communication. Having a clean API with the required fields avoids passing a fully allocated XML schema. We could, for example, start moving the schema structure as the API between the handler and the launcher. This should initially keep the same memory consumption on the handler side. Afterwards, we could drop the fields that aren't necessary there.

As far as it regards the launcher, is this a concern? I mean, we do set some memory limits on the pod. But I guess we can calculate the maximum size that a fully allocated Domain in libvirtxml can be. Would this be a sufficient metric for you? 

Please, also keep in mind that we are getting and unmarshalling the full domain in case of the migration, so we are already doing this. It has been added in [3].

 

[Adi Aloni]

Regarding the concerns about memory usage, I recall there was an effort to compare virt-handler’s memory consumption with and without libvirtxml in order to provide data that could influence a consensus on the topic[1]. At the time, however, we lacked memory metrics for virt-handler. This might be a good opportunity to revisit that analysis.

[1] https://github.com/kubevirt/kubevirt/pull/13056#issuecomment-2428948769

To view this discussion visit https://groups.google.com/d/msgid/kubevirt-dev/CA%2BNAZzwbsxJ1A465nQ2DT73VkpGa%3DNobXWOCi%2BvfVH7-CLbRVg%40mail.gmail.com.

[Victor Toso]

Hi,

Awesome discussion, sorry for joining late.

I actually think we should use libvirtxml in virt-launcher as
much as possible even if increases Pod memory. My argument is
that we rely on libvirt for everything and the mix state of using
schema.go to set libvirt and storing back what it provides in
schema.go leads to inconsistency on KubeVirt side.

Half mirroring libvirtxml leads to one more place to debug when
we have issues. I saw missing 'omitempty' tags that were a
problem for instance. I myself introduced ACPI table that should
have been an array (and will have to fix that when extending it)

https://github.com/victortoso/kubevirt/commit/47a0dfc8e340e180623eb

The memory increase is not significant imho considering the
potential benefits. The libvirtxml struct averages 20% more, for
example 119KiB (libvirtxml) against 93KiB (schema.go).

https://github.com/kubevirt/kubevirt/issues/10844#issuecomment-1889644678

The goal here should be to reduce the burden of maintaining a
separate struct.

Regarding the question of schema.go being a good way to spot what
KubeVirt supports or not, I don't think it is worth the issues
that it brings and we can potentially find another way to do
that.

As we will have to implement some filter mechanism for
cache/communication between virt-launcher and virt-handler, it
could also act as gatekeeper of valid fields we could set.

Advanced users should use the hook api and sidecars to customize
KubeVirt to what it does not yet support nor plans to support.
But even that nowadays is problematic because we will store their
changes in schema.go, see:

https://github.com/kubevirt/kubevirt/issues/11276#issuecomment-1956362690

Cheers,
Victor

[Alice Frosi]

Hi everyone,

sorry to refresh this again, but I would really love to have a clear answer on this, or at least know the path forward. What are the things to clarify, if there are any?

Where is the proper place to discuss this? The community meeting?

Many thanks,

Alice

[Vladik Romanovsky]

Hey Alice,

From my side, I would like to understand the memory usage impact on the virt-launcher and the virt-handler first. 

Once we have all the data, we can schedule an ad-hoc meeting to discuss this - as we did in the past with various proposals. 

What do you think?

Vladik

[Alice Frosi]

Hi Vladik,

On Fri, May 16, 2025 at 1:09 PM Vladik Romanovsky <vrom...@redhat.com> wrote:

Hey Alice,

From my side, I would like to understand the memory usage impact on the virt-launcher and the virt-handler first. 

This has always been a bit unclear to me.

If we keep the schema.go as the API of the domain passed between the handler and the launcher, can we drop the virt-handler evaluation? So, at first, the refactoring shouldn't have any impact on the handler.

About the launcher, what is the memory usage evaluation that you would like to see? Victor has already done some evaluation and it is available in the issue: https://github.com/kubevirt/kubevirt/issues/10844#issuecomment-1889644678

There are a couple of comments also from Andrea and Daniel. Is this enough? Or do we want to evaluate the worst case scenario?

[Finn Snape]

any news on this? thanks :)

[Victor Toso]

Hi,

On Wed, Nov 19, 2025 at 03:36:08AM -0800, 'Finn Snape' via kubevirt-dev wrote:
> any news on this? thanks :)

Thanks for the ping but curious also interest/motivation for it,
does the current codebase affects you somehow?

I have not had the time to work on this but I'm starting to get
back to improve the pkg/hook and I plan to integrate a bit better
with libvirtxml there.

Besides that, the plan/goal was to improve virt-launcher as
mentioned before here:

https://groups.google.com/g/kubevirt-dev/c/7p2-TFsa5IY/m/xEZR1S0SEwAJ

If someone has the time to pick this up, I'm happy to help with
reviews and such.

I'd say that the main issue with libvirtxml in comparison to
schema.go is that the latter supports DeepCopy() from
deepcopy-gen which is included in libvirtxml and complicates the
refactor.

Cheers,
Victor

[Victor Toso]

On Thu, Nov 20, 2025 at 10:48:47AM +0100, 'Victor Toso' via kubevirt-dev wrote:
> I'd say that the main issue with libvirtxml in comparison to
> schema.go is that the latter supports DeepCopy() from
> deepcopy-gen which is included in libvirtxml and complicates the
> refactor.

Sorry, I mean that DeepCopy() *isn't* included in libvirtxml.

[Finn Snape]

yeah we are having issues with the fact that kubevirt will strip extra XML passed into the onDefineDomain hook if it is not supported by your API schema. we would like to be able to pass whatever we like straight into libvirt.

i think this specifically may have been a regression introduced in v1.6.0 (though i can't quite pinpoint where, but it seemed to work fine for us on v1.5.1). related: https://github.com/kubevirt/kubevirt/issues/15747

[Victor Toso]

Hi,

On Thu, Nov 20, 2025 at 02:03:52AM -0800, 'Finn Snape' via kubevirt-dev wrote:
> yeah we are having issues with the fact that kubevirt will strip extra XML
> passed into the onDefineDomain hook if it is not supported by your API schema.

To be clear, it should not on the resulting XML but internally,
yes and I agree that moving virt-launcher to use libvirtxml types
might help with issues of missing fields from sidecars.

The relevant code:

https://github.com/kubevirt/kubevirt/blob/main/pkg/virt-launcher/virtwrap/util/libvirt_helper.go#L144

domainSpec returned by OnDefineDomain() is passed to
SetDomainSpecStr() as wantedSpec but what virt-launcher keeps is
the subset of fields set, that is, what is supported in
schema.go.

I'll follow-up in the issue upstream after I understand the
regression. Thanks for filing it.

Cheers,
Victor

> --
> You received this message because you are subscribed to the Google Groups
> "kubevirt-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to kubevirt-dev...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/kubevirt-dev/

> 30b7538d-ade4-4f37-926f-d9e82ca4703dn%40googlegroups.com.