virt-launcher panic with EFI bootloader

Pramodh.Pallapothu

unread,

Oct 31, 2025, 8:15:04 PMOct 31

to kubevirt-dev

Hi,

When I start Windows and Ubuntu VMs with EFI bootloader. Apps run fine for few hours and then we see panic in virt-launcher pods and they get restarted and in the process VMs get restarted too.

The backend storage class is longhorn.

longhorn version: v1.9.1

kubevirt version: v1.6.0

k3s version : v1.33.3+k3s1

Sample log from virt-launcher pod when this issue occurs. Has anyone seen this issue and any suggestions to recover from this ?

{"component":"virt-launcher","kind":"","level":"info","msg":"Synced vmi","name":"win-fml-18099xsc2r","namespace":"eve-kube-app","pos":"server.go:208","timestamp":"2025-10-31T23:59:22.618706Z","uid":"67b33e26-d398-45ed-a22c-b28ed6d9458a"}

{"component":"virt-launcher","level":"info","msg":"No DRA GPU devices found for vmi eve-kube-app/win-fml-18099xsc2r","pos":"gpu_hostdev.go:42","timestamp":"2025-10-31T23:59:37.141641Z"}

{"component":"virt-launcher","kind":"","level":"info","msg":"Synced vmi","name":"win-fml-18099xsc2r","namespace":"eve-kube-app","pos":"server.go:208","timestamp":"2025-10-31T23:59:37.143659Z","uid":"67b33e26-d398-45ed-a22c-b28ed6d9458a"}

{"component":"virt-launcher","level":"info","msg":"No DRA GPU devices found for vmi eve-kube-app/win-fml-18099xsc2r","pos":"gpu_hostdev.go:42","timestamp":"2025-10-31T23:59:41.431008Z"}

{"component":"virt-launcher","kind":"","level":"info","msg":"Synced vmi","name":"win-fml-18099xsc2r","namespace":"eve-kube-app","pos":"server.go:208","timestamp":"2025-10-31T23:59:41.433024Z","uid":"67b33e26-d398-45ed-a22c-b28ed6d9458a"}

{"component":"virt-launcher","level":"warning","msg":"Cannot start job (query, none, none) in API remoteDispatchConnectGetAllDomainStats for domain eve-kube-app_win-fml-18099xsc2r; current job is (query, none, none) owned by (28 remoteDispatchConnectGetAllDomainStats, 0 <null>, 0 <null> (flags=0x0)) for (35s, 0s, 0s)","pos":"virDomainObjBeginJobInternal:434","subcomponent":"libvirt","thread":"29","timestamp":"2025-10-31T23:59:52.173000Z"}

{"component":"virt-launcher","level":"error","msg":"Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchConnectGetAllDomainStats)","pos":"virDomainObjBeginJobInternal:468","subcomponent":"libvirt","thread":"29","timestamp":"2025-10-31T23:59:52.173000Z"}

panic: runtime error: invalid memory address or nil pointer dereference

[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x18cda04]

{"component":"virt-launcher-monitor","level":"info","msg":"Reaped Launcher main pid","pos":"virt-launcher-monitor.go:128","timestamp":"2025-11-01T00:00:02.670887Z"}

{"component":"virt-launcher-monitor","level":"info","msg":"Reaped pid 13 with status 512","pos":"virt-launcher-monitor.go:131","timestamp":"2025-11-01T00:00:02.671000Z"}

{"component":"virt-launcher-monitor","level":"error","msg":"dirty virt-launcher shutdown: exit-code 2","pos":"virt-launcher-monitor.go:145","timestamp":"2025-11-01T00:00:02.671042Z"}

{"component":"virt-launcher-monitor","level":"info","msg":"Killing QEMU gracefully.","pos":"virt-launcher-monitor.go:162","timestamp":"2025-11-01T00:00:02.676365Z"}

{"component":"virt-launcher-monitor","level":"info","msg":"Reaped pid 74 with status 0","pos":"virt-launcher-monitor.go:131","timestamp":"2025-11-01T00:00:02.991805Z"}

{"component":"virt-launcher-monitor","level":"info","msg":"virt-launcher-monitor: Exiting...","pos":"virt-launcher-monitor.go:91","timestamp":"2025-11-01T00:00:11.482144Z"}

Thanks,

Pramodh

Luboslav Pivarc

unread,

Nov 1, 2025, 12:45:56 PMNov 1

to Pramodh.Pallapothu, Edward Haas, Orel Misan, kubevirt-dev

Hi

On Sat, Nov 1, 2025 at 1:15 AM Pramodh.Pallapothu <pramodh.p...@gmail.com> wrote:

Hi,

When I start Windows and Ubuntu VMs with EFI bootloader. Apps run fine for few hours and then we see panic in virt-launcher pods and they get restarted and in the process VMs get restarted too.

The backend storage class is longhorn.
longhorn version: v1.9.1
kubevirt version: v1.6.0

k3s version : v1.33.3+k3s1

Sample log from virt-launcher pod when this issue occurs. Has anyone seen this issue and any suggestions to recover from this ?

{"component":"virt-launcher","kind":"","level":"info","msg":"Synced vmi","name":"win-fml-18099xsc2r","namespace":"eve-kube-app","pos":"server.go:208","timestamp":"2025-10-31T23:59:22.618706Z","uid":"67b33e26-d398-45ed-a22c-b28ed6d9458a"}

{"component":"virt-launcher","level":"info","msg":"No DRA GPU devices found for vmi eve-kube-app/win-fml-18099xsc2r","pos":"gpu_hostdev.go:42","timestamp":"2025-10-31T23:59:37.141641Z"}

{"component":"virt-launcher","kind":"","level":"info","msg":"Synced vmi","name":"win-fml-18099xsc2r","namespace":"eve-kube-app","pos":"server.go:208","timestamp":"2025-10-31T23:59:37.143659Z","uid":"67b33e26-d398-45ed-a22c-b28ed6d9458a"}

{"component":"virt-launcher","level":"info","msg":"No DRA GPU devices found for vmi eve-kube-app/win-fml-18099xsc2r","pos":"gpu_hostdev.go:42","timestamp":"2025-10-31T23:59:41.431008Z"}

{"component":"virt-launcher","kind":"","level":"info","msg":"Synced vmi","name":"win-fml-18099xsc2r","namespace":"eve-kube-app","pos":"server.go:208","timestamp":"2025-10-31T23:59:41.433024Z","uid":"67b33e26-d398-45ed-a22c-b28ed6d9458a"}

{"component":"virt-launcher","level":"warning","msg":"Cannot start job (query, none, none) in API remoteDispatchConnectGetAllDomainStats for domain eve-kube-app_win-fml-18099xsc2r; current job is (query, none, none) owned by (28 remoteDispatchConnectGetAllDomainStats, 0 <null>, 0 <null> (flags=0x0)) for (35s, 0s, 0s)","pos":"virDomainObjBeginJobInternal:434","subcomponent":"libvirt","thread":"29","timestamp":"2025-10-31T23:59:52.173000Z"}

{"component":"virt-launcher","level":"error","msg":"Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchConnectGetAllDomainStats)","pos":"virDomainObjBeginJobInternal:468","subcomponent":"libvirt","thread":"29","timestamp":"2025-10-31T23:59:52.173000Z"}

panic: runtime error: invalid memory address or nil pointer dereference

[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x18cda04]

@Edward Haas @Orel Misan

This seems to be related to netlink,
github.com/vishvananda/netlink.deserializeRoute
kubevirt/vendor/github.com/vishvananda/netlink/route_linux.go:1441

Can you have a look?

-Lubo

{"component":"virt-launcher-monitor","level":"info","msg":"Reaped Launcher main pid","pos":"virt-launcher-monitor.go:128","timestamp":"2025-11-01T00:00:02.670887Z"}

{"component":"virt-launcher-monitor","level":"info","msg":"Reaped pid 13 with status 512","pos":"virt-launcher-monitor.go:131","timestamp":"2025-11-01T00:00:02.671000Z"}

{"component":"virt-launcher-monitor","level":"error","msg":"dirty virt-launcher shutdown: exit-code 2","pos":"virt-launcher-monitor.go:145","timestamp":"2025-11-01T00:00:02.671042Z"}

{"component":"virt-launcher-monitor","level":"info","msg":"Killing QEMU gracefully.","pos":"virt-launcher-monitor.go:162","timestamp":"2025-11-01T00:00:02.676365Z"}

{"component":"virt-launcher-monitor","level":"info","msg":"Reaped pid 74 with status 0","pos":"virt-launcher-monitor.go:131","timestamp":"2025-11-01T00:00:02.991805Z"}

{"component":"virt-launcher-monitor","level":"info","msg":"virt-launcher-monitor: Exiting...","pos":"virt-launcher-monitor.go:91","timestamp":"2025-11-01T00:00:11.482144Z"}

Thanks,
Pramodh

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/kubevirt-dev/CAFXUsz%2BeLr32_2SKrO%3DtebD5oWDYvoMDZHAyua0ZNePV%3DLMERg%40mail.gmail.com.

Orel Misan

unread,

Nov 2, 2025, 8:42:13 AMNov 2

to Luboslav Pivarc, Pramodh.Pallapothu, Edward Haas, kubevirt-dev

Hi Lubo,

Could you elaborate on why you suspect netlink might be involved?
Thanks,

Orel

Luboslav Pivarc

unread,

Nov 3, 2025, 9:08:14 AMNov 3

to Orel Misan, Pramodh.Pallapothu, Edward Haas, kubevirt-dev

Hi Orel,

This is what you get from addr2line:
"""
github.com/vishvananda/netlink.deserializeRoute
kubevirt/vendor/github.com/vishvananda/netlink/route_linux.go:1441
"""

-Lubo

To view this discussion visit https://groups.google.com/d/msgid/kubevirt-dev/CALzEmMcNb06Ryufn1aeW3pXhW6ETGoN%2BaHqVNdPNVCeMFRGBPg%40mail.gmail.com.

Orel Misan

unread,

Nov 9, 2025, 7:17:03 AMNov 9

to Luboslav Pivarc, Pramodh.Pallapothu, Edward Haas, kubevirt-dev

Hi Lubo,

Apologies for the delayed response, I was out of office for the majority of last week.

Could you please provide additional context on where you suspect the failure occurs (e.g., in virt-launcher or libvirt)?

Thanks,

Orel

Or Shoval

unread,

Nov 17, 2025, 8:37:00 AM (13 days ago) Nov 17

to kubevirt-dev

Hi

If we take the binary from v1.6.0, it points to "GetDomainDirtyRate"

$ echo 0x18cda04 | go tool addr2line /tmp/virt-launcher-v1.6.0 kubevirt.io/kubevirt/pkg/virt-launcher/virtwrap/cli.(*LibvirtConnection).GetDomainDirtyRate.func2 pkg/virt-launcher/virtwrap/cli/libvirt.go:397

We can see that there were some errors indeed about "GetAllDomainStats" area by libvirt (can't start job and cant acquire state change lock etc).

If we look on libvirt, we can see that there might be entries that aren't empty but with nil fields, for example DirtyRate
https://github.com/kubevirt/kubevirt/blob/d53baa7f84687d156bd8a3a80d7ea2fbd8e0a993/vendor/libvirt.org/go/libvirt/connect.go#L3623
it will be nil, in a non nil domStats entry if unpack returned count == 0

If we get back to Kubevirt's GetDomainDirtyRate, we can see it didn't check for nil on 1.6.0
but it got fixed on 1.7.0
https://github.com/kubevirt/kubevirt/pull/15547

Note, this is as close as i can get from the logs, it might be not accurate, but seems related.

Thanks

Or Shoval

unread,

Nov 17, 2025, 8:43:03 AM (13 days ago) Nov 17

to kubevirt-dev

On Monday, November 17, 2025 at 3:37:00 PM UTC+2 Or Shoval wrote:

Hi

If we take the binary from v1.6.0, it points to "GetDomainDirtyRate"

$ echo 0x18cda04 | go tool addr2line /tmp/virt-launcher-v1.6.0 kubevirt.io/kubevirt/pkg/virt-launcher/virtwrap/cli.(*LibvirtConnection).GetDomainDirtyRate.func2 pkg/virt-launcher/virtwrap/cli/libvirt.go:397

We can see that there were some errors indeed about "GetAllDomainStats" area by libvirt (can't start job and cant acquire state change lock etc).

If we look on libvirt, we can see that there might be entries that aren't empty but with nil fields, for example DirtyRate
https://github.com/kubevirt/kubevirt/blob/d53baa7f84687d156bd8a3a80d7ea2fbd8e0a993/vendor/libvirt.org/go/libvirt/connect.go#L3623
it will be nil, in a non nil domStats entry if unpack returned count == 0

If we get back to Kubevirt's GetDomainDirtyRate, we can see it didn't check for nil on 1.6.0
but it got fixed on 1.7.0
https://github.com/kubevirt/kubevirt/pull/15547

and was backported to 1.6.z

https://github.com/kubevirt/kubevirt/pull/15549

Or Shoval

unread,

Nov 19, 2025, 6:38:12 AM (11 days ago) Nov 19

to kubevirt-dev

Hi Pramodh,

Can you please use Kubevirt v.1.6.3 [1] ?

It should be fixed there.

If you can, please update us in case it returns, or if it looks better.

Thank you

[1] https://github.com/kubevirt/kubevirt/releases/tag/v1.6.3

Pramodh.Pallapothu

unread,

Nov 19, 2025, 1:59:31 PM (11 days ago) Nov 19

to Or Shoval, kubevirt-dev

Sure I will test and update it here. Is that a stable release ?

Thanks,

Pramodh

To view this discussion visit https://groups.google.com/d/msgid/kubevirt-dev/0fde926e-533d-4363-9226-991d93c45ae0n%40googlegroups.com.

Or Shoval

unread,

Nov 20, 2025, 1:18:45 AM (10 days ago) Nov 20

to kubevirt-dev

On Wednesday, November 19, 2025 at 8:59:31 PM UTC+2 Pramodh.Pallapothu wrote:

Sure I will test and update it here. Is that a stable release ?

Yes,

It is the one that you get atm with

$ curl -L https://storage.googleapis.com/kubevirt-prow/release/kubevirt/kubevirt/stable.txt

Thank you

Reply all

Reply to author

Forward