virt-launcher panic with EFI bootloader

[Pramodh.Pallapothu]

Hi,

When I start Windows and Ubuntu VMs with EFI bootloader. Apps run fine for few hours and then we see panic in virt-launcher pods and they get restarted and in the process VMs get restarted too.

The backend storage class is longhorn.

longhorn version: v1.9.1

kubevirt version: v1.6.0

k3s version : v1.33.3+k3s1

Sample  log from virt-launcher pod when this issue occurs. Has anyone seen this issue and any suggestions to recover from this ?

{"component":"virt-launcher","kind":"","level":"info","msg":"Synced vmi","name":"win-fml-18099xsc2r","namespace":"eve-kube-app","pos":"server.go:208","timestamp":"2025-10-31T23:59:22.618706Z","uid":"67b33e26-d398-45ed-a22c-b28ed6d9458a"}

{"component":"virt-launcher","level":"info","msg":"No DRA GPU devices found for vmi eve-kube-app/win-fml-18099xsc2r","pos":"gpu_hostdev.go:42","timestamp":"2025-10-31T23:59:37.141641Z"}

{"component":"virt-launcher","kind":"","level":"info","msg":"Synced vmi","name":"win-fml-18099xsc2r","namespace":"eve-kube-app","pos":"server.go:208","timestamp":"2025-10-31T23:59:37.143659Z","uid":"67b33e26-d398-45ed-a22c-b28ed6d9458a"}

{"component":"virt-launcher","level":"info","msg":"No DRA GPU devices found for vmi eve-kube-app/win-fml-18099xsc2r","pos":"gpu_hostdev.go:42","timestamp":"2025-10-31T23:59:41.431008Z"}

{"component":"virt-launcher","kind":"","level":"info","msg":"Synced vmi","name":"win-fml-18099xsc2r","namespace":"eve-kube-app","pos":"server.go:208","timestamp":"2025-10-31T23:59:41.433024Z","uid":"67b33e26-d398-45ed-a22c-b28ed6d9458a"}

{"component":"virt-launcher","level":"warning","msg":"Cannot start job (query, none, none) in API remoteDispatchConnectGetAllDomainStats for domain eve-kube-app_win-fml-18099xsc2r; current job is (query, none, none) owned by (28 remoteDispatchConnectGetAllDomainStats, 0 <null>, 0 <null> (flags=0x0)) for (35s, 0s, 0s)","pos":"virDomainObjBeginJobInternal:434","subcomponent":"libvirt","thread":"29","timestamp":"2025-10-31T23:59:52.173000Z"}

{"component":"virt-launcher","level":"error","msg":"Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchConnectGetAllDomainStats)","pos":"virDomainObjBeginJobInternal:468","subcomponent":"libvirt","thread":"29","timestamp":"2025-10-31T23:59:52.173000Z"}

panic: runtime error: invalid memory address or nil pointer dereference

[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x18cda04]

{"component":"virt-launcher-monitor","level":"info","msg":"Reaped Launcher main pid","pos":"virt-launcher-monitor.go:128","timestamp":"2025-11-01T00:00:02.670887Z"}

{"component":"virt-launcher-monitor","level":"info","msg":"Reaped pid 13 with status 512","pos":"virt-launcher-monitor.go:131","timestamp":"2025-11-01T00:00:02.671000Z"}

{"component":"virt-launcher-monitor","level":"error","msg":"dirty virt-launcher shutdown: exit-code 2","pos":"virt-launcher-monitor.go:145","timestamp":"2025-11-01T00:00:02.671042Z"}

{"component":"virt-launcher-monitor","level":"info","msg":"Killing QEMU gracefully.","pos":"virt-launcher-monitor.go:162","timestamp":"2025-11-01T00:00:02.676365Z"}

{"component":"virt-launcher-monitor","level":"info","msg":"Reaped pid 74 with status 0","pos":"virt-launcher-monitor.go:131","timestamp":"2025-11-01T00:00:02.991805Z"}

{"component":"virt-launcher-monitor","level":"info","msg":"virt-launcher-monitor: Exiting...","pos":"virt-launcher-monitor.go:91","timestamp":"2025-11-01T00:00:11.482144Z"}

Thanks,

Pramodh

[Luboslav Pivarc]

Hi

On Sat, Nov 1, 2025 at 1:15 AM Pramodh.Pallapothu <pramodh.p...@gmail.com> wrote:

Hi,

When I start Windows and Ubuntu VMs with EFI bootloader. Apps run fine for few hours and then we see panic in virt-launcher pods and they get restarted and in the process VMs get restarted too.

The backend storage class is longhorn.

longhorn version: v1.9.1

kubevirt version: v1.6.0

k3s version : v1.33.3+k3s1

Sample  log from virt-launcher pod when this issue occurs. Has anyone seen this issue and any suggestions to recover from this ?

{"component":"virt-launcher","kind":"","level":"info","msg":"Synced vmi","name":"win-fml-18099xsc2r","namespace":"eve-kube-app","pos":"server.go:208","timestamp":"2025-10-31T23:59:22.618706Z","uid":"67b33e26-d398-45ed-a22c-b28ed6d9458a"}

{"component":"virt-launcher","level":"info","msg":"No DRA GPU devices found for vmi eve-kube-app/win-fml-18099xsc2r","pos":"gpu_hostdev.go:42","timestamp":"2025-10-31T23:59:37.141641Z"}

{"component":"virt-launcher","kind":"","level":"info","msg":"Synced vmi","name":"win-fml-18099xsc2r","namespace":"eve-kube-app","pos":"server.go:208","timestamp":"2025-10-31T23:59:37.143659Z","uid":"67b33e26-d398-45ed-a22c-b28ed6d9458a"}

{"component":"virt-launcher","level":"info","msg":"No DRA GPU devices found for vmi eve-kube-app/win-fml-18099xsc2r","pos":"gpu_hostdev.go:42","timestamp":"2025-10-31T23:59:41.431008Z"}

{"component":"virt-launcher","kind":"","level":"info","msg":"Synced vmi","name":"win-fml-18099xsc2r","namespace":"eve-kube-app","pos":"server.go:208","timestamp":"2025-10-31T23:59:41.433024Z","uid":"67b33e26-d398-45ed-a22c-b28ed6d9458a"}

{"component":"virt-launcher","level":"warning","msg":"Cannot start job (query, none, none) in API remoteDispatchConnectGetAllDomainStats for domain eve-kube-app_win-fml-18099xsc2r; current job is (query, none, none) owned by (28 remoteDispatchConnectGetAllDomainStats, 0 <null>, 0 <null> (flags=0x0)) for (35s, 0s, 0s)","pos":"virDomainObjBeginJobInternal:434","subcomponent":"libvirt","thread":"29","timestamp":"2025-10-31T23:59:52.173000Z"}

{"component":"virt-launcher","level":"error","msg":"Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchConnectGetAllDomainStats)","pos":"virDomainObjBeginJobInternal:468","subcomponent":"libvirt","thread":"29","timestamp":"2025-10-31T23:59:52.173000Z"}

panic: runtime error: invalid memory address or nil pointer dereference

[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x18cda04]

@Edward Haas @Orel Misan 

This seems to be related to netlink, 
github.com/vishvananda/netlink.deserializeRoute
kubevirt/vendor/github.com/vishvananda/netlink/route_linux.go:1441

Can you have a look?

-Lubo

 

 

{"component":"virt-launcher-monitor","level":"info","msg":"Reaped Launcher main pid","pos":"virt-launcher-monitor.go:128","timestamp":"2025-11-01T00:00:02.670887Z"}

{"component":"virt-launcher-monitor","level":"info","msg":"Reaped pid 13 with status 512","pos":"virt-launcher-monitor.go:131","timestamp":"2025-11-01T00:00:02.671000Z"}

{"component":"virt-launcher-monitor","level":"error","msg":"dirty virt-launcher shutdown: exit-code 2","pos":"virt-launcher-monitor.go:145","timestamp":"2025-11-01T00:00:02.671042Z"}

{"component":"virt-launcher-monitor","level":"info","msg":"Killing QEMU gracefully.","pos":"virt-launcher-monitor.go:162","timestamp":"2025-11-01T00:00:02.676365Z"}

{"component":"virt-launcher-monitor","level":"info","msg":"Reaped pid 74 with status 0","pos":"virt-launcher-monitor.go:131","timestamp":"2025-11-01T00:00:02.991805Z"}

{"component":"virt-launcher-monitor","level":"info","msg":"virt-launcher-monitor: Exiting...","pos":"virt-launcher-monitor.go:91","timestamp":"2025-11-01T00:00:11.482144Z"}

Thanks,

Pramodh

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/kubevirt-dev/CAFXUsz%2BeLr32_2SKrO%3DtebD5oWDYvoMDZHAyua0ZNePV%3DLMERg%40mail.gmail.com.

[Orel Misan]

Hi Lubo,

Could you elaborate on why you suspect netlink might be involved?
Thanks,

Orel

[Luboslav Pivarc]

Hi Orel,

This is what you get from addr2line:
"""
github.com/vishvananda/netlink.deserializeRoute
kubevirt/vendor/github.com/vishvananda/netlink/route_linux.go:1441
"""

-Lubo

To view this discussion visit https://groups.google.com/d/msgid/kubevirt-dev/CALzEmMcNb06Ryufn1aeW3pXhW6ETGoN%2BaHqVNdPNVCeMFRGBPg%40mail.gmail.com.