Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory

[Prince Sarvaiya]

Hi team,

I am using "CIVO" 3rd party cloud provider to enable with my task, as it was not working with VMware ESXi, as we had to do an urgent assessment.

I am able to launch a dummy/demo VM of Cirros and access with console, but i started getting issue when i used CDI.

I took fedora.qcow2 image, and imported into PVC using Virtctl, but checking virt-launcher pod logs I get 

{"component":"virt-launcher","level":"info","msg":"Collected all requested hook sidecar sockets","pos":"manager.go:77","timestamp":"2022-09-06T13:53:52.268421Z"}
{"component":"virt-launcher","level":"info","msg":"Sorted all collected sidecar sockets per hook point based on their priority and name: map[]","pos":"manager.go:80","timestamp":"2022-09-06T13:53:52.269673Z"}
{"component":"virt-launcher","level":"info","msg":"Connecting to libvirt daemon: qemu:///system","pos":"libvirt.go:497","timestamp":"2022-09-06T13:53:52.276891Z"}
{"component":"virt-launcher","level":"info","msg":"Connecting to libvirt daemon failed: virError(Code=38, Domain=7, Message='Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory')","pos":"libvirt.go:505","timestamp":"2022-09-06T13:53:52.277787Z"}
{"component":"virt-launcher","level":"info","msg":"libvirt version: 8.0.0, package: 2.module_el8.6.0+1087+b42c8331 (CentOS Buildsys \u003...@centos.org\u003e, 2022-02-08-22:20:52, )","subcomponent":"libvirt","thread":"38","timestamp":"2022-09-06T13:53:52.299000Z"}
{"component":"virt-launcher","level":"info","msg":"hostname: vm1","subcomponent":"libvirt","thread":"38","timestamp":"2022-09-06T13:53:52.299000Z"}
{"component":"virt-launcher","level":"error","msg":"internal error: Child process (dmidecode -q -t 0,1,2,3,4,11,17) unexpected exit status 1: /dev/mem: No such file or directory","pos":"virCommandWait:2752","subcomponent":"libvirt","thread":"38","timestamp":"2022-09-06T13:53:52.299000Z"}
{"component":"virt-launcher","level":"info","msg":"Connected to libvirt daemon","pos":"libvirt.go:513","timestamp":"2022-09-06T13:53:52.781749Z"}
{"component":"virt-launcher","level":"info","msg":"Registered libvirt event notify callback","pos":"client.go:510","timestamp":"2022-09-06T13:53:52.788271Z"}
{"component":"virt-launcher","level":"info","msg":"Marked as ready","pos":"virt-launcher.go:74","timestamp":"2022-09-06T13:53:52.790094Z"}
panic: timed out waiting for domain to be defined

goroutine 1 [running]:
main.waitForDomainUUID(0xc00056d560, 0xc00056d3e0, 0xc0000912c0, {0x1d0a4e0, 0xc00009c410})
        cmd/virt-launcher/virt-launcher.go:244 +0x43a
main.main()
        cmd/virt-launcher/virt-launcher.go:475 +0x107a
{"component":"virt-launcher-monitor","level":"info","msg":"Reaped pid 11 with status 512","pos":"virt-launcher-monitor.go:125","timestamp":"2022-09-06T13:58:34.804660Z"}
{"component":"virt-launcher-monitor","level":"error","msg":"dirty virt-launcher shutdown: exit-code 2","pos":"virt-launcher-monitor.go:143","timestamp":"2022-09-06T13:58:34.806200Z"}

----

node

  default                     virt-launcher-vm1-56bxj             100m (5%)     0 (0%)      1274512512 (32%)  0 (0%)         40s
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource                       Requests          Limits
  --------                       --------          ------
  cpu                            275m (13%)        750m (37%)
  memory                         2991262592 (76%)  600M (15%)
  ephemeral-storage              50M (0%)          0 (0%)
  hugepages-1Gi                  0 (0%)            0 (0%)
  hugepages-2Mi                  0 (0%)            0 (0%)
  devices.kubevirt.io/kvm        1                 1
  devices.kubevirt.io/tun        1                 1
  devices.kubevirt.io/vhost-net  1                 1

and   kubevirt.io/schedulable=true

Can you please assist me, I am stuck since 1 week. 

So, I posted 2 queries in group 1 after another.

Regards,

Prince A. Sarvaiya

[Jed Lejosne]

Hi Prince,

The error you highlighted, "failed to connect socket", is a red herring. Libvirtd was just not ready the first time virt-launcher tried to reach it. It worked the second time.

The issue here is most likely that qemu died shortly after starting. That can happen for various reasons, and unfortunately the virt-launcher logs don't include the qemu logs by default.

More logs can be obtained by adding a "debugLogs" label to the VMI with a value of 1 and/or "logVerbosity" to a value equal or higher than 5.

However, if the cause of the issue is a crash that happened inside the guest, you still won't get much debug information. In that case, there's probably a problem with the disk image.

I hope that helps...

Regards,

Jed

[andy wang]

Hi Jed , I am using kubevirt 0.56.0.  I encoutered the same problem with Prince. According to your instruction ,I added the  " debugLogs " and " logVerbosity  " label. the detailed log info is as below:

could u please help me, to figure out what's the problem. thanks very much

{"component":"virt-launcher","level":"info","msg":"Connected to libvirt daemon","pos":"libvirt.go:513","timestamp":"2022-09-07T03:11:53.705889Z"}
{"component":"virt-launcher","level":"info","msg":"dom=(nil), (VM: name=\u003cnull\u003e, uuid=), conn=0x7f94700161b0, eventID=0, cb=0x55cad2ddc5d0, opaque=0x7f94b4002660, freecb=0x55cad2dd69b0","pos":"virConnectDomainEventRegisterAny:9465","subcomponent":"libvirt","thread":"28","timestamp":"2022-09-07T03:11:53.710000Z"}
{"component":"virt-launcher","level":"info","msg":"dom=(nil), (VM: name=\u003cnull\u003e, uuid=), conn=0x7f94700161b0, eventID=19, cb=0x55cad2ddaa10, opaque=0x7f94ac002ab0, freecb=0x55cad2dd69b0","pos":"virConnectDomainEventRegisterAny:9465","subcomponent":"libvirt","thread":"30","timestamp":"2022-09-07T03:11:53.710000Z"}
{"component":"virt-launcher","level":"info","msg":"dom=(nil), (VM: name=\u003cnull\u003e, uuid=), conn=0x7f94700161b0, eventID=15, cb=0x55cad2ddaec0, opaque=0x7f94b0005c30, freecb=0x55cad2dd69b0","pos":"virConnectDomainEventRegisterAny:9465","subcomponent":"libvirt","thread":"32","timestamp":"2022-09-07T03:11:53.711000Z"}
{"component":"virt-launcher","level":"info","msg":"dom=(nil), (VM: name=\u003cnull\u003e, uuid=), conn=0x7f94700161b0, eventID=18, cb=0x55cad2ddab20, opaque=0x7f94a4003f30, freecb=0x55cad2dd69b0","pos":"virConnectDomainEventRegisterAny:9465","subcomponent":"libvirt","thread":"33","timestamp":"2022-09-07T03:11:53.711000Z"}
{"component":"virt-launcher","level":"info","msg":"Registered libvirt event notify callback","pos":"client.go:510","timestamp":"2022-09-07T03:11:53.711420Z"}
{"component":"virt-launcher","level":"info","msg":"Marked as ready","pos":"virt-launcher.go:74","timestamp":"2022-09-07T03:11:53.711790Z"}
{"component":"virt-launcher","level":"info","msg":"Periodically checking for domain with UUID","pos":"virt-launcher.go:246","timestamp":"2022-09-07T03:12:03.712320Z"}
{"component":"virt-launcher","level":"info","msg":"conn=0x7f94700161b0, domains=0x7f94b98a4920, flags=0x3","pos":"virConnectListAllDomains:6669","subcomponent":"libvirt","thread":"35","timestamp":"2022-09-07T03:12:03.712000Z"}
{"component":"virt-launcher","level":"info","msg":"Periodically checking for domain with UUID","pos":"virt-launcher.go:246","timestamp":"2022-09-07T03:12:13.712299Z"}
{"component":"virt-launcher","level":"info","msg":"conn=0x7f94700161b0, domains=0x7f94bb8a8920, flags=0x3","pos":"virConnectListAllDomains:6669","subcomponent":"libvirt","thread":"28","timestamp":"2022-09-07T03:12:13.712000Z"}
{"component":"virt-launcher","level":"info","msg":"Periodically checking for domain with UUID","pos":"virt-launcher.go:246","timestamp":"2022-09-07T03:12:23.712718Z"}
{"component":"virt-launcher","level":"info","msg":"conn=0x7f94700161b0, domains=0x7f94bb0a7920, flags=0x3","pos":"virConnectListAllDomains:6669","subcomponent":"libvirt","thread":"30","timestamp":"2022-09-07T03:12:23.713000Z"}
{"component":"virt-launcher","level":"info","msg":"Periodically checking for domain with UUID","pos":"virt-launcher.go:246","timestamp":"2022-09-07T03:12:33.713004Z"}
{"component":"virt-launcher","level":"info","msg":"conn=0x7f94700161b0, domains=0x7f949fffe920, flags=0x3","pos":"virConnectListAllDomains:6669","subcomponent":"libvirt","thread":"38","timestamp":"2022-09-07T03:12:33.713000Z"}
{"component":"virt-launcher","level":"info","msg":"Periodically checking for domain with UUID","pos":"virt-launcher.go:246","timestamp":"2022-09-07T03:12:43.712120Z"}
{"component":"virt-launcher","level":"info","msg":"conn=0x7f94700161b0, domains=0x7f949effc920, flags=0x3","pos":"virConnectListAllDomains:6669","subcomponent":"libvirt","thread":"40","timestamp":"2022-09-07T03:12:43.712000Z"}
{"component":"virt-launcher","level":"info","msg":"Periodically checking for domain with UUID","pos":"virt-launcher.go:246","timestamp":"2022-09-07T03:12:53.712153Z"}
{"component":"virt-launcher","level":"info","msg":"conn=0x7f94700161b0, domains=0x7f94b98a4920, flags=0x3","pos":"virConnectListAllDomains:6669","subcomponent":"libvirt","thread":"35","timestamp":"2022-09-07T03:12:53.712000Z"}
{"component":"virt-launcher","level":"info","msg":"Periodically checking for domain with UUID","pos":"virt-launcher.go:246","timestamp":"2022-09-07T03:13:03.712099Z"}
{"component":"virt-launcher","level":"info","msg":"conn=0x7f94700161b0, domains=0x7f94b90a3920, flags=0x3","pos":"virConnectListAllDomains:6669","subcomponent":"libvirt","thread":"36","timestamp":"2022-09-07T03:13:03.712000Z"}
{"component":"virt-launcher","level":"info","msg":"Periodically checking for domain with UUID","pos":"virt-launcher.go:246","timestamp":"2022-09-07T03:13:13.712012Z"}
{"component":"virt-launcher","level":"info","msg":"conn=0x7f94700161b0, domains=0x7f94bb0a7920, flags=0x3","pos":"virConnectListAllDomains:6669","subcomponent":"libvirt","thread":"30","timestamp":"2022-09-07T03:13:13.712000Z"}
{"component":"virt-launcher","level":"info","msg":"Periodically checking for domain with UUID","pos":"virt-launcher.go:246","timestamp":"2022-09-07T03:13:23.712804Z"}
{"component":"virt-launcher","level":"info","msg":"conn=0x7f94700161b0, domains=0x7f949fffe920, flags=0x3","pos":"virConnectListAllDomains:6669","subcomponent":"libvirt","thread":"38","timestamp":"2022-09-07T03:13:23.713000Z"}
{"component":"virt-launcher","level":"info","msg":"Periodically checking for domain with UUID","pos":"virt-launcher.go:246","timestamp":"2022-09-07T03:13:33.712958Z"}
{"component":"virt-launcher","level":"info","msg":"conn=0x7f94700161b0, domains=0x7f94ba0a5920, flags=0x3","pos":"virConnectListAllDomains:6669","subcomponent":"libvirt","thread":"33","timestamp":"2022-09-07T03:13:33.713000Z"}
{"component":"virt-launcher","level":"info","msg":"Periodically checking for domain with UUID","pos":"virt-launcher.go:246","timestamp":"2022-09-07T03:13:43.711973Z"}
{"component":"virt-launcher","level":"info","msg":"conn=0x7f94700161b0, domains=0x7f94b98a4920, flags=0x3","pos":"virConnectListAllDomains:6669","subcomponent":"libvirt","thread":"35","timestamp":"2022-09-07T03:13:43.712000Z"}
{"component":"virt-launcher","level":"info","msg":"Periodically checking for domain with UUID","pos":"virt-launcher.go:246","timestamp":"2022-09-07T03:13:53.712245Z"}
{"component":"virt-launcher","level":"info","msg":"conn=0x7f94700161b0, domains=0x7f94b90a3920, flags=0x3","pos":"virConnectListAllDomains:6669","subcomponent":"libvirt","thread":"36","timestamp":"2022-09-07T03:13:53.712000Z"}
{"component":"virt-launcher","level":"info","msg":"Periodically checking for domain with UUID","pos":"virt-launcher.go:246","timestamp":"2022-09-07T03:14:03.712594Z"}
{"component":"virt-launcher","level":"info","msg":"conn=0x7f94700161b0, domains=0x7f94bb0a7920, flags=0x3","pos":"virConnectListAllDomains:6669","subcomponent":"libvirt","thread":"30","timestamp":"2022-09-07T03:14:03.713000Z"}

[Luboslav Pivarc]

Hi,

It's a long time since you encountered this problem, hopefully, you got this working. If this is not the case I would suggest having a look into virt-handler logs and I suspect you could find error there.

Regards,

Lubo

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/66496e99-b106-4414-9380-c2a483a8051dn%40googlegroups.com.