VM stuck at "Booting from Hard Disk"

[uraben...@gmail.com]

Hi all,

I was following the quickstart guide here and the VM is coming up but stuck at booting. I can connect to it via 

./virtctl vnc testvm

and see the following. It is stuck in this state for about three days.

I am using Ubuntu 16.04.4 and kubernetes v1.10.4 in a VM with 10GM RAM and 10 processors.

Attached you can find the output of the following commands and the specification of the VM. I didn't change the latter compared to the quickstart tutorial. 

kubectl get -o yaml vm testvm  > kubectl-get-yaml-vm-testvm.txt
kubectl describe pod virt-launcher-testvm-x9kzn > kubectl-describe-pod-virt-launcher-testvm.txt
kubectl logs virt-launcher-testvm-x9kzn -c compute > kubectl-logs-virt-launcher-testvm.txt

Please let me know, if I can provide any further information!

Thanks for your help,

Ulrich

[Marc Sluiter]

Hi Ulrich,

I will try to reproduce your problem. How do you run k8s, with minikube?

Regards,

Marc

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev+unsubscribe@googlegroups.com.
To post to this group, send email to kubevi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/559c1022-deb3-403d-91c3-7ab0343d8232%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MARC SLUITER

SENIOR SOFTWARE ENGINEER

Red Hat GmbH

Werner-von-Siemens-Ring 14, 85630 Grasbrunn, Germany

Commercial register: Amtsgericht München, HRB 153243

mslu...@redhat.com    T: +49-211-95439679    M: +49-174-9902337    

TRIED. TESTED. TRUSTED.

@redhatjobs   redhatjobs   @redhatjobs  

[uraben...@gmail.com]

Hi Marc,

thanks for the quick reply! I have installed k8s using kubeadm but also tried using minikube. Same result for both approaches. The logs in this post are from kubernetes via kubeadm.

Regards,

Ulrich

On Monday, June 18, 2018 at 11:38:01 AM UTC+2, Marc Sluiter wrote:

Hi Ulrich,

I will try to reproduce your problem. How do you run k8s, with minikube?

Regards,

Marc

On Mon, Jun 18, 2018 at 10:15 AM, <uraben...@gmail.com> wrote:

Hi all,

I was following the quickstart guide here and the VM is coming up but stuck at booting. I can connect to it via 

./virtctl vnc testvm

and see the following. It is stuck in this state for about three days.

I am using Ubuntu 16.04.4 and kubernetes v1.10.4 in a VM with 10GM RAM and 10 processors.

Attached you can find the output of the following commands and the specification of the VM. I didn't change the latter compared to the quickstart tutorial. 

kubectl get -o yaml vm testvm  > kubectl-get-yaml-vm-testvm.txt
kubectl describe pod virt-launcher-testvm-x9kzn > kubectl-describe-pod-virt-launcher-testvm.txt
kubectl logs virt-launcher-testvm-x9kzn -c compute > kubectl-logs-virt-launcher-testvm.txt

Please let me know, if I can provide any further information!

Thanks for your help,

Ulrich

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.

To post to this group, send email to kubevi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/559c1022-deb3-403d-91c3-7ab0343d8232%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[uraben...@gmail.com]

I tried the whole thing on two other ways

1. with centos instead of Ubuntu

2. with Ubuntu but in Azure instead of using local VMs

It's been the same for both attempts... 

I am attaching my way to setup k8s on Ubuntu as well - just in case it matters.

[dvo...@redhat.com]

On Monday, June 18, 2018 at 8:13:45 AM UTC-4, uraben...@gmail.com wrote:

I tried the whole thing on two other ways

1. with centos instead of Ubuntu

This is the cirros image you're using. https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img

It would be interesting to see if you can launch the image directly with libvirt locally using something like virt-manager. If that get's you past the boot phase, please post the libvirt domain xml (virsh dumpxml <name of vm>)

[Marcus Sorensen]

On Monday, June 18, 2018 at 11:49:14 AM UTC-4, dvo...@redhat.com wrote:

On Monday, June 18, 2018 at 8:13:45 AM UTC-4, uraben...@gmail.com wrote:

I tried the whole thing on two other ways

1. with centos instead of Ubuntu

This is the cirros image you're using. https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img

It would be interesting to see if you can launch the image directly with libvirt locally using something like virt-manager. If that get's you past the boot phase, please post the libvirt domain xml (virsh dumpxml <name of vm>)

This may be a KVM vs non-KVM thing in combination with the disk type being used. I've seen VMs hang (or be unreasonably slow) at grub boot using domain type 'qemu' in the past. It may be informative to get the domain XML of the non-working VMs as well.

[Fabian Deutsch]

Good point Marcus.

Ulrich, are you actually using emulation mode?

- fabian

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/47ad0ad5-2565-43cf-9798-94fa3b64ccaf%40googlegroups.com.

[uraben...@gmail.com]

Starting the VM using virt-manager worked without any problems and with a reasonable speed. The domain xml is attached. 

How can I retrieve the domain XML of the non-working KubeVirt VM? It's not listed via virsh. 

Fabian, what exactly do you mean by emulation mode? In the domain xml, it says that it is using qemu as an emulator. Which one is virtctl using? And which one performs better according to your experience, kvm or qemu? I have nested virtualization activated for the VMs if that matters.  

Sorry for that many questions, I am not experienced in that field :-)

[ssch...@redhat.com]

inline 

On Tuesday, June 19, 2018 at 10:47:35 AM UTC+3, uraben...@gmail.com wrote:

Starting the VM using virt-manager worked without any problems and with a reasonable speed. The domain xml is attached. 

How can I retrieve the domain XML of the non-working KubeVirt VM? It's not listed via virsh. 

Please follow this procedure to retrieve the domain XML

kubectl get po

find the pod running your vm.

kubectl exec -it <Pod-Name> -c compute bash
cat /etc/libvirt/qemu/default_*

and attach the file

Thanks!

Sebastian

[uraben...@gmail.com]

Thanks for the instructions! I have attached the domain XML

[Marc Sluiter]

reposting to list...

---------- Forwarded message ---------
From: Marc Sluiter <mslu...@redhat.com>
Date: Tue, Jun 19, 2018 at 5:18 PM
Subject: Re: [kubevirt-dev] VM stuck at "Booting from Hard Disk"
To: <uraben...@gmail.com>

I have exactly the same domain.xml on my Ubuntu guest for the testvm (and both use qemu-kvm).

Can you please run virt-host-validate? Should look like this:

test@test-desktop:~$ virt-host-validate

  QEMU: Checking for hardware virtualization                                 : PASS

  QEMU: Checking if device /dev/kvm exists                                   : PASS

  QEMU: Checking if device /dev/kvm is accessible                            : PASS

  QEMU: Checking if device /dev/vhost-net exists                             : PASS

  QEMU: Checking if device /dev/net/tun exists                               : PASS

  QEMU: Checking for cgroup 'memory' controller support                      : PASS

  QEMU: Checking for cgroup 'memory' controller mount-point                  : PASS

  QEMU: Checking for cgroup 'cpu' controller support                         : PASS

  QEMU: Checking for cgroup 'cpu' controller mount-point                     : PASS

  QEMU: Checking for cgroup 'cpuacct' controller support                     : PASS

  QEMU: Checking for cgroup 'cpuacct' controller mount-point                 : PASS

  QEMU: Checking for cgroup 'devices' controller support                     : PASS

  QEMU: Checking for cgroup 'devices' controller mount-point                 : PASS

  QEMU: Checking for cgroup 'net_cls' controller support                     : PASS

  QEMU: Checking for cgroup 'net_cls' controller mount-point                 : PASS

  QEMU: Checking for cgroup 'blkio' controller support                       : PASS

  QEMU: Checking for cgroup 'blkio' controller mount-point                   : PASS

  QEMU: Checking for device assignment IOMMU support                         : WARN (No ACPI DMAR table found, IOMMU either disabled in BIOS or not supported by this hardware platform)

   LXC: Checking for Linux >= 2.6.26                                         : PASS

   LXC: Checking for namespace ipc                                           : PASS

   LXC: Checking for namespace mnt                                           : PASS

   LXC: Checking for namespace pid                                           : PASS

   LXC: Checking for namespace uts                                           : PASS

   LXC: Checking for namespace net                                           : PASS

   LXC: Checking for namespace user                                          : PASS

   LXC: Checking for cgroup 'memory' controller support                      : PASS

   LXC: Checking for cgroup 'memory' controller mount-point                  : PASS

   LXC: Checking for cgroup 'cpu' controller support                         : PASS

   LXC: Checking for cgroup 'cpu' controller mount-point                     : PASS

   LXC: Checking for cgroup 'cpuacct' controller support                     : PASS

   LXC: Checking for cgroup 'cpuacct' controller mount-point                 : PASS

   LXC: Checking for cgroup 'devices' controller support                     : PASS

   LXC: Checking for cgroup 'devices' controller mount-point                 : PASS

   LXC: Checking for cgroup 'net_cls' controller support                     : PASS

   LXC: Checking for cgroup 'net_cls' controller mount-point                 : PASS

   LXC: Checking for cgroup 'freezer' controller support                     : PASS

   LXC: Checking for cgroup 'freezer' controller mount-point                 : PASS

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/cadfee18-7478-4de5-9068-2c61fef5136a%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Ben Warren]

Is your host virtualized?  We've seen this happen using nested virtualization when the inner VM (Kubevirt workload in this case) uses virtio as the disk type.  Setting it to SCSI allows Linux to boot.  Unfortunately I've not done any debugging to figure out why this is the case.

Set the VM spec as follows:

    devices:

      disks:

      - name: mydisk

        volumeName: myvolume

        disk:

          dev: vda

          bus: scsi

--Ben

[uraben...@gmail.com]

That solved the problem, thank you very much! 

And yes, my host is virtualized. 

[uraben...@gmail.com]

I am receiving the exact same output. Ben's hint solved the problem - it must have been a disk issue. Thank you very much!

[Fabian Deutsch]

On Thu, Jun 21, 2018 at 1:54 AM 'Ben Warren' via kubevirt-dev <kubevi...@googlegroups.com> wrote:

Is your host virtualized?  We've seen this happen using nested virtualization when the inner VM (Kubevirt workload in this case) uses virtio as the disk type.  Setting it to SCSI allows Linux to boot.  Unfortunately I've not done any debugging to figure out why this is the case.

Set the VM spec as follows:

    devices:

      disks:

      - name: mydisk

        volumeName: myvolume

        disk:

          dev: vda

          bus: scsi

Thanks for pointing this out! Good to see that it now works for Ulrich.

Now, I wonder why this actually happens.

IIUIC then you, Ben, said that in a nested environment using virtio as a disk (bus) type sometimes prevents the VM from booting.

Michal, Martin, Dan, are you aware of such a behavior?

- fabian

 

--Ben

On Monday, June 18, 2018 at 1:15:13 AM UTC-7, uraben...@gmail.com wrote:

Hi all,

I was following the quickstart guide here and the VM is coming up but stuck at booting. I can connect to it via 

./virtctl vnc testvm

and see the following. It is stuck in this state for about three days.

I am using Ubuntu 16.04.4 and kubernetes v1.10.4 in a VM with 10GM RAM and 10 processors.

Attached you can find the output of the following commands and the specification of the VM. I didn't change the latter compared to the quickstart tutorial. 

kubectl get -o yaml vm testvm  > kubectl-get-yaml-vm-testvm.txt
kubectl describe pod virt-launcher-testvm-x9kzn > kubectl-describe-pod-virt-launcher-testvm.txt
kubectl logs virt-launcher-testvm-x9kzn -c compute > kubectl-logs-virt-launcher-testvm.txt

Please let me know, if I can provide any further information!

Thanks for your help,

Ulrich

--

You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.
To post to this group, send email to kubevi...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/39de88c8-977a-4437-8e20-66e17bc26acf%40googlegroups.com.

[Daniel P. Berrangé]

On Thu, Jun 21, 2018 at 11:32:47AM +0200, Fabian Deutsch wrote:
> On Thu, Jun 21, 2018 at 1:54 AM 'Ben Warren' via kubevirt-dev <
> kubevi...@googlegroups.com> wrote:
>
> > Is your host virtualized? We've seen this happen using nested
> > virtualization when the inner VM (Kubevirt workload in this case) uses
> > virtio as the disk type. Setting it to SCSI allows Linux to boot.
> > Unfortunately I've not done any debugging to figure out why this is the
> > case.
> >
> > Set the VM spec as follows:
> >
> > devices:
> >
> > disks:
> >
> > - name: mydisk
> >
> > volumeName: myvolume
> >
> > disk:
> >
> > dev: vda
> >
> > bus: scsi
> >
> >
> Thanks for pointing this out! Good to see that it now works for Ulrich.
>
> Now, I wonder why this actually happens.
> IIUIC then you, Ben, said that in a nested environment using virtio as a
> disk (bus) type sometimes prevents the VM from booting.
>
> Michal, Martin, Dan, are you aware of such a behavior?

Never heard of this problem before. It would certainly be considered a
significant bug if virtio-blk didn't work in nested. I honestly can't
think of what could conceivably even cause such a problem. It probably
would not be related to virtio-blk at all - eg more likely would be
that virtio-blk is exposing a bug in nested-KVM itself.

IOW, if anyone can produce the problem, please file appropriate bugs.

Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

[Fabian Deutsch]

Yes ... I've also never heared of this. But it seems to be easily trackable to be related to the bus attribute.

I've filed https://github.com/kubevirt/kubevirt/issues/1199 to track this.

Nelly, maybe we can reproduce this ..

- fabian

[Benjamin Warren]

I'm quite sure this is particular to kubevirt, since I (and probably everybody else here) have booted nested VMs with virtio disks many times on vanilla libvirt.  It may even be a recent regression.  I'll try to bisect today if there's time.

--Ben

[Nelly Credi]

Hi Ben,

Could you please share more information -

What do you see exactly?

Does the vm stay in scheduling or does it claim to be running?

Can you please share your vm yaml?

--

Thanks,

Nelly