Unable to install qemu-kvm and related packages with CoreOS

[Laurent Le Moux]

Hi all,

I use a Docker container with 4.9.16-coreos-r1 on a DigitalOcean host.
Virtualization is available as shown below :
Model name: Intel(R) Xeon(R) CPU E5-2650L v3 @ 1.80GHz
Virtualization: VT-x
Hypervisor vendor: KVM
Virtualization type: full

I issue : apt-get install qemu-kvm libvirt-bin virtinst bridge-utils
At some point, the installation complains :
Configuring kvm qemu-kvm
modprobe: ERROR: ../libkmod/libkmod.c:586 kmodsearchmoddep() could not open moddep file '/lib/modules/4.9.16-coreos-r1/modules.dep.bin'
modprobe: FATAL: Module kvm_intel not found in directory /lib/modules/4.9.16-coreos-r1
...done.

Surprisingly, there is actually no '/lib/modules/4.9.16-coreos-r1' directory...
Nor does '/lib/modules' exist...
Is this specific to a CoreOS container ?

From what I read in several posts, I need to install Linux headers. Then, the missing directory will be created and it will be possible to get 'kvm-intel.ko' module properly installed in a subdirectory and loaded.

But, when I issue : apt-get install linux-headers-'uname -r'
I get the following error :
Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package linux-headers-4.9.16-coreos-r1
E: Couldn't find any package by glob 'linux-headers-4.9.16-coreos-r1'
E: Couldn't find any package by regex 'linux-headers-4.9.16-coreos-r1'

Why are the headers not available ? How can I get them ?

I tried to use another kernel by following this tutorial :
https://www.digitalocean.com/community/tutorials/how-to-update-a-digitalocean-server-s-kernel

Then, 'kvm-intel.ko' is avalaible but can not be loaded.
Using 'insmod -f' fails with an 'invalid module format'.

I even tried to to execute the new kernel without reboot using 'kexec' command.
But it also fails with an “invalid argument” error.

I am not an expert at all. And I am stuck for days now.
I would really appreciate if somebody could help me figure out what is wrong...

Kind regards,

Laurent Le Moux

[paul...@coreos.com]

Hi Laurent, are you running CoreOS _inside_ a container? Or is CoreOS the host OS? If the latter, then I have two thoughts.

Googling, I found a thread from 2015 where a dev answered a similar question:

"You can provide qemu, libvirt, or any other vm management tools you 

need in containers. The only extra bit you need is to give the 

container running qemu access to the /dev/kvm device."

In the same thread, someone else linked to a docker image they'd built following the advice. (Warning: now stale).

https://github.com/Ulexus/docker-qemu

[David Michael]

On Fri, Apr 21, 2017 at 5:24 AM, Laurent Le Moux
<laurent...@gmail.com> wrote:
> Hi all,
>
> I use a Docker container with 4.9.16-coreos-r1 on a DigitalOcean host.
> Virtualization is available as shown below :
> Model name: Intel(R) Xeon(R) CPU E5-2650L v3 @ 1.80GHz
> Virtualization: VT-x
> Hypervisor vendor: KVM
> Virtualization type: full
>
> I issue : apt-get install qemu-kvm libvirt-bin virtinst bridge-utils
> At some point, the installation complains :
> Configuring kvm qemu-kvm
> modprobe: ERROR: ../libkmod/libkmod.c:586 kmodsearchmoddep() could not open
> moddep file '/lib/modules/4.9.16-coreos-r1/modules.dep.bin'
> modprobe: FATAL: Module kvm_intel not found in directory
> /lib/modules/4.9.16-coreos-r1
> ...done.

How are you running the Docker container? It needs KVM access, so it
should be started with "docker run --device /dev/kvm". This works for
me:

docker run --device /dev/kvm --volume /usr/boot:/boot -it fedora /bin/bash -l
dnf -y install qemu-kvm
qemu-kvm -cpu host -kernel /boot/vmlinuz -curses

Thanks.

David

[Laurent Le Moux]

Hi guys,

Thanks for your answers. I am NOT running the docker instance.
I use an instance provided by Gitlab CI and DigitalOcean.
So far, I execute the following '.gitlab-ci.yml' script :

image: ubuntu

before_script:
  - lscpu # Display virtualization details
  - find / -name policy-rc.d -exec sed -ie 's/101/0/g' {} \; # Allow virtualization deamons startup later on...
  - export DEBIAN_FRONTEND=noninteractive
  - apt-get --quiet update --yes
  - apt-get --quiet install --yes qemu-kvm libvirt-bin ubuntu-vm-builder bridge-utils
  - ls /lib # No modules directory... !
  - find / -name "kvm-intel.ko" -print -exec insmod -f {} \; # Nothing found... !
  - virt-host-validate # Fails... !
...

virt-host-validate fails with the following message :
$ virt-host-validate
  QEMU: Checking for hardware virtualization                                 : PASS
  QEMU: Checking if device /dev/kvm exists                                   : PASS
  QEMU: Checking if device /dev/kvm is accessible                            : PASS
  QEMU: Checking if device /dev/vhost-net exists                             : PASS
  QEMU: Checking if device /dev/net/tun exists                               : PASS
  QEMU: Checking for cgroup 'memory' controller support                      : PASS
  QEMU: Checking for cgroup 'memory' controller mount-point                  : PASS
  QEMU: Checking for cgroup 'cpu' controller support                         : PASS
  QEMU: Checking for cgroup 'cpu' controller mount-point                     : PASS
  QEMU: Checking for cgroup 'cpuacct' controller support                     : PASS
  QEMU: Checking for cgroup 'cpuacct' controller mount-point                 : PASS
  QEMU: Checking for cgroup 'devices' controller support                     : PASS
  QEMU: Checking for cgroup 'devices' controller mount-point                 : PASS
  QEMU: Checking for cgroup 'net_cls' controller support                     : PASS
  QEMU: Checking for cgroup 'net_cls' controller mount-point                 : PASS
  QEMU: Checking for cgroup 'blkio' controller support                       : PASS
  QEMU: Checking for cgroup 'blkio' controller mount-point                   : PASS
  QEMU: Checking for device assignment IOMMU support                         : WARN (No ACPI DMAR table found, IOMMU either disabled in BIOS or not supported by this hardware platform)
   LXC: Checking for Linux >= 2.6.26                                         : PASS
   LXC: Checking for namespace ipc                                           : PASS
   LXC: Checking for namespace mnt                                           : PASS
   LXC: Checking for namespace pid                                           : PASS
   LXC: Checking for namespace uts                                           : PASS
   LXC: Checking for namespace net                                           : PASS
   LXC: Checking for namespace user                                          : PASS
   LXC: Checking for cgroup 'memory' controller support                      : PASS
   LXC: Checking for cgroup 'memory' controller mount-point                  : PASS
   LXC: Checking for cgroup 'cpu' controller support                         : PASS
   LXC: Checking for cgroup 'cpu' controller mount-point                     : PASS
   LXC: Checking for cgroup 'cpuacct' controller support                     : PASS
   LXC: Checking for cgroup 'cpuacct' controller mount-point                 : PASS
   LXC: Checking for cgroup 'devices' controller support                     : PASS
   LXC: Checking for cgroup 'devices' controller mount-point                 : PASS
   LXC: Checking for cgroup 'net_cls' controller support                     : PASS
   LXC: Checking for cgroup 'net_cls' controller mount-point                 : PASS
   LXC: Checking for cgroup 'freezer' controller support                     : PASS
   LXC: Checking for cgroup 'freezer' controller mount-point                 : PASS
ERROR: Job failed: exit code 1

If I comment out this command, the script execution continues until the creation of an Android virtual device (my use case) which fails to start :
$ /sdk/tools/emulator64-x86 -avd test -camera-back emulated -sdcard sdcard.img -no-audio -gpu off -no-boot-anim -no-window &
emulator: WARNING: Classic qemu does not support SMP. The hw.cpu.ncore option from your config file is ignored.
emulator: ERROR: x86 emulation currently requires hardware acceleration!
Please ensure KVM is properly installed and usable.
CPU acceleration status: Could not open /dev/kvm : No such device

Is there a way to specify the '--device /dev/kvm' option or something equal ?

Regards,

Laurent

[Seán C. McCord]

So you say you're not running in Docker, and your commands (apt-get) are obviously not running in Container Linux/CoreOS.  Does this have anything whatever to do with Container Linux?

The kernel modules are certainly available in Container Linux.  The following is taken from one of my nodes which runs my https://github.com/Ulexus/docker-qemu (with rkt):

$ cat /proc/version 

Linux version 4.10.9-coreos (jenkins@worker-1) (gcc version 4.9.4 (Gentoo Hardened 4.9.4 p1.0, pie-0.6.4) ) #1 SMP Wed Apr 12 22:54:16 UTC 2017

$ sudo find /usr -name kvm-intel.ko

/usr/lib64/modules/4.10.9-coreos/kernel/arch/x86/kvm/kvm-intel.ko

$ lsmod |grep kvm_intel

kvm_intel             184320  0

kvm                   479232  1 kvm_intel

--
You received this message because you are subscribed to the Google Groups "CoreOS User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to coreos-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Seán C McCord

CyCore Systems, Inc

+1 888 240 0308

PGP/GPG: http://cycoresys.com/scm.asc

[Laurent Le Moux]

Hi Seán,

Sorry for the misunderstanding.
My script is executed in a Docker instance but I have no control on it.
I can not issue a "docker run --device /dev/kvm" as suggested by David.
The docker instance is automatically run by Gitlab CI. Not by me...

And, among other things, I did try to use your 'ulexus/qemu' image that I found on Docker hub.
Unfortunately, I got the following error :
Running with gitlab-ci-multi-runner 9.0.1 (a3da309)
  on docker-auto-scale (e11ae361)
Using Docker executor with image ulexus/qemu ...
Using docker image sha256:dc4840fadbb39c787b8baa3daf67733fd0a4ed0465798d6887e65512cd3a7ad9 ID=sha256:dc4840fadbb39c787b8baa3daf67733fd0a4ed0465798d6887e65512cd3a7ad9 for predefined container...
Pulling docker image ulexus/qemu ...
Using docker image ulexus/qemu ID=sha256:9fdcc06fda2e2be5401c3c74aa97282fbac6db031acfbe6b61113244375d3bd5 for build container...
Running on runner-e11ae361-project-2840585-concurrent-0 via runner-e11ae361-machine-1493041932-62e22a83-digital-ocean-2gb...
Cloning repository...
Cloning into '/builds/llemoux/AndroidStudio'...
Checking out 8e8532b8 as master...
Skipping Git submodules setup
mknod: missing operand after '10'
Try 'mknod --help' for more information.
qemu-system-x86_64: -c: invalid option
mknod: missing operand after '10'
Try 'mknod --help' for more information.
qemu-system-x86_64: -c: invalid option

ERROR: Job failed: exit code 1

Your description says one should specify '--privileged' and some options.
But as I am not the one that launches the container...

I am not at all a specialist about LXC and there are a few things I really do not understand.
Why do I get "uname -ir = 4.9.16-coreos-r1 x86_64" when I asked for an Ubuntu image on top of my script ?

How comes I can install a bunch of available linux headers except the coreos ones if they are needed for kvm-intel.ko to be properly loaded as suggested in many forums ?

For now, I try to switch from kernel but it fails for whatever reason with : 'kexec failed: Invalid argument'.
My script is now :
image: ubuntu

before_script:
  - cat /proc/version # Linux version 4.9.16-coreos-r1 (jenkins@localhost) (gcc version 4.9.3 (Gentoo Hardened 4.9.3 p1.5, pie-0.6.4) ) #1 SMP Fri Mar 31 02:07:42 UTC 2017
  - find / -name kvm-intel.ko -print # Nothing found...

  - lscpu # Display virtualization details
  - find / -name policy-rc.d -exec sed -ie 's/101/0/g' {} \; # Allow virtualization deamons startup later on...
  - export DEBIAN_FRONTEND=noninteractive
  - apt-get --quiet update --yes

  - apt-get --quiet install linux-virtual --yes # Install 4.4.0-72-generic image & headers
  - apt-get --quiet install kexec-tools --yes
  - kexec -l /boot/vmlinuz-4.4.0-72-generic --initrd=/boot/initrd.img-4.4.0-72-generic
  - kexec -e # Failing here with 'Invalid argument'...

  - apt-get --quiet install --yes qemu-kvm libvirt-bin ubuntu-vm-builder bridge-utils

  - virt-host-validate
...

Regards,

Laurent

[Seán C. McCord]

That helps to clear things up, then, yes.  If you don't have any control of the launch of the container or the environment in which that occurs, I'm not sure there is much that you can do.   At least two things are critical here that you don't have access to:

  a) `/dev/kvm` in the container

  b) escalated privileges within the container

The first could be solved within the container if you had the second, but the first is not even _sufficient_ without the second.  That is, even if you made `/dev/kvm` available to the container, you likely would not have access rights to it.

The kernel module is a triviality, really.  Container Linux ships with it, so that can be loaded simply from outside the container.

[Laurent Le Moux]

Hi Sean,

Thank you for the clarification.
I will edit the issue I opened in "gitlab ci multi runner" project to ask whether the team could load the virtualization packages by default...

Kind regards,

Laurent

[Sibabalwe Mvelo]

@Laurent 

How did you manage to fix your gitlab issue?