nested virtualization (virtualbox>kvm)

[Guido Serra]

this is probably more a question for OSU-OSL people
who created the vagrant-ganeti project on github...

    how did u achieved nested virtualization?

I'm trying to replicate your setup starting with a gentoo vagrant box
obtained using https://github.com/d11wtq/gentoo-packer.git
and manually adding bits and pieces to have drbd and bridged net

now, I'm stuck on starting a VM, since it fails complaining about:

node1 ganeti # gnt-instance start instance1
Waiting for job 16 for instance1.v ...
Job 16 for instance1.v has failed: Failure: prerequisites not met for this operation:
Could not start instance '%s': Hypervisor error: Failed to start instance instance1.v: exited with exit code 1 (Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory
)

obviously I have no hardware acceleration available
to be provided to the nested virtual machines...

  as your ubuntu machines:

root@node1:~# kvm-ok
INFO: Your CPU does not support KVM extensions
KVM acceleration can NOT be used

so, did I missed a qemu compilation flag? or ganeti cluster conf?

mine is trying to use:

  kvm_path: /usr/bin/qemu-system-x86_64

this is not set for your machines... is it this that makes the difference?
(I found no option to drop it, I tried even to manually forge the json config files and restart ganeti daemons)

This is the version currently running...

2.11.2-r3(17:13:03 07/15/14)(drbd ipv6 kvm -haskell-daemons -htools -lxc -rbd -syslog -test -xen PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7")

regards,
G.

p.s. WORK ADV... we have an office in Berlin, and we need Gentoo people with Ganeti experience
                       * * * https://gaikai.com/   * * *  https://www.playstation.com/en-us/explore/psnow/

      (we also have offices in San Mateo and Orange County, CA - USA)

[Thomas Ott]

Hallo Guido,

do you have passed the VT-x for Intel or AMD-V for AMD Processors to
your virtualbox?

Greetz
Thomas

[Guido Serra]

Hi Thomas,
I didn't explicitly called anything in the Vagrantfile,
neither I see anything specific in the one from the vagrant-ganeti project

If I bring up the VirtualBox interface I can tell you that in both situations
I see this -> Acceleration: VT-x/AMD-V, Nested Paging, PAE/NX
...but I feel it related to the virtualbox image, it has nothing to do with
how KVM is handling the ganeti images... it shall run in "emulation" if I understood correctly

   slow, yes, but it should ran

tnx,
G.

[Gilberto Nunes]

Hi

accepted

KVM requires VT-X/AMD-V, but VirtualBox does not pass VT-X/AMD-V to the guest operating system. Therefore, KVM can't run in VirtualBox (yet). Please track bug ticket #4032. VT-X and AMD-V (so-called virtualization extensions) run the guest operating system natively in the CPU. Without them, the virtualization software must interpret the operating system opcodes in software, which is very slow. For now, you can either run KVM inside a hypervisor which passes virtualization extensions to the guest (e.g. KVM insideVMware Workstation 8), or run another hypervisor inside VirtualBox (e.g. VirtualBox inside VirtualBox). This option will be very slow, because the guest will miss VT-X/AMD-V.

--
Gilberto Ferreira

[Guido Serra]

I don’t get it… I’m not trying to setup something new
the guys from OSL already have such setup working

…I just need to understand how they achieved it

cheers,
--
Guido Serra

[candlerb]

(1) If you want to run

    physical host with virtualbox ---> ganeti ---> instance

Because virtualbox doest not support nested virtualisation, you will have to run with pure QEMU software emulation instead of KVM.

That is: I'm assuming you're using some system running virtualbox as the physical platform, and inside that you're running Debian+Ganeti as a VM, and you want to run VMs inside that, correct? If so, then in the ganeti VMs you should do

   apt-get install qemu-system
   gnt-cluster modify -H kvm:kvm_path=/usr/bin/qemu-system-x86_64 

With that setup, guests will run something like 10-20 times slower than under KVM, but they do work.

(2) if the physical host is running ganeti (not virtualbox), and you want to run a ganeti VM inside that and inside that run more VMs, then you need to pass through the VT-x flag.

    physical host with ganeti ---> ganeti ---> instance

On the physical host machine, do:

   gnt-cluster modify -H kvm:cpu_type="qemu64\,+vmx"

If what you're trying to do is neither of those things, then please explain more clearly.

HTH,

Brian.

[Guido Serra]

Hi candlerb,
I'm in ur first scenario, and it doesn't work

   kvm_path=/usr/bin/qemu-system-x86_64

this is my cluster as of now, if I compare it with
the cluster created by vagrant-ganeti this is not even set

regards,
G.

[Guido Serra]

worth mentioning….
kvm_path seems to be mandatory in the version I’m attempting 2.11.3
while, the version which is working is 2.6.3 (might be related)
--
Guido Serra

[Guido Serra]

k, it was the version of ganeti... with 2.7.2 it worked

I'll report it to the vagrant-ganeti project

[candlerb]

On Thursday, 17 July 2014 12:36:17 UTC+1, Guido Serra wrote:

Hi candlerb,
I'm in ur first scenario, and it doesn't work

   kvm_path=/usr/bin/qemu-system-x86_64

this is my cluster as of now, if I compare it with
the cluster created by vagrant-ganeti this is not even set

It works for me with:

* Virtualbox 4.3.12 under OSX 10.9

* Three Debian Wheezy VMs, with qemu-kvm installed from wheezy-backports (gives qemu version 2.0.0 instead of 1.1.2)

* ganeti 2.10.4

It's unlikely that the kvm_path setting has changed much in ganeti versions. However the behaviour certainly does depend on the operating system; in particular, qemu-kvm from wheezy-backports is different than qemu-kvm from standard wheezy.

In standard Debian wheezy: the package is called "qemu-system" which contains /usr/bin/qemu-system-x86_64. In backports the package is called "qemu-system-x86"

In standard Wheezy, /usr/bin/kvm is a binary executable. In wheezy-backports, it is just a shell script wrapper:

$ cat /usr/bin/kvm
#! /bin/sh
exec qemu-system-x86_64 -enable-kvm "$@" 

So basically it's up to you to find a qemu binary which you can run from the command line, e.g. "/usr/bin/qemu-system-x86_64"; once you've found one that works, tell ganeti to use that with the kvm_path hypervisor setting. You can either set this globally at the cluster level, or on individual instances.