Nested VM inside VM should work with enough hardware, right?

763 views
Skip to first unread message

blong

unread,
May 9, 2014, 2:56:53 PM5/9/14
to vagra...@googlegroups.com
I'm assuming this isn't recommended, but stay with me!  I'm trying to configure a CentOS environment to run VirtualBox and Vagrant without any issues.  A while back, I was able to successfully nest the  "hashicorp/precise32" VM running in VirtualBox within another "hashicorp/precise32" VM running in VirtualBox (as long as VT-x/AMD-V is enabled).  It's been so long, that I can't exactly remember, but I might have been running CentOS inside the precise32 VM, or vice-versa.

I've installed both VirtualBox and Vagrant in CentOS (multiple experiments using various versions of each), but no matter what I do my inner VM runs slowly. I don't expect this, since I gave the CentOS VM more than 7GB of memory, and 2 cores from a 2.7GHz Core i7 (real hardware). When I try to startup my inner VM's (with or without Vagrant) they run slowly, and don't seem to allocate much memory. As the VM is booting, I see CPU 1 & 2 spike a bit, then eventually drop, but the total memory usage by CentOS doesn't rise above 1GB (via CentOS' System Monitor). When trying to start the "precise32" VM, it times out like this:

[me@localhost hashicorp-precise32]$ vagrant destroy
    default: Are you sure you want to destroy the 'default' VM? [y/N] y
==> default: Destroying VM and associated drives...
[me@localhost hashicorp-precise32]$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'hashicorp/precise32'...
==> default: Matching MAC address for NAT networking...
==> default: Checking if box 'hashicorp/precise32' is up to date...
==> default: Setting the name of the VM: hashicorp-precise32_default_1399644995759_24359
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
==> default: Forwarding ports...
    default: 22 => 2222 (adapter 1)
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:2222
    default: SSH username: vagrant
    default: SSH auth method: private key
    default: Warning: Connection timeout. Retrying...
    default: Warning: Connection timeout. Retrying...
    default: Warning: Connection timeout. Retrying...

... # Omitted for brevity

    default: Warning: Connection timeout. Retrying...
Timed out while waiting for the machine to boot. This means that
Vagrant was unable to communicate with the guest machine within
the configured ("config.vm.boot_timeout" value) time period.

If you look above, you should be able to see the error(s) that
Vagrant had when attempting to connect to the machine. These errors
are usually good hints as to what may be wrong.

If you're using a custom box, make sure that networking is properly
working and you're able to connect to the machine. It is a common
problem that networking isn't setup properly in these boxes.
Verify that authentication configurations are also setup properly,
as well.

If the box appears to be booting properly, you may want to increase
the timeout ("config.vm.boot_timeout") value.


I posted a comment on StackOverflow about this, but I'm not sure if that will drum up a response from the guy who appears to have had success with a nested VM arrangement: http://stackoverflow.com/questions/17175696/running-vagrant-inside-vmware-vm/22931930#comment36172153_22931930

Would anyone be able to help me get this working?

Thanks in advance!

Mitchell Hashimoto

unread,
May 9, 2014, 3:16:57 PM5/9/14
to vagra...@googlegroups.com
VirtualBox itself doesn't support VirtualBox running in VirtualBox.

VMware this works fine.
> --
> You received this message because you are subscribed to the Google Groups
> "Vagrant" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to vagrant-up+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

blong

unread,
May 9, 2014, 3:27:30 PM5/9/14
to vagra...@googlegroups.com
Mitchell,
Thanks for getting back to me so quickly & thanks for the info.  I am trying to run VirtualBox inside VMWare Fusion.  Later on, the goal is to move the VMWare fusions instance to VCenter.  I'm not sure if it's helpful, but my "VAGRANT_LOG=debug" output is attached.

Thanks,
Brian
debugging_vm.log

Brian Long

unread,
May 11, 2014, 5:23:40 PM5/11/14
to vagra...@googlegroups.com

Is there a trick to getting VirtualBox to run smoothly inside a linux VM?  I've followed a few different tutorials for installing VirtualBox in CentOS, which typically include installing DKMS, kernel-devel & kernel-headers.  The latest advice I saw indicated that IPv6 networking might be the slowness culprit, but even with that disabled my VMs still run slowly.  Any tutorials, pointers or even someone's bash history would be greatly appreciated :)

You received this message because you are subscribed to a topic in the Google Groups "Vagrant" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vagrant-up/6YHdRupCKuI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to vagrant-up+...@googlegroups.com.

Mitchell Hashimoto

unread,
May 11, 2014, 5:25:39 PM5/11/14
to vagra...@googlegroups.com
Brian,

VirtualBox doesn't support running in VirtualBox. You can run
VirtualBox in another VM (VMware) usually, but it is very expensive.

Best,
Mitchell

Brian Long

unread,
May 11, 2014, 5:30:02 PM5/11/14
to vagra...@googlegroups.com
Mitchell,
Sorry, I should have clarified, I'm not running VirtualBox inside VirtualBox.  I'm running VirtualBox inside VMWare Fusion.  Do you think throwing more memory (e.g. 10 GB?) at it will help?  I'm a bit desperate at this point :)  We really like Vagrant and want to build out an environment for testing new builds.

- Brian

Mitchell Hashimoto

unread,
May 11, 2014, 5:36:30 PM5/11/14
to vagra...@googlegroups.com
Brian,

Make sure you check the box (or enable the setting) to enable
virtualization extensions inside the VM. It is not on by default.

Best,
Mitchel

Brian Long

unread,
May 11, 2014, 5:48:09 PM5/11/14
to vagra...@googlegroups.com
Thanks Mitchell,  I've done that (see the attached screenshot), but still the performance seems really bad.  The reason for double-virtualization at this point is to build out the CentOS host that can then be exported and moved to a vShpere environment. Once I'm sure the CentOS host is configured properly, it'll be moved to the server hypervisor and no longer be run it in VMWare Fusion.  My strategy was to configure and check the CentOS host before exporting it, but maybe I'd be better off moving to vSphere first?
screenshot.tiff

Alvaro Miranda Aguilera

unread,
May 11, 2014, 7:03:46 PM5/11/14
to vagra...@googlegroups.com
Can I ask why you are suing 2 expensive virtualizations? the performance should not be a surprise here.

Why you need Virtualbox running inside vmware on the firstplace?

Are other virtualization that are  light, like LXC or Docker that should run faster/better.

Other may be XEN or KVM, but basically a good understanding on "what you need" will be helpful to be able to suggest "how I should do it" if I have to do the same as you want..

Brian Long

unread,
May 11, 2014, 7:40:50 PM5/11/14
to vagra...@googlegroups.com
@Alvaro - regarding the 2 expensive virtualizers, please see the details earlier in this thread (it is not the desired end state).

We have been targeting VirtualBox with our users & box files, although we're not using provider specific configuration in our Vagrantfile.  Is there a simple way to set LXC as the provider / default virtualizer and try starting my Vagrant VM with LXC?  I'm not a Linux guru, but if it's as simple as installing a few packages and modifying some configuration files, I'm certainly willing to try LXC.

- Brian  

Alvaro Miranda Aguilera

unread,
May 11, 2014, 9:42:18 PM5/11/14
to vagra...@googlegroups.com
At some point the future, based on my personal experience, my very own point of view, I will say the order of things will be:

- Docker
- LXC
- Lightvirtualization (xen/kvm)
- PVM/HVM (qemu,vmware,virtualbox, any other )

If you app/task can be run in a docker container, go for it, is the best and quickest way to do it.

LXC works pretty good if you app can't be run in a docker container, or you need some kind of isolation from the host machine (specific set of packages, or different OS/bit)

As for now, if your host is ubuntu, (or the vm to be used for lxc) , there is a plugin for this vagrant-lxc and I have read it works and it have traction.

https://github.com/fgrehm/vagrant-lxc

I use Docker and lxc at command line level as for know, haven't get into those 2 with Vagrant, but I have read they work, so I suggest have a look at them as time permits.

Hope this helps.

Alvaro

dragon788

unread,
May 28, 2015, 10:18:47 AM5/28/15
to vagra...@googlegroups.com
As I'm trying to do something similar myself (actually Vagrant/VMware hosting Vagrant/Vbox) I have attempted all the optimizations I can think of, giving my "host" VM a large amount of RAM (24GB) and 6 cores, disabling swapping VMs to disk (this KILLS things on Windows when it happens) by setting Fit all VM memory into reserved host memory, and allowing per VM page files (otherwise they live in the system page file which limits how many VMs you can run at once).

My biggest issue right now is networking within the inner VM, apparently Vbox isn't honoring some of the network settings I'm defining, and this may be because the outer VM is being managed by Vagrant and doesn't have the right combination of network devices enabled.

dragon788

unread,
May 29, 2015, 12:16:28 PM5/29/15
to vagra...@googlegroups.com
And apparently what I was doing worked perfectly, the networking issue was due to the corporate proxy I'm behind. Once I configured that my VM got internet access and all was right with the world.

I did have to manually set --natbindip1 and --natnet1 via the Vagrantfile in addition to the natdnsproxy1 and naddnshostresolver1 that were already set in my example (Virtualbox) Vagrantfile. These settings can be found in the Virtualbox documentation for the correct usage.
Reply all
Reply to author
Forward
0 new messages