mysterious errors with concurrent vagrant cmds: Could not find a registered machine with UUID

[Chris Burroughs]

I'm using vagrant with the virtualbox driver in a CI environment
(jenkins + test-kitchen). I'm running into problems with spooky errors
when multiple jobs are running at the same time (and there are thus
concurrent vagrant commands running). For example with running 4 jobs
at a time, about 6% of jobs will fail with a transient error (it is
unlikely to occur for the same job if only that job is run again in
isolation). I *think* based on the logs it is a problem with vagrant
or virtualbox, but it could be induced by one of the many other moving
pieces.

Specifically what I am seeing is:
* Test kitchen runs: vagrant up --no-provision --provider virtualbox
* At some point during that vagrant calls VBoxManage showvminfo UUID
* This fails:
STDERR: There was an error while executing `VBoxManage`, a CLI used by
Vagrant for controlling VirtualBox. The command and stderr is shown below.

Command: ["showvminfo", "e622fd71-8aa1-409f-90e0-4df89891454f",
"--machinereadable"]

Stderr: VBoxManage: error: Could not find a registered machine with UUID
{e622fd71-8aa1-409f-90e0-4df89891454f}
VBoxManage: error: Details: code VBOX_E_OBJECT_NOT_FOUND (0x80bb0001),
component VirtualBox, interface IVirtualBox, callee nsISupports
VBoxManage: error: Context: "FindMachine(Bstr(VMNameOrUuid).raw(),
machine.asOutParam())" at line 2611 of file VBoxManageInfo.cpp

When inspecting ~/VirtualBox\ VMs I find a .vbox, .vbox-prev, & .vmdk
file, but no Logs directory. Without the logs I'm unsure where to look
further.

One possible oddity is that the UUID in stdout is
e622fd71-8aa1-409f-90e0-4df89891454f (not in `vms list`) but
.kitchen/kitchen-vagrant/foo-default-centos-66/.vagrant/machines/default/virtualbox/id
has 07b9e760-1fc2-44ac-b70a-fca527111b10 which *is* in `vms list`.
Since I think the problem occurs during a single vagrant provision
command I'm unsure what to make of that.

Versions:
* centos: 2.6.32-504.23.4.el6.x86_64
* test-kitchen: 1.4.0
* vagrant: 1.7.2
* vitualbox: 4.3.28r100309

Example Vagrantfile:
Vagrant.configure("2") do |c|
c.berkshelf.enabled = false if Vagrant.has_plugin?("vagrant-berkshelf")
c.vm.box = "internal-centos-6.6-base"
c.vm.box_url = "http://example.com.foo.box"
c.vm.hostname = "default-centos-66"
c.vm.synced_folder ".", "/vagrant", disabled: true
c.vm.provider :virtualbox do |p|
p.customize ["modifyvm", :id, "--memory", "512"]
p.customize ["modifyvm", :id, "--cpus", "1"]
end
end


Log snippet:

+ chef exec 'kitchen test --destroy=always'
-----> Starting Kitchen (v1.4.0)
-----> Cleaning up any prior instances of <default-centos-66>
-----> Destroying <default-centos-66>...
Finished destroying <default-centos-66> (0m0.00s).
-----> Testing <default-centos-66>
-----> Creating <default-centos-66>...
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'addthis-centos-6.6-iad-base-v1'...

Progress: 10%
Progress: 20%
Progress: 30%
Progress: 40%
Progress: 50%
Progress: 60%
Progress: 70%
Progress: 90%
==> default: Matching MAC address for NAT networking...
==> default: Setting the name of the VM:
kitchen-addthis-hastebin-default-centos-66_default_1436302236546_22072
There was an error while executing `VBoxManage`, a CLI used by
Vagrant
for controlling VirtualBox. The command and stderr is shown below.

Command: ["showvminfo", "e622fd71-8aa1-409f-90e0-4df89891454f",
"--machinereadable"]

Stderr: VBoxManage: error: Could not find a registered machine
with UUID {e622fd71-8aa1-409f-90e0-4df89891454f}
VBoxManage: error: Details: code VBOX_E_OBJECT_NOT_FOUND
(0x80bb0001), component VirtualBox, interface IVirtualBox, callee
nsISupports
VBoxManage: error: Context:
"FindMachine(Bstr(VMNameOrUuid).raw(), machine.asOutParam())" at line
2611 of file VBoxManageInfo.cpp

[Chris Burroughs]

On 07/07/2015 08:50 PM, Chris Burroughs wrote:
> I'm using vagrant with the virtualbox driver in a CI environment
> (jenkins + test-kitchen). I'm running into problems with spooky errors
> when multiple jobs are running at the same time (and there are thus
> concurrent vagrant commands running). For example with running 4 jobs
> at a time, about 6% of jobs will fail with a transient error (it is
> unlikely to occur for the same job if only that job is run again in
> isolation).

Gating so that only one 'vagrant' command runs at a time was
unsuccessful in reducing the error rates. I suppose the 'critical
section' could include async operations after `vagrant up` returns.
This is a surprising result to me since we have a box with > 50 VMs spun
up by humans and no one has reported a similar error (so >> VMS, but <<
start/stop operations)

I ran the same job in serial > 100 times with no failures which I
believe confirms is a problem with concurrent operations and not just a
rare error.

[Alvaro Miranda Aguilera]

Hello

What you can try is create a 100% isolated environment for each run

since when you import a box using virtualbox, it first will import the
same default folder ~/Virtualbox VMs/<same_name_if_same_base_box> this
is base on my own personal tests.

then will rename it to the default name per vagrant setup or vagrant default

So at each environment, you can add

JOBID="<some_unique_id>"
VBOX_USER_HOME="~/ci/$JOBID/vbox"

mkdir -p $VBOX_USER_HOME

VBoxManage setproperty machinefolder VBOX_USER_HOME="~/ci/$JOBID/vbox"


That will create the structure that will avoid such collisions

JOBID can be an id from jenkins or some time stamp generated with date

Hope this helps
Alvaro

> --
> This mailing list is governed under the HashiCorp Community Guidelines -
> https://www.hashicorp.com/community-guidelines.html. Behavior in violation
> of those guidelines may result in your removal from this mailing list.
>
> GitHub Issues: https://github.com/mitchellh/vagrant/issues
> IRC: #vagrant on Freenode
> --- You received this message because you are subscribed to the Google
> Groups "Vagrant" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to vagrant-up+...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/vagrant-up/559E6BAD.2030008%40gmail.com.
> For more options, visit https://groups.google.com/d/optout.

[Chris Burroughs]

I'm currently trying a 'sharded' jenkins slave where n jenkins slaves
running as different users are on the same physical host. This is
similar to the technique suggested by Alvaro (thanks!) and initial
results are promising.

[gseux....@gmail.com]

Hello Chris,

have you succeed to solve your issue with the sharded setup ? We encounter exactly the same issue in our organization.

I'd be glad to hear from your progress,

--
Grégoire

[Alvaro Miranda Aguilera]

Hello,

That error (from the subject) can happen in several cases.

- When you copy a folder and that include a .vagrant folder, you can
delete the folder and do vagrant up and that will create a new box.
This apply to projectd where they don't ignore this folder.

- You have some shared folder between users or computers, and a 2nd
user tries to do vagrant up.

They will end overwriting that .vagrant folder and make some boxes
orphans later on.

Can you describe how you do to get that project and what command are
you using when you get that error?

Alvaro.

> --
> This mailing list is governed under the HashiCorp Community Guidelines -
> https://www.hashicorp.com/community-guidelines.html. Behavior in violation
> of those guidelines may result in your removal from this mailing list.
>
> GitHub Issues: https://github.com/mitchellh/vagrant/issues
> IRC: #vagrant on Freenode
> ---
> You received this message because you are subscribed to the Google Groups
> "Vagrant" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to vagrant-up+...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/vagrant-up/37b87a3d-398c-4786-a8ee-2ea92c177dfd%40googlegroups.com.

[gseux....@gmail.com]

Thanks Alvarro for your answer.

We have the same setup as Chris, a jenkins cluster launching `kitchen
test --concurrency=2`.
The jenkins cluster uses mesos plugin to launch jenkins slave on our
mesos cluster. There are multiple jenkins slaves running on a physical
mesos slave and they all uses the same user (jenkins).

I am not sure to see in which of the cases you've described that fell on.

--
Grégoire

[Alvaro Miranda Aguilera]

Hello.

Do you use Virtualbox?

I think I know the answer.

It was some messages back on the same email thread,

https://groups.google.com/d/msg/vagrant-up/UeGrB82PSTs/ad6JTz6AsQkJ

If you are using the same user same source box, etc, you can have some
collisions:

Try to setup this as part of the job

JOBID="<some_unique_id>"

VBOX_USER_HOME="~/ci/$JOBID/vbox"

mkdir -p $VBOX_USER_HOME

VBoxManage setproperty machinefolder VBOX_USER_HOME="~/ci/$JOBID/vbox"


That will create the structure that will avoid such collisions

JOBID can be an id from jenkins or some time stamp generated with date

Let me know if this helps.

Other users have setup test kitchen to run with different users

Alvaro

> --
> This mailing list is governed under the HashiCorp Community Guidelines -
> https://www.hashicorp.com/community-guidelines.html. Behavior in violation
> of those guidelines may result in your removal from this mailing list.
>
> GitHub Issues: https://github.com/mitchellh/vagrant/issues
> IRC: #vagrant on Freenode
> ---
> You received this message because you are subscribed to the Google Groups
> "Vagrant" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to vagrant-up+...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/vagrant-up/2f338a6f-f99b-4068-8169-7a4798f791f9%40googlegroups.com.

[gseux....@gmail.com]

Hello,

About the solution you suggest, I am concerned the setproperty is a global operation that will not isolate builds if several tests are running in parallel. Could you infirm that hypothesis?

Anyway the command you gave has some weird answer:

> VBoxManage: error: Given default machine folder 'VBOX_USER_HOME=~/ci/toto/vbox' is not fully qualified
> VBoxManage: error: Details: code NS_ERROR_INVALID_ARG (0x80070057), component SystemProperties, interface ISystemProperties, callee nsISupports
> VBoxManage: error: Context: "COMSETTER(DefaultMachineFolder)(Bstr(a->argv[1]).raw())" at line 838 of file VBoxManageMisc.cpp

thanks a lot for your help

--
Grégoire

[Alvaro Miranda Aguilera]

Hello,

My suggestion is to create one VBOX_USER_HOME and run the other command per job

so each job will have his own VBOX_USER_HOME folder, in this way
parallel runs will be run on separate VBOX folders.


and setproperty will be run and configure that VBOX_USER_HOME config,
so need to be unique per run.

Can you try a full path? the path I gave was just an example.

so you can try

mkdir -p /some/path
VBOX_USER_HOME=/some/path

and then run the setproperty

Thanks!
Alvaro.

> --
> This mailing list is governed under the HashiCorp Community Guidelines -
> https://www.hashicorp.com/community-guidelines.html. Behavior in violation
> of those guidelines may result in your removal from this mailing list.
>
> GitHub Issues: https://github.com/mitchellh/vagrant/issues
> IRC: #vagrant on Freenode
> ---
> You received this message because you are subscribed to the Google Groups
> "Vagrant" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to vagrant-up+...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/vagrant-up/7704970a-b8ee-49ac-8ce5-0cca4590c4a3%40googlegroups.com.

[gseux....@gmail.com]

Thanks Alvaro for your answer.

I have succeeded to set a machinefolder using the following command:
> VBoxManage setproperty machinefolder "/home/jenkins/vbox/"

This modifies the .config/VirtualBox/VirtualBox.xml to contain:
>     <SystemProperties defaultMachineFolder="/home/jenkins/vbox/" defaultHardDiskFormat="VDI" VRDEAuthLibrary="VBoxAuth" webServiceAuthLibrary="VBoxAuth" LogHistoryCount="3" exclusiveHwVirt="true"/>

I could run the setproperty command before each job uses "kitchen
test" command and specify a per job directory.

I am not sure to understand however how this will solve the issue I
encounter (which I am not sure to understand by the way).
I understand that each new box launched by kitchen will be run in a
folder that will (hopefully) be its job directory.
Since the setproperty seems to act at the user level, there could be a
race conditions if a job B set the machinefolder property after job A
sets his but before job A start the box.
This is unlikely if jobs have only one box to launch but our kitchen
tests usually launch several of them sequentially.

My manual tests so far show that vbox can starts and destroy boxses
even if machinefolder have changed since creation.


Can you help to understand if it will help to solve my issue ?

Thanks again for the time you spend to help me

--
Gregoire

[Alvaro Miranda Aguilera]

Hello,

When the problem happen?

When Vagrant create the box from the template, and you try to do more than 1 at the same time.

Usually goes like this

mkdir myawesomeproject

cd myawesomeproject

vagrant init -m myawesomebox

here, on first vagrant up

Vagrant will read myawesomebox and will create myawesomebox inside the default virtualbox folder

once that happen, it will rename myawesomebox into myawesomeproject_<some_numbers>

So, when you do concurrent vagrant up under the same folder you may hit a collision on the default virtualbox folder.

That's why, the best is

a. use different users, each user for each run

b.

Create a virtualbox default folder for each run

Create a vagrant folder

vagrant up

Hope this helps to see what the problem is, and what the solution is trying to avoid.

Thanks

Alvaro.

--

This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/mitchellh/vagrant/issues
IRC: #vagrant on Freenode
---
You received this message because you are subscribed to the Google Groups "Vagrant" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vagrant-up+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/vagrant-up/b9306f7b-4c84-4d68-b28f-cdea9ee4ff51%40googlegroups.com.

[gseux....@gmail.com]

I understand better the issue you've described.

The solution you offer though does not protect concurrent runs since setproperty is global.

I've ended up patching the vagrant driver for kitchen to include a lock (per user) around vagrant up.

Thanks a lot for your help.

--
Grégoire

[Alvaro Miranda Aguilera]

On Wed, Dec 2, 2015 at 3:47 AM, <gseux....@gmail.com> wrote:

he solution you offer though does not protect concurrent runs since setproperty is global.

Hello,

Just to close this look.

a. New proccess.

b. You set the HOME VBOX for that process

c. You use setproperty that will overrride the GLOBAL file in step b

If you do that per job, each job will get his own VirtualBox home and his own global Virtualbox folder.

Thanks

Alvaro.

[Srikanth Viswanathan]

Hi folks,

Sorry to open this old email chain. I'm facing the exact same problem as the OP and wanted to make sure I understood the workaround. I couldn't quite parse the last message in this thread

a. New proccess.
b. You set the HOME VBOX for that process
c. You use setproperty that will overrride the GLOBAL file in step b

What does "no proccess" mean? How can setproperty be used to override the "GLOBAL file" in step b? It does not appear there is a way to locally override the virtualbox machinefolder, is there? Absent this workaround, what is the suggested way around this problem?

I appreciate all the information in this thread, and want to make sure the workaround (if any) is well understood.

Thanks,
Srikanth

[Alvaro Miranda Aguilera]

hello

this has been answered before, so perhaps you just got the high level reply

check this reply:

https://groups.google.com/forum/#!searchin/vagrant-up/setproperty%7Csort:date/vagrant-up/-Ky73Bev2W0/d9i2cFNCpPcJ

your question is more a virtualbox question rather than vagrant, but let me help you abit.

if you want to have unique locations for boxes, you can create a virtualbox home in a separate folder this will be used by virtualbox and then you can override the machine folder

set VBOX_USER_HOME="C:\VirtualBoxVMs" 
"C:\program files\oracle\virtualbox\VBoxManage.exe" setproperty 
machinefolder C:\VirtualBoxVMs 

if you just want to have the machine location moved, just run something like this:

"C:\program files\oracle\virtualbox\VBoxManage.exe" setproperty 
machinefolder C:\VirtualBoxVMs 

by default virtualbox assume VBOX_USER_HOME as your home