Jenkins slave node management with VM snapshots

[Yves Goergen]

We're running a Jenkins server on one machine that monitors the SVN repository and performs the complete build of the project. It's using a custom PowerShell script which basically just finds and runs MSBuild, Dotfuscator and InnoSetup. The resulting setup executable shall then be transfered to a number of slave nodes which are Hyper-V virtual machines, for GUI testing under different operating systems. I've already set up the slave node and after a lot of guesswork and try&error, it finally copies over the setup.exe and runs it. We don't have a GUI test tool yet, so that's still left to be done.

Now the problem lies in the VM management. I've just tried it once to revert the slave VM to a clean snapshot - with the computer in running state and the Jenkins agent also running - and then wanted the Jenkins slave work to start right afterwards. But the problem is that after reverting the VM, while the Jenkins agent is still running, all external resources like network connections are broken and nobody knows. And it takes a lot of time for both to recover from that. The server still thinks for a while that the client is still there, and the client still thinks that it's connected. But nothing works and the job eventually fails with a Jenkins internal exception that seems to come from the broken network connection. A while after that, the connection recovers and the node comes back available again.

This doesn't exactly seem to work.

I've found some hint on the web to tell the Jenkins server via its web API to "doDisconnect" a certain slave node, but that URL is only 404'd. Since there is virtually no API documentation of Jenkins at all, I cannot correct this method or even tell whether it was entirely made up.

So, with the workflow:

* Build on the master
* Revert the slave VM (Hyper-V) to a running snapshot
* Start another job on a slave

What would be the suggested process? How can I get Jenkins to work in this environment? Does it even work at all?

We're all Windows, with scripting in CMD or PowerShell, no Bash supported. I still need to find an automation script for Hyper-V control, but that should be possible.

[Matthias]

Hi, have you managed to get this working?

[Yves Goergen]

Yes, it's working now. I figured out that I had some URL entered wrong. But I can't remember the details anymore. We're using two separate jobs after the build has completed: Prepare-Test and test. The first one restores the VM and disconnects the client from the Jenkins server. The VMM API is synchronous, but a little more delay was necessary to stabilise it. Then, the slave is started on that VM through remote execution of a Windows scheduled task. The second job then does the actual work.

Transferring the build files (setup etc.) to the slave is still complicated though. I played with diverse settings and plug-ins and eventually made it work with the help of additional batch script copying of files through the public userContent directory. Not nice but working. A truly integrated solution would offer me this out of the box.

About the VMM control: Nothing else worked reliably or satisfactory than installing the VMM console on the Jenkins host and using the PowerShell commands locally. It takes about 1 GB disk space, which is more bloat than anything else on that machine, but at least it works easily.

(I hope this reply goes in the right direction. I don't know where I originally posted it and just found this message in my inbox, using the Gmail reply function...)

2013/10/31 Matthias <nap...@gmail.com>

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/Cuhykr4bXco/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-use...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Matthias Napalowski]

If I try to start a Hyper-V VM with a powershell script all i get is:

Could not load file or assembly 'TraceWrapper, Version=1.0.523.0, Culture=neutr
al, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. An attempt was
 made to load a program with an incorrect format.

The script looks like this:

Import-Module "C:\Program Files\Microsoft System Center 2012\Virtual Machine Manager\bin\psModules\virtualmachinemanager\virtualmachinemanager.psd1";

Get-VMMServer "hades"
$VM = Get-SCVirtualMachine -Name "Windows Server 2008 R2"
Start-SCVirtualMachine -VM $VM

Why does Jenkins fail with this script? if i execute it in a normal powershell, started as admin and also non admin, it works just fine.


Greetings
Matthias

2013/10/31 Yves Goergen <yves.g...@gmail.com>

[Yves Goergen]

"Incorrect format" sounds a bit like "BadImageFormatException" which usually means that you're mixing 32- and 64-bit modules. That's all I know.

Here's my VMM PowerShell script that works just fine in Jenkins:

param($VMMServer, $VMName)

Import-Module -Name "virtualmachinemanager"

Get-SCVirtualMachine -Name $VMName -VMMServer $VMMServer `
| Get-SCVMCheckpoint -MostRecent `
| Restore-SCVMCheckpoint

2013/10/31 Matthias Napalowski <nap...@gmail.com>