libvirt-slave plugin getting "Broken pipe" error

[Bernie Duggan]

Hi all,
I'm in the process of setting up a Jenkins build environment for our
system, using a bunch of VMs to do different configuration builds. Our
architecture is:
Machine A running jenkins with the libvirt-slave plugin.
Machine B hosting KVM VMs, accessible through a NAT network, with the
SSH port on each VM forwarded to from a unique port on B so that they
can be reached from A.

For the most part it's all working quite nicely (so, thanks to all the
Jenkins people for a good system). I am running into a problem,
however, with starting up the slave nodes. When I start up Jenkins on
the master node, and manually or automatically start each slave node,
all goes well. I can do builds on them and everyone is happy. However,
I have configured the nodes to be shut down after a period of being
idle. When they try to start back up (either manually or after being
triggered by a build), I get this error:

Getting connection to the virtual datacenter
org.libvirt.LibvirtException: cannot send data: Broken pipe
at org.libvirt.ErrorHandler.processError(Unknown Source)
at org.libvirt.Connect.processError(Unknown Source)
at org.libvirt.Connect.numOfDefinedDomains(Unknown Source)
at org.libvirt.Connect.listDefinedDomains(Unknown Source)
at hudson.plugins.libvirt.Hypervisor.getDomains(Hypervisor.java:146)
at hudson.plugins.libvirt.VirtualMachineLauncher.launch(VirtualMachineLauncher.java:99)

The virtual machine itself is running (indeed, the VM never shut down,
which I thought it was meant to do as part of the idle-offline
feature) and I can successfully ssh from A to the VM. I can also use
virsh to connect from A to the hypervisor on B and list running VMs
with no problems (using virsh --connect qemu+ssh://B/system), so short
of assuming a bug in either the libvirt java bindings or the
libvirt-slave plugin, I'm not really sure where else to look.

I'm running Jenkins 1.451 and libvirt-slave 1.6 on RHEL6.

If anyone has any suggestions or pointers I'd be very grateful.

Cheers,

Bernard

[Bernie Duggan]

Just an additional bit of information: It appears that, not only did
the VM not shut down, but the slave jenkins process on the VM never
actually stopped, even though the slave is marked as offline. Killing
the slave process, however, does not fix the problem.

The only thing I've found which does fix the problem is to restart
jenkins on machine A.

Cheers,

Bernard