Jobs fail due to lost slave connection since upgrade 1.560

70 views
Skip to first unread message

Marius Gedminas

unread,
May 6, 2014, 7:19:31 AM5/6/14
to jenkins...@googlegroups.com
I've an Ubuntu 12.04 LTS server and upgrade Jenkins to the latest
upstream deb package whenever it comes out. Jenkins runs some of the
jobs on an Ubuntu 10.04 LTS slave on the same LAN, controlled over SSH.

I've had four failures (out of 16 runs) since Apr 27, all with

FATAL: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:41)
at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:34)
at hudson.remoting.Request.call(Request.java:174)
at hudson.remoting.Channel.call(Channel.java:738)
at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:168)
at com.sun.proxy.$Proxy64.join(Unknown Source)
at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:951)
at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:137)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:97)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:740)
at hudson.model.Build$BuildExecution.build(Build.java:198)
at hudson.model.Build$BuildExecution.doRun(Build.java:159)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:519)
at hudson.model.Run.execute(Run.java:1703)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:231)
Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.Request.abort(Request.java:299)
at hudson.remoting.Channel.terminate(Channel.java:801)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69)
Caused by: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50)
Caused by: java.io.EOFException
at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2323)
at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2792)
at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:800)
at java.io.ObjectInputStream.<init>(ObjectInputStream.java:298)
at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:40)
at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

I never had this kind of problem before. Either my Ethernet suddenly became
less reliable, or it's a software issue.

I upgraded Jenkins to 1.560 on Apr 22, and to 1.561 on Apr 29. The
first failure of this kind happened on Apr 27.

Advice appreciated.

Marius Gedminas
--
OK, so you're a Ph.D. Just don't touch anything.
signature.asc

Matthew...@diamond.ac.uk

unread,
May 6, 2014, 7:28:57 AM5/6/14
to jenkins...@googlegroups.com
Before you upgrade Jenkins, it's always worth looking at http://jenkins-ci.org/changelog, and in particular looking at "Community Ratings" to see if there are any reported issues with a particular release.

In this case, I suspect that you have encountered https://issues.jenkins-ci.org/browse/JENKINS-22734. This problem was introduced in Jenkins 1.560, and will be fixed in 1.563.

Hope that helps.
Matthew
--
This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom




Marius Gedminas

unread,
May 6, 2014, 11:51:16 AM5/6/14
to jenkins...@googlegroups.com
On Tue, May 06, 2014 at 11:28:57AM +0000, Matthew...@Diamond.ac.uk wrote:
> Before you upgrade Jenkins, it's always worth looking at
> http://jenkins-ci.org/changelog, and in particular looking at
> "Community Ratings" to see if there are any reported issues with a
> particular release.

Ooh, I hadn't noticed those before!

> In this case, I suspect that you have encountered
> https://issues.jenkins-ci.org/browse/JENKINS-22734.

Thank you. I'd've never guessed this was it, since my problem did not
happen during file transfers, and I didn't see OutOfMemoryErrors in the
system log on the web interface. I did find a

java.lang.OutOfMemoryError: Java heap space

near the end of /var/log/jenkins/logs/slaves/*/slave.log.1 on my
Jenkins master.

> This problem was introduced in Jenkins 1.560, and will be fixed in 1.563.

I'll wait for 563 then. The releases are weekly, aren't they? (Another piece
of information I somehow didn't know about until today.)

> Hope that helps.

Yes, thank you.

Marius Gedminas
--
(Pdb) operationerr.w_value.w_value.w_value.w_value.w_value.w_value
<pypy.interpreter.executioncontext.OperationError instance at 0x5eee30>
-- one of the clearer PyPy debugging sessions
(seen in Michael Hudson's sig)
signature.asc
Reply all
Reply to author
Forward
0 new messages