Jenkins exit code 143 - due to Unexpected termination of the channel

[BobTheBuilder]

Hello,

I have a long running build that never runs to completion on the slave
computer, it gets "Maven JVM terminated unexpectedly with exit code 143". It
does not terminate in the same spot, some times it terminates in 5 minutes
others 30+ minutes (usually 21-22). If I run the Maven build on my machine,
it runs just fine, completing in 82-92 minutes.

The Jenkins master log says
INFO: PMMT-SystemTest-Milestone3 #814 main build action completed: FAILURE

Jan 17, 2013 1:38:37 PM hudson.remoting.Channel$ReaderThread run
SEVERE: I/O error in channel Channel to Maven [/usr/lib/jvm/java/bin/java,
-Xrs, -Djava.awt.headless=true, -cp,
/opt/ltsapps/jenkins/maven-agent.jar:/opt/ltsapps/jenkins/classworlds.jar,
hudson.maven.agent.Main, /opt/ltsapps/maven,
/opt/ltsapps/jenkins-1.424.2/slave.jar,
/opt/ltsapps/jenkins/maven-interceptor.jar, 46746,
/opt/ltsapps/jenkins/maven2.1-interceptor.jar]
java.io.IOException: Unexpected termination of the channel
at hudson.remoting.Channel$ReaderThread.run(Channel.java:1037)
Caused by: java.io.EOFException
at
java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2553)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1296)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:1031)

Various Google searches say to check the cpu and memory usage and possibly
add more. However from what I've observed, our machine has 4 CPUs and lots
of free memory that never seems to get used, so I don't think thats it. I've
also tried the -Xms500m -Xmx550m options with no impact.

I've also found sites that referrence problems with Jenkins clients using
SSH ("http://jenkins.361315.n4.nabble.com/fai ... 36397.html") to
communicate with the master but we're using "Lauch slave agents via Java Web
Start". Please note, I will not be able to switch to SSH without an approval
from Congress. :)

Our Maven goals and options are: clean verify -U -e
-Dtest.machine.qualifier=Z -Dtarget.environment.alias=dev1 -Drest.skip=true
-Dfirefox.skip=true -Dchrome.skip=true -Drunner.timeout.in.seconds=25200
-Drunner.threads.to.use=1 -Ppmmt -DtestFailureIgnore=true
-Dstory.paths.to.include="stories/sf/baf/pmmt/stories/rules/**/*.story"

MAVEN_OPTS is set to: -Xrs -Djava.awt.headless=true

Any help would be appreciated.

Thank you.



--
View this message in context: http://jenkins.361315.n4.nabble.com/Jenkins-exit-code-143-due-to-Unexpected-termination-of-the-channel-tp4652347.html
Sent from the Jenkins users mailing list archive at Nabble.com.

[Richard Mortimer]

Hi,

On 22/01/2013 00:02, BobTheBuilder wrote:
> Hello,
>
> I have a long running build that never runs to completion on the slave
> computer, it gets "Maven JVM terminated unexpectedly with exit code 143".

Exit codes over 128 signify that the process died due to a signal.

exitcode = 128 + signal number.

That would be signal 25. Assuming that you have a standard linux
distribution that would be SIGXFSZ.
http://www.lindevdoc.org/wiki/SIGXFSZ

Does your build create large files? If so then maybe your jenkins master
or slave is running in an environment where an rlimit has been placed on
file size.

Regards

Richard

[Richard Lavoie]

128+25=153 not 143 the signal is 15 not 25

Richard

[Richard Mortimer]

On 22/01/2013 10:47, Richard Lavoie wrote:
> 128+25=153 not 143 the signal is 15 not 25
>

Damn! Indeed it is. That'll teach me to send messages before first
coffee of the day.

Well signal 15 is SIGTERM.

My initial thought was that the issue would have been due to the kernel
OOM (out of memory) killer killing things.

https://wiki.jenkins-ci.org/display/JENKINS/I%27m+getting+OutOfMemoryError

That normally uses SIGKILL (9) but maybe it can use SIGTERM in some
scenarios.

A quick search showed SIGTERM being used in kernel 2.4 but that doesn't
seem to be done in the latest linux kernels.

ftp://ftp.kernel.org/pub/linux/kernel/people/marcelo/linux-2.4/mm/oom_kill.c

Anyhow the solution will likely lie in finding the cause/source of the
SIGTERM.

Regards

Richard

[BobTheBuilder]

Hello,

Given the significant variation in time range that the JBehave test runs, I
wouldn't expect the problem to be file size. If it was, it would fail
around the same spot. However wanting to investigate all possibilities, I
ran a "ulimit -a" and here's what I came up with:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 106496
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 4096
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 106496
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

Earlier in the day, I pulled down the heap dump from the master.
Unfortunately it was over 770 megs and I don't have an editor capable of
opening a file that large (my laptop is Windows). I'm in the process of
requesting some different text editing software. Given the possibility of
the heap dump containing confidential information, I'm not posting it to the
site. When I'm able to view the output then I'll worry about it. :)



--
View this message in context: http://jenkins.361315.n4.nabble.com/Jenkins-exit-code-143-due-to-Unexpected-termination-of-the-channel-tp4652347p4652500.html