Hudson slave terminates mid-job with SocketException

[svisagie]

I'm having a problem with a long-running job (roughly 4-5 hours) where
the slave terminates unexpectedly at different stages of the job. I'm
running the slave through a shell script that executes "java.exe -jar
slave.jar -jnlpUrl <url>"

In terms of logging I've had some difficulty trying to obtain the
stdout and/or stderr using pipe commands whereby the file is blank and
the slave just logs to the console window. Previously the process
started a new shell session which then closed after the slave
terminated, I've changed this and am currently waiting to see if I can
get more information. In the meantime, the only information in the
node log file on the host:

<===[HUDSON REMOTING CAPACITY]===>���Slave.jar version: 1.395
This is a Windows slave
Slave successfully connected and online
ERROR: Connection terminated
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source)
at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown
Source)
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown
Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.readObject(Unknown Source)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:947)

Thanks.

[svisagie]

This is the stack on the slave:

SEVERE: I/O error in channel channel
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)

at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at java.io.ObjectInputStream$PeekInputStream.peek(Unknown
Source)
at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown
Source)
at java.io.ObjectInputStream
$BlockDataInputStream.peekByte(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.readObject(Unknown Source)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:947)

These nodes are running on an ESXi host without any firewall or router
between them.

[svisagie]

SEVERE: I/O error in channel channel
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)

at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at java.io.ObjectInputStream$PeekInputStream.peek(Unknown
Source)
at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown
Source)
at java.io.ObjectInputStream
$BlockDataInputStream.peekByte(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.readObject(Unknown Source)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:947)

On Feb 22, 10:39 am, svisagie <steven.visa...@gmail.com> wrote:

[svisagie]

SEVERE: I/O error in channel channel
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)

at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at java.io.ObjectInputStream$PeekInputStream.peek(Unknown
Source)
at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown
Source)
at java.io.ObjectInputStream
$BlockDataInputStream.peekByte(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.readObject(Unknown Source)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:947)

On Feb 22, 10:39 am, svisagie <steven.visa...@gmail.com> wrote:

[svisagie]

SEVERE: I/O error in channel channel
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)

at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at java.io.ObjectInputStream$PeekInputStream.peek(Unknown
Source)
at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown
Source)
at java.io.ObjectInputStream
$BlockDataInputStream.peekByte(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.readObject(Unknown Source)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:947)

On Feb 22, 10:39 am, svisagie <steven.visa...@gmail.com> wrote:

[svisagie]

This is the error occuring on the slave:

SEVERE: I/O error in channel channel
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)

at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at java.io.ObjectInputStream$PeekInputStream.peek(Unknown
Source)
at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown
Source)
at java.io.ObjectInputStream
$BlockDataInputStream.peekByte(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.readObject(Unknown Source)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:947)

These machines are running on an ESXi host system with no router of
firewall between them.

On Feb 22, 10:39 am, svisagie <steven.visa...@gmail.com> wrote:

[Jminne]

I have experienced periodic SocketTimeoutException when the jenkin's
master is on an ESXi host. After moving to a physical master the
problem went away. I am able to use esxi slaves though.
The root cause likely is a network connectivity problem, but that's
how I dealt with it.

-John

[Swindells, Thomas]

Posting the same stack trace 4 times is unlikely to make a positive difference in people responding!

Are you sure your network is healthy and doesn't suffer from glitches?

Thomas

**************************************************************************************
This message is confidential and intended only for the addressee. If you have received this message in error, please immediately notify the postm...@nds.com and delete it from your system as well as any copies. The content of e-mails as well as traffic data may be monitored by NDS for employment and security purposes. To protect the environment please do not print this e-mail unless necessary.

NDS Limited. Registered Office: One London Road, Staines, Middlesex, TW18 4EX, United Kingdom. A company registered in England and Wales. Registered no. 3080780. VAT no. GB 603 8808 40-00
**************************************************************************************

[svisagie]

Sorry about multiple posts, I submitted the reply but it didn't
display even after refreshing! Wasn't trying to push for a reply :)

Basically I'm trying to find out if it is network related and then I
can escalate it to the necessary department.

If other folks have experienced the same problem, will be great to
hear from you and how you resolved it.

Thanks, Steven.

> This message is confidential and intended only for the addressee. If you have received this message in error, please immediately notify the postmas...@nds.com and delete it from your system as well as any copies. The content of e-mails as well as traffic data may be monitored by NDS for employment and security purposes. To protect the environment please do not print this e-mail unless necessary.