[JIRA] [core] (JENKINS-18781) Configurable channel timeout for slaves

Rick Liu this issue has been solved in remoting-2.62 (~~JENKINS-22853~~)
There was also a fix of SocketTimeoutException in remoting-2.62 (~~JENKINS-22722~~), which makes remoting tolerant against SocketTimeout exceptions.

So the remoting layer should be more stable now

This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)

o.v.nenashev@gmail.com (JIRA)

unread,

Aug 19, 2016, 4:10:07 PM8/19/16

to jenkinsc...@googlegroups.com

Oleg Nenashev started work on

Change By:	Oleg Nenashev
Status:	Open In Progress

o.v.nenashev@gmail.com (JIRA)

unread,

Aug 19, 2016, 4:10:07 PM8/19/16

to jenkinsc...@googlegroups.com

Oleg Nenashev assigned an issue to Oleg Nenashev

Jenkins /

Configurable channel timeout for slaves

Change By:	Oleg Nenashev
Assignee:	Oleg Nenashev

stefan.moebius@actix.com (JIRA)

unread,

Sep 12, 2016, 4:42:10 AM9/12/16

to jenkinsc...@googlegroups.com

Stefan Möbius commented on

Oleg Nenashev: The reference to ~~JENKINS-22853~~ seems to be unrelated. Did you type the wrong number by any chance?
Also, ~~JENKINS-22722~~ states it was fixed in remoting-2.60 (although we still have pretty bad problems with broken connections)

All: Are you running Jenkins on VMs? We noticed that VMware moving VMs between hosts can cause a brief packet loss which can cause Jenkins to loose connection.

elliott.jones@sas.com (JIRA)

unread,

Sep 13, 2016, 4:54:04 AM9/13/16

to jenkinsc...@googlegroups.com

Elliott Jones commented on

We have slave disconnect issues and are running on VMware (both master and slave). From the recent available data, the 'Tasks & Events' history does NOT show a 'Migrate virtual machine' entry at the time of disconnect (for either master or the slave or involved).

We'll continue to monitor, though we've not had any disconnects since our upgrade to Jenkins 2.7.2 and we used to get 1 or 2 a week.

maciej.kusz@gmail.com (JIRA)

unread,

Sep 13, 2016, 7:00:04 AM9/13/16

to jenkinsc...@googlegroups.com

Maciej Kusz commented on

We;ve got similar problem when our master was on VMware. After migration to Hyper-V from Microsoft problem has been solved. I think that this is some problem with VMware configuration or it's network switch virtualization.

muppy.cwa@gmail.com (JIRA)

unread,

Sep 19, 2016, 9:22:04 AM9/19/16

to jenkinsc...@googlegroups.com

Markus Niklasson commented on

Hi,

We have recently also encountered disconnection issues. Slave is a Windows 7 (x64) PC with enough of RAM and CPU to run heavy applications. The Jenkins master is a Enterprise Redhat 7 (3.10.0-327.18.2.el7.x86_64) also with enough memory and so on to run Jenkins. Both running Java 8 update 102. The slave are connected through JNLP. Network can be a bit unstable at times.

The following intermittent error occurs very frequently during builds:

Agent went offline during the build
ERROR: Connection was broken: java.io.IOException: Connection aborted: org.jenkinsci.remoting.nio.NioChannelHub$MonoNioTransport@69c08f2a[name=Buildserver]
at org.jenkinsci.remoting.nio.NioChannelHub$NioTransport.abort(NioChannelHub.java:208)
at org.jenkinsci.remoting.nio.NioChannelHub.run(NioChannelHub.java:629)

at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

Caused by: java.io.IOException: Connection timed out
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(Unknown Source)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source)
at sun.nio.ch.IOUtil.read(Unknown Source)
at sun.nio.ch.SocketChannelImpl.read(Unknown Source)
at org.jenkinsci.remoting.nio.FifoBuffer$Pointer.receive(FifoBuffer.java:137)
at org.jenkinsci.remoting.nio.FifoBuffer.receive(FifoBuffer.java:310)
at org.jenkinsci.remoting.nio.NioChannelHub.run(NioChannelHub.java:561)
... 6 more

I have unticked "Response Time" from "Preventive Node Monitoring" and Slaves has -Dhudson.slaves.ChannelPinger.pingInterval=1 set.

Any other workaround available?

muppy.cwa@gmail.com (JIRA)

unread,

Sep 19, 2016, 9:23:05 AM9/19/16

to jenkinsc...@googlegroups.com

Markus Niklasson edited a comment on

Hi,

We have recently also encountered disconnection issues. Slave is a Windows 7 (x64) PC with enough of RAM and CPU to run heavy applications. The Jenkins master is a Enterprise Redhat 7 (3.10.0-327.18.2.el7.x86_64) running Jenkins 2.23 also with enough memory and so on to run Jenkins. Both running Java 8 update 102. The slave are connected through JNLP. Network can be a bit unstable at times.

joe@externl.com (JIRA)

unread,

Sep 19, 2016, 9:41:12 AM9/19/16

to jenkinsc...@googlegroups.com

Joe George commented on

According to the documentation (https://wiki.jenkins-ci.org/display/JENKINS/Ping+Thread) -Dhudson.slaves.ChannelPinger.pingInterval=1 should be set on Master? You should also try setting -Dhudson.remoting.Launcher.pingIntervalSec=-1 on the Slave.

I haven't experience any issues since disabling pinging this way. Next is to start testing different timeout values.

joe@externl.com (JIRA)

unread,

Sep 19, 2016, 9:41:26 AM9/19/16

to jenkinsc...@googlegroups.com

Joe George edited a comment on

According to the documentation (https://wiki.jenkins-ci.org/display/JENKINS/Ping+Thread) {{-Dhudson.slaves.ChannelPinger.pingInterval=1}} should be set on Master? _Master_. You should also try setting {{-Dhudson.remoting.Launcher.pingIntervalSec=-1}} on the Slave.

I haven't experience any issues since disabling pinging this way. Next is to start testing different timeout values.

muppy.cwa@gmail.com (JIRA)

unread,

Sep 21, 2016, 2:40:03 AM9/21/16

to jenkinsc...@googlegroups.com

Markus Niklasson commented on

Thanks for the tip!

By disabling the ping completely it made it more stable. However, I still experience intermittent connectivity problems. During an execution, the slave computer went offline for a couple of seconds and then reconnects to Jenkins Master as seen in the system log:
—
Accepted connection #7 from /10.31.43.49:52692

Sep 21, 2016 8:14:21 AM INFO jenkins.slaves.DefaultJnlpSlaveReceiver handle

Disconnecting Buildserver as we are reconnected from the current peer

Sep 21, 2016 8:29:49 AM WARNING org.jenkinsci.remoting.nio.NioChannelHub run

Communication problem

java.io.IOException: Connection timed out
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(Unknown Source)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source)
at sun.nio.ch.IOUtil.read(Unknown Source)
at sun.nio.ch.SocketChannelImpl.read(Unknown Source)
at org.jenkinsci.remoting.nio.FifoBuffer$Pointer.receive(FifoBuffer.java:137)
at org.jenkinsci.remoting.nio.FifoBuffer.receive(FifoBuffer.java:310)
at org.jenkinsci.remoting.nio.NioChannelHub.run(NioChannelHub.java:561)

at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

Sep 21, 2016 8:29:49 AM WARNING jenkins.slaves.JnlpSlaveAgentProtocol$Handler$1 onClosed

NioChannelHub keys=3 gen=842933: Computer.threadPoolForRemoting 2 for Buildserver terminated
java.io.IOException: Connection aborted: org.jenkinsci.remoting.nio.NioChannelHub$MonoNioTransport@28b1969e[name=Buildserver]

at org.jenkinsci.remoting.nio.NioChannelHub$NioTransport.abort(NioChannelHub.java:208)
at org.jenkinsci.remoting.nio.NioChannelHub.run(NioChannelHub.java:629)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: Connection timed out
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(Unknown Source)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source)
at sun.nio.ch.IOUtil.read(Unknown Source)
at sun.nio.ch.SocketChannelImpl.read(Unknown Source)
at org.jenkinsci.remoting.nio.FifoBuffer$Pointer.receive(FifoBuffer.java:137)
at org.jenkinsci.remoting.nio.FifoBuffer.receive(FifoBuffer.java:310)
at org.jenkinsci.remoting.nio.NioChannelHub.run(NioChannelHub.java:561)
... 6 more

Any ideas how I can prevent the Master from disconnecting the slave (use the reconnected session instead)?

o.v.nenashev@gmail.com (JIRA)

unread,

Jan 2, 2019, 5:33:08 AM1/2/19

to jenkinsc...@googlegroups.com

Oleg Nenashev commented on