[JIRA] [core] (JENKINS-28155) Job fails with [An existing connection was forcibly closed by the remote host]

161 views
Skip to first unread message

vasilena.treneva@softwareag.com (JIRA)

unread,
Apr 29, 2015, 9:19:24 AM4/29/15
to jenkinsc...@googlegroups.com
Issue Type: Bug Bug
Assignee: Unassigned
Components: core, slave-utilization
Created: 29/Apr/15 1:17 PM
Description:

Our Windows slaves are JNLP connected using scheduled tasks (like this: https://wiki.jenkins-ci.org/display/JENKINS/Launch+Java+Web+Start+slave+agent+via+Windows+Scheduler).

When a job starts working with a slave it suddenly fails because the connection was lost. It says something on slave side killed the connection. Few minutes afterwards the slave is back on-line. I am sure it is not a network issue as I tried pinging the network from system start-up till the problem occurs. I could not find find what kills the connection and disable TCP chimney (advice from google) did not work.

The exception is below.

I think proper behaviour would be not to fail the job but attempt to reconnect. Any hints of additional configuration that will help me resolve this issue is appreciated.

Building remotely on vmbam32.eur.ad.sag (R2 Server 2012 Enterprise Windows) in workspace c:\jenkins\workspace\optimize-install
FATAL: java.io.IOException: Connection aborted: org.jenkinsci.remoting.nio.NioChannelHub$MonoNioTransport@b7add1[name=vmbam32.eur.ad.sag]
hudson.remoting.RequestAbortedException: java.io.IOException: Connection aborted: org.jenkinsci.remoting.nio.NioChannelHub$MonoNioTransport@b7add1[name=vmbam32.eur.ad.sag]
at hudson.remoting.Request.abort(Request.java:296)
at hudson.remoting.Channel.terminate(Channel.java:815)
at hudson.remoting.Channel$2.terminate(Channel.java:492)
at hudson.remoting.AbstractByteArrayCommandTransport$1.terminate(AbstractByteArrayCommandTransport.java:72)
at org.jenkinsci.remoting.nio.NioChannelHub$NioTransport.abort(NioChannelHub.java:208)
at org.jenkinsci.remoting.nio.NioChannelHub.run(NioChannelHub.java:628)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
at ......remote call to vmbam32.eur.ad.sag(Native Method)
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1361)
at hudson.remoting.Request.call(Request.java:171)
at hudson.remoting.Channel.call(Channel.java:752)
at hudson.FilePath.act(FilePath.java:980)
at hudson.FilePath.act(FilePath.java:969)
at hudson.FilePath.mkdirs(FilePath.java:1152)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1269)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:610)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:532)
at hudson.model.Run.execute(Run.java:1744)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:374)
Caused by: java.io.IOException: Connection aborted: org.jenkinsci.remoting.nio.NioChannelHub$MonoNioTransport@b7add1[name=vmbam32.eur.ad.sag]
at org.jenkinsci.remoting.nio.NioChannelHub$NioTransport.abort(NioChannelHub.java:208)
at org.jenkinsci.remoting.nio.NioChannelHub.run(NioChannelHub.java:628)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: An existing connection was forcibly closed by the remote host
at sun.nio.ch.SocketDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(Unknown Source)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source)
at sun.nio.ch.IOUtil.read(Unknown Source)
at sun.nio.ch.SocketChannelImpl.read(Unknown Source)
at org.jenkinsci.remoting.nio.FifoBuffer$Pointer.receive(FifoBuffer.java:136)
at org.jenkinsci.remoting.nio.FifoBuffer.receive(FifoBuffer.java:306)
at org.jenkinsci.remoting.nio.NioChannelHub.run(NioChannelHub.java:561)
... 6 more
Environment: Slave Windows Server 2012 R
Master Windows Server 2012 R
Jenkins version 1.611
Project: Jenkins
Priority: Major Major
Reporter: Vassilena Treneva
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira

vasilena.treneva@softwareag.com (JIRA)

unread,
Apr 29, 2015, 9:21:20 AM4/29/15
to jenkinsc...@googlegroups.com
Change By: Vassilena Treneva (29/Apr/15 1:20 PM)
Component/s: windows-slaves-plugin

daniel@beckweb.net (JIRA)

unread,
May 4, 2015, 6:16:01 AM5/4/15
to jenkinsc...@googlegroups.com

I am sure it is not a network issue as I tried pinging the network from system start-up till the problem occurs.

Why are you so sure about this? Pinging does not keep a connection open.
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265)
Atlassian logo

vasilena.treneva@softwareag.com (JIRA)

unread,
May 4, 2015, 6:19:01 AM5/4/15
to jenkinsc...@googlegroups.com

Hi Daniel,

I will rephrase - I am sure that pinging the master from the slave is fine as I initially suspect some kind of network failure.
Do you have some suggestion that can help me debug or resolve my problem?

vasilena.treneva@softwareag.com (JIRA)

unread,
May 7, 2015, 3:16:02 AM5/7/15
to jenkinsc...@googlegroups.com
Vassilena Treneva updated an issue
 
Jenkins / Bug JENKINS-28155

SQL server that I have installed on this slave, has some TCP settings. By default it listens to 1433. Could that be some kind of a problem? Any ideas? SQL killing some connection?

The post here: http://www.redmountainsw.com/wordpress/864/tcp-provider-error-0-an-existing-connection-was-forcibly-closed-by-the-remote-host/ provides a lot of information, but my error is not on SQL side and I don’t want to turn connection pooling off…
Change By: Vassilena Treneva
Attachment: sql-server-manager.jpg

vasilena.treneva@softwareag.com (JIRA)

unread,
May 25, 2015, 7:55:05 AM5/25/15
to jenkinsc...@googlegroups.com
 
Re: Job fails with [An existing connection was forcibly closed by the remote host]

On my slave where I see this error in the server management console I see this:

TCP/IP failed to establish an outgoing connection because the selected local endpoint was recently used to connect to the same remote endpoint. This error typically occurs when outgoing connections are opened and closed at a high rate, causing all available local ports to be used and forcing TCP/IP to reuse a local port for an outgoing connection. To minimize the risk of data corruption, the TCP/IP standard requires a minimum time period to elapse between successive connections from a given local endpoint to a given remote endpoint.

I guess Jenkins is abusing the connection somehow Any ideas how to resolve it?

vasilena.treneva@softwareag.com (JIRA)

unread,
May 25, 2015, 7:56:05 AM5/25/15
to jenkinsc...@googlegroups.com
Vassilena Treneva updated an issue
Change By: Vassilena Treneva
Component/s: remoting
Priority: Major Critical

jan.mueller@janitza.de (JIRA)

unread,
Jul 2, 2015, 8:31:01 AM7/2/15
to jenkinsc...@googlegroups.com
Jan Müller commented on Bug JENKINS-28155
 
Re: Job fails with [An existing connection was forcibly closed by the remote host]

We may have a similar problem. We start a windows vm and connect it via jnlp to the master, then it starts some work and after some time, usually 3-5h the slave gets disconnected unintentionally:

java.io.IOException: Connection aborted: org.jenkinsci.remoting.nio.NioChannelHub$MonoNioTransport@7aa84678[name=QF-WinXP]
	at org.jenkinsci.remoting.nio.NioChannelHub$NioTransport.abort(NioChannelHub.java:211)
	at org.jenkinsci.remoting.nio.NioChannelHub.run(NioChannelHub.java:631)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
	at sun.nio.ch.IOUtil.read(IOUtil.java:197)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
	at org.jenkinsci.remoting.nio.FifoBuffer$Pointer.receive(FifoBuffer.java:136)
	at org.jenkinsci.remoting.nio.FifoBuffer.receive(FifoBuffer.java:306)
	at org.jenkinsci.remoting.nio.NioChannelHub.run(NioChannelHub.java:564)

jan.mueller@janitza.de (JIRA)

unread,
Jul 2, 2015, 8:33:01 AM7/2/15
to jenkinsc...@googlegroups.com
Jan Müller edited a comment on Bug JENKINS-28155
We may have a similar problem. We start a windows vm and connect it via jnlp to the master, then it starts some work and after some time, usually 3-5h the slave gets disconnected unintentionally:
{code}

java.io.IOException: Connection aborted: org.jenkinsci.remoting.nio.NioChannelHub$MonoNioTransport@7aa84678[name=QF-WinXP]
at org.jenkinsci.remoting.nio.NioChannelHub$NioTransport.abort(NioChannelHub.java:211)
at org.jenkinsci.remoting.nio.NioChannelHub.run(NioChannelHub.java:631)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at org.jenkinsci.remoting.nio.FifoBuffer$Pointer.receive(FifoBuffer.java:136)
at org.jenkinsci.remoting.nio.FifoBuffer.receive(FifoBuffer.java:306)
at org.jenkinsci.remoting.nio.NioChannelHub.run(NioChannelHub.java:564)
{code}

We have only one windows machine and this machine shows this behavior. All the linux slaves are connected constantly without any problems.

w.maleska@gmail.com (JIRA)

unread,
Jul 21, 2015, 9:13:01 AM7/21/15
to jenkinsc...@googlegroups.com
Waldek M commented on Bug JENKINS-28155

Same here. Slave started on WIndows 7, on an AD account, as a service.

java.io.IOException: Connection aborted: org.jenkinsci.remoting.nio.NioChannelHub$MonoNioTransport@240c0b4e[name=ST_windows_slave]
	at org.jenkinsci.remoting.nio.NioChannelHub$NioTransport.abort(NioChannelHub.java:208)
	at org.jenkinsci.remoting.nio.NioChannelHub.run(NioChannelHub.java:628)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:724)
Caused by: java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
	at sun.nio.ch.IOUtil.read(IOUtil.java:197)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
	at org.jenkinsci.remoting.nio.FifoBuffer$Pointer.receive(FifoBuffer.java:136)
	at org.jenkinsci.remoting.nio.FifoBuffer.receive(FifoBuffer.java:306)
	at org.jenkinsci.remoting.nio.NioChannelHub.run(NioChannelHub.java:561)
	... 6 more

Found some more in slave's log. My master instance is using https and although the certificate
is installed in Windows and browsers recognize it, slave complains about it - as below.
Any ideas would be welcome.

Exception in thread "main" java.io.IOException: Failed to validate a server certificate. If you are using a self-signed certificate, you can use the -noCertificateCheck option to bypass this check.
	at hudson.remoting.Launcher.parseJnlpArguments(Launcher.java:327)
	at hudson.remoting.Launcher.run(Launcher.java:219)
	at hudson.remoting.Launcher.main(Launcher.java:192)
Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.ssl.Alerts.getSSLException(Unknown Source)
	at sun.security.ssl.SSLSocketImpl.fatal(Unknown Source)
	at sun.security.ssl.Handshaker.fatalSE(Unknown Source)
	at sun.security.ssl.Handshaker.fatalSE(Unknown Source)
	at sun.security.ssl.ClientHandshaker.serverCertificate(Unknown Source)
	at sun.security.ssl.ClientHandshaker.processMessage(Unknown Source)
	at sun.security.ssl.Handshaker.processLoop(Unknown Source)
	at sun.security.ssl.Handshaker.process_record(Unknown Source)
	at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
	at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
	at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
	at sun.net.www.protocol.https.HttpsClient.afterConnect(Unknown Source)
	at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(Unknown Source)
	at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(Unknown Source)
	at hudson.remoting.Launcher.parseJnlpArguments(Launcher.java:269)
	... 2 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.validator.PKIXValidator.doBuild(Unknown Source)
	at sun.security.validator.PKIXValidator.engineValidate(Unknown Source)
	at sun.security.validator.Validator.validate(Unknown Source)
	at sun.security.ssl.X509TrustManagerImpl.validate(Unknown Source)
	at sun.security.ssl.X509TrustManagerImpl.checkTrusted(Unknown Source)
	at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(Unknown Source)
	... 14 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.provider.certpath.SunCertPathBuilder.build(Unknown Source)
	at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(Unknown Source)
	at java.security.cert.CertPathBuilder.build(Unknown Source)
	... 20 more

w.maleska@gmail.com (JIRA)

unread,
Jul 21, 2015, 9:14:02 AM7/21/15
to jenkinsc...@googlegroups.com
Waldek M edited a comment on Bug JENKINS-28155
Same here  - although reason might be different . Slave started on WIndows 7, on an AD account, as a service.
 Drops almost immediately.

{noformat}

java.io.IOException: Connection aborted: org.jenkinsci.remoting.nio.NioChannelHub$MonoNioTransport@240c0b4e[name=ST_windows_slave]
at org.jenkinsci.remoting.nio.NioChannelHub$NioTransport.abort(NioChannelHub.java:208)
at org.jenkinsci.remoting.nio.NioChannelHub.run(NioChannelHub.java:628)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at org.jenkinsci.remoting.nio.FifoBuffer$Pointer.receive(FifoBuffer.java:136)
at org.jenkinsci.remoting.nio.FifoBuffer.receive(FifoBuffer.java:306)
at org.jenkinsci.remoting.nio.NioChannelHub.run(NioChannelHub.java:561)
... 6 more
{noformat}


    
  
Found some more in slave's log. My master instance is using https and although the certificate
is installed in Windows and browsers recognize it, slave complains about it - as below.
Any ideas would be welcome.


{noformat}
{noformat}

w.maleska@gmail.com (JIRA)

unread,
Jul 21, 2015, 12:19:02 PM7/21/15
to jenkinsc...@googlegroups.com
Waldek M edited a comment on Bug JENKINS-28155
*UPDATE*: It turned out my issue was related to the recent switch to https (so probably not related to this one).
After providing Java trustStore with RootCA and Intermediate CA - issue resolved.

====

 Same here - although reason might be different. Slave started on WIndows 7, on an AD account, as a service.

vasilena.treneva@softwareag.com (JIRA)

unread,
Jul 23, 2015, 2:45:02 AM7/23/15
to jenkinsc...@googlegroups.com
Vassilena Treneva updated an issue
Change By: Vassilena Treneva
Attachment: Jenkins Slave.xml

vasilena.treneva@softwareag.com (JIRA)

unread,
Jul 23, 2015, 2:45:02 AM7/23/15
to jenkinsc...@googlegroups.com
Vassilena Treneva commented on Bug JENKINS-28155
 
Re: Job fails with [An existing connection was forcibly closed by the remote host]

I have no actual solution for this problem but after doing many different experiments we managed to configure our windows slaves with a scheduled task (Jenkins Slave.xml attached here) and so far they are quite stable. A prerequisite is to have the test user automatically login at OS boot.

vasilena.treneva@softwareag.com (JIRA)

unread,
Jul 23, 2015, 2:46:03 AM7/23/15
to jenkinsc...@googlegroups.com
Vassilena Treneva edited a comment on Bug JENKINS-28155
I have no actual solution for this problem but after doing many different experiments we managed to configure our windows slaves  (win server 2012 R2 standart)  with a scheduled task (Jenkins Slave.xml attached here) and so far they are quite stable. A prerequisite is to have the test user automatically  login  logged  at OS boot.

dan_albu_ro@yahoo.com (JIRA)

unread,
Aug 26, 2015, 2:27:04 AM8/26/15
to jenkinsc...@googlegroups.com
Dan Albu commented on Bug JENKINS-28155

I have started to have the same issues with the same configuration. I am using the LTS version of Jenkins 1.609.1

o.v.nenashev@gmail.com (JIRA)

unread,
Dec 16, 2015, 11:50:03 AM12/16/15
to jenkinsc...@googlegroups.com
Oleg Nenashev assigned an issue to Oleg Nenashev
 
Change By: Oleg Nenashev
Assignee: Oleg Nenashev

lostisok@gmail.com (JIRA)

unread,
Feb 22, 2016, 12:27:03 PM2/22/16
to jenkinsc...@googlegroups.com

george@fischhof.hu (JIRA)

unread,
Jul 12, 2016, 3:47:02 AM7/12/16
to jenkinsc...@googlegroups.com

I have the same issue:
used: jenkins 2.09 (bitbnami stack)
master: win 7 64 bit, java 8.91 32 bit
slave: win 2008R2 64 bit, java 8.91 32 bit, 64 bit
This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
Atlassian logo

robin.bonaudo@thecosmocompany.com (JIRA)

unread,
Jan 25, 2017, 11:27:02 AM1/25/17
to jenkinsc...@googlegroups.com

I had the same issue, also using the Windows' task scheduler to attach the slave.

For me, the issue seems to be due to the "Stop task if it runs longer than x time" option being activated and set to 3 days by default.
Windows kills the task after 3 days, stopping the connection between Jenkins and the slave.
This option is set in the properties of the task, inside Windows' task scheduler.

Hope this can help someone, I was blocked by this problem for a very long time myself.

alokkumar.mahajan@gmail.com (JIRA)

unread,
Feb 7, 2017, 8:37:02 AM2/7/17
to jenkinsc...@googlegroups.com

Hello,
We are also getting this exception with Jenkins ver. 2.38 inconsistently, has anyone found any working solution for this issue? Found https://issues.jenkins-ci.org/browse/JENKINS-27251 where i see a comment saying "It was likely SocketTimeoutException on the slave side in this case (fixed in 2.60 & released in 2.9)." We see "SocketTimeoutException" on slave side. So, does upgrading to 2.9 will solve this issue?

Thanks in advance!

george@fischhof.hu (JIRA)

unread,
Feb 13, 2017, 10:31:02 AM2/13/17
to jenkinsc...@googlegroups.com

Hi,

my working solution is the following:

I use windows for master and for the slave as well.

On the slave I have an administrator user with autologin, when the user logged in it waits 1 minute, then starts jenkins slave with java webstart with the jnlp file:
javaws.lnk c:\CI\slave-agent.jnlp
(javaws.lnk is created to be on path)

I use java 1.7.1 because the javaws needs 1.7, but 1.7.89 has a security enhancement, and it does not allow to start the javaws automatically, it needs manual confirmation. (I did not check the versions between 1.7.1 and 1.7.89) 1.7.1 allows the starting.

This way the connection is stable.

BR,
George

o.v.nenashev@gmail.com (JIRA)

unread,
Mar 11, 2018, 7:13:02 PM3/11/18
to jenkinsc...@googlegroups.com
Oleg Nenashev updated an issue
 
Change By: Oleg Nenashev
Component/s: slave-utilization
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
Atlassian logo

o.v.nenashev@gmail.com (JIRA)

unread,
Mar 13, 2018, 9:20:03 AM3/13/18
to jenkinsc...@googlegroups.com
Oleg Nenashev assigned an issue to Unassigned
Change By: Oleg Nenashev
Assignee: Oleg Nenashev

o.v.nenashev@gmail.com (JIRA)

unread,
Mar 13, 2018, 9:24:02 AM3/13/18
to jenkinsc...@googlegroups.com
Oleg Nenashev commented on Bug JENKINS-28155
 
Re: Job fails with [An existing connection was forcibly closed by the remote host]

Still seems like a generic infrastructure issue symptom, which may be caused by Remoting defects in some cases. Investigation from pjdarton is definitely correct, but I'd guess the most of reporters/voters here see quite another issue. Maybe it should be reported as a separate ticket since it may be worked around somehow.

I do not anticipate to have any time for Remoting maintenance anytime soon, so I am going to unassign this issue.

pjdarton@gmail.com (JIRA)

unread,
Mar 13, 2018, 10:02:02 AM3/13/18
to jenkinsc...@googlegroups.com
pjdarton commented on Bug JENKINS-28155

FYI we got to the bottom of why our Windows slaves were disconnecting - it would appear that the Windows DHCP Client is incompatible with the Windows Time Service.

Our slaves were VMs created within OpenStack, and what we were seeing was a failure to renew the DHCP lease correctly. When OpenStack detects that the guest OS has failed to renew the DHCP lease on time, it (briefly) drops the network link in order to prompt a lease renewal. However this causes Windows to panic and kill all TCP connections (due to the way Windows mishandles network layers).
It seems that the DHCP client is not calculating the renewal time in a manner that's independent of the system's idea of "real time", and so it all goes wrong when the date/time gets changed (by the Windows Time Service), triggering OpenStack to bounce the physical layer, which Windows cascades in to an application-layer network outage killing the TCP connection that the slave relies on.

We "fixed" this by forcing our slaves to:

  1. run "w32tm /resync" until they'd got the time synchronized,
  2. turn off the Windows Time Service entirely,
  3. ipconfig /release /renew to update the DHCP lease time
  4. start the Jenkins JNLP slave process

This ensured that Windows would not update its clock while the slave's TCP connection was live, meaning that we weren't affected by the DHCP client's inability to keep the network alive after clock changes.
Since doing that we've not had any further problems of this nature (and we're quite pleased with that!)

Note: I've also seen Windows 10 report unpredictable (and incorrect) DHCP lease renewal times on other (non-OpenStack) machines - it lies.

dan_albu_ro@yahoo.com (JIRA)

unread,
Mar 13, 2018, 10:03:02 AM3/13/18
to jenkinsc...@googlegroups.com
Dan Albu commented on Bug JENKINS-28155

After i did mass refactoring on my INFRA i do not see this type of errors anymore, except when the network in the lab goes high wire.
Reply all
Reply to author
Forward
0 new messages