Amazon slaves automatically disconnect from Master on longer builds

[George Cimpoies]

Environment: Jenkins 2.187 Amazon EC2 1.44.1 Swarm 3.13  

I have set up the connection between Jenkins and AWS via Amazon EC2 plugin. Jenkins master cloud config:

The node connects via the Amazon plugin and then creates a new connection via Swarm plugin and the job ends up running on the connection made through swarm. (This is because my jobs include TestComplete & FlaUI and winRM is not quite suited for their requirements).

Jobs that take under 25 min run successfully, anything that goes over 25-26 min fails with the following:

12:49:46 java.io.IOException: Backing channel 'JNLP4-connect connection from 10.230.0.101/10.230.0.101:49724' is disconnected.
12:49:46    at hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:214)
12:49:46    at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:283)
12:49:46    at com.sun.proxy.$Proxy89.isAlive(Unknown Source)
12:49:46    at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1172)
12:49:46    at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1164)
12:49:46    at hudson.Launcher$ProcStarter.join(Launcher.java:492)
12:49:46    at hudson.plugins.gradle.Gradle.performTask(Gradle.java:333)
12:49:46    at hudson.plugins.gradle.Gradle.perform(Gradle.java:225)
12:49:46    at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
12:49:46    at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:741)
12:49:46    at hudson.model.Build$BuildExecution.build(Build.java:206)
12:49:46    at hudson.model.Build$BuildExecution.doRun(Build.java:163)
12:49:46    at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504)
12:49:46    at hudson.model.Run.execute(Run.java:1815)
12:49:46    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
12:49:46    at hudson.model.ResourceController.execute(ResourceController.java:97)
12:49:46    at hudson.model.Executor.run(Executor.java:429)
12:49:46 Caused by: java.nio.channels.ClosedChannelException
12:49:46    at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154)
12:49:46    at org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:179)
12:49:46    at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:795)
12:49:46    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
12:49:46    at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
12:49:46    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
12:49:46    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
12:49:46    at java.lang.Thread.run(Thread.java:748)

On the Master's log I see:

ouch, stdout exception for java  -jar C:\Windows\Temp\remoting.jar -workDir C:\Users\cluj
java.lang.NumberFormatException: For input string: "4294967295"
    at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
    at java.lang.Integer.parseInt(Integer.java:583)
    at java.lang.Integer.parseInt(Integer.java:615)
    at hudson.plugins.ec2.win.winrm.WinRMClient.slurpOutput(WinRMClient.java:151)
    at hudson.plugins.ec2.win.winrm.WindowsProcess$1.run(WindowsProcess.java:99)

Sep 27, 2019 10:45:27 AM INFO hudson.remoting.SynchronousCommandTransport$ReaderThread run
I/O error in channel EC2 (Itiviti AWS) - Windows Jenkins node autoconnecting to deb-jenkins-prd using Swarm plugin (i-03c88c3229acf1e7a)
java.io.EOFException
    at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2681)
    at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3156)
    at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:862)
    at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358)
    at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
    at hudson.remoting.Command.readFrom(Command.java:140)
    at hudson.remoting.Command.readFrom(Command.java:126)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:36)
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
Caused: java.io.IOException: Unexpected termination of the channel
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)

Sep 27, 2019 10:45:27 AM SEVERE hudson.plugins.ec2.win.winrm.WinRMClient sendRequest
I/O Exception in HTTP POST
java.io.IOException: Attempted read from closed stream.
    at org.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:165)
    at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135)
    at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
    at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
    at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
    at java.io.InputStreamReader.read(InputStreamReader.java:184)
    at java.io.Reader.read(Reader.java:140)
    at org.apache.http.util.EntityUtils.toString(EntityUtils.java:227)
    at org.apache.http.util.EntityUtils.toString(EntityUtils.java:308)
    at hudson.plugins.ec2.win.winrm.WinRMClient.sendRequest(WinRMClient.java:261)
    at hudson.plugins.ec2.win.winrm.WinRMClient.sendRequest(WinRMClient.java:188)
    at hudson.plugins.ec2.win.winrm.WinRMClient.sendInput(WinRMClient.java:120)
    at hudson.plugins.ec2.win.winrm.WindowsProcess$2.run(WindowsProcess.java:134)

Sep 27, 2019 10:45:27 AM WARNING hudson.plugins.ec2.win.winrm.WindowsProcess$2 run
ouch, STDIN exception for java  -jar C:\Windows\Temp\remoting.jar -workDir C:\Users\cluj
java.io.IOException: Attempted read from closed stream.
    at org.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:165)
    at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135)
    at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
    at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
    at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
    at java.io.InputStreamReader.read(InputStreamReader.java:184)
    at java.io.Reader.read(Reader.java:140)
    at org.apache.http.util.EntityUtils.toString(EntityUtils.java:227)
    at org.apache.http.util.EntityUtils.toString(EntityUtils.java:308)
    at hudson.plugins.ec2.win.winrm.WinRMClient.sendRequest(WinRMClient.java:261)
Caused: hudson.plugins.ec2.win.winrm.RuntimeIOException: I/O Exception Attempted read from closed stream.
    at hudson.plugins.ec2.win.winrm.WinRMClient.sendRequest(WinRMClient.java:276)
    at hudson.plugins.ec2.win.winrm.WinRMClient.sendRequest(WinRMClient.java:188)
    at hudson.plugins.ec2.win.winrm.WinRMClient.sendInput(WinRMClient.java:120)
    at hudson.plugins.ec2.win.winrm.WindowsProcess$2.run(WindowsProcess.java:134)

Any help is much appreciated!

--

George Cimpoies
Research & Development, Itiviti
Software engineer
george....@itiviti.com

______________________________

itiviti.com
         

The information contained in or attached to this email is strictly confidential. If you are not the intended recipient, please notify us immediately by telephone and return the message to us. Email communications by definition contain personal information. The Itiviti group of companies is subject to European data protection regulations.

Itiviti’s Privacy Notice is available at www.itiviti.com. Itiviti expects the recipient of this email to be compliant with Itiviti’s Privacy Notice and applicable regulations. Please advise us immediately at dataprote...@Itiviti.com if you are not compliant with these.

[George Cimpoies]

The actual log on the master is:

Idle timeout of EC2 (Itiviti AWS) - Windows Jenkins node autoconnecting to deb-jenkins-prd using Swarm plugin (i-000908b57bb5d82a7) after 30 idle minutes, instance statusRUNNING
Sep 30, 2019 8:40:45 AM INFO hudson.plugins.ec2.EC2AbstractSlave idleTimeout
EC2 instance idle time expired: i-000908b57bb5d82a7
Sep 30, 2019 8:40:46 AM INFO hudson.plugins.ec2.EC2OndemandSlave terminate
Terminated EC2 instance (terminated): i-000908b57bb5d82a7
Sep 30, 2019 8:40:46 AM INFO jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed
IOHub#1: Worker[channel:java.nio.channels.SocketChannel[connected local=/172.17.0.2:40440 remote=10.230.0.71/10.230.0.71:49735]] / Computer.threadPoolForRemoting [#85772] for ec2amaz-glc1084 terminated: java.nio.channels.ClosedChannelException
Sep 30, 2019 8:40:46 AM INFO hudson.model.Run execute
aws-ul-trader-extension-master-desk-uitests-listorders #22 main build action completed: FAILURE
Sep 30, 2019 8:40:46 AM INFO hudson.plugins.ec2.EC2OndemandSlave terminate
Removed EC2 instance from jenkins master: i-000908b57bb5d82a7

So there is a timeout of 30 minutes. Where is this set and how can I increase it?

[Gavin]

So just FYI, you've contacted the development list (questions about developing core and plugins) with a usage/support question. 

most have trained themselves to not reply to off topic posts, so you'll get better responses from the users list.

Gavin

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CABE7CDfCACH4wtYYhzgG0WVaQe-PLasgSDgMADh5D2ZCQj8Gpw%40mail.gmail.com.