[JIRA] (JENKINS-59885) "Agent went offline" or "Channel alread closed" during the build

[nico@nicovandekamp.nl (JIRA)]

Nico van de Kamp created an issue

Jenkins / JENKINS-59885 "Agent went offline" or "Channel alread closed" during the build Issue Type: Bug Assignee: Praqma Support Attachments: Configuration Jenkins node.jpg, Jenkinsslave-C__Jenkins service stopping .jpg Components: component Created: 2019-10-22 13:25 Environment: We have a Linux master with 77 windows 2012R2 slaves and a few linux node's. On this moment we have Jenkins 2.9. In the near future we will update Jenkins to the latest version and split up the environment into DEV, ACC and PRD. So than we have three masters with each approx. 25 windows 2012R2 nodes (slaves) Labels: jenkins configuration slave windows Priority: Major Reporter: Nico van de Kamp The main job is started, which is starting a some other (sub)jobs. Mostly at the same sub-job, after the first four jobs has been executed without any problem, a job is started. When that job is started or if this job runs one or three minutes we get the issue "Agent went offline". When it runs like say 4 min. than it runs to the end without any issue! That job runs for 1,5 hour or even more! After it went wrong with "Agent went offliner" or with "Channel already closed" we start the main job again and than it works always without a issue!!!!????? WHY????? Analyzing this: Windows event viewer (see screenshot), I can see that the Jenkinsslave-C_Jenkins is "normally" stopped. But by whom? The Job is not doing this or at least so far as I can see the Job is not doing this and should not do this. See my screenshot. In this screenshot it's a few seconds later the service is stopped. But I have other circumstances and then it was in the exact same second! Jenkins logging "Jenkins-slave.err". There I see issue's with JNLP connection but on this moment I don't know what to do about uit! What do I have to change or were can I find documentation.  INFO: Trying protocol: JNLP3-connect Oct 21, 2019 7:01:43 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Server didn't accept the handshake: Unknown protocol:Protocol:JNLP3-connect Oct 21, 2019 7:01:43 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Connecting to <ip adress>:<fixed port adress> Oct 21, 2019 7:01:43 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Trying protocol: JNLP2-connect Oct 21, 2019 7:01:43 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Connected Oct 21, 2019 7:12:21 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Terminated Oct 21, 2019 7:12:31 PM jenkins.slaves.restarter.JnlpSlaveRestarterInstaller$2$1 onReconnect INFO: Restarting agent via jenkins.slaves.restarter.WinswSlaveRestarter@48eae634 Oct 21, 2019 7:12:34 PM hudson.remoting.jnlp.Main createEngine INFO: Setting up slave: nodename Oct 21, 2019 7:12:34 PM hudson.remoting.jnlp.Main$CuiListener <init> INFO: Jenkins agent is running in headless mode. Oct 21, 2019 7:12:34 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Locating server among http://<ip adres>/jenkins/ Oct 21, 2019 7:12:34 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Handshaking Oct 21, 2019 7:12:34 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Connecting to <ip adress>:<fixed port adress> Oct 21, 2019 7:12:34 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Trying protocol: JNLP3-connect Oct 21, 2019 7:12:34 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Server didn't accept the handshake: Unknown protocol:Protocol:JNLP3-connect Oct 21, 2019 7:12:34 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Connecting to <ip adress>:<fixed port adress> Oct 21, 2019 7:12:34 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Trying protocol: JNLP2-connect Oct 21, 2019 7:12:34 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Connected Oct 21, 2019 7:13:58 PM hudson.remoting.jnlp.Main createEngine INFO: Setting up slave: nodename Oct 21, 2019 7:13:58 PM hudson.remoting.jnlp.Main$CuiListener <init> INFO: Jenkins agent is running in headless mode. Oct 21, 2019 7:13:58 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Locating server among http://<ip adress>jenkins/ Oct 21, 2019 7:13:58 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Handshaking Oct 21, 2019 7:13:58 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Connecting to <ip adress>:<fixed port adress> Oct 21, 2019 7:13:58 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Trying protocol: JNLP3-connect Oct 21, 2019 7:13:58 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Server didn't accept the handshake: Unknown protocol:Protocol:JNLP3-connect Oct 21, 2019 7:13:58 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Connecting to <ip adress>:<fixed port adress> Oct 21, 2019 7:13:58 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Trying protocol: JNLP2-connect Oct 21, 2019 7:13:58 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Connected Failed to load native POSIX impl; falling back on Java impl. Unsupported OS. Oct 21, 2019 10:51:41 PM hudson.plugins.tmpcleaner.TmpCleanTask call INFO: end TmpCleanTask Oct 22, 2019 4:51:41 AM hudson.plugins.tmpcleaner.TmpCleanTask call INFO: end TmpCleanTask Oct 22, 2019 10:51:41 AM hudson.plugins.tmpcleaner.TmpCleanTask call INFO: end TmpCleanTask The console log Agent went offline at the same time the JNLP is TERMINATED in Jenkins-slave.err file!: 19:12:10 2019-10-21 19:12:10.2303|INFO|Deploy.Program|File upload successfully done: xxxx.json 19:12:10 2019-10-21 19:12:10.2303|INFO|Deploy.Program|File upload successfully done: xxxx.json 19:12:10 2019-10-21 19:12:10.3153|INFO|Deploy.Program|File upload successfully done: yyy.json 19:12:21 Agent went offline during the build19:12:21 ERROR: Connection was broken: java.io.IOException: Connection aborted: org.jenkinsci.remoting.nio.NioChannelHub$MonoNioTransport@2838043[name=<node_name>] 19:12:21 at org.jenkinsci.remoting.nio.NioChannelHub$NioTransport.abort(NioChannelHub.java:208) 19:12:21 at org.jenkinsci.remoting.nio.NioChannelHub.run(NioChannelHub.java:629)19:12:21 at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) 19:12:21 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)19:12:21 at java.util.concurrent.FutureTask.run(FutureTask.java:266)19:12:21 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 19:12:21 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 19:12:21 at java.lang.Thread.run(Thread.java:745) 19:12:21 Caused by: java.io.IOException: Connection timed out19:12:21 at sun.nio.ch.FileDispatcherImpl.read0(Native Method) 19:12:21 at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) 19:12:21 at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) 19:12:21 at sun.nio.ch.IOUtil.read(IOUtil.java:197)19:12:21 at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) 19:12:21 at org.jenkinsci.remoting.nio.FifoBuffer$Pointer.receive(FifoBuffer.java:137) 19:12:21 at org.jenkinsci.remoting.nio.FifoBuffer.receive(FifoBuffer.java:310) 19:12:21 at org.jenkinsci.remoting.nio.NioChannelHub.run(NioChannelHub.java:561) 19:12:21 ... 6 more19:12:21  Screenshot configuring the node   I hope that someone can help me out this issue. When go DuckDuckGo than I find out I'm not the only one who have this issue but I can't find a solution.  Regards, Nico Add Comment

This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)

[nico@nicovandekamp.nl (JIRA)]

Nico van de Kamp updated an issue

Jenkins / JENKINS-59885 "Agent went offline" or "Channel alread closed" during the build

Change By: Nico van de Kamp

The main job is started, which is starting a some other (sub)jobs. Mostly at the same sub-job, after the first four jobs has been executed without any problem, a job is started. When that job is started or if this job runs one or three minutes we get the issue "Agent went offline". When it runs like say 4 min. than it runs to the end without any issue! That job runs for 1,5 hour or even more! After it went wrong with "Agent went offliner" or with "Channel already closed" we start the main job again and than it works always without a issue!!!!????? WHY????? Analyzing this:

* *Windows event viewer* (see screenshot), I can see that the Jenkinsslave-C_Jenkins is "normally" stopped. But by whom? The Job is not doing this or at least so far as I can see the Job is not doing this and should not do this. See my screenshot. In this screenshot it's a few seconds later the service is stopped. But I have other circumstances and then it was in the exact same second! * *Jenkins logging "Jenkins-slave.err"*. There I see issue's with JNLP connection but on this moment I don't know what to do about uit it ! What do I have to change or were can I find documentation.

INFO: Trying protocol: JNLP3-connect Oct 21, 2019 7:01:43 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Server didn't accept the handshake: Unknown protocol:Protocol:JNLP3-connect Oct 21, 2019 7:01:43 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Connecting to <ip adress>:<fixed port adress> Oct 21, 2019 7:01:43 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Trying protocol: JNLP2-connect Oct 21, 2019 7:01:43 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Connected

*{color:# FF0000 ff0000 }Oct 21, 2019 7:12:21 PM hudson.remoting.jnlp.Main$CuiListener status{color}* *{color:# FF0000 ff0000 }INFO: Terminated{color}*

Oct 21, 2019 7:12:31 PM jenkins.slaves.restarter.JnlpSlaveRestarterInstaller$2$1 onReconnect INFO: Restarting agent via jenkins.slaves.restarter.WinswSlaveRestarter@48eae634 Oct 21, 2019 7:12:34 PM hudson.remoting.jnlp.Main createEngine INFO: Setting up slave: nodename Oct 21, 2019 7:12:34 PM hudson.remoting.jnlp.Main$CuiListener <init> INFO: Jenkins agent is running in headless mode. Oct 21, 2019 7:12:34 PM hudson.remoting.jnlp.Main$CuiListener status

INFO: Locating server among [http://<ip adres>/jenkins/ |http://%3Cip%20adres%3E/jenkins/ ]

Oct 21, 2019 7:12:34 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Handshaking Oct 21, 2019 7:12:34 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Connecting to <ip adress>:<fixed port adress> Oct 21, 2019 7:12:34 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Trying protocol: JNLP3-connect Oct 21, 2019 7:12:34 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Server didn't accept the handshake: Unknown protocol:Protocol:JNLP3-connect Oct 21, 2019 7:12:34 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Connecting to <ip adress>:<fixed port adress> Oct 21, 2019 7:12:34 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Trying protocol: JNLP2-connect Oct 21, 2019 7:12:34 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Connected Oct 21, 2019 7:13:58 PM hudson.remoting.jnlp.Main createEngine INFO: Setting up slave: nodename Oct 21, 2019 7:13:58 PM hudson.remoting.jnlp.Main$CuiListener <init> INFO: Jenkins agent is running in headless mode. Oct 21, 2019 7:13:58 PM hudson.remoting.jnlp.Main$CuiListener status

INFO: Locating server among [http://<ip adress>jenkins/ |http://%3Cip%20adress%3Ejenkins/ ]

Oct 21, 2019 7:13:58 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Handshaking Oct 21, 2019 7:13:58 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Connecting to <ip adress>:<fixed port adress> Oct 21, 2019 7:13:58 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Trying protocol: JNLP3-connect Oct 21, 2019 7:13:58 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Server didn't accept the handshake: Unknown protocol:Protocol:JNLP3-connect Oct 21, 2019 7:13:58 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Connecting to <ip adress>:<fixed port adress> Oct 21, 2019 7:13:58 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Trying protocol: JNLP2-connect Oct 21, 2019 7:13:58 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Connected Failed to load native POSIX impl; falling back on Java impl. Unsupported OS. Oct 21, 2019 10:51:41 PM hudson.plugins.tmpcleaner.TmpCleanTask call INFO: end TmpCleanTask Oct 22, 2019 4:51:41 AM hudson.plugins.tmpcleaner.TmpCleanTask call INFO: end TmpCleanTask Oct 22, 2019 10:51:41 AM hudson.plugins.tmpcleaner.TmpCleanTask call INFO: end TmpCleanTask

* *The console log* Agent went offline at the same time the JNLP is TERMINATED in Jenkins-slave.err file!:

19:12:10 2019-10-21 19:12:10.2303|INFO|Deploy.Program|File upload successfully done: xxxx.json 19:12:10 2019-10-21 19:12:10.2303|INFO|Deploy.Program|File upload successfully done: xxxx.json 19:12:10 2019-10-21 19:12:10.3153|INFO|Deploy.Program|File upload successfully done: yyy.json

*{color:# FF0000 ff0000 }19:12:21 Agent went offline during the build19:12:21{color}* ERROR: Connection was broken: java.io.IOException: Connection aborted: org.jenkinsci.remoting.nio.NioChannelHub$MonoNioTransport@2838043[name=<node_name>]

19:12:21 at org.jenkinsci.remoting.nio.NioChannelHub$NioTransport.abort(NioChannelHub.java:208) 19:12:21 at org.jenkinsci.remoting.nio.NioChannelHub.run(NioChannelHub.java:629)19:12:21 at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) 19:12:21 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)19:12:21 at java.util.concurrent.FutureTask.run(FutureTask.java:266)19:12:21 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 19:12:21 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 19:12:21 at java.lang.Thread.run(Thread.java:745) 19:12:21 Caused by: java.io.IOException: Connection timed out19:12:21 at sun.nio.ch.FileDispatcherImpl.read0(Native Method) 19:12:21 at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) 19:12:21 at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) 19:12:21 at sun.nio.ch.IOUtil.read(IOUtil.java:197)19:12:21 at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) 19:12:21 at org.jenkinsci.remoting.nio.FifoBuffer$Pointer.receive(FifoBuffer.java:137) 19:12:21 at org.jenkinsci.remoting.nio.FifoBuffer.receive(FifoBuffer.java:310) 19:12:21 at org.jenkinsci.remoting.nio.NioChannelHub.run(NioChannelHub.java:561) 19:12:21 ... 6 more19:12:21

* *Screenshot configuring the node*   *I hope that someone can help me out this issue. When go DuckDuckGo than I find out I'm not the only one who have this issue but I can't find a solution.*  *Regards,* *Nico*

Add Comment

[nico@nicovandekamp.nl (JIRA)]

Nico van de Kamp updated an issue

Jenkins / JENKINS-59885 "Agent went offline" or "Channel alread closed" during the build Change By: Nico van de Kamp

The main job is started, which is starting a some other (sub)jobs. Mostly at the same sub-job, after the first four jobs has been executed without any problem, a job is started. When that job is started or if this job runs one or three minutes we get the issue "Agent went offline". When it runs like say 4 min. than it runs to the end without any issue! That job runs for 1,5 hour or even more! After it went wrong with "Agent went offliner" or with "Channel already closed" we start the main job again and than it works always without a issue!!!!????? WHY????? Analyzing this: * *Windows event viewer* (see screenshot), I can see that the Jenkinsslave-C_Jenkins is "normally" stopped. But by whom? The Job is not doing this or at least so far as I can see the Job is not doing this and should not do this. See my screenshot. In this screenshot it's a few seconds later the service is stopped. But I have other circumstances and then it was in the exact same second!

* *Jenkins logging "Jenkins-slave.err"*. There I see issue's with JNLP connection but on this moment I don't know what to do about it! What do I have to change or were can I find documentation.

INFO: Trying protocol: JNLP3-connect Oct 21, 2019 7:01:43 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Server didn't accept the handshake: Unknown protocol:Protocol:JNLP3-connect Oct 21, 2019 7:01:43 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Connecting to <ip adress>:<fixed port adress> Oct 21, 2019 7:01:43 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Trying protocol: JNLP2-connect Oct 21, 2019 7:01:43 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Connected

*{color:#ff0000}Oct 21, 2019 7:12:21 PM hudson.remoting.jnlp.Main$CuiListener status{color}* *{color:#ff0000}INFO: Terminated{color}*

*{color:#ff0000}19:12:21 Agent went offline during the build19 build {color}*{color : #172b4d}19: 12:21{color} * ERROR: Connection was broken: java.io.IOException: Connection aborted: org.jenkinsci.remoting.nio.NioChannelHub$MonoNioTransport@2838043[name=<node_name>]

19:12:21 at org.jenkinsci.remoting.nio.NioChannelHub$NioTransport.abort(NioChannelHub.java:208) 19:12:21 at org.jenkinsci.remoting.nio.NioChannelHub.run(NioChannelHub.java:629)19:12:21 at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) 19:12:21 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)19:12:21 at java.util.concurrent.FutureTask.run(FutureTask.java:266)19:12:21 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 19:12:21 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 19:12:21 at java.lang.Thread.run(Thread.java:745) 19:12:21 Caused by: java.io.IOException: Connection timed out19:12:21 at sun.nio.ch.FileDispatcherImpl.read0(Native Method) 19:12:21 at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) 19:12:21 at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) 19:12:21 at sun.nio.ch.IOUtil.read(IOUtil.java:197)19:12:21 at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) 19:12:21 at org.jenkinsci.remoting.nio.FifoBuffer$Pointer.receive(FifoBuffer.java:137) 19:12:21 at org.jenkinsci.remoting.nio.FifoBuffer.receive(FifoBuffer.java:310) 19:12:21 at org.jenkinsci.remoting.nio.NioChannelHub.run(NioChannelHub.java:561) 19:12:21 ... 6 more19:12:21  * *Screenshot configuring the node*   *I hope that someone can help me out this issue. When go DuckDuckGo than I find out I'm not the only one who have this issue but I can't find a solution.*  *Regards,* *Nico*

Add Comment

[nico@nicovandekamp.nl (JIRA)]

Nico van de Kamp updated an issue

Jenkins / JENKINS-59885 "Agent went offline" or "Channel alread closed" during the build

Change By: Nico van de Kamp

Environment:

We have a Linux master with 77 windows 2012R2 slaves and a few linux node's. On this moment we have Jenkins 2.9. In the near future we will update Jenkins to the latest version and split up the environment into DEV, ACC and PRD. So than we have three masters with each approx. 25 windows 2012R2 nodes (slaves)

Under Manage Jenkins --> Globaal security we have enabled security and "TCP port for JNLP agents" is Fixed with a port number.

Add Comment

[nico@nicovandekamp.nl (JIRA)]

Nico van de Kamp commented on JENKINS-59885

Re: "Agent went offline" or "Channel alread closed" during the build Ok, I understand now that, to take these  JVM options into account by the slave I do have to remove the slave first on the Windows server and then install that slave again. Than these options are taken into account by slave in the JNLP file. I was expecting that what I have filled in here, when I stop/start the Jenkins service on the slave, that these options are taken into account by the slave! I have not set up the environment, so this is new to me. I have created a jobs, and a lot of scripting but not configured the master and slaves. And I hope that we go asap to the latest Jenkins version but till than, I have this issue. And I don't know what to do than try a lot of things.

Add Comment