[JIRA] (JENKINS-44001) Jenkins crash with ClosedChannelException

1,995 views
Skip to first unread message

o.v.nenashev@gmail.com (JIRA)

unread,
Mar 13, 2018, 10:33:35 PM3/13/18
to jenkinsc...@googlegroups.com
Oleg Nenashev assigned an issue to Unassigned
 

Unfortunately I have no capacity to work on Remoting in medium term, so I will unassign it and let others to take it. If somebody is interested to submit a pull request, I will be happy to help to get it reviewed and released.

Jenkins / Bug JENKINS-44001
Jenkins crash with ClosedChannelException
Change By: Oleg Nenashev
Assignee: Oleg Nenashev
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
Atlassian logo

sourabhgaikwad@gmail.com (JIRA)

unread,
May 30, 2018, 6:07:02 PM5/30/18
to jenkinsc...@googlegroups.com
Sourabh Gaikwad commented on Bug JENKINS-44001
 
Re: Jenkins crash with ClosedChannelException

Arun S Thanks, Increasing timeout of ELB worked for me. I was getting this error.
FATAL: command execution failed
java.nio.channels.ClosedChannelException
at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154)
at org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:142)
at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:789)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused: java.io.IOException: Backing channel 'JNLP4-connect connection from 10.11.112.1/10.11.112.1:51938' is disconnected.

eslam.husseiny@gmail.com (JIRA)

unread,
Aug 9, 2018, 5:56:06 AM8/9/18
to jenkinsc...@googlegroups.com

I'm having the same issue:

 

WARNING: IOHub#1: Worker[channel:java.nio.channels.SocketChannel[connected local=/172.17.0.6:50000 remote=0974272363aa38ddf.blah.example.net/10.195.74.13:34503]] / Computer.threadPoolForRemoting 237 for swarm-5d1940f807e4-1a0f2d4a terminated java.nio.channels.ClosedChannelException at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154) at org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:142) at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:721) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Aug 08, 2018 8:13:50 AM jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed WARNING: IOHub#1: Worker[channel:java.nio.channels.SocketChannel[connected local=/172.17.0.6:50000 remote=0974272363aa38ddf.blah.example.net/10.195.74.13:34501]] / Computer.threadPoolForRemoting 238 for swarm-f39e88a85dad-28c72ae9 terminated java.nio.channels.ClosedChannelException at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154) at org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:142) at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:721) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Aug 08, 2018 8:13:50 AM jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed WARNING: IOHub#1: Worker[channel:java.nio.channels.SocketChannel[connected local=/172.17.0.6:50000 remote=05fc2dab9bd28bc23.blah.example.net/10.195.73.6:58646]] / Computer.threadPoolForRemoting 237 for swarm-ab530d51c71c-41b03616 terminated java.nio.channels.ClosedChannelException at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154) at org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:142) at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:721) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Aug 08, 2018 8:13:50 AM jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed WARNING: IOHub#1: Worker[channel:java.nio.channels.SocketChannel[connected local=/172.17.0.6:50000 remote=05fc2dab9bd28bc23.blah.example.net/10.195.73.6:58648]] / Computer.threadPoolForRemoting 238 for swarm-0d9d00d3aadd-e5776da7 terminated java.nio.channels.ClosedChannelException at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154) at org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:142) at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:721) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Aug 08, 2018 8:14:06 AM winstone.Logger logInternal INFO: JVM is terminating. Shutting down Winstone

 

I'm running Jenkins version: 2.73.2

With swarm plugin version 3.6

Jenkins agents with Swarm agents version: 3.13

Both Jenkins master and Jenkins agents (Swarm nodes) are running on ECS and Docker engine version: 17.12.1.ce-1

the ELB idle timeout for Jenkins master is: 1800 seconds

Both Jenkins master and Jenkins agents are Linux based with one extra Windows agent

 

 

This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396)

todd@boxboat.com (JIRA)

unread,
Jul 3, 2019, 12:29:02 PM7/3/19
to jenkinsc...@googlegroups.com

I am experiencing this issue too. My pipeline fails using Kaniko on Kubernetes. 

This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

todd@boxboat.com (JIRA)

unread,
Jul 3, 2019, 12:32:02 PM7/3/19
to jenkinsc...@googlegroups.com
Todd Densmore edited a comment on Bug JENKINS-44001
I am experiencing this issue too. My pipeline fails using Kaniko on Kubernetes. Looks like this might have to do with pod resource limits...  

[https://medium.com/@garunski/closedchannelexception-in-jenkins-with-kubernetes-plugin-a7788f1c62a9]

chetuiyer@gmail.com (JIRA)

unread,
Aug 21, 2019, 4:43:09 AM8/21/19
to jenkinsc...@googlegroups.com

Hi guys,

Did anyone of you get this issue resolved ? I have a similar issue -

One of my jenkins job calls windows Task Scheduler to trigger network connection emulator tool.  The job does its work, opens the network emultor tool, but the immediately the agent communication terminates.

I cant even restart the jenkins service when the network emulator tool is opened.

Jenkins version - 2.176.2   Java - 1.8.0_211 

OS - windows 2012 R2 ( both master and agent machines)

Note - When the same agent is configured for our old jenkins V2.76 with all the same set up, it does not fail.

I'm not able to figure out if its a remoting issue or if I'm missing some configuration/ settings

 

Error in jenkins job console output - 

7:27:56 Network latency in network emulator will be adjusted with the value of 86 based in a median latency of 27.5019407272338877:27:56 Network latency in network emulator will be adjusted with the value of 86 based in a median latency of 27.50194072723388707:27:56 SUCCESS: Attempted to run the scheduled task "StartNetworkEmulator".07:28:16 FATAL: java.nio.channels.ClosedChannelException07:28:16 java.nio.channels.ClosedChannelException07:28:16 Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 10.XXX.XXX.XXX/10.XXX.XXX.XXX:6083307:28:16 at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743)07:28:16 at hudson.remoting.Request.call(Request.java:202)07:28:16 at hudson.remoting.Channel.call(Channel.java:956)07:28:16 at hudson.Launcher$RemoteLauncher.kill(Launcher.java:1086)07:28:16 at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:510)07:28:16 at hudson.model.Run.execute(Run.java:1818)07:28:16 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)07:28:16 at hudson.model.ResourceController.execute(ResourceController.java:97)07:28:16 at hudson.model.Executor.run(Executor.java:429)07:28:16 Caused: hudson.remoting.RequestAbortedException07:28:16 at hudson.remoting.Request.abort(Request.java:340)07:28:16 at hudson.remoting.Channel.terminate(Channel.java:1040)07:28:16 at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:209)07:28:16 at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:222)07:28:16 at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:816)07:28:16 at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287)07:28:16 at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:172)07:28:16 at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:816)07:28:16 at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154)07:28:16 at org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:179)07:28:16 at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:795)07:28:16 at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)07:28:16 at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)07:28:16 at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)07:28:16 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)07:28:16 at java.lang.Thread.run(Unknown Source)07:28:16 Discard old builds...07:28:16 #581 is removed because it is older than daysToKeep07:28:16 #580 is removed because it is older than daysToKeep07:28:16 Finished: FAILURE

 

Looking forward for help.

 

Thanks,

Chethan

chetuiyer@gmail.com (JIRA)

unread,
Aug 21, 2019, 5:01:04 AM8/21/19
to jenkinsc...@googlegroups.com
Hi guys,

Did anyone of you get this issue resolved ? I have a similar issue -

One of my jenkins job calls windows Task Scheduler to trigger network connection emulator tool.  The job does its work, opens the network emultor tool, but the immediately the agent communication terminates.

I cant even restart the jenkins service when the network emulator tool is opened.

Jenkins version - 2.176.2   Java - 1.8.0_211 

OS - windows 2012 R2 ( both master and agent machines)

Note - When the same agent is configured for our old jenkins V2.76 with all the same set up, it does not fail.

I'm not able to figure out if its a remoting issue or if I'm missing some configuration/ settings

 

Error in jenkins job console output - 

7:27:56 Network latency in network emulator will be adjusted with the value of 86 based in a median latency of 27. 5019407272338877:27:56 Network 501940727233887Network latency in network emulator will be adjusted with the value of 86 based in a median latency of 27. 50194072723388707 501940727233887SUCCESS : 27:56 SUCCESS: Attempted to run the scheduled task "StartNetworkEmulator". 07:28:16 FATAL: java.nio.channels. ClosedChannelException07:28:16 java ClosedChannelExceptionjava .nio.channels. ClosedChannelException07 ClosedChannelExceptionAlso : 28:16 Also:    hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 10.XXX.XXX.XXX/10.XXX.XXX.XXX: 6083307:28:16 60833 at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743) 07:28:16 at hudson.remoting.Request.call(Request.java:202) 07:28:16 at hudson.remoting.Channel.call(Channel.java:956) 07:28:16 at hudson.Launcher$RemoteLauncher.kill(Launcher.java:1086) 07:28:16 at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:510) 07:28:16 at hudson.model.Run.execute(Run.java:1818) 07:28:16 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) 07:28:16 at hudson.model.ResourceController.execute(ResourceController.java:97) 07:28:16 at hudson.model.Executor.run(Executor.java:429) 07:28:16 Caused: hudson.remoting. RequestAbortedException07:28:16 RequestAbortedException at hudson.remoting.Request.abort(Request.java:340) 07:28:16 at hudson.remoting.Channel.terminate(Channel.java:1040) 07:28:16 at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:209) 07:28:16 at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:222) 07:28:16 at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:816) 07:28:16 at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287) 07:28:16 at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:172) 07:28:16 at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:816) 07:28:16 at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154) 07:28:16 at org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:179) 07:28:16 at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:795) 07:28:16 at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) 07:28:16 at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59) 07:28:16 at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) 07:28:16 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) 07:28:16 at java.lang.Thread.run(Unknown Source) 07:28:16

Discard old builds... 07:28:16

#581 is removed because it is older than daysToKeep07:28:16 daysToKeep

#580 is removed because it is older than daysToKeep07:28:16 daysToKeep

Finished: FAILURE

 

Looking forward for help.

 

Thanks,

Chethan

todd@boxboat.com (JIRA)

unread,
Aug 21, 2019, 10:34:02 AM8/21/19
to jenkinsc...@googlegroups.com
Todd Densmore edited a comment on Bug JENKINS-44001
I am experiencing this issue too. My pipeline fails using Kaniko on Kubernetes. Looks like this might have to do with pod resource limits... 

[https://medium.com/@garunski/closedchannelexception-in-jenkins-with-kubernetes-plugin-a7788f1c62a9]


 

EDIT: Yes I resolved the issue. Resource limits seemed to prevent my pods from spinning up completely. 
Reply all
Reply to author
Forward
0 new messages