JNLP java.nio.channels.ClosedChannelException with kubernetes-plugin

[fabio...@singlepoint.ie]

I'm using Kubernetes plugin 1.12.3

The plugin works well, but whenever the agent does streams the log to the master, it causes a connection exception as follows.

To reproduce, simply create a free style job, restrict to run on the kubernetes pod, and add a step to sleep for 300 seconds.

FATAL: command execution failed
java.nio.channels.ClosedChannelException
	at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154)
	at org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:142)
	at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:795)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused: java.io.IOException: Backing channel 'JNLP4-connect connection from ip-192-168-247-3.us-west-2.compute.internal/192.168.247.3:23684' is disconnected.
	at hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:214)
	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:283)
	at com.sun.proxy.$Proxy110.isAlive(Unknown Source)
	at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1137)
	at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1129)
	at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:155)
	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:109)
	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
	at hudson.model.Build$BuildExecution.build(Build.java:206)
	at hudson.model.Build$BuildExecution.doRun(Build.java:163)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504)
	at hudson.model.Run.execute(Run.java:1798)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:429)
FATAL: Unable to delete script file /tmp/jenkins5788393309854271460.sh
java.nio.channels.ClosedChannelException
	at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154)
	at org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:142)
	at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:795)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused: hudson.remoting.ChannelClosedException: Channel "unknown": Remote call on JNLP4-connect connection from ip-192-168-247-3.us-west-2.compute.internal/192.168.247.3:23684 failed. The channel is closing down or has closed down
	at hudson.remoting.Channel.call(Channel.java:948)
	at hudson.FilePath.act(FilePath.java:1036)
	at hudson.FilePath.act(FilePath.java:1025)
	at hudson.FilePath.delete(FilePath.java:1511)
	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:123)
	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
	at hudson.model.Build$BuildExecution.build(Build.java:206)
	at hudson.model.Build$BuildExecution.doRun(Build.java:163)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504)
	at hudson.model.Run.execute(Run.java:1798)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:429)
Build step 'Execute shell' marked build as failure

[Mark Waite]

Does the pod template meet the Kubernetes plugin constraints mentioned in readme? Specifically, is ttyEnabled and is there a long-running process that will allow the pod to continue running until the job completes?

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/7f88b2e6-0344-4b1f-b719-d270c89cb307%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[fabio...@singlepoint.ie]

Hi Mark, thanks for the response.

Yes, the pod meets the requirements:

- I haven't provided a jnlp container

- cat is the command

- tty is enabled

- Additional default container: maven:3.5.4-jdk-8 (tried different ones)

What I've observed:

- A build step with a sleep of 60 seconds always works

- A build step with a sleep of 70 seconds always fails

I was wondering, could be  a 60 seconds timeout in either the JNLP or somewhere else?

[fabio...@singlepoint.ie]

Interesting that if I have a loop with some activity as follows, the job completes successfully, so it seems to be a constraint of 60 seconds of inactivity perhaps between the agent and the jnlp?

for i in 1 2 3 4 5

do

  echo "Looping ... number $i"

  sleep 50

done

[roey....@cybereason.com]

Fabio,

Did you find the solution for this issue or the root cause?

Thanks

Roey

[Gopi Naidu]

I have got into the same issue. Could you please post the solution if anything is found?

[Carlos Sanchez]

what kubernetes cluster are you using and with what networking?

my guess is that the underlying network layer is killing inactive connections

--

You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/cb6a14e9-c8d0-411a-bd3a-1b1481482df4%40googlegroups.com.

[Gopi]

Hi

Using latest Kubernetes 1.13 with 3 master nodes.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CALHFn6PefYwsUuZHwaaGONXkQ3C45xC8nmi%2BeJ-bS6SCH_4JFA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

        Thanks & Regards
        Gopi. Ch

[Carlos Sanchez]

what flavor of kubernetes ? cloud? on prem? how was it built? what networking is it using?

there are a lot of variations so the best chance to find out is to relate with other people's environment and try to isolate

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CACSzXNtm6uuZfirZ3sRT1reLpF6yXhYgiY%2Bn1L%3DPADw0u-1%2Bdw%40mail.gmail.com.

[Art]

Hi Carlos,

I am running into this same issue when I try to run pods in parallel within the same job. As the number of pods launched by the job increases, the number of timeouts/disconnects increases.

See attached images for the failure details.

I believe it is the same issue.

Regards,

Art

To unsubscribe from this group and stop receiving emails from it, send an email to jenkins...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/cb6a14e9-c8d0-411a-bd3a-1b1481482df4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkins...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CALHFn6PefYwsUuZHwaaGONXkQ3C45xC8nmi%2BeJ-bS6SCH_4JFA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

        Thanks & Regards
        Gopi. Ch

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkins...@googlegroups.com.