[JIRA] (JENKINS-55138) Exec Failure: HTTP:0. Message:No response

2 views
Skip to first unread message

juha.tiensyrja@arm.com (JIRA)

unread,
Dec 12, 2018, 3:25:02 AM12/12/18
to jenkinsc...@googlegroups.com
Juha Tiensyrjä created an issue
 
Jenkins / Bug JENKINS-55138
Exec Failure: HTTP:0. Message:No response
Issue Type: Bug Bug
Assignee: Carlos Sanchez
Components: kubernetes-plugin
Created: 2018-12-12 08:24
Environment: Jenkins version: 2.150.1
Kubernetes plugin version: 1.13.7
Kubernetes cluster version: 1.10.11
JNLP container: jenkins/jnlp-slave:3.27-1
Priority: Minor Minor
Reporter: Juha Tiensyrjä

Since upgrading our Jenkins version from 2.138.3 to 2.150.1 and K8s plugin from 1.13.5 to 1.13.7, some of our jobs have started to fail with the following error present in the job console log:

 

14:56:27 java.net.SocketException: Socket closed
14:56:27 at java.net.SocketInputStream.socketRead0(Native Method)
14:56:27 at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) 
14:56:27 at java.net.SocketInputStream.read(SocketInputStream.java:171) 
14:56:27 at java.net.SocketInputStream.read(SocketInputStream.java:141) 
14:56:27 at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) 
14:56:27 at sun.security.ssl.InputRecord.read(InputRecord.java:503) 
14:56:27 at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983) 
14:56:27 at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940) 
14:56:27 at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) 
14:56:27 at okio.Okio$2.read(Okio.java:139) 
14:56:27 at okio.AsyncTimeout$2.read(AsyncTimeout.java:237) 
14:56:27 at okio.RealBufferedSource.request(RealBufferedSource.java:67) 
14:56:27 at okio.RealBufferedSource.require(RealBufferedSource.java:60) 
14:56:27 at okio.RealBufferedSource.readByte(RealBufferedSource.java:73) 
14:56:27 at okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:113) 
14:56:27 at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:97) 
14:56:27 at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:265) 
14:56:27 at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:204) 
14:56:27 at okhttp3.RealCall$AsyncCall.execute(RealCall.java:153) 
14:56:27 at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) 
14:56:27 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
14:56:27 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
14:56:27 at java.lang.Thread.run(Thread.java:748)

 

The job continues until the end of the stage and fails there after this message is encountered.

In Jenkins master logs we have found these kinds of errors (usually more than one at exactly the same time) which correspond to the timing in the job console logs:

 

Dec 12, 2018 1:22:08 AM io.fabric8.kubernetes.client.dsl.internal.ExecWebSocketListener onFailure
SEVERE: Exec Failure: HTTP:0. Message:No response
java.net.SocketException: Socket closed
  at java.net.SocketInputStream.socketRead0(Native Method)
  at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
  at java.net.SocketInputStream.read(SocketInputStream.java:171)
  at java.net.SocketInputStream.read(SocketInputStream.java:141)
  at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
  at sun.security.ssl.InputRecord.read(InputRecord.java:503)
  at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
  at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
  at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
  at okio.Okio$2.read(Okio.java:139)
  at okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
  at okio.RealBufferedSource.request(RealBufferedSource.java:67)
  at okio.RealBufferedSource.require(RealBufferedSource.java:60)
  at okio.RealBufferedSource.readByte(RealBufferedSource.java:73)
  at okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:113)
  at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:97)
  at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:265)
  at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:204)
  at okhttp3.RealCall$AsyncCall.execute(RealCall.java:153)
  at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:748)

Both the Jenkins master and the slave pods are running in the same K8s cluster in the default namespace.

 
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

juha.tiensyrja@arm.com (JIRA)

unread,
Dec 12, 2018, 6:02:02 AM12/12/18
to jenkinsc...@googlegroups.com
Juha Tiensyrjä updated an issue
Change By: Juha Tiensyrjä
Since upgrading our Jenkins version from 2.138.3 to 2.150.1 and K8s plugin from 1.13.5 to 1.13.7, some of our jobs have started to fail with the following error present in the job console log:
 
{noformat}

14:56:27 java.net.SocketException: Socket closed
14:56:27 at java.net.SocketInputStream.socketRead0(Native Method)
14:56:27 at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
14:56:27 at java.net.SocketInputStream.read(SocketInputStream.java:171)
14:56:27 at java.net.SocketInputStream.read(SocketInputStream.java:141)
14:56:27 at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
14:56:27 at sun.security.ssl.InputRecord.read(InputRecord.java:503)
14:56:27 at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
14:56:27 at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
14:56:27 at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
14:56:27 at okio.Okio$2.read(Okio.java:139)
14:56:27 at okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
14:56:27 at okio.RealBufferedSource.request(RealBufferedSource.java:67)
14:56:27 at okio.RealBufferedSource.require(RealBufferedSource.java:60)
14:56:27 at okio.RealBufferedSource.readByte(RealBufferedSource.java:73)
14:56:27 at okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:113)
14:56:27 at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:97)
14:56:27 at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:265)
14:56:27 at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:204)
14:56:27 at okhttp3.RealCall$AsyncCall.execute(RealCall.java:153)
14:56:27 at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
14:56:27 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
14:56:27 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
14:56:27 at java.lang.Thread.run(Thread.java:748)
{noformat}
 

 The job continues until the end of the stage and fails there after this message is encountered.

In Jenkins master logs we have found these kinds of errors (usually more than one at exactly the same time) which correspond to the timing in the job console logs:
 
{noformat}
 ..
 Dec 12, 2018 1:22: 05 AM org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1 doLaunch
INFO: Created process inside pod: [clitest-rcc-30831-mhzxr-xrsf6], container: [clitest] with pid:[-1]
Dec 12, 2018 1:22:06 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Excess workload after pending Kubernetes agents: 1
Dec 12, 2018 1:22:06 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Template for label py36: Kubernetes Pod Template
Dec 12, 2018 1:22:06 AM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
INFO: Started provisioning Kubernetes Pod Template from kubernetes with 1 executors. Remaining excess workload: 0
Dec 12, 2018 1:22: 08 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
INFO: Terminating Kubernetes instance for agent clitest-3415-8zdzg-646kj
io.fabric8.kubernetes.client.dsl.internal.ExecWebSocketListener onFailure
  at java.lang.Thread.run(Thread.java:748){noformat}

Both the Jenkins master and the slave pods are running in the same K8s cluster in the default namespace.

 

juha.tiensyrja@arm.com (JIRA)

unread,
Dec 12, 2018, 6:03:04 AM12/12/18
to jenkinsc...@googlegroups.com
Juha Tiensyrjä updated an issue
Change By: Juha Tiensyrjä
Attachment: jenkins-master.log

juha.tiensyrja@arm.com (JIRA)

unread,
Dec 12, 2018, 6:04:01 AM12/12/18
to jenkinsc...@googlegroups.com
Juha Tiensyrjä commented on Bug JENKINS-55138
 
Re: Exec Failure: HTTP:0. Message:No response

I attached full Jenkins master log since the previous restart.

juha.tiensyrja@arm.com (JIRA)

unread,
Dec 13, 2018, 2:08:02 AM12/13/18
to jenkinsc...@googlegroups.com

I downgraded Jenkins master to version 2.138.4, still seeing this issue.

stephan@blackbuild.com (JIRA)

unread,
Dec 13, 2018, 7:06:02 AM12/13/18
to jenkinsc...@googlegroups.com

we are encountering this error about 10 times a day which is quite critical, since it also breaks jobs which run for over an hour.

 

As with OP, this occured after updating to 1.13.7. Before the upgrade, we had a lot of memory issues in OpenShift, therefore I would like to prevent having to roll back the version.

 

One more thing that I noticed: Once this happens, it seems to affect more than one job.

 

Also, after that there is always the exception later:

 

java.util.concurrent.RejectedExecutionException: Task okhttp3.RealCall$AsyncCall@3300fdcc rejected from java.util.concurrent.ThreadPoolExecutor@2e02dc25[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 258]
{{ at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)}}
{{ at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)}}
{{ at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)}}
{{ at okhttp3.Dispatcher.enqueue(Dispatcher.java:130)}}
{{ at okhttp3.RealCall.enqueue(RealCall.java:100)}}
{{ at okhttp3.internal.ws.RealWebSocket.connect(RealWebSocket.java:183)}}
{{ at okhttp3.OkHttpClient.newWebSocket(OkHttpClient.java:436)}}
{{ at io.fabric8.kubernetes.client.dsl.internal.PodOperationsImpl.exec(PodOperationsImpl.java:267)}}
{{ at io.fabric8.kubernetes.client.dsl.internal.PodOperationsImpl.exec(PodOperationsImpl.java:61)}}
{{ at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.doLaunch(ContainerExecDecorator.java:318)}}
{{ at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:236)}}

So obviously, the error seems to not really affect the execution, but leads to the TRhreadPoll bein terminated.

 

stephan@blackbuild.com (JIRA)

unread,
Dec 13, 2018, 7:06:02 AM12/13/18
to jenkinsc...@googlegroups.com
Stephan Pauxberger updated an issue
 
Change By: Stephan Pauxberger
Priority: Minor Major

juha.tiensyrja@arm.com (JIRA)

unread,
Dec 13, 2018, 10:09:02 AM12/13/18
to jenkinsc...@googlegroups.com
Juha Tiensyrjä edited a comment on Bug JENKINS-55138
I downgraded Jenkins master to version 2.138.4, still seeing this issue.


Upgraded back to 2.150.1 and downgraded the K8s plugin to 1.13.6, it still reproduces. Will try 1.13.5 next.

vincent@latombe.net (JIRA)

unread,
Dec 13, 2018, 10:51:02 AM12/13/18
to jenkinsc...@googlegroups.com
Vincent Latombe assigned an issue to Vincent Latombe
 
Change By: Vincent Latombe
Assignee: Carlos Sanchez Vincent Latombe

vincent@latombe.net (JIRA)

unread,
Dec 13, 2018, 10:51:03 AM12/13/18
to jenkinsc...@googlegroups.com
Vincent Latombe started work on Bug JENKINS-55138
 
Change By: Vincent Latombe
Status: Open In Progress

vincent@latombe.net (JIRA)

unread,
Dec 13, 2018, 10:51:03 AM12/13/18
to jenkinsc...@googlegroups.com
Change By: Vincent Latombe
Status: In Progress Review

stephan@blackbuild.com (JIRA)

unread,
Dec 13, 2018, 11:11:03 AM12/13/18
to jenkinsc...@googlegroups.com
Stephan Pauxberger edited a comment on Bug JENKINS-55138
we are encountering this error about 10 times a day which is quite critical, since it also breaks jobs which run for over an hour.

 

As with OP, this occured after updating to 1.13.7. Before the upgrade, we had a lot of memory issues in OpenShift, therefore I would like to prevent having to roll back the version.

 

One more thing that I noticed: Once this happens, it seems to affect more than one job.

 

Also, after that there is always the exception later:

 
{ quote noformat } {{
java.util.concurrent.RejectedExecutionException: Task okhttp3.RealCall$AsyncCall@ 3300fdcc 512db46f rejected from java.util.concurrent.ThreadPoolExecutor@2e02dc25[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 258] }}
{{ at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) }}
{{ at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) }}
{{ at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) }}
{{ at okhttp3.Dispatcher.enqueue(Dispatcher.java:130) }}
{{ at okhttp3.RealCall.enqueue(RealCall.java:100) }}
{{ at okhttp3.internal.ws.RealWebSocket.connect(RealWebSocket.java:183) }}
{{ at okhttp3.OkHttpClient.newWebSocket(OkHttpClient.java:436) }}
{{ at io.fabric8.kubernetes.client.dsl.internal.PodOperationsImpl.exec(PodOperationsImpl.java:267) }}
{{ at io.fabric8.kubernetes.client.dsl.internal.PodOperationsImpl.exec(PodOperationsImpl.java:61) }}
{{ at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.doLaunch(ContainerExecDecorator.java:318) }}
{{ at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:236) }}

at hudson.Launcher$ProcStarter.start(Launcher.java:455)
 { quote noformat }


 So obviously, the error seems to not really affect the execution, but leads to the TRhreadPoll bein terminated.

 

stephan@blackbuild.com (JIRA)

unread,
Dec 13, 2018, 1:42:02 PM12/13/18
to jenkinsc...@googlegroups.com
Stephan Pauxberger edited a comment on Bug JENKINS-55138
we are encountering this error about 10 times a day which is quite critical, since it also breaks jobs which run for over an hour.

 

As with OP, this occured after updating to 1.13.7. Before the upgrade, we had a lot of memory issues in OpenShift, therefore I would like to prevent having to roll back the version.

 

One more thing that I noticed: Once this happens, it seems to affect more than one job.

 

Also, after that there is always the exception later:

 
{noformat}
java.util.concurrent.RejectedExecutionException: Task okhttp3.RealCall$AsyncCall@512db46f rejected from java.util.concurrent.ThreadPoolExecutor@2e02dc25[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 258]

at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
at okhttp3.Dispatcher.enqueue(Dispatcher.java:130)
at okhttp3.RealCall.enqueue(RealCall.java:100)
at okhttp3.internal.ws.RealWebSocket.connect(RealWebSocket.java:183)
at okhttp3.OkHttpClient.newWebSocket(OkHttpClient.java:436)
at io.fabric8.kubernetes.client.dsl.internal.PodOperationsImpl.exec(PodOperationsImpl.java:267)
at io.fabric8.kubernetes.client.dsl.internal.PodOperationsImpl.exec(PodOperationsImpl.java:61)
at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.doLaunch(ContainerExecDecorator.java:318)
at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:236)
at hudson.Launcher$ProcStarter.start(Launcher.java:455)
{noformat}

So obviously, the error seems to not really affect the execution, but leads to the TRhreadPoll ThreadPool bein terminated.

 

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Dec 13, 2018, 1:52:02 PM12/13/18
to jenkinsc...@googlegroups.com

Downgrade to 1.13.5 for now to solve the issue

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Dec 13, 2018, 4:57:02 PM12/13/18
to jenkinsc...@googlegroups.com

should be fixed in 1.13.8

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Dec 13, 2018, 4:57:03 PM12/13/18
to jenkinsc...@googlegroups.com
Change By: Carlos Sanchez
Status: In Review Resolved
Resolution: Fixed

yoanlin93@gmail.com (JIRA)

unread,
Dec 13, 2018, 10:32:02 PM12/13/18
to jenkinsc...@googlegroups.com
Yo-An Lin commented on Bug JENKINS-55138
 
Re: Exec Failure: HTTP:0. Message:No response

Same here, 1.13.7 is broken.  this issue is very critical, it happens 10+ times, 80% builds failed due to this issue.  Now Trying 1.13.8.

yoanlin93@gmail.com (JIRA)

unread,
Dec 13, 2018, 10:32:02 PM12/13/18
to jenkinsc...@googlegroups.com
Yo-An Lin edited a comment on Bug JENKINS-55138
Same here, 1.13.7 is broken.  this issue is very critical, it happens 10+ times a day , 80% builds failed due to this issue.  Now Trying 1.13.8.

jmukhtar@divaaco.com (JIRA)

unread,
Sep 9, 2019, 5:22:04 AM9/9/19
to jenkinsc...@googlegroups.com

We upgraded the plugin to 1.18.3 but are still hitting the same issue although a bit different error message this time

 

```Cannot contact ugc-test-ff643992-51da-478c-b63c-36f16f9ab9eb-0kz5p-v4642: hudson.remoting.RequestAbortedException: java.nio.channels.ClosedChannelException```
Reply all
Reply to author
Forward
0 new messages