[JIRA] (JENKINS-43038) Intermittent error "Cannot contact node123: java.lang.InterruptedException " in jenkins

[manish.savlani@gmail.com (JIRA)]

Manish Sawlani created an issue

Jenkins / JENKINS-43038 Intermittent error "Cannot contact node123: java.lang.InterruptedException " in jenkins Issue Type: Bug Assignee: Unassigned Components: core Created: 2017/Mar/22 3:36 PM Environment: Jenkins Version : 2.48 OS on Master : RHEL 5.4 OS on Salve : RHEL 6.6 Java version on salve : jdk1.7.0_80 Priority: Minor Reporter: Manish Sawlani We face below connection errors intermittently while running jobs on node123. Error which we see in build log is : Cannot contact node123: java.lang.InterruptedException I dont see any error in thread dump or any other logs related to this node. Also i see there was not connection drop between Master and node. Slave is see is running since more than 24 hrs now.     Add Comment

This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)

[o.v.nenashev@gmail.com (JIRA)]

Oleg Nenashev commented on JENKINS-43038

Re: Intermittent error "Cannot contact node123: java.lang.InterruptedException " in jenkins Please provide a full exception stacktrace at least

Add Comment

[o.v.nenashev@gmail.com (JIRA)]

Oleg Nenashev assigned an issue to Oleg Nenashev

Jenkins / JENKINS-43038

Intermittent error "Cannot contact node123: java.lang.InterruptedException " in jenkins

Change By: Oleg Nenashev Assignee: Oleg Nenashev

Add Comment

[levtar@gmail.com (JIRA)]

Lev Tartakovsky commented on JENKINS-43038

Re: Intermittent error "Cannot contact node123: java.lang.InterruptedException " in jenkins The problem persist also in Ubuntu 16.04, Jenkins 2.32.3. Unfortunately, I cannot find any evidence of exception stack trace. As part of the above problem troubleshooting I've used SSH Jenkins slave running at the same server as master.I've managed to workaround the problem by

Add Comment

[levtar@gmail.com (JIRA)]

Lev Tartakovsky edited a comment on JENKINS-43038

Re: Intermittent error "Cannot contact node123: java.lang.InterruptedException " in jenkins

The problem persist also in Ubuntu 16.04, Jenkins 2.32.3. Unfortunately, I cannot find any evidence of exception stack trace.

As part of the above problem troubleshooting I've used SSH Jenkins slave running at the same server as master.I've managed to workaround the problem by switching my jobs to run at master and not slave.

Add Comment

[o.v.nenashev@gmail.com (JIRA)]

Oleg Nenashev commented on JENKINS-43038

Re: Intermittent error "Cannot contact node123: java.lang.InterruptedException " in jenkins

OK. If you see no exception, please provide full Jenkins System logs at least. Without such information I cannot triangulate the issue

Add Comment

[levtar@gmail.com (JIRA)]

Lev Tartakovsky updated an issue

Jenkins / JENKINS-43038

Intermittent error "Cannot contact node123: java.lang.InterruptedException " in jenkins

Change By: Lev Tartakovsky Attachment: jenkins.log.gz

Add Comment

[levtar@gmail.com (JIRA)]

Lev Tartakovsky commented on JENKINS-43038

Re: Intermittent error "Cannot contact node123: java.lang.InterruptedException " in jenkins I've just uploaded my Jenkins log. Please note that most exceptions in the log are referring to disconnect/connect of slave.

Add Comment

[levtar@gmail.com (JIRA)]

Lev Tartakovsky commented on JENKINS-43038

Re: Intermittent error "Cannot contact node123: java.lang.InterruptedException " in jenkins

The problem could be related to another problem that I've reported recently https://issues.jenkins-ci.org/browse/JENKINS-43106 At the problem description you may find more logs, including thread dump that may shed some lite at the root cause of the problem.

Add Comment

[levtar@gmail.com (JIRA)]

Lev Tartakovsky updated an issue

Jenkins / JENKINS-43038

Intermittent error "Cannot contact node123: java.lang.InterruptedException " in jenkins

Change By: Lev Tartakovsky Attachment: pipeline_log.txt

Add Comment

[levtar@gmail.com (JIRA)]

Lev Tartakovsky commented on JENKINS-43038

Re: Intermittent error "Cannot contact node123: java.lang.InterruptedException " in jenkins I've managed to catch the exception which may shed some light at the problem. Please review the attached pipeline_log.txt

Add Comment

[levtar@gmail.com (JIRA)]

Lev Tartakovsky edited a comment on JENKINS-43038

Re: Intermittent error "Cannot contact node123: java.lang.InterruptedException " in jenkins

I've managed to catch the exception which may shed some light at on the problem.

Please review the attached pipeline_log.txt

At the same time build log printed: [Pipeline] stage [Pipeline] \{ (Create GIT TAG) [Pipeline] sh 10:45:26 [CISystem_generic@2] Running shell script 10:45:37 Cannot contact ##############: java.lang.InterruptedException 10:45:47 Cannot contact ##############: java.lang.InterruptedException 10:45:57 Cannot contact ##############: java.lang.InterruptedException 10:46:07 Cannot contact ##############: java.lang.InterruptedException 10:46:18 Cannot contact ##############: java.lang.InterruptedException 10:46:28 Cannot contact ##############: java.lang.InterruptedException 10:46:38 Cannot contact ##############: java.lang.InterruptedException 10:46:48 Cannot contact ##############: java.lang.InterruptedException 10:46:59 Cannot contact ##############: java.lang.InterruptedException 10:47:09 Cannot contact ##############: java.lang.InterruptedException 10:47:19 Cannot contact ##############: java.lang.InterruptedException 10:47:29 Cannot contact ##############: java.lang.InterruptedException 10:47:40 Cannot contact ##############: java.lang.InterruptedException 10:47:50 Cannot contact ##############: java.lang.InterruptedException 10:48:00 Cannot contact ##############: java.lang.InterruptedException 10:48:10 Cannot contact ##############: java.lang.InterruptedException 10:48:21 Cannot contact ##############: java.lang.InterruptedException 10:48:31 Cannot contact ##############: java.lang.InterruptedException 10:48:41 Cannot contact ##############: java.lang.InterruptedException 10:48:51 Cannot contact ##############: java.lang.InterruptedException 10:49:02 Cannot contact ##############: java.lang.InterruptedException 10:49:12 Cannot contact ##############: java.lang.InterruptedException 10:49:22 Cannot contact ##############: java.lang.InterruptedException 10:49:32 Cannot contact ##############: java.lang.InterruptedException 10:49:43 Cannot contact ##############: java.lang.InterruptedException 10:49:53 Cannot contact ##############: java.lang.InterruptedException 10:50:03 Cannot contact ##############: java.lang.InterruptedException 10:50:13 Cannot contact ##############: java.lang.InterruptedException 10:50:24 Cannot contact ##############: java.lang.InterruptedException 10:50:34 Cannot contact ##############: java.lang.InterruptedException 10:50:44 Cannot contact ##############: java.lang.InterruptedException 10:50:54 Cannot contact ##############: java.lang.InterruptedException 10:51:00 + git tag -a ############## -m Created by Jenkins 10:51:00 + git tag -a ############## -m Created by Jenkins 10:51:00 + git tag -a ############## -m Created by Jenkins 10:51:00 + git tag -a ############## -m Created by Jenkins 10:51:00 + git tag -a ############## -m Created by Jenkins 10:51:00 + git tag -a ############## -m Created by Jenkins 10:51:00 + git tag -a ############## -m Created by Jenkins 10:51:00 + git tag -a ############## -m Created by Jenkins 10:51:00 + git tag -a ############## -m Created by Jenkins 10:51:00 + git tag -a ############## -m Created by Jenkins 10:51:00 + git tag -a ############## -m Created by Jenkins 10:51:00 + git tag -a ############## -m Created by Jenkins 10:51:00 + git tag -a ############## -m Created by Jenkins 10:51:00 + git tag -a ############## -m Created by Jenkins 10:51:00 + git tag -a ############## -m Created by Jenkins 10:51:00 + git tag -a ############## -m Created by Jenkins 10:51:00 + git tag -a ############## -m Created by Jenkins 10:51:00 + git tag -a ############## -m Created by Jenkins 10:51:00 + git tag -a ############## -m Created by Jenkins 10:51:00 + git tag -a ############## -m Created by Jenkins 10:51:00 + git tag -a ############## -m Created by Jenkins 10:51:00 + git tag -a ############## -m Created by Jenkins 10:51:00 + git tag -a ############## -m Created by Jenkins 10:51:00 + git tag -a ############## -m Created by Jenkins 10:51:00 + git tag -a ############## -m Created by Jenkins 10:51:00 + git tag -a ############## -m Created by Jenkins 10:51:00 + git tag -a ############## -m Created by Jenkins 10:51:00 + git tag -a ############## -m Created by Jenkins 10:51:00 + git tag -a ############## -m Created by Jenkins 10:51:00 + git tag -a ############## -m Created by Jenkins 10:51:00 + git tag -a ############## -m Created by Jenkins 10:51:00 + git tag -a ############## -m Created by Jenkins 10:51:00 + git tag -a ############## -m Created by Jenkins

Add Comment

[ttmost@gmail.com (JIRA)]

Tsvi Mostovicz commented on JENKINS-43038

Re: Intermittent error "Cannot contact node123: java.lang.InterruptedException " in jenkins

I see this issue as well during testing which can take about 10-20 minutes of running a single shell script. I suppose it happens when the agent gets disconnected for a split second. Is there anyway to create a workaround protecting the shell script from this. At the moment I have to manually abort the running test. Thanks, Tsvi

Add Comment

[shahmishal@gmail.com (JIRA)]

mishal shah commented on JENKINS-43038

Re: Intermittent error "Cannot contact node123: java.lang.InterruptedException " in jenkins

Renjith Pillai Did you find a workaround for the x4 slowness?

Add Comment

[dor982@gmail.com (JIRA)]

Dor Segal updated an issue

Jenkins / JENKINS-43038

Intermittent error "Cannot contact node123: java.lang.InterruptedException " in jenkins

Change By: Dor Segal Priority: Minor Major

Add Comment

[o.v.nenashev@gmail.com (JIRA)]

Oleg Nenashev assigned an issue to Unassigned

Unfortunately I have no capacity to work on Remoting in medium term, so I will unassign it and let others to take it. If somebody is interested to submit a pull request, I will be happy to help to get it reviewed and released.

Jenkins / JENKINS-43038 Intermittent error "Cannot contact node123: java.lang.InterruptedException " in jenkins

Change By: Oleg Nenashev Assignee: Oleg Nenashev

Add Comment

[svanoort@cloudbees.com (JIRA)]

Sam Van Oort closed an issue as Fixed

Manish Sawlani mishal shah Tsvi Mostovicz If you update to the latest Pipeline plugins and especially support-core plugin and use the suggested GC settings (https://jenkins.io/blog/2016/11/21/gc-tuning/) you should find that the InterruptedExceptions are pretty much gone – they are the result of timeouts in remoting-related operations generally. The only cases they should happen now I believe are actual hardware/system/network issues. In the last quarter of 2017 we did a big change to the way Pipeline's durable tasks interact with remoting that should avoid many of these issues. Explanation: support-core plugin in version 2.42 added heap histogram analysis for diagnostics but this had the side effect of introducing periodic catastrophically long GC pauses that made the Jenkins master unresponsive for long periods and triggered timeouts (and thus the InterruptedException here when Timeouts kick in). Please see https://issues.jenkins-ci.org/browse/JENKINS-49931 for more details of that. For now I'm going to transition this to "closed" because when working with several users showing this among other symptoms, the suggestions above successfully resolved the issues – but I'm happy to re-open this if you all still experience problems after applying the above (please reply to note the same).

Jenkins / JENKINS-43038 Intermittent error "Cannot contact node123: java.lang.InterruptedException " in jenkins

Change By: Sam Van Oort Status: Open Closed Resolution: Fixed

Add Comment

[svanoort@cloudbees.com (JIRA)]

Sam Van Oort assigned an issue to Sam Van Oort

Jenkins / JENKINS-43038 Intermittent error "Cannot contact node123: java.lang.InterruptedException " in jenkins

Change By: Sam Van Oort

Assignee: Sam Van Oort

Add Comment

[svanoort@cloudbees.com (JIRA)]

Sam Van Oort edited a comment on JENKINS-43038

Re: Intermittent error "Cannot contact node123: java.lang.InterruptedException " in jenkins [~msavlani1] [~shahmishal] [~tsvi]  If you update to the latest Pipeline plugins and *especially* support-core plugin and use the suggested GC settings (https://jenkins.io/blog/2016/11/21/gc-tuning/) you should find that the InterruptedExceptions are pretty much gone -- they are the result of timeouts in remoting-related operations generally.  The only cases they should happen now I believe are actual hardware/system/network issues.

In the last quarter of 2017 we did a big change to the way Pipeline's durable tasks interact with remoting that should avoid many of these issues.

Explanation Edit : There was an additional issue fixed around support-core that caused problems and was recently fixed.  Specifically, support-core plugin in version 2.42 added heap histogram analysis for diagnostics but this had the unexpected side effect of introducing periodic catastrophically long GC pauses that made the Jenkins master unresponsive for long periods and triggered timeouts (and thus the InterruptedException here when Timeouts kick in).

Please see https://issues.jenkins-ci.org/browse/JENKINS-49931 for more details of that.

For now I'm going to transition this to "closed" because when working with several users showing this among other symptoms, the suggestions above successfully resolved the issues -- but I'm happy to re-open this if you all still experience problems after applying the above (please reply to note the same).

Add Comment

[joe.barber@genband.com (JIRA)]

Joe Barber commented on JENKINS-43038

Re: Intermittent error "Cannot contact node123: java.lang.InterruptedException " in jenkins

Hi I am recently seeing the same "Cannot contact node123: java.lang.InterruptedException" error but only during parallel stages in a pipeline job. I have created a brand new Jenkins environment (Jenkins version 2.121.1) with all updated plugins and have the GC settings according to the gc-tuning page from the above comment. This issue is intermittent (about 1 every 8 builds or so). Support-Core version 2.48 Pipeline version 2.5 Any other advice?   Thanks,

Add Comment

This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396)

[svanoort@cloudbees.com (JIRA)]

Sam Van Oort commented on JENKINS-43038

Re: Intermittent error "Cannot contact node123: java.lang.InterruptedException " in jenkins

Joe Barber What you describe sounds a lot like https://issues.jenkins-ci.org/browse/JENKINS-46507 but we have not had a consistent way to reproduce the issue, so it's very hard to debug. If you can provide a simple, self-contained sample Pipeline in the comments of that ticket that will reproduce the issue, that would be very helpful. Thanks!

Add Comment

[oxygenxo@gmail.com (JIRA)]

Andrey Babushkin commented on JENKINS-43038

Re: Intermittent error "Cannot contact node123: java.lang.InterruptedException " in jenkins

We're experiencing the same issue when our java agent get killed my OOM or machine on which agent is running is rebooted. Is there any way to reduce amount of time Jenkins will wait till the build will be mark as failed?

Add Comment

This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)