[JIRA] (JENKINS-55106) Build stuck on final "exit 0"

[jenkins.jira@wikipiedi.it (JIRA)]

Michele Ippolito created an issue

Jenkins / JENKINS-55106 Build stuck on final "exit 0" Issue Type: Bug Assignee: Unassigned Components: core Created: 2018-12-10 14:21 Environment: Jenkins 2.150 Windows 7 cygwin Labels: regression Priority: Minor Reporter: Michele Ippolito After executing successfully the shell script the workers remain stuck for 10 minutes on the final "exit 0". There hasn't been any other failure that I could find: all the jobs run exactly as planned, they just don't seem to exit. The fact that the jobs remain stuck for exactly 600 seconds makes me think of a timeout of some sort. Reverting to 2.138 fixed the issue, that's why I am marking it as a regression. Add Comment

This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

[vamsi_ch@rocketmail.com (JIRA)]

Guru Vamsi CHINTALA commented on JENKINS-55106

Re: Build stuck on final "exit 0" We are seeing the same issue in regular builds and pull requests, build stucks on exit 0 for 10 minutes and reports the status.

Add Comment

[vamsi_ch@rocketmail.com (JIRA)]

Guru Vamsi CHINTALA edited a comment on JENKINS-55106

Re: Build stuck on final "exit 0"

We are seeing the same issue in regular builds and pull requests, build stucks on exit 0 for 10 more than 5 minutes and reports the status.

Add Comment

[skline@precisiontele.com (JIRA)]

Sean Kline commented on JENKINS-55106

Re: Build stuck on final "exit 0"

We are seeing this issue as well on version 2.150.1 running on Windows Server 2012 R2. Builds that took 4 minutes prior to the upgrade were taking 18 minutes afterward. We have reverted to version 2.138.3, which resolved the issue. If there's information that I can provide to help pin this down, please let me know.

Add Comment

[vamsi_ch@rocketmail.com (JIRA)]

Guru Vamsi Chintala commented on JENKINS-55106

Re: Build stuck on final "exit 0"

Reverting the jenkins version to 2.138.3 fixed the issue. Hope it is fixed in next Jenkins LTS version. Thank you  Sean.

Add Comment

[anatoly.shirokov@gmail.com (JIRA)]

Anatoly Shirokov commented on JENKINS-55106

Re: Build stuck on final "exit 0"

The same issue:

Add Comment

[anatoly.shirokov@gmail.com (JIRA)]

Anatoly Shirokov edited a comment on JENKINS-55106

Re: Build stuck on final "exit 0"

Confirmed. The same issue with 2.150.1. As you see exactly 10 minutes before finished : ``` 19:59:42 D:\Jenkins\jobs\product\workspace>echo done 19:59:42 done 19:59:42 19:59:42 D:\Jenkins\jobs\product\workspace>exit 0 20:09:44 Finished: SUCCESS ``` We have reverted to the 2.138.2 version.

Add Comment

[anatoly.shirokov@gmail.com (JIRA)]

Anatoly Shirokov edited a comment on JENKINS-55106

Re: Build stuck on final "exit 0"

Confirmed. The same issue with 2.150.1. As you see exactly 10 minutes before finished the finish :   {code:java}

19:59:42 D:\Jenkins\jobs\product\workspace>echo done 19:59:42 done 19:59:42 19:59:42 D:\Jenkins\jobs\product\workspace>exit 0 20:09:44 Finished: SUCCESS

{code}

We have reverted to the 2.138.2 version.

Add Comment

[anatoly.shirokov@gmail.com (JIRA)]

Anatoly Shirokov edited a comment on JENKINS-55106

Re: Build stuck on final "exit 0"

Confirmed. The same issue with 2.150.1. As you see exactly 10 minutes before finished: ```

{code:java} 19:59:42 D:\Jenkins\jobs\product\workspace>echo done 19:59:42 done 19:59:42 19:59:42 D:\Jenkins\jobs\product\workspace>exit 0 20:09:44 Finished: SUCCESS {code} ``` We have reverted to the 2.138.2 version.

Add Comment

[anatoly.shirokov@gmail.com (JIRA)]

Anatoly Shirokov edited a comment on JENKINS-55106

Re: Build stuck on final "exit 0"

Confirmed. The same issue with 2.150.1 on Windows 2003, JDK 8 . As you see exactly 10 minutes before the finish:

{code:java} 19:59:42 D:\Jenkins\jobs\product\workspace>echo done 19:59:42 done 19:59:42 19:59:42 D:\Jenkins\jobs\product\workspace>exit 0 20:09:44 Finished: SUCCESS {code} We have reverted to the 2.138.2 version.

Add Comment

[anatoly.shirokov@gmail.com (JIRA)]

Anatoly Shirokov edited a comment on JENKINS-55106

Re: Build stuck on final "exit 0"

Confirmed. The same issue with 2.150.1 on Windows Server 2003, JDK 8. As you see exactly 10 minutes before the finish:

{code:java} 19:59:42 D:\Jenkins\jobs\product\workspace>echo done 19:59:42 done 19:59:42 19:59:42 D:\Jenkins\jobs\product\workspace>exit 0 20:09:44 Finished: SUCCESS {code} We have reverted to the 2.138.2 version.

Add Comment

[anatoly.shirokov@gmail.com (JIRA)]

Anatoly Shirokov updated an issue

Jenkins / JENKINS-55106

Build stuck on final "exit 0"

Change By: Anatoly Shirokov Priority: Minor Critical

Add Comment

[scb147@bakerpage.com (JIRA)]

Shawn Baker commented on JENKINS-55106

Re: Build stuck on final "exit 0" I can also confirmed this on Windows Server 2012 R2, JDK 8 on Jenkins 2.150.2.  After the Build portion of the configuration has completed, there is a 10 minute delay before the Post-build Actions begin. I reverted back to 2.138.4.

Add Comment

[jenkins.jira@wikipiedi.it (JIRA)]

Michele Ippolito commented on JENKINS-55106

Re: Build stuck on final "exit 0"

I tried upgrading my instance to 2.164 and I can still reproduce it. I'll revert again to 2.138 for the moment. I'll lose access to this instance soon (~3 weeks) so if anyone needs me to try stuff, now's the time.

Add Comment

[vamsi_ch@rocketmail.com (JIRA)]

Guru Vamsi Chintala commented on JENKINS-55106

Re: Build stuck on final "exit 0"

Added "-DSoftKillWaitSeconds=0 " in jenkins.xml before the -jar option. Now jobs execute normally with 2.150.2 version Reference: https://stackoverflow.com/questions/54039226/jenkins-hangs-between-build-and-post-build/54072987#54072987                  https://issues.jenkins-ci.org/browse/JENKINS-55422?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel

Add Comment

[skline@precisiontele.com (JIRA)]

Sean Kline commented on JENKINS-55106

Re: Build stuck on final "exit 0"

Thank you very much for pointing this out, Guru. We tried this and are now running 2.150.2 without the delay. Have a great day!

Add Comment

[kredens@outlook.com (JIRA)]

K D commented on JENKINS-55106

Re: Build stuck on final "exit 0"

Still doesn't work properly, had to roll back from 2.164.1 LTS to 2.138.4 LTS. Workaround mentioned above doesn't work for me either, builds are getting stuck.

Add Comment

[dbeck@cloudbees.com (JIRA)]

Daniel Beck commented on JENKINS-55106

Re: Build stuck on final "exit 0"

Workaround mentioned above doesn't work for me either Are you sure you applied it correctly? Check the /systemInfo to see whether the system property is defined in Jenkins?

Add Comment

[kredens@outlook.com (JIRA)]

kredens commented on JENKINS-55106

Re: Build stuck on final "exit 0"

Daniel Beck yes, it's applied properly and still various builds are randomly getting stuck.

Add Comment

[dbeck@cloudbees.com (JIRA)]

Daniel Beck commented on JENKINS-55106

Re: Build stuck on final "exit 0"

kredens While you're waiting for the build to finish, check what Jenkins is doing: https://wiki.jenkins.io/display/JENKINS/Obtaining+a+thread+dump

Add Comment

[josh.schreuder@gmail.com (JIRA)]

Josh Schreuder commented on JENKINS-55106

Re: Build stuck on final "exit 0"

Does this work for Jenkins slaves? This fixed our master instance, but it seems like a similar issue is present with jobs that run on slaves. Adding `-DSoftKillWaitSeconds=0` to jenkins-slave.xml and restarting the service adds it to the command line, but it doesn't seem to have any effect. Any ideas?

Add Comment

[kredens@outlook.com (JIRA)]

kredens commented on JENKINS-55106

Re: Build stuck on final "exit 0"

I should add I also run the jobs on slaves, not on the master node.

Add Comment

[rob.anderson@nomadgames.co.uk (JIRA)]

Rob Anderson commented on JENKINS-55106

Re: Build stuck on final "exit 0"

We also experience this but only on slave machines, and adding -DSoftKillWaitSeconds=0 has no affect on slave nodes.

Add Comment

[kwlamb@gmail.com (JIRA)]

Ken Lamb commented on JENKINS-55106

Re: Build stuck on final "exit 0"

We also experience this issue with jobs that run on slave machines. Adding -DSoftKillWaitSeconds did not affect the issue.

Add Comment

[kwlamb@gmail.com (JIRA)]

Ken Lamb edited a comment on JENKINS-55106

Re: Build stuck on final "exit 0"

We also experience this issue with jobs that run on slave machines. Adding -DSoftKillWaitSeconds did not affect the issue.

Tried rolling back to Jenkins version 2.150.3, but the bug was still there. Then, rolled back to Jenkins version 2.138.3, and the bug is now gone. We will have to stay on 2.138.3 until this bug is resolved.

Add Comment

[kwlamb@gmail.com (JIRA)]

Ken Lamb edited a comment on JENKINS-55106

Re: Build stuck on final "exit 0"

We also experience this issue with jobs that run on slave machines. Adding -DSoftKillWaitSeconds did not affect the issue. Tried rolling back to Jenkins version 2.150.3, but the bug was still there.

Then, rolled back to Jenkins version 2.138. 3 4 , and the bug is now gone. We will have to stay on 2.138. 3 4 until this bug is resolved.

Add Comment

[dbeck@cloudbees.com (JIRA)]

Daniel Beck commented on JENKINS-55106

Re: Build stuck on final "exit 0"

To clarify, are you setting the system property on agent processes? I.e. as additional launch arguments to java -jar agent.jar?

Add Comment

[kwlamb@gmail.com (JIRA)]

Ken Lamb commented on JENKINS-55106

Re: Build stuck on final "exit 0"

I only set it on the master launch process. From what I have read, it has no effect on slaves.

Add Comment

[dbeck@cloudbees.com (JIRA)]

Daniel Beck commented on JENKINS-55106

Re: Build stuck on final "exit 0"

Right, Josh wrote that. Would still like explicit confirmation from someone affected that setting it doesn't work, including confirmation that it appears correctly on the URL /computer/name_here/systemInfo in the list of system properties, since it's easy to get the Java invocation wrong.

Add Comment

[josh.schreuder@gmail.com (JIRA)]

Josh Schreuder updated an issue

Jenkins / JENKINS-55106

Build stuck on final "exit 0"

Change By: Josh Schreuder Attachment: image-2019-05-03-12-51-09-161.png

Add Comment

[josh.schreuder@gmail.com (JIRA)]

Josh Schreuder updated an issue

Jenkins / JENKINS-55106 Build stuck on final "exit 0"

Change By: Josh Schreuder Attachment: image-2019-05-03-12-51-45-572.png

Add Comment

[josh.schreuder@gmail.com (JIRA)]

Josh Schreuder commented on JENKINS-55106

Re: Build stuck on final "exit 0" Daniel Beck Here's the slave command line: And from jenkins-slave.xml I'm pretty confident that this invocation is correct, as it's copied from our master agent where this parameter is working fine.

Add Comment

[scb147@bakerpage.com (JIRA)]

Shawn Baker commented on JENKINS-55106

Re: Build stuck on final "exit 0"

Any progress with this? I understand that there is a workaround, but shouldn't the issue that broke it be looked at to at least see why it's broken?

Add Comment

[scb147@bakerpage.com (JIRA)]

Shawn Baker edited a comment on JENKINS-55106

Re: Build stuck on final "exit 0"

Any progress with this? I understand that there is a workaround, but shouldn't the issue commit that broke it be looked at to at least see why it's broken?

Add Comment

[P.Mascha@osram.com (JIRA)]

Philipp Mascha commented on JENKINS-55106

Re: Build stuck on final "exit 0"

Can you please fix this?

Add Comment

[fsteff@gmail.com (JIRA)]

Flemming Steffensen commented on JENKINS-55106

Re: Build stuck on final "exit 0"

We've had this problem for 6 months or more, and have been searching high and low for a solution, without finding this issue. Just applied the workaround on one of our agents, and immediately cut down the build-time of one of our jobs by 25 minutes!!!!!!!!!!  I can't wait to see how much server time will be freed by this, but it looks like a LOT!

Add Comment

[anlin@adobe.com (JIRA)]

Andy Lin commented on JENKINS-55106

Re: Build stuck on final "exit 0"

This issue is preventing me from upgrading my Jenkins, and the plugin to Jenkins version gap is getting harder and harder to deal with. Is this issue going to be looked at? And has anyone had success with a workaround for a Jenkins instance that uses only slave machines?

Add Comment

This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)

[kwlamb@gmail.com (JIRA)]

Ken Lamb commented on JENKINS-55106

Re: Build stuck on final "exit 0"

Just wanted to add a "me too" to Andy Lin's comment. I used to be very diligent about keeping my Jenkins and all the plugins up to date. This bug, however, has everything stuck with what works using Jenkins 2.138.4.

Add Comment

[Rocha@Stratovan.com (JIRA)]

John Rocha commented on JENKINS-55106

Re: Build stuck on final "exit 0"

Andy Lin, what scenario are you encountering this under? I had a similar problem when using MS Visual Studio on a slave. In my case the problem is that the slave waits for remote processes to close, and has a timeout of ~2 minutes per process. I found that I had parallel compiles enabled and 6 remote VS compile session on the slave. When it finished, those VS processes would not go away, and every 2 minutes jenknis would kill one of them. I learned that MS causes compile processes to stick around once they are started. The idea being that when a new compile is needed it can grab one of the idle processes. However, in my case Jenkins doesn't need/want any more compiles and is stuck waiting for the VS processes to go away. There is a flag that can be used at the command line that informs VS to not keep the processes alive. Details for this can be found in a similar issue I logged JENKINS-59400

Add Comment

[anlin@adobe.com (JIRA)]

Andy Lin commented on JENKINS-55106

Re: Build stuck on final "exit 0"

John Rocha, I do use MS Visual Studio on some of my slave machines, but I don't think it does parallel compilation. I'll be sure to try out what you suggested. Do you experience the issue if you don't do parallel compiles? I have Mac slave machines for the other half. The solution you had might apply in some way so I'll have to investigate if xcodebuild also does something similar with lingering processes. Thanks!

Add Comment

[Rocha@Stratovan.com (JIRA)]

John Rocha commented on JENKINS-55106

Re: Build stuck on final "exit 0"

Andy Lin, I didn't seem to notice it when I did simple compiles without parallel compilation. Although I honestly would expect there to be at least a 2 minute delay even if there is just one compile process. But I don't know. Good luck.

Add Comment

[kredens@outlook.com (JIRA)]

kredens commented on JENKINS-55106

Re: Build stuck on final "exit 0"

I don't see the issue anymore using v2.204.2 and newest set of BlueOcean plugins.

Add Comment

[kredens@outlook.com (JIRA)]

kredens edited a comment on JENKINS-55106

Re: Build stuck on final "exit 0"

I don can 't see the issue anymore using v2.204.2 and newest set of BlueOcean plugins.

Add Comment

[kredens@outlook.com (JIRA)]

kredens updated an issue

Jenkins / JENKINS-55106

Build stuck on final "exit 0"

Change By: kredens Comment: I can't see the issue anymore using v2.204.2 and newest set of BlueOcean plugins.

Add Comment

This message was sent by Atlassian Jira (v7.13.12#713012-sha1:6e07c38)

[kredens@outlook.com (JIRA)]

kredens commented on JENKINS-55106

Re: Build stuck on final "exit 0" Well, time passes, and some jobs still get stuck on the FINAL stage for about two minutes before finally letting go. How hard can it be to fix this?

Add Comment

[dbeck@cloudbees.com (JIRA)]

Daniel Beck commented on JENKINS-55106

Re: Build stuck on final "exit 0"

kredens If it's so easy, submit a PR that does it.

Add Comment

[kredens@outlook.com (JIRA)]

kredens commented on JENKINS-55106

Re: Build stuck on final "exit 0"

Daniel Beck I'm not a developer on this project, neither am I using it by choice. Also - it used to work fine until someone changed something and can't be bothered to fix it.

Add Comment