[JIRA] (JENKINS-59083) Builds stuck on "is already in progress" forever

Issue Type:	Bug
Assignee:	Unassigned
Attachments:	plugins.txt
Components:	workflow-job-plugin
Created:	2019-08-26 12:58
Environment:	Centos 7.2.1511 (master and slaves) Jenkins 2.176.2 Jenkins master - open-jdk 1.8.0_212 Jenkins slaves - Oracle jdk 1.8.0_60 Jenkins running as docker container (standalone) Attached plugin versions as file
Labels:	regression
Priority:	Major
Reporter:	Roy Arnon

After upgrading our jenkins master and plugins, some of our build jobs - specifically it seemed like most pipeline jobs - were stuck waiting for previous build to complete, even though it already completed.

This seems to have been caused by ~~JENKINS-46076~~.

After donwngrading the plugin to version 2.32, all build were working correctly.

From some testing, it seems that calling

 
                                                                Jenkins.instance.getItemByFullName(...).getBuildByNumber(..).isLogUpdated()

On the previous build (in the sample case build #185) returns true, although the build is completed and the log seems to have been completed.

This issue happened immediately after restarting the jenkins instance with any of our pipeline jobs

This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

royarnon@gmail.com (JIRA)

unread,

Aug 26, 2019, 9:03:03 AM8/26/19

to jenkinsc...@googlegroups.com

Roy Arnon updated an issue

Jenkins /

!https://files.slack.com/files-pri/T0255EC3D-FMP3HKY56/image.png!

Change By:	Roy Arnon

After upgrading our jenkins master and plugins, some of our build jobs - specifically it seemed like most pipeline jobs - were stuck waiting for previous build to complete, even though it already completed.

This seems to have been caused by JENKINS-46076.

After donwngrading the plugin to version 2.32, all build builds were working correctly.

From some testing, it seems that calling

{code:java}
Jenkins.instance.getItemByFullName(...).getBuildByNumber(..).isLogUpdated()
{code}

On the previous build (in the sample case build #185) returns true, although the build is completed and the log seems to have been completed.

This issue happened immediately after restarting the jenkins instance with any of our pipeline jobs

jglick@cloudbees.com (JIRA)

unread,

Aug 26, 2019, 9:16:02 AM8/26/19

to jenkinsc...@googlegroups.com

Jesse Glick commented on

Re: Builds stuck on "is already in progress" forever

The image is not loadable anonymously. Use Attach files to copy it to this report.

If logUpdated is true on a build which appears to be complete, then something went wrong with that build. I really cannot speculate what that might be without steps to reproduce from scratch.

dnusbaum@cloudbees.com (JIRA)

unread,

Aug 26, 2019, 9:48:01 AM8/26/19

to jenkinsc...@googlegroups.com

Devin Nusbaum commented on

Re: Builds stuck on "is already in progress" forever

Perhaps related to issues like JENKINS-45571, ~~JENKINS-53223~~, and ~~JENKINS-50199~~, where Pipeline builds appear to have completed but are still running in some sense (symptoms include flyweight executors sticking around, and that the "completed" builds resume when Jenkins restarts). I am suspicious of a bug in Pipeline shutdown/cleanup code, but I'm not really sure.

Roy Arnon Can you reproduce the problem consistently or is it intermittent? Do you see any messages in your Jenkins system logs that seem relevant? Do the old builds that look like they completed resume if you restart Jenkins? What durability setting are you using? If you could upload the full build folder ($JENKINS_HOME/jobs/$JOB_NAME/builds/$BUILD_NUMBER) of one of the jobs that is hung, it might help diagnose the problem.

jglick@cloudbees.com (JIRA)

unread,

Aug 26, 2019, 10:01:04 AM8/26/19

to jenkinsc...@googlegroups.com

Jesse Glick commented on

Re: Builds stuck on "is already in progress" forever

Yes, I suspect this is not so much an introduced bug as a change which made an existing bug more visible by treating termination conditions more consistently.

royarnon@gmail.com (JIRA)

unread,

Aug 26, 2019, 10:48:02 AM8/26/19

to jenkinsc...@googlegroups.com

Roy Arnon updated an issue

Jenkins /

Change By:	Roy Arnon
Attachment:	image.png

jglick@cloudbees.com (JIRA)

unread,

Aug 26, 2019, 11:10:02 AM8/26/19

to jenkinsc...@googlegroups.com

Jesse Glick commented on

Re: Builds stuck on "is already in progress" forever

Judging by the cause of blockage, I presume your project is set to disable concurrent builds. (The default for Pipeline is to permit them.)

royarnon@gmail.com (JIRA)

unread,

Aug 26, 2019, 12:03:02 PM8/26/19

to jenkinsc...@googlegroups.com

Roy Arnon commented on

Re: Builds stuck on "is already in progress" forever

Hi,

I can reproduce the error, I am not near a computer right now so I will have that for you by tomorrow.
Yes, in this specific project - and probably all stuck builds - we have disabled concurrent builds.

royarnon@gmail.com (JIRA)

unread,

Aug 26, 2019, 4:13:02 PM8/26/19

to jenkinsc...@googlegroups.com

Roy Arnon updated an issue

Jenkins /

Change By:	Roy Arnon
Attachment:	stuck-build.tar.gz

royarnon@gmail.com (JIRA)

unread,

Aug 26, 2019, 4:14:01 PM8/26/19

to jenkinsc...@googlegroups.com

Roy Arnon updated an issue

Jenkins /

Change By:	Roy Arnon
Attachment:	Capture.PNG

royarnon@gmail.com (JIRA)

unread,

Aug 26, 2019, 4:18:04 PM8/26/19

to jenkinsc...@googlegroups.com

Roy Arnon updated an issue

Jenkins /

Change By:	Roy Arnon
Attachment:	tdump

royarnon@gmail.com (JIRA)

unread,

Aug 26, 2019, 4:18:05 PM8/26/19

to jenkinsc...@googlegroups.com

Roy Arnon commented on

Re: Builds stuck on "is already in progress" forever

Hi,

I've uploaded a build that is stuck and an image of the state in jenkins. The build I uploaded is build #1913 - I aborted it (the process it was testing was just taking too much time, some misconfiguration probably), but jenkins still does not allow next builds to start - stuck-build.tar.gz

I'v also attached a thread dump - tdump

After a restart, these "stuck" builds immediately start. The jenkins logs do not seem to contain anything relevant.

Let me know if you need anything else.

stuck-build.tar.gz

royarnon@gmail.com (JIRA)

unread,

Aug 26, 2019, 4:18:06 PM8/26/19

to jenkinsc...@googlegroups.com

Roy Arnon edited a comment on

Re: Builds stuck on "is already in progress" forever

Hi,

I've uploaded a build that is stuck and an image of the state in jenkins. The build I uploaded is build #1913 - I aborted it (the process it was testing was just taking too much time, some misconfiguration probably), but jenkins still does not allow next builds to start - [^stuck-build.tar.gz]

I'v also attached a thread dump - [^tdump]

After a restart, these "stuck" builds immediately start. The jenkins logs do not seem to contain anything relevant.

Let me know if you need anything else.

[^stuck-build !Capture . tar.gz] PNG|thumbnail!

royarnon@gmail.com (JIRA)

unread,

Aug 26, 2019, 4:23:03 PM8/26/19

to jenkinsc...@googlegroups.com

Roy Arnon edited a comment on

Re: Builds stuck on "is already in progress" forever

Hi,

I've uploaded a build that is stuck and an image of the state in jenkins. The build I uploaded is build #1913 - I aborted it (the process it was testing was just taking too much time, some misconfiguration probably), but jenkins still does not allow next builds to start - [^stuck-build.tar.gz]

I'v also attached a thread dump - [^tdump]

After a restart, these "stuck" builds immediately start. The jenkins logs do not seem to contain anything relevant.

Regarding durability settings - this job is using the default (PERFORMANCE) but I've seen this issue on jobs that are set to DURABILITY as well.

Let me know if you need anything else.

!Capture.PNG|thumbnail!

jglick@cloudbees.com (JIRA)

unread,

Aug 26, 2019, 4:51:03 PM8/26/19

to jenkinsc...@googlegroups.com

Jesse Glick commented on

Re: Builds stuck on "is already in progress" forever

Unfortunately I see nothing problematic in the screenshot or either of the attachments, so unless you have a known way to reproduce the problem from scratch I doubt this is going anywhere.

royarnon@gmail.com (JIRA)

unread,

Aug 26, 2019, 5:33:02 PM8/26/19

to jenkinsc...@googlegroups.com

Roy Arnon updated an issue

Jenkins /

Change By:	Roy Arnon
Attachment:	Capture2.PNG

olegspace@gmail.com (JIRA)

unread,

Aug 26, 2019, 5:35:02 PM8/26/19

to jenkinsc...@googlegroups.com

Oleg Kalugin commented on

Re: Builds stuck on "is already in progress" forever

Hey, I can confirm experiencing the same problem of builds stuck in "already in progress" condition. I had to delete the previous build for it to move on every time.

Unfortunately I didn't grab any logs or screenshots before downgrading because I had a bunch of work dependent on this, so my main concern was trying to get it fixed.

The common traits with the builds that I noticed were:

All of them were multibranch pipeline jobs
Concurrent builds were disabled
And this one seemed the most outstanding (pure speculation on my part) - each build was kicking off downstream jobs that are defined though properties.pipelineTriggers.upstream.upstreamProjects in those downstream projects

royarnon@gmail.com (JIRA)

unread,

Aug 26, 2019, 5:35:03 PM8/26/19

to jenkinsc...@googlegroups.com

Roy Arnon commented on

Re: Builds stuck on "is already in progress" forever

I do not get what you mean nothing problematic in the screenshot.

This jenkins instance is still stuck forever on this build. I have tried restarting it and it reproduces again after completing one build. Is that not enough of a reproduction?

royarnon@gmail.com (JIRA)

unread,

Aug 26, 2019, 5:45:03 PM8/26/19

to jenkinsc...@googlegroups.com

Roy Arnon updated an issue

Jenkins /

Change By:	Roy Arnon
Attachment:	failed.PNG

royarnon@gmail.com (JIRA)

unread,

Aug 26, 2019, 5:47:03 PM8/26/19

to jenkinsc...@googlegroups.com

Roy Arnon updated an issue

Jenkins /

Change By:	Roy Arnon
Attachment:	stuck-build.tar.gz

royarnon@gmail.com (JIRA)

unread,

Aug 26, 2019, 5:50:09 PM8/26/19

to jenkinsc...@googlegroups.com

Roy Arnon commented on

Re: Builds stuck on "is already in progress" forever

I went ahead and reproduced it again.

This time I simulated a failure in the pipeline job - you can see build #1916 waiting for the failed build #1915.

I have attached the builds folder again - stuck-build-2.tar.gz

royarnon@gmail.com (JIRA)

unread,

Aug 26, 2019, 5:50:09 PM8/26/19

to jenkinsc...@googlegroups.com

Roy Arnon updated an issue

Jenkins /

Change By:	Roy Arnon
Attachment:	stuck-build-2.tar.gz

jglick@cloudbees.com (JIRA)

unread,

Aug 26, 2019, 6:51:41 PM8/26/19

to jenkinsc...@googlegroups.com

Jesse Glick commented on

Re: Builds stuck on "is already in progress" forever

I do not get what you mean nothing problematic in the screenshot.

Sorry, I meant nothing problematic other than the new build waiting, which is probably just due to the incorrect return value from isLogUpdated mentioned to begin with. Everything else looks normal enough.

each build was kicking off downstream jobs

A plausible lead, though apparently not the explanation for Roy Arnon.

jglick@cloudbees.com (JIRA)

unread,

Aug 26, 2019, 6:52:02 PM8/26/19

to jenkinsc...@googlegroups.com

jglick@cloudbees.com (JIRA)

unread,

Aug 26, 2019, 7:02:02 PM8/26/19

to jenkinsc...@googlegroups.com

Jesse Glick commented on

Re: Builds stuck on "is already in progress" forever

Assuming workflow-job #137 passes, anyone affected should please try upgrading to the build which should be linked from the “incrementals” status. Purely speculative, but in the absence of any known steps to reproduce that is the best we can do.

jglick@cloudbees.com (JIRA)

unread,

Aug 27, 2019, 10:19:02 AM8/27/19

to jenkinsc...@googlegroups.com

Jesse Glick edited a comment on

Re: Builds stuck on "is already in progress" forever

Assuming workflow-job #137 passes, anyone Anyone affected should please try upgrading to the [this build which should be linked from the “ |https://repo.jenkins-ci.org/ incrementals ” status /org/jenkins-ci/plugins/workflow/workflow-job/2 . 35-rc1005.3c1093c31d53/workflow-job-2.35-rc1005.3c1093c31d53.hpi]. Purely speculative, but in the absence of any known steps to reproduce that is the best we can do.

royarnon@gmail.com (JIRA)

unread,

Aug 27, 2019, 2:25:04 PM8/27/19

to jenkinsc...@googlegroups.com

Roy Arnon commented on

Re: Builds stuck on "is already in progress" forever

Hi,

I can confirm that the build you've uploaded fixes the issue for us. Thanks!

jglick@cloudbees.com (JIRA)

unread,

Aug 27, 2019, 2:52:03 PM8/27/19

to jenkinsc...@googlegroups.com

Jesse Glick assigned an issue to Jesse Glick

Jenkins /

Change By:	Jesse Glick
Assignee:	Jesse Glick

jglick@cloudbees.com (JIRA)

unread,

Aug 27, 2019, 2:53:04 PM8/27/19

to jenkinsc...@googlegroups.com

Jesse Glick updated

Jenkins /

Change By:	Jesse Glick
Status:	Open Fixed but Unreleased
Resolution:	Fixed

jglick@cloudbees.com (JIRA)

unread,

Aug 27, 2019, 2:54:02 PM8/27/19

to jenkinsc...@googlegroups.com

Jesse Glick commented on

Re: Builds stuck on "is already in progress" forever

Roy Arnon good; now do you see any stack traces in your system log starting with

 
                                                                java.lang.IllegalStateException: trying to open a build log on … after it has completed

? If so, we need to see the stack trace to try to understand the root cause.

royarnon@gmail.com (JIRA)

unread,

Aug 27, 2019, 3:51:02 PM8/27/19

to jenkinsc...@googlegroups.com

Roy Arnon commented on

Re: Builds stuck on "is already in progress" forever

Yes, I do see a stack trace like that in the log:

 
                                                                Aug 27, 2019 7:37:09 PM org.jenkinsci.plugins.workflow.job.WorkflowRun getListener
WARNING: null
java.lang.IllegalStateException: trying to open a build log on Lint.Hooks #1923 after it has completed
        at org.jenkinsci.plugins.workflow.job.WorkflowRun.getListener(WorkflowRun.java:219)
        at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$300(WorkflowRun.java:133)
        at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.getListener(WorkflowRun.java:955)
        at org.jenkinsci.plugins.workflow.cps.EnvActionImpl.getListener(EnvActionImpl.java:77)
        at org.jenkinsci.plugins.workflow.cps.EnvActionImpl.getEnvironment(EnvActionImpl.java:69)
        at org.jenkinsci.plugin.taboola.ElasticSearchManager.getConsoleEnvs(ElasticSearchManager.java:277)
        at org.jenkinsci.plugin.taboola.ElasticSearchManager.addCommonDataAndSend(ElasticSearchManager.java:110)
        at org.jenkinsci.plugin.taboola.ElasticSearchManager.sendStageData(ElasticSearchManager.java:179)
        at org.jenkinsci.plugin.taboola.events.TaboolaStagesListener.onNewHead(TaboolaStagesListener.java:145)
        at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1463)
        at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:458)
        at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:37)
        at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
        at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
        at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

 
                                                            

royarnon@gmail.com (JIRA)

unread,

Aug 27, 2019, 3:52:02 PM8/27/19

to jenkinsc...@googlegroups.com

Roy Arnon commented on

Re: Builds stuck on "is already in progress" forever

Actually, taking a look at it, it seems it has to do with one of our own plugins - I will look into that.

jglick@cloudbees.com (JIRA)

unread,

Aug 27, 2019, 4:26:04 PM8/27/19

to jenkinsc...@googlegroups.com

Jesse Glick commented on

Re: Builds stuck on "is already in progress" forever

workflow-job #140 should suppress the stack trace, which at least in this case seems to be innocuous.

jglick@cloudbees.com (JIRA)

unread,

Aug 27, 2019, 4:26:05 PM8/27/19

to jenkinsc...@googlegroups.com

Jesse Glick updated

Jenkins /

Change By:	Jesse Glick
Resolution:	Fixed
Status:	Fixed but Unreleased Reopened

jglick@cloudbees.com (JIRA)

unread,

Aug 27, 2019, 4:26:05 PM8/27/19

to jenkinsc...@googlegroups.com

Jesse Glick updated

Jenkins /

Change By:	Jesse Glick
Status:	Reopened Open

jglick@cloudbees.com (JIRA)

unread,

Aug 27, 2019, 4:26:07 PM8/27/19

to jenkinsc...@googlegroups.com

Jesse Glick updated

Jenkins /

Change By:	Jesse Glick
Status:	In Progress Review

jglick@cloudbees.com (JIRA)

unread,

Aug 27, 2019, 4:26:09 PM8/27/19

to jenkinsc...@googlegroups.com

Jesse Glick started work on

Change By:	Jesse Glick
Status:	Open In Progress

dnusbaum@cloudbees.com (JIRA)

unread,

Sep 10, 2019, 4:18:04 PM9/10/19

to jenkinsc...@googlegroups.com

Devin Nusbaum updated

Ok, at least one way that this could be happening should be fixed as of Pipeline: Job Plugin 2.34. If you end up finding this ticket when running Pipeline: Job Plugin 2.34 or higher because you are seeing the following error in your logs, please file a new ticket, and include the full stack trace:

 
                                                                java.lang.IllegalStateException: trying to open a build log on … after it has completed

Note that if you are not running Pipeline: Groovy Plugin 2.74 or later, you will see this error message for one case that has already been fixed, so I would recommend upgrading to Pipeline: Job Plugin 2.35 or newer which will automatically update Pipeline: Groovy Plugin as well.

Jenkins /

Change By:	Devin Nusbaum
Status:	In Review Resolved
Resolution:	Fixed
Released As:	workflow-job 2.34

This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)

jglick@cloudbees.com (JIRA)

unread,

Sep 10, 2019, 5:45:03 PM9/10/19

to jenkinsc...@googlegroups.com

Jesse Glick commented on

Re: Builds stuck on "is already in progress" forever

upgrading to Pipeline: Job Plugin 2.35 or newer which will automatically update Pipeline: Groovy Plugin as well

No it will not—this is just a test dependency.

 
                                                                Plugin-Dependencies: workflow-api:2.36,workflow-step-api:2.20,workflow-support:3.3

dnusbaum@cloudbees.com (JIRA)

unread,

Sep 13, 2019, 11:37:03 AM9/13/19

to jenkinsc...@googlegroups.com

Devin Nusbaum updated an issue

Jenkins /

Change By:	Devin Nusbaum
Released As:	workflow-job 2. 34 35

dnusbaum@cloudbees.com (JIRA)

unread,

Sep 13, 2019, 11:39:02 AM9/13/19

to jenkinsc...@googlegroups.com

Devin Nusbaum edited a comment on

Re: Builds stuck on "is already in progress" forever

Ok, at least one way that this could be happening should be fixed as of Pipeline: Job Plugin 2. 34 35 . If you end up finding this ticket when running Pipeline: Job Plugin 2. 34 35 or higher because you are seeing the following error in your logs, please file a new ticket, and include the full stack trace:
{code :java }

java.lang.IllegalStateException: trying to open a build log on … after it has completed

{code}

EDIT: Note that if you are not running Pipeline: Groovy Plugin 2.74 or later, you will see this error message for one case that has already been fixed, so I would recommend upgrading to Pipeline: Job Groovy Plugin 2. 35 74 or newer which will automatically update Pipeline: Groovy Plugin as well later .