[JIRA] (JENKINS-49365) Resuming of pipelines vs. docker.image(...).inside(...)?

7 views
Skip to first unread message

r.fuereder@xortex.com (JIRA)

unread,
Feb 15, 2018, 5:47:01 AM2/15/18
to jenkinsc...@googlegroups.com
Reinhold Füreder updated an issue
 
Jenkins / Bug JENKINS-49365
Resuming of pipelines vs. docker.image(...).inside(...)?
Change By: Reinhold Füreder
Priority: Critical Blocker
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
Atlassian logo

r.fuereder@xortex.com (JIRA)

unread,
Mar 9, 2018, 6:45:02 AM3/9/18
to jenkinsc...@googlegroups.com
Reinhold Füreder commented on Bug JENKINS-49365
 
Re: Resuming of pipelines vs. docker.image(...).inside(...)?

I guess there are (at least) two different problems: (1) the Docker container presumably is terminated and (2) loading the build state fails because de-serialization does not work or is maybe intentionally not supported because of issue #1?

Sam Van Oort Just a naive thought, but do you think there is a workaround possibility related/similar to your recently introduced "pipeline speed/durability level" code: basically another durability level that only considers finished stages (i.e. persists only after finishing a stage); ideally linked with "Manage Jenkins > Prepare for Shutdown" (= http://<jenkins-server>/quietDown) action to not just waiting for the finishing of the currently running step, but only to stop at the end of the current stage to allow a gentle shutdown possibility (in between stages, once the build state was persisted.

Actually:

  1. the documentation for "Manage Jenkins > Prepare for Shutdown" (= http://<jenkins-server>/quietDown) action is not really correct, as it says to "stops executing new builds" although it seems to pause running pipelines after the completion of the currently running pipeline step
  2. after cancelling the shutdown (cancelQuietDown) I then have to pause + resume the running pipeline (to kind of wake it up?) that was in the middle of a sleep step, when I triggered "Manage Jenkins > Prepare for Shutdown"!?

I am not sure if my imagined use case is rather theoretical or irrelevant: in case I have many pipelines AND they are longer running I may have great difficulties to find a time slot for e.g. installing Jenkins plug-in updates including restarting the Jenkins master subsequently?

(Thus I am now ignoring e.g. pipeline resumability because of "unplanned Jenkins outages" like crashes or network connection problems. I think.)

r.fuereder@xortex.com (JIRA)

unread,
Mar 14, 2018, 2:41:03 AM3/14/18
to jenkinsc...@googlegroups.com

JENKINS-38316 covers the (surprising) pipeline pausing/freezing instead of the expected finish the job due to "Manage Jenkins > Prepare for Shutdown"

r.fuereder@xortex.com (JIRA)

unread,
Mar 14, 2018, 2:42:02 AM3/14/18
to jenkinsc...@googlegroups.com
Reinhold Füreder edited a comment on Bug JENKINS-49365
JENKINS-38316 covers both "actually items", that is the (surprising) pipeline pausing/freezing instead of the expected finish the job due to "Manage Jenkins > Prepare for Shutdown" , as well as the waking up...

jglick@cloudbees.com (JIRA)

unread,
May 18, 2018, 9:49:01 AM5/18/18
to jenkinsc...@googlegroups.com

Yes it is supposed to work. I have not seen the error you mention before; offhand I would guess it has something to do with the use of a library. Best to avoid use of the DSL and call the withDockerContainer step directly. (Better still to not use this step at all and call sh steps directly, running docker build or docker-compose or whatever would make sense for your task if you had never heard of Jenkins.)

r.fuereder@xortex.com (JIRA)

unread,
May 22, 2018, 5:46:01 AM5/22/18
to jenkinsc...@googlegroups.com

Jesse Glick Thanks to parts of your comment ("Yes it is supposed to work. I have not seen the error you mention before") I started to re-produce the problem in a small context in order to later on try your suggestions (e.g. "avoid use of DSL" or even "calling sh steps directly") more or less full of hope... BUT ... and this is indeed very good news: I not only failed to re-produce the problem in the small context (even without "use of a library"), but also in my original real-world scenario!!!

And in the latter I actually remembered having no problems to re-produce it back then...

(Sam Van Oort And even better, during my reproduction efforts I also never had to use the pause + resume workaround after cancelling the shutdown...)

Not sure if it is relevant or could explain the problem, but of course there were a range of Jenkins (core) and especially (workflow/cps related) Jenkins plugins updates since then:

  • Jenkins core 2.124 (was: 2.103)
  • Plugins installed:
        workflow-api:2.27 (was: 2.25)
        workflow-cps:2.53 (was: 2.43)
        workflow-durable-task-step:2.19 (was: 2.18)
        workflow-job:2.21 (was: 2.17)
        workflow-support:2.18 (was: 2.17)
    

=> I'll resolve this issue with "cannot reproduce" (or so)...

r.fuereder@xortex.com (JIRA)

unread,
May 22, 2018, 5:46:01 AM5/22/18
to jenkinsc...@googlegroups.com
Reinhold Füreder resolved as Cannot Reproduce
 
Change By: Reinhold Füreder
Status: Open Resolved
Resolution: Cannot Reproduce

svanoort@cloudbees.com (JIRA)

unread,
May 22, 2018, 9:38:01 AM5/22/18
to jenkinsc...@googlegroups.com
Sam Van Oort commented on Bug JENKINS-49365
 
Re: Resuming of pipelines vs. docker.image(...).inside(...)?

Reinhold Füreder Any issue that can no longer be reproduced after apply updates is one I'll count as a win!

r.fuereder@xortex.com (JIRA)

unread,
May 22, 2018, 9:42:02 AM5/22/18
to jenkinsc...@googlegroups.com

r.fuereder@xortex.com (JIRA)

unread,
Jun 28, 2018, 2:57:03 AM6/28/18
to jenkinsc...@googlegroups.com

Unfortunately it ("java.lang.ClassNotFoundException: org.jenkinsci.plugins.docker.workflow.Docker$Image") happend now (once) again:

...
[Pipeline] stage
[Pipeline] { (docker-other-tests)
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/workspace/ACME-Pipeline
[Pipeline] {
[Pipeline] ws
Running in /var/lib/jenkins/workspace/ACME-Pipeline-WS
[Pipeline] {
[Pipeline] sh
[ACME-Pipeline-WS] Running shell script
+ find . -mindepth 1 -delete
[Pipeline] unstash
[Pipeline] sh
[ACME-Pipeline-WS] Running shell script
+ unzip -q acme.zip
[Pipeline] pwd
[Pipeline] unstash
[Pipeline] sh
[ACME-Pipeline-WS] Running shell script
+ unzip -q acme-unittests.zip
[Pipeline] pwd
[Pipeline] sh
[ACME-Pipeline-WS] Running shell script
+ docker inspect -f . acme/phpunit:5
.
[Pipeline] withDockerContainer
Jenkins does not seem to be running inside a container
$ docker run -t -d -u 10112:10005 --dns=10.1.114.5 -w /var/lib/jenkins/workspace/ACME-Pipeline-WS -v /var/lib/jenkins/workspace/ACME-Pipeline-WS:/var/lib/jenkins/workspace/ACME-Pipeline-WS:rw,z -v /var/lib/jenkins/workspace/ACME-Pipeline-WS@tmp:/var/lib/jenkins/workspace/ACME-Pipeline-WS@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** acme/phpunit:5 cat
$ docker top 5e3d99376e810429e9d57ce61201af771f2dc3d2feb3cd21335398eadbfd3370 -eo pid,comm
[Pipeline] {
...
[Pipeline] sh
[ACME-Pipeline-WS] Running shell script
+ ./gradlew --no-daemon -PphpUnitXml=phpunit-result.xml executeOtherPHPUnitTests
> Task :deleteDependencyCache UP-TO-DATE

> Task :executeOtherPHPUnitTests
Execution according to PHPUnit XML configuration file './unittests/AllOtherPHPUnitTests.xml'...
PHPUnit 6.5.8 by Sebastian Bergmann and contributors.

.............................................                     45 / 45 (100%)

Time: 1.62 minutes, Memory: 82.00MB

OK (45 tests, 89 assertions)

BUILD SUCCESSFUL in 1m 43s
2 actionable tasks: 1 executed, 1 up-to-date
Resuming build at Thu Jun 28 08:29:44 CEST 2018 after Jenkins restart
Waiting to resume part of ACME-Pipeline 20180628-082349-rev163009: ???
Waiting to resume part of ACME-Pipeline 20180628-082349-rev163009: Jenkins is about to shut down
Waiting to resume part of ACME-Pipeline 20180628-082349-rev163009: Jenkins is about to shut down
Waiting to resume part of ACME-Pipeline 20180628-082349-rev163009: Jenkins is about to shut down
Waiting to resume part of ACME-Pipeline 20180628-082349-rev163009: Jenkins is about to shut down
Waiting to resume part of ACME-Pipeline 20180628-082349-rev163009: Jenkins is about to shut down
Waiting to resume part of ACME-Pipeline 20180628-082349-rev163009: Jenkins is about to shut down
Waiting to resume part of ACME-Pipeline 20180628-082349-rev163009: Jenkins is about to shut down
Waiting to resume part of ACME-Pipeline 20180628-082349-rev163009: Jenkins is about to shut down
Waiting to resume part of ACME-Pipeline 20180628-082349-rev163009: Jenkins is about to shut down
Waiting to resume part of ACME-Pipeline 20180628-082349-rev163009: Jenkins is about to shut down
Resuming build at Thu Jun 28 08:32:27 CEST 2018 after Jenkins restart
Waiting to resume part of ACME-Pipeline 20180628-082349-rev163009: ???
Waiting to resume part of ACME-Pipeline 20180628-082349-rev163009: Jenkins is about to shut down
Waiting to resume part of ACME-Pipeline 20180628-082349-rev163009: Jenkins is about to shut down
Waiting to resume part of ACME-Pipeline 20180628-082349-rev163009: Jenkins is about to shut down
Waiting to resume part of ACME-Pipeline 20180628-082349-rev163009: Jenkins is about to shut down
Waiting to resume part of ACME-Pipeline 20180628-082349-rev163009: Jenkins is about to shut down
Waiting to resume part of ACME-Pipeline 20180628-082349-rev163009: Jenkins is about to shut down
Waiting to resume part of ACME-Pipeline 20180628-082349-rev163009: Jenkins is about to shut down
Waiting to resume part of ACME-Pipeline 20180628-082349-rev163009: Jenkins is about to shut down
Waiting to resume part of ACME-Pipeline 20180628-082349-rev163009: Jenkins is about to shut down
Resuming build at Thu Jun 28 08:34:59 CEST 2018 after Jenkins restart
Waiting to resume part of ACME-Pipeline 20180628-082349-rev163009: ???
Waiting to resume part of ACME-Pipeline 20180628-082349-rev163009: Jenkins is about to shut down
Waiting to resume part of ACME-Pipeline 20180628-082349-rev163009: Jenkins is about to shut down
Waiting to resume part of ACME-Pipeline 20180628-082349-rev163009: Jenkins is about to shut down
[Pipeline] End of Pipeline
java.lang.ClassNotFoundException: org.jenkinsci.plugins.docker.workflow.Docker$Image
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:677)
	at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787)
	at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:775)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at org.jboss.marshalling.AbstractClassResolver.loadClass(AbstractClassResolver.java:131)
	at org.jboss.marshalling.AbstractClassResolver.resolveClass(AbstractClassResolver.java:112)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadClassDescriptor(RiverUnmarshaller.java:948)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1255)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:276)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:224)
	at org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1752)
	at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1665)
	at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1613)
	at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1613)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1292)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:276)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:224)
	at org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1752)
	at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1665)
	at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1613)
	at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1613)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1292)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:276)
	at org.jboss.marshalling.river.BlockUnmarshaller.readObject(BlockUnmarshaller.java:149)
	at org.jboss.marshalling.river.BlockUnmarshaller.readObject(BlockUnmarshaller.java:135)
	at org.jboss.marshalling.MarshallerObjectInputStream.readObjectOverride(MarshallerObjectInputStream.java:53)
	at org.jboss.marshalling.river.RiverObjectInputStream.readObjectOverride(RiverObjectInputStream.java:307)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
	at java.util.HashMap.readObject(HashMap.java:1409)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.jboss.marshalling.reflect.SerializableClass.callReadObject(SerializableClass.java:309)
	at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1644)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1292)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:276)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:224)
	at org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1752)
	at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1665)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1292)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:276)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
	at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)
	at org.jenkinsci.plugins.workflow.support.pickles.serialization.RiverReader$SandboxedUnmarshaller.lambda$readObject$0(RiverReader.java:250)
	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
	at org.jenkinsci.plugins.workflow.support.pickles.serialization.RiverReader$SandboxedUnmarshaller.sandbox(RiverReader.java:237)
	at org.jenkinsci.plugins.workflow.support.pickles.serialization.RiverReader$SandboxedUnmarshaller.readObject(RiverReader.java:250)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$2.onSuccess(CpsFlowExecution.java:782)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$2.onSuccess(CpsFlowExecution.java:775)
	at org.jenkinsci.plugins.workflow.support.concurrent.Futures$1.run(Futures.java:150)
	at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)
	at com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:149)
	at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:134)
	at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:170)
	at org.jenkinsci.plugins.workflow.support.concurrent.ChainingListenableFuture.access$000(ChainingListenableFuture.java:33)
	at org.jenkinsci.plugins.workflow.support.concurrent.ChainingListenableFuture$1.run(ChainingListenableFuture.java:196)
	at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)
	at com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:149)
	at com.google.common.util.concurrent.ExecutionList.add(ExecutionList.java:105)
	at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:155)
	at org.jenkinsci.plugins.workflow.support.concurrent.ChainingListenableFuture.run(ChainingListenableFuture.java:189)
	at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)
	at com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:149)
	at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:134)
	at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:170)
	at org.jenkinsci.plugins.workflow.support.concurrent.ChainingListenableFuture.access$000(ChainingListenableFuture.java:33)
	at org.jenkinsci.plugins.workflow.support.concurrent.ChainingListenableFuture$1.run(ChainingListenableFuture.java:196)
	at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)
	at com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:149)
	at com.google.common.util.concurrent.ExecutionList.add(ExecutionList.java:105)
	at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:155)
	at org.jenkinsci.plugins.workflow.support.concurrent.ChainingListenableFuture.run(ChainingListenableFuture.java:189)
	at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)
	at com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:149)
	at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:134)
	at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:170)
	at org.jenkinsci.plugins.workflow.support.concurrent.ListFuture.setOneValue(ListFuture.java:158)
	at org.jenkinsci.plugins.workflow.support.concurrent.ListFuture.access$000(ListFuture.java:40)
	at org.jenkinsci.plugins.workflow.support.concurrent.ListFuture$2.run(ListFuture.java:107)
	at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)
	at com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:149)
	at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:134)
	at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:170)
	at org.jenkinsci.plugins.workflow.support.concurrent.ChainingListenableFuture.access$000(ChainingListenableFuture.java:33)
	at org.jenkinsci.plugins.workflow.support.concurrent.ChainingListenableFuture$1.run(ChainingListenableFuture.java:196)
	at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)
	at com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:149)
	at com.google.common.util.concurrent.ExecutionList.add(ExecutionList.java:105)
	at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:155)
	at org.jenkinsci.plugins.workflow.support.concurrent.ChainingListenableFuture.run(ChainingListenableFuture.java:189)
	at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)
	at com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:149)
	at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:134)
	at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:170)
	at org.jenkinsci.plugins.workflow.support.pickles.TryRepeatedly.access$500(TryRepeatedly.java:48)
	at org.jenkinsci.plugins.workflow.support.pickles.TryRepeatedly$1.run(TryRepeatedly.java:112)
	at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
Caused: java.io.IOException: Failed to load build state
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$3.onSuccess(CpsFlowExecution.java:854)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$3.onSuccess(CpsFlowExecution.java:852)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$4$1.run(CpsFlowExecution.java:906)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:35)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Finished: FAILURE

Is that a kind of timing issue: depending on which point during the sh step (that is running inside the Docker container) the Jenkins master restart is happening (of course I had used the "Manage Jenkins > Prepare for Shutdown" before)!?

This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396)

jglick@cloudbees.com (JIRA)

unread,
Jul 20, 2018, 4:11:01 PM7/20/18
to jenkinsc...@googlegroups.com

r.fuereder@xortex.com (JIRA)

unread,
Jul 24, 2018, 2:15:01 AM7/24/18
to jenkinsc...@googlegroups.com
Reinhold Füreder reopened an issue
 

As commented above this issue has re-occurred => re-opening this issue...

Change By: Reinhold Füreder
Resolution: Cannot Reproduce
Status: Resolved Reopened

r.fuereder@xortex.com (JIRA)

unread,
Dec 18, 2019, 2:05:05 AM12/18/19
to jenkinsc...@googlegroups.com
Reinhold Füreder commented on Bug JENKINS-49365
 
Re: Resuming of pipelines vs. docker.image(...).inside(...)?

This problem has definitely re-occurred, because I accidentally restarted Jenkins master without waiting for pipelines to complete (cf. JENKINS-60434):

...
[Pipeline] withEnv
[Pipeline] {
[Pipeline] withDockerRegistry
[Pipeline] {
[Pipeline] isUnix
[Pipeline] sh
07:34:43  + docker inspect -f . acme/mytool:13
07:34:43  
07:34:43  Error: No such object: acme/mytool:13
[Pipeline] isUnix
[Pipeline] sh
07:34:43  + docker inspect -f . dockerregistry.acme.com/acme/mytool:13
07:34:43  .
[Pipeline] withDockerContainer
07:34:43  Jenkins does not seem to be running inside a container
07:34:43  $ docker run -t -d -u 10112:10005 --network=none -w /var/lib/jenkins/workspace/MyProject/MyProject-Pipeline-WS -v /var/lib/jenkins/workspace/MyProject/MyProject-Pipeline-WS:/var/lib/jenkins/workspace/MyProject/MyProject-Pipeline-WS:rw,z -v /var/lib/jenkins/workspace/MyProject/MyProject-Pipeline-WS@tmp:/var/lib/jenkins/workspace/MyProject/MyProject-Pipeline-WS@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** dockerregistry.acme.com/acme/mytool:13 cat
07:34:44  $ docker top 0a46f91e3a89720908a64eef2c8ac3688115a6472cb34161abec37b6bfc68e67 -eo pid,comm
[Pipeline] {
[Pipeline] lock
07:34:44  Trying to acquire lock on [Label: MyTool, Quantity: 1]
07:34:44  Lock acquired on [Label: MyTool, Quantity: 1]
[Pipeline] {
[Pipeline] pwd
[Pipeline] sh
07:34:44  + ... /opt/mytool/bin/...sh /var/lib/jenkins/workspace/MyProject/MyProject-Pipeline-WS ...
...
07:34:48  Starting up MyTool  ...done.
Resuming build at Wed Dec 18 07:35:36 CET 2019 after Jenkins restart
Waiting to resume part of MyProject » MyProject-Pipeline 20191218-073417-rev4122d3ea: Finished waiting
[Pipeline] End of Pipeline
[lockable-resources] released lock on [MyTool#1, MyProject-Pipeline-mytool]
java.lang.ClassNotFoundException: org.jenkinsci.plugins.docker.workflow.Docker$Image
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:677)
	at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787)
	at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:775)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at org.jboss.marshalling.AbstractClassResolver.loadClass(AbstractClassResolver.java:123)
	at org.jboss.marshalling.AbstractClassResolver.resolveClass(AbstractClassResolver.java:104)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadClassDescriptor(RiverUnmarshaller.java:1022)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1355)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:272)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:220)
	at org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1853)
	at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1767)
	at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1715)
	at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1715)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1395)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:272)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:220)
	at org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1853)
	at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1767)
	at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1715)
	at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1715)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1395)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:272)
	at org.jboss.marshalling.river.BlockUnmarshaller.readObject(BlockUnmarshaller.java:149)
	at org.jboss.marshalling.river.BlockUnmarshaller.readObject(BlockUnmarshaller.java:135)
	at org.jboss.marshalling.MarshallerObjectInputStream.readObjectOverride(MarshallerObjectInputStream.java:53)
	at org.jboss.marshalling.river.RiverObjectInputStream.readObjectOverride(RiverObjectInputStream.java:307)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
	at java.util.HashMap.readObject(HashMap.java:1412)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.jboss.marshalling.reflect.JDKSpecific$SerMethods.callReadObject(JDKSpecific.java:179)
	at org.jboss.marshalling.reflect.SerializableClass.callReadObject(SerializableClass.java:212)
	at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1746)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1395)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:272)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:220)
	at org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1853)
	at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1767)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1395)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:272)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:205)
	at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)
	at org.jenkinsci.plugins.workflow.support.pickles.serialization.RiverReader$SandboxedUnmarshaller.lambda$readObject$0(RiverReader.java:250)
	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:237)
	at org.jenkinsci.plugins.workflow.support.pickles.serialization.RiverReader$SandboxedUnmarshaller.sandbox(RiverReader.java:237)
	at org.jenkinsci.plugins.workflow.support.pickles.serialization.RiverReader$SandboxedUnmarshaller.readObject(RiverReader.java:250)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$2.onSuccess(CpsFlowExecution.java:783)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$2.onSuccess(CpsFlowExecution.java:776)
	at org.jenkinsci.plugins.workflow.support.concurrent.Futures$1.run(Futures.java:150)
	at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)
	at com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:149)
	at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:134)
	at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:170)
	at org.jenkinsci.plugins.workflow.support.concurrent.ChainingListenableFuture.access$000(ChainingListenableFuture.java:33)
	at org.jenkinsci.plugins.workflow.support.concurrent.ChainingListenableFuture$1.run(ChainingListenableFuture.java:196)
	at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)
	at com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:149)
	at com.google.common.util.concurrent.ExecutionList.add(ExecutionList.java:105)
	at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:155)
	at org.jenkinsci.plugins.workflow.support.concurrent.ChainingListenableFuture.run(ChainingListenableFuture.java:189)
	at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)
	at com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:149)
	at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:134)
	at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:170)
	at org.jenkinsci.plugins.workflow.support.concurrent.ChainingListenableFuture.access$000(ChainingListenableFuture.java:33)
	at org.jenkinsci.plugins.workflow.support.concurrent.ChainingListenableFuture$1.run(ChainingListenableFuture.java:196)
	at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)
	at com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:149)
	at com.google.common.util.concurrent.ExecutionList.add(ExecutionList.java:105)
	at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:155)
	at org.jenkinsci.plugins.workflow.support.concurrent.ChainingListenableFuture.run(ChainingListenableFuture.java:189)
	at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)
	at com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:149)
	at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:134)
	at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:170)
	at org.jenkinsci.plugins.workflow.support.concurrent.ListFuture.setOneValue(ListFuture.java:158)
	at org.jenkinsci.plugins.workflow.support.concurrent.ListFuture.access$000(ListFuture.java:40)
	at org.jenkinsci.plugins.workflow.support.concurrent.ListFuture$2.run(ListFuture.java:107)
	at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)
	at com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:149)
	at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:134)
	at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:170)
	at org.jenkinsci.plugins.workflow.support.concurrent.ChainingListenableFuture.access$000(ChainingListenableFuture.java:33)
	at org.jenkinsci.plugins.workflow.support.concurrent.ChainingListenableFuture$1.run(ChainingListenableFuture.java:196)
	at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)
	at com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:149)
	at com.google.common.util.concurrent.ExecutionList.add(ExecutionList.java:105)
	at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:155)
	at org.jenkinsci.plugins.workflow.support.concurrent.ChainingListenableFuture.run(ChainingListenableFuture.java:189)
	at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)
	at com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:149)
	at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:134)
	at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:170)
	at org.jenkinsci.plugins.workflow.support.pickles.TryRepeatedly.access$500(TryRepeatedly.java:48)
	at org.jenkinsci.plugins.workflow.support.pickles.TryRepeatedly$1.run(TryRepeatedly.java:112)
	at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
Caused: java.io.IOException: Failed to load build state
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$3.onSuccess(CpsFlowExecution.java:855)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$3.onSuccess(CpsFlowExecution.java:853)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$4$1.run(CpsFlowExecution.java:907)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:38)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Finished: FAILURE

Please note that the Docker container was still running and I therefore had to stop it...

$ docker ps -a
CONTAINER ID        IMAGE                                          COMMAND                  CREATED             STATUS              PORTS                    NAMES
0a46f91e3a89        dockerregistry.acme.com/acme/mytool:13   "cat"                    17 minutes ago      Up 17 minutes                                nifty_heisenberg
...
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

jglick@cloudbees.com (JIRA)

unread,
Dec 18, 2019, 10:19:03 AM12/18/19
to jenkinsc...@googlegroups.com

No idea as to the root cause. Again the workaround would be simple: avoid the DSL and use withDockerContainer directly. (Or, better yet, avoid the whole plugin and run docker CLI commands from sh.)

Reply all
Reply to author
Forward
0 new messages