Resuming of pipelines vs. docker.image(...).inside(...)?
Reinhold Füreder
Hi,
when playing around with resuming of pipelines after restarting the Jenkins master (note – not sure if of relevance though: it is a single jenkins master without any slaves setup) during pipeline execution, I stumbled over the the following problem when the restart happens in the course of executing a docker.image(...).inside(...) closure:
java.lang.ClassNotFoundException: org.jenkinsci.plugins.docker.workflow.Docker$Image
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:677)
at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787)
at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:775)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.jboss.marshalling.AbstractClassResolver.loadClass(AbstractClassResolver.java:131)
at org.jboss.marshalling.AbstractClassResolver.resolveClass(AbstractClassResolver.java:112)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadClassDescriptor(RiverUnmarshaller.java:948)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1255)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:276)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:224)
at org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1752)
at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1665)
at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1613)
at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1613)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1292)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:276)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:224)
at org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1752)
at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1665)
at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1613)
at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1613)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1292)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:276)
at org.jboss.marshalling.river.BlockUnmarshaller.readObject(BlockUnmarshaller.java:149)
at org.jboss.marshalling.river.BlockUnmarshaller.readObject(BlockUnmarshaller.java:135)
at org.jboss.marshalling.MarshallerObjectInputStream.readObjectOverride(MarshallerObjectInputStream.java:53)
at org.jboss.marshalling.river.RiverObjectInputStream.readObjectOverride(RiverObjectInputStream.java:307)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
at java.util.HashMap.readObject(HashMap.java:1409)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.jboss.marshalling.reflect.SerializableClass.callReadObject(SerializableClass.java:309)
at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1644)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1292)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:276)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:224)
at org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1752)
at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1665)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1292)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:276)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$2.onSuccess(CpsFlowExecution.java:742)
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$2.onSuccess(CpsFlowExecution.java:735)
at org.jenkinsci.plugins.workflow.support.concurrent.Futures$1.run(Futures.java:150)
at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)
at com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:149)
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:134)
at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:170)
at org.jenkinsci.plugins.workflow.support.concurrent.ChainingListenableFuture.access$000(ChainingListenableFuture.java:33)
at org.jenkinsci.plugins.workflow.support.concurrent.ChainingListenableFuture$1.run(ChainingListenableFuture.java:196)
at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)
at com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:149)
at com.google.common.util.concurrent.ExecutionList.add(ExecutionList.java:105)
at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:155)
at org.jenkinsci.plugins.workflow.support.concurrent.ChainingListenableFuture.run(ChainingListenableFuture.java:189)
at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)
at com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:149)
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:134)
at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:170)
at org.jenkinsci.plugins.workflow.support.concurrent.ChainingListenableFuture.access$000(ChainingListenableFuture.java:33)
at org.jenkinsci.plugins.workflow.support.concurrent.ChainingListenableFuture$1.run(ChainingListenableFuture.java:196)
at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)
at com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:149)
at com.google.common.util.concurrent.ExecutionList.add(ExecutionList.java:105)
at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:155)
at org.jenkinsci.plugins.workflow.support.concurrent.ChainingListenableFuture.run(ChainingListenableFuture.java:189)
at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)
at com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:149)
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:134)
at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:170)
at org.jenkinsci.plugins.workflow.support.concurrent.ListFuture.setOneValue(ListFuture.java:158)
at org.jenkinsci.plugins.workflow.support.concurrent.ListFuture.access$000(ListFuture.java:40)
at org.jenkinsci.plugins.workflow.support.concurrent.ListFuture$2.run(ListFuture.java:107)
at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)
at com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:149)
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:134)
at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:170)
at org.jenkinsci.plugins.workflow.support.concurrent.ChainingListenableFuture.access$000(ChainingListenableFuture.java:33)
at org.jenkinsci.plugins.workflow.support.concurrent.ChainingListenableFuture$1.run(ChainingListenableFuture.java:196)
at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)
at com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:149)
at com.google.common.util.concurrent.ExecutionList.add(ExecutionList.java:105)
at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:155)
at org.jenkinsci.plugins.workflow.support.concurrent.ChainingListenableFuture.run(ChainingListenableFuture.java:189)
at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)
at com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:149)
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:134)
at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:170)
at org.jenkinsci.plugins.workflow.support.pickles.TryRepeatedly.access$500(TryRepeatedly.java:48)
at org.jenkinsci.plugins.workflow.support.pickles.TryRepeatedly$1.run(TryRepeatedly.java:112)
at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
Caused: java.io.IOException: Failed to load build state
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$3.onSuccess(CpsFlowExecution.java:814)
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$3.onSuccess(CpsFlowExecution.java:812)
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$4$1.run(CpsFlowExecution.java:861)
at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:35)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Here is the build log right before the problem and including the resume logs:
[Pipeline] sh
[ACME-Pipeline (Branch)-WS] Running shell script
+ docker inspect -f . ACME/phpunit:5
.
[Pipeline] withDockerContainer
Jenkins does not seem to be running inside a container
$ docker run -t -d -u 10112:10005 --entrypoint -w "/var/lib/jenkins/workspace/ACME-Pipeline (Branch)-WS" -v "/var/lib/jenkins/workspace/ACME-Pipeline (Branch)-WS:/var/lib/jenkins/workspace/ACME-Pipeline (Branch)-WS:rw,z" -v "/var/lib/jenkins/workspace/ACME-Pipeline (Branch)-WS@tmp:/var/lib/jenkins/workspace/ACME-Pipeline (Branch)-WS@tmp:rw,z" -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** --entrypoint cat ACME/phpunit:5
[Pipeline] {
[Pipeline] sh
[ACME-Pipeline (Branch)-WS] Running shell script
...
[Pipeline] sh
[ACME-Pipeline (Branch)-WS] Running shell script
+ ./gradlew --no-daemon -PphpUnitXml=phpunit-result.xml executePlainPHPUnitTests
:deleteDependencyCache UP-TO-DATE
:executePlainPHPUnitTests
PHPUnit 6.5.5 by Sebastian Bergmann and contributors.
............................................................... 63 / 484 ( 13%)
............................................................... 126 / 484 ( 26%)
............................................................... 189 / 484 ( 39%)
............................................................... 252 / 484 ( 52%)
............................................................... 315 / 484 ( 65%)
............................................................... 378 / 484 ( 78%)
............................................................... 441 / 484 ( 91%)
........................................... 484 / 484 (100%)
Time: 10.73 seconds, Memory: 78.00MB
OK (484 tests, 927 assertions)
BUILD SUCCESSFUL in 15s
2 actionable tasks: 1 executed, 1 up-to-date
Resuming build at Wed Jan 24 15:57:45 CET 2018 after Jenkins restart
Waiting to resume part of ACME-Pipeline (Branch) 20180124-155522-revUNKNOWN: Jenkins is about to shut down
Waiting to resume part of ACME-Pipeline (Branch) 20180124-155522-revUNKNOWN: Jenkins is about to shut down
Waiting to resume part of ACME-Pipeline (Branch) 20180124-155522-revUNKNOWN: Jenkins is about to shut down
[Pipeline] End of Pipeline
java.lang.ClassNotFoundException: org.jenkinsci.plugins.docker.workflow.Docker$Image
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
Here is a thread dump after “Manage Jenkins > Prepare for Shutdown” + waiting for the finishing of the “sh” step using gradle to execute PHPUnit and right before restarting the Jenkins master.
Thread #18
at com.ACME.php.PHPUnitViaGradle.execute(file:/var/lib/jenkins/jobs/ACME-Pipeline%20(Branch)/builds/660/libs/ACME-shared-library/src/com/ACME/php/PHPUnitViaGradle.groovy:30)
at org.jenkinsci.plugins.docker.workflow.Docker$Image.inside(jar:file:/var/lib/jenkins/plugins/docker-workflow/WEB-INF/lib/docker-workflow.jar!/org/jenkinsci/plugins/docker/workflow/Docker.groovy:129)
at DSL.withDockerContainer(Native Method)
at org.jenkinsci.plugins.docker.workflow.Docker$Image.inside(jar:file:/var/lib/jenkins/plugins/docker-workflow/WEB-INF/lib/docker-workflow.jar!/org/jenkinsci/plugins/docker/workflow/Docker.groovy:128)
at org.jenkinsci.plugins.docker.workflow.Docker.node(jar:file:/var/lib/jenkins/plugins/docker-workflow/WEB-INF/lib/docker-workflow.jar!/org/jenkinsci/plugins/docker/workflow/Docker.groovy:63)
at org.jenkinsci.plugins.docker.workflow.Docker$Image.inside(jar:file:/var/lib/jenkins/plugins/docker-workflow/WEB-INF/lib/docker-workflow.jar!/org/jenkinsci/plugins/docker/workflow/Docker.groovy:116)
at com.ACME.php.PHPUnitViaGradle.execute(file:/var/lib/jenkins/jobs/ACME-Pipeline%20(Branch)/builds/660/libs/ACME-shared-library/src/com/ACME/php/PHPUnitViaGradle.groovy:26)
at ACME.executePHPUnitTests(/var/lib/jenkins/jobs/ACME-Pipeline (Branch)/builds/660/libs/ACME-shared-library/vars/ACME.groovy:191)
at WorkflowScript.run(WorkflowScript:65)
at com.ACME.Workspace.prepare(file:/var/lib/jenkins/jobs/ACME-Pipeline%20(Branch)/builds/660/libs/ACME-shared-library/src/com/ACME/Workspace.groovy:53)
at DSL.ws(Native Method)
at com.ACME.Workspace.prepare(file:/var/lib/jenkins/jobs/ACME-Pipeline%20(Branch)/builds/660/libs/ACME-shared-library/src/com/ACME/Workspace.groovy:30)
at ACME.wsWithArtifacts(/var/lib/jenkins/jobs/ACME-Pipeline (Branch)/builds/660/libs/ACME-shared-library/vars/ACME.groovy:164)
at WorkflowScript.run(WorkflowScript:64)
at DSL.node(running on )
at WorkflowScript.run(WorkflowScript:55)
at ACME.skipableStage(/var/lib/jenkins/jobs/ACME-Pipeline (Branch)/builds/660/libs/ACME-shared-library/vars/ACME.groovy:76)
at DSL.stage(Native Method)
at ACME.skipableStage(/var/lib/jenkins/jobs/ACME-Pipeline (Branch)/builds/660/libs/ACME-shared-library/vars/ACME.groovy:67)
at ACME.skipableStage(/var/lib/jenkins/jobs/ACME-Pipeline (Branch)/builds/660/libs/ACME-shared-library/vars/ACME.groovy:57)
at ACME.skipableStage(/var/lib/jenkins/jobs/ACME-Pipeline (Branch)/builds/660/libs/ACME-shared-library/vars/ACME.groovy:52)
at WorkflowScript.run(WorkflowScript:53)
at com.ACME.Build.build(file:/var/lib/jenkins/jobs/ACME-Pipeline%20(Branch)/builds/660/libs/ACME-shared-library/src/com/ACME/Build.groovy:42)
at ACME.build(/var/lib/jenkins/jobs/ACME-Pipeline (Branch)/builds/660/libs/ACME-shared-library/vars/ACME.groovy:48)
at WorkflowScript.run(WorkflowScript:19)
Not sure if it is relevant, but the Jenkins setup is very up to date: Jenins v2.103 with very recent update of these plugins (
---> Plugins installed:
...
workflow-api:2.25
workflow-cps:2.43
workflow-durable-task-step:2.18
workflow-job:2.17
workflow-multibranch:2.17
workflow-support:2.17
…, and thus this output at the top of the build log:
Running in Durability level: MAX_SURVIVABILITY
Is this supposed to work? If so, what am I missing?
Or can it work at all?
Thanks,
Reinhold
Reinhold Füreder - XORTEX eBusiness GmbH
Testautomatisierung und Qualitätssicherung
Veldner Str. 29 | A-4120 Neufelden
Tel: +43 7282 20797 4425 | Fax: +43 7282 20797 4490
Mail: r.fue...@xortex.com
Web: https://xortex.com
Reinhold Füreder
I now dared to file https://issues.jenkins-ci.org/browse/JENKINS-49365