[JIRA] (JENKINS-38769) Pipeline job hangs when agent loses network connectivity

18 views
Skip to first unread message

yotamshapiraspam@gmail.com (JIRA)

unread,
Oct 6, 2016, 12:49:04 AM10/6/16
to jenkinsc...@googlegroups.com
Yotam Shapira created an issue
 
Jenkins / Bug JENKINS-38769
Pipeline job hangs when agent loses network connectivity
Issue Type: Bug Bug
Assignee: Unassigned
Components: pipeline, workflow-support-plugin
Created: 2016/Oct/06 4:48 AM
Environment: Jenkins 2.7.2
Pipeline: Supporting APIs 2.8

Other plugins that may be relevant:
Pipeline: API 2.4
Pipeline: Basic Steps 2.2
Pipeline: Build Step 2.3
Pipeline: Groovy 2.18
Pipeline: Input Step 2.1
Pipeline: Job 2.7
Pipeline: Milestone Step 1.0
Pipeline: Multibranch 2.9
Pipeline: Nodes and Processes 2.5
Pipeline: REST API Plugin 2.1
Pipeline: SCM Step 2.2
Pipeline: Shared Groovy Libraries 2.3
Pipeline: Stage Step 2.2
Pipeline: Stage View Plugin 2.1
Pipeline: Step API 2.4
Self-Organizing Swarm Plug-in Modules 2.2 (Note: the client is using 2.0)
Priority: Critical Critical
Reporter: Yotam Shapira

When running a Pipeline job (see end of this description) while the agent (swarm client v2.0) loses network connectivity (in this case connectivity is lost for a long while), the Pipeline hangs, showing the following in the console log:

Could not connect to <AGENT NAME> to send interrupt signal to process

After clicking the abort button (/stop), I get the same message printed in the log again and then the "Click here to terminate message":

Could not connect to <AGENT NAME> to send interrupt signal to process
Click here to forcibly terminate running steps

After forcing termination, I see the following in the console log:

Terminating sh
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Publish Results)
[Pipeline] step
09:51:54.766 Required context class hudson.Launcher is missing
09:51:54.766 Perhaps you forgot to surround the code with a step that provides this, such as: node
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withDockerContainer
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
java.lang.RuntimeException: Failed to serialize org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage$Tag#actions for class org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage$Tag
	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:256)
	at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224)
	at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
	at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209)
	at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150)
	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)
	at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:82)
	at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37)
	at com.thoughtworks.xstream.XStream.marshal(XStream.java:1026)
	at com.thoughtworks.xstream.XStream.marshal(XStream.java:1015)
	at com.thoughtworks.xstream.XStream.toXML(XStream.java:988)
	at hudson.XmlFile.write(XmlFile.java:178)
	at org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage.saveActions(SimpleXStreamFlowNodeStorage.java:109)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.saveActions(CpsFlowExecution.java:887)
	at org.jenkinsci.plugins.workflow.graph.FlowNode.save(FlowNode.java:315)
	at org.jenkinsci.plugins.workflow.graph.FlowNode$1.persist(FlowNode.java:302)
	at org.jenkinsci.plugins.workflow.graph.FlowNode$1.add(FlowNode.java:278)
	at org.jenkinsci.plugins.workflow.graph.FlowNode$1.add(FlowNode.java:269)
	at java.util.AbstractList.add(AbstractList.java:108)
	at hudson.model.Actionable.addAction(Actionable.java:129)
	at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$FailureAdapter.receive(CpsBodyExecution.java:273)
	at com.cloudbees.groovy.cps.impl.ContinuationGroup.throwException(ContinuationGroup.java:180)
	at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:54)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
	at sun.reflect.GeneratedMethodAccessor1031.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
	at com.cloudbees.groovy.cps.impl.CollectionLiteralBlock$ContinuationImpl.dispatch(CollectionLiteralBlock.java:55)
	at com.cloudbees.groovy.cps.impl.CollectionLiteralBlock$ContinuationImpl.item(CollectionLiteralBlock.java:45)
	at sun.reflect.GeneratedMethodAccessor1034.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:103)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
	at sun.reflect.GeneratedMethodAccessor1031.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
	at performance.Script1.runScript(Script1.groovy:74)
	at ___cps.transform___(Native Method)
	at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:50)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
	at sun.reflect.GeneratedMethodAccessor1031.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
	at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
	at com.cloudbees.groovy.cps.Next.step(Next.java:58)
	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:154)
	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:163)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:324)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:78)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:236)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:224)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:63)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:791)
	Suppressed: java.lang.NullPointerException: while trying to invoke the method hudson.Launcher.getListener() of a null object loaded from field org.jenkinsci.plugins.docker.workflow.client.DockerClient.launcher of an object loaded from local variable 'this'
		at org.jenkinsci.plugins.docker.workflow.client.DockerClient.launch(DockerClient.java:251)
		at org.jenkinsci.plugins.docker.workflow.client.DockerClient.launch(DockerClient.java:247)
		at org.jenkinsci.plugins.docker.workflow.client.DockerClient.launch(DockerClient.java:244)
		at org.jenkinsci.plugins.docker.workflow.client.DockerClient.stop(DockerClient.java:139)
		at org.jenkinsci.plugins.docker.workflow.WithContainerStep.destroy(WithContainerStep.java:104)
		at org.jenkinsci.plugins.docker.workflow.WithContainerStep.access$400(WithContainerStep.java:71)
		at org.jenkinsci.plugins.docker.workflow.WithContainerStep$Callback.finished(WithContainerStep.java:281)
		at org.jenkinsci.plugins.workflow.steps.BodyExecutionCallback$TailCall.onFailure(BodyExecutionCallback.java:124)
		at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$FailureAdapter.receive(CpsBodyExecution.java:278)
		at com.cloudbees.groovy.cps.impl.ThrowBlock$1.receive(ThrowBlock.java:68)
		... 19 more
Caused by: java.lang.RuntimeException: Failed to serialize org.jenkinsci.plugins.workflow.actions.ErrorAction#error for class org.jenkinsci.plugins.workflow.actions.ErrorAction
	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:256)
	at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224)
	at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
	at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209)
	at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150)
	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)
	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:88)
	at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:64)
	at com.thoughtworks.xstream.converters.collections.ArrayConverter.marshal(ArrayConverter.java:45)
	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)
	at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:265)
	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:252)
	at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224)
	at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
	at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209)
	at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150)
	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)
	at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:82)
	at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37)
	at com.thoughtworks.xstream.XStream.marshal(XStream.java:1026)
	at com.thoughtworks.xstream.XStream.marshal(XStream.java:1015)
	at com.thoughtworks.xstream.XStream.toXML(XStream.java:988)
	at hudson.XmlFile.write(XmlFile.java:178)
	at org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage.saveActions(SimpleXStreamFlowNodeStorage.java:109)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.saveActions(CpsFlowExecution.java:887)
	at org.jenkinsci.plugins.workflow.graph.FlowNode.save(FlowNode.java:315)
	at org.jenkinsci.plugins.workflow.graph.FlowNode$1.persist(FlowNode.java:302)
	at org.jenkinsci.plugins.workflow.graph.FlowNode$1.add(FlowNode.java:278)
	at org.jenkinsci.plugins.workflow.graph.FlowNode$1.add(FlowNode.java:269)
	at java.util.AbstractList.add(AbstractList.java:108)
	at hudson.model.Actionable.addAction(Actionable.java:129)
	at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$FailureAdapter.receive(CpsBodyExecution.java:273)
	at com.cloudbees.groovy.cps.impl.ContinuationGroup.throwException(ContinuationGroup.java:180)
	at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:54)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
	at sun.reflect.GeneratedMethodAccessor1031.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
	at com.cloudbees.groovy.cps.impl.CollectionLiteralBlock$ContinuationImpl.dispatch(CollectionLiteralBlock.java:55)
	at com.cloudbees.groovy.cps.impl.CollectionLiteralBlock$ContinuationImpl.item(CollectionLiteralBlock.java:45)
	at sun.reflect.GeneratedMethodAccessor1034.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:103)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
	at sun.reflect.GeneratedMethodAccessor1031.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
	... 26 more
Caused by: java.lang.RuntimeException: Failed to serialize java.lang.Throwable#cause for class java.lang.IllegalArgumentException
	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:256)
	at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224)
	at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
	at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209)
	at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150)
	at com.thoughtworks.xstream.converters.extended.ThrowableConverter.marshal(ThrowableConverter.java:62)
	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)
	at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:265)
	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:252)
	... 83 more
Caused by: java.lang.UnsupportedOperationException: Refusing to marshal org.codehaus.groovy.runtime.InvokerInvocationException for security reasons
	at hudson.util.XStream2$BlacklistedTypesConverter.marshal(XStream2.java:442)
	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)
	at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:265)
	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:252)
	... 93 more
Finished: ABORTED

Pipeline is defined like this:
Main Script:

def pipelineScript
node('my_node') {
    git changelog: false, credentialsId: '...', poll: false, url: 'ssh://...'
    pipelineScript = load '.....groovy'
}

pipelineScript.runScript()

Script1:

def runScript() {
    node("my_other_node") {

        timestamps {

            stage('S1') {
                seleniumHost = sh script: 'hostname --all-fqdns', returnStdout: true
                seleniumHost = seleniumHost.stripIndent().trim()

                env.PATH = '/usr/lib/jvm/jdk1.8.0_60//bin:$PATH:/usr/local/bin/'
            }

            stage('Checkout Git') {
                checkout(
                        [
                                $class           : 'GitSCM', branches: [[name: "${GERRIT_BRANCH}"]],
                                userRemoteConfigs: [[credentialsId: '...', url: "ssh://...", refspec: "${REFSPEC}"]],
                        ])
            }
            stage('S3') {
                sh """Some script""".stripIndent().trim()

                sh """Some other script""".stripIndent().trim()
            }

            docker.image('...').inside('') {

                try {
                    stage('S4') {
                        sh """
                        ...
                        npm install
                  """.stripIndent().trim()
                    }

                    stage('Run Tests") {
                        timeout(20) {
                            sh """grunt run_tests""".stripIndent().trim()
                        }
                    }
                } finally {
                    stage('Publish Results') {
                        archiveArtifacts artifacts: "**/${resultsFileName}"
                        junit "**/${resultsFileName}"
                    }

                }
            }

        }
    }
}

return this

Thanks!

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
Atlassian logo

jglick@cloudbees.com (JIRA)

unread,
Oct 19, 2016, 5:13:01 PM10/19/16
to jenkinsc...@googlegroups.com
Jesse Glick updated an issue
Change By: Jesse Glick
Component/s: workflow-durable-task-step-plugin
Component/s: pipeline
Component/s: workflow-support-plugin

jglick@cloudbees.com (JIRA)

unread,
Oct 19, 2016, 5:14:04 PM10/19/16
to jenkinsc...@googlegroups.com
Jesse Glick updated an issue

The failure to abort when the agent is hung—the summary issue—is a known limitation that I would like to improve.

The exceptions are probably all duplicates of JENKINS-34488, though the surface symptoms are different.

Change By: Jesse Glick
Priority: Critical Major

jglick@cloudbees.com (JIRA)

unread,
Dec 29, 2016, 3:11:02 PM12/29/16
to jenkinsc...@googlegroups.com
Jesse Glick started work on Bug JENKINS-38769
 
Change By: Jesse Glick
Status: Open In Progress

jglick@cloudbees.com (JIRA)

unread,
Dec 29, 2016, 4:28:03 PM12/29/16
to jenkinsc...@googlegroups.com

ryan.campbell@gmail.com (JIRA)

unread,
Dec 30, 2016, 7:58:01 AM12/30/16
to jenkinsc...@googlegroups.com
recampbell updated an issue
Change By: recampbell
Labels: pipeline-hangs robustness

jglick@cloudbees.com (JIRA)

unread,
Jan 5, 2017, 12:41:02 PM1/5/17
to jenkinsc...@googlegroups.com
Reply all
Reply to author
Forward
0 new messages