[JIRA] (JENKINS-50020) CpsFlowExecution unresponsive and Failed to serialize SimpleXStreamFlowNodeStorage

henti@geekware.co.za (JIRA)

unread,

Mar 8, 2018, 10:31:04 AM3/8/18

to jenkinsc...@googlegroups.com

Henti Smith created an issue

Jenkins /

JENKINS-50020

CpsFlowExecution unresponsive and Failed to serialize SimpleXStreamFlowNodeStorage

Issue Type:	Bug
Assignee:	Unassigned
Components:	workflow-support-plugin
Created:	2018-03-08 15:30
Environment:	Operating System (including version, and whether it's 32/64 bit) of all systems involved (your client's, the Jenkins server's, all slave nodes'), OS : Debian GNU/Linux 8.10 Kernel : 3.16.51-3+deb8u1 openjdk version "1.8.0_131" OpenJDK Runtime Environment (build 1.8.0_131-8u131-b11-1~bpo8+1-b11) OpenJDK 64-Bit Server VM (build 25.131-b11, mixed mode) JAVA_ARGS="-Djava.awt.headless=true -server -XX:+AlwaysPreTouch -Xloggc:/var/log/jenkins/gc-%t.log -XX:NumberOfGCLogFiles=5 -XX:+UseGCLogFileRotation -XX:GCLogFileSize=20m -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintGCCause -XX:+PrintTenuringDistribution -XX:+PrintReferenceGC -XX:+PrintAdaptiveSizePolicy -XX:+UseG1GC -XX:+ExplicitGCInvokesConcurrent -XX:+ParallelRefProcEnabled -XX:+UseStringDeduplication -XX:+UnlockDiagnosticVMOptions -XX:G1SummarizeRSetStatsPeriod=1 -Xms2g -Xmx4g" Jenkins : 2.89.4 accessed directly and running directly. Installed from https://pkg.jenkins.io/debian/ metrics 3.1.2.10 pam-auth 1.3 pipeline-model-api 1.2.7 run-condition 1.0 pipeline-utility-steps 2.0.1 pipeline-github-lib 1.0 antisamy-markup-formatter 1.5 jquery 1.12.4-0 authentication-tokens 1.3 credentials-binding 1.15 icon-shim 2.0.3 pipeline-input-step 2.8 jquery-detached 1.2.1 github-branch-source 2.3.2 command-launcher 1.0 handlebars 1.1.1 matrix-project 1.12 resource-disposer 0.8 build-timeout 1.19 pipeline-model-definition 1.2.7 jackson2-api 2.8.11.1 copyartifact 1.39 scm-api 2.2.6 pipeline-stage-tags-metadata 1.2.7 PrioritySorter 3.6.0 pipeline-stage-step 2.3 azure-commons 0.2.4 email-ext 2.61 workflow-multibranch 2.17 gradle 1.28 workflow-cps 2.45 workflow-aggregator 2.5 workflow-api 2.25 script-security 1.41 cloudbees-folder 6.3 msbuild 1.29 git-server 1.7 ssh-agent 1.15 branch-api 2.0.18 promoted-builds 2.31.1 matrix-auth 2.2 pipeline-stage-view 2.9 workflow-cps-global-lib 2.9 ssh-credentials 1.13 ssh-slaves 1.25.1 timestamper 1.8.9 job-dsl 1.68 ant 1.8 workflow-scm-step 2.6 pipeline-graph-analysis 1.6 external-monitor-job 1.7 mailer 1.20 github-api 1.90 azure-credentials 1.5.0 windows-slaves 1.3.1 throttle-concurrents 2.0.1 conditional-buildstep 1.3.6 jsch 0.1.54.2 nodelabelparameter 1.7.2 maven-plugin 3.1 workflow-basic-steps 2.6 token-macro 2.3 rebuild 1.27 ws-cleanup 0.34 pipeline-rest-api 2.9 workflow-step-api 2.14 credentials 2.1.16 structs 1.14 ldap 1.19 git-client 2.7.1 docker-workflow 1.15.1 plain-credentials 1.4 mapdb-api 1.0.9.0 javadoc 1.4 github-organization-folder 1.6 ghprb 1.39.1-0-Semmle github-oauth 0.29 azure-vm-agents 0.6.1 github 1.29.0 display-url-api 2.2.0 subversion 2.10.2 momentjs 1.1.1 parameterized-trigger 2.35.2 pipeline-model-declarative-agent 1.1.1 workflow-durable-task-step 2.19 pipeline-build-step 2.7 workflow-support 2.18 pipeline-milestone-step 1.3.1 groovy 2.0 jobConfigHistory 2.18 ace-editor 1.1 durable-task 1.18 pipeline-model-extensions 1.2.7 apache-httpcomponents-client-4-api 4.5.3-2.1 junit 1.24 docker-commons 1.11 build-alias-setter 0.4 git 3.7.0 workflow-job 2.17 bouncycastle-api 2.16.2 cloud-stats 0.17 lockable-resources 2.1
Labels:	jenkins
Priority:	Major
Reporter:	Henti Smith

This happens to multiple jobs in our queues, but the logs are related to one job. the job stop processing and the master logs contains the following

 
                                                                Mar 07, 2018 4:51:52 PM org.jenkinsci.plugins.workflow.support.concurrent.Timeout lambda$ping$0
INFO: Running CpsFlowExecution[Owner[Language-Tests/JavaScript/2290:Language-Tests/JavaScript #2290]] unresponsive for 5 sec
-- SNIP --
Mar 07, 2018 4:55:18 PM org.jenkinsci.plugins.workflow.support.concurrent.Timeout lambda$ping$0
INFO: Running CpsFlowExecution[Owner[Language-Tests/JavaScript/2290:Language-Tests/JavaScript #2290]] unresponsive for 3 min 30 sec
 
                                                            

Shortly thereafter we see an exception.

 
                                                                Mar 07, 2018 4:55:22 PM
org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService reportProblem
WARNING: Unexpected exception in CPS VM thread:
CpsFlowExecution[Owner[Language-Tests/JavaScript/2290:Language-Tests/JavaScript
#2290]]
java.lang.RuntimeException: Failed to serialize
org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage$Tag#actions
for class org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage$Tag
        at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:256)
        at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224)
        at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
        at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209)
        at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150)
        at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
        at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
        at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)
        at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:82)
        at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37)
        at com.thoughtworks.xstream.XStream.marshal(XStream.java:1026)
        at com.thoughtworks.xstream.XStream.marshal(XStream.java:1015)
        at com.thoughtworks.xstream.XStream.toXML(XStream.java:988)
        at hudson.XmlFile.write(XmlFile.java:181)
        at org.jenkinsci.plugins.workflow.support.PipelineIOUtils.writeByXStream(PipelineIOUtils.java:30)
        at org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage.writeNode(SimpleXStreamFlowNodeStorage.java:182)
        at org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage.flushNode(SimpleXStreamFlowNodeStorage.java:151)
        at org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage.autopersist(SimpleXStreamFlowNodeStorage.java:136)
        at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$TimingFlowNodeStorage.autopersist(CpsFlowExecution.java:1691)
        at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.maybeAutoPersistNode(CpsFlowExecution.java:1149)
        at org.jenkinsci.plugins.workflow.cps.FlowHead.setNewHead(FlowHead.java:120)
        at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution.addBodyEndFlowNode(CpsBodyExecution.java:405)
        at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution.access$800(CpsBodyExecution.java:64)
        at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$FailureAdapter.receive(CpsBodyExecution.java:343)
        at com.cloudbees.groovy.cps.impl.ContinuationGroup.throwException(ContinuationGroup.java:157)
        at com.cloudbees.groovy.cps.impl.ClosureBlock.eval(ClosureBlock.java:49)
        at com.cloudbees.groovy.cps.Next.step(Next.java:83)
        at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
        at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
        at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:122)
        at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:261)
        at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
        at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:19)
        at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:35)
        at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:32)
        at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
        at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:32)
        at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:174)
        at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:331)
        at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:82)
        at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:243)
        at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:231)
        at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
        at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Failed to serialize
org.jenkinsci.plugins.workflow.actions.ErrorAction#error for class
org.jenkinsci.plugins.workflow.actions.ErrorAction
        at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:256)
        at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224)
        at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
        at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209)
        at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150)
        at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
        at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
        at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)
        at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:88)
        at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:64)
        at com.thoughtworks.xstream.converters.collections.ArrayConverter.marshal(ArrayConverter.java:45)
        at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
        at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
        at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)
        at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:265)
        at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:252)
        ... 50 more
Caused by: java.lang.RuntimeException: Failed to serialize
java.lang.Throwable#cause for class java.lang.RuntimeException
        at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:256)
        at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224)
        at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
        at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209)
        at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150)
        at com.thoughtworks.xstream.converters.extended.ThrowableConverter.marshal(ThrowableConverter.java:62)
        at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
        at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
        at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)
        at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:265)
        at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:252)
        ... 65 more
Caused by: java.lang.RuntimeException: Failed to serialize
java.lang.Throwable#stackTrace for class
com.thoughtworks.xstream.io.StreamException
        at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:256)
        at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224)
        at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
        at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209)
        at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150)
        at com.thoughtworks.xstream.converters.extended.ThrowableConverter.marshal(ThrowableConverter.java:62)
        at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
        at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
        at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)
        at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:265)
        at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:252)
        ... 75 more
Caused by: com.thoughtworks.xstream.io.StreamException:  : null
        at com.thoughtworks.xstream.core.util.QuickWriter.flush(QuickWriter.java:77)
        at com.thoughtworks.xstream.core.util.QuickWriter.write(QuickWriter.java:49)
        at com.thoughtworks.xstream.io.xml.PrettyPrintWriter.writeText(PrettyPrintWriter.java:279)
        at com.thoughtworks.xstream.io.xml.PrettyPrintWriter.writeText(PrettyPrintWriter.java:231)
        at com.thoughtworks.xstream.io.xml.PrettyPrintWriter.setValue(PrettyPrintWriter.java:214)
        at com.thoughtworks.xstream.io.WriterWrapper.setValue(WriterWrapper.java:45)
        at com.thoughtworks.xstream.converters.SingleValueConverterWrapper.marshal(SingleValueConverterWrapper.java:45)
        at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
        at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
        at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)
        at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:88)
        at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:64)
        at com.thoughtworks.xstream.converters.collections.ArrayConverter.marshal(ArrayConverter.java:45)
        at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
        at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
        at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)
        at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:265)
        at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:252)
        ... 85 more
Caused by: java.nio.channels.ClosedByInterruptException
        at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
        at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:216)
        at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
        at java.nio.channels.Channels.writeFully(Channels.java:101)
        at java.nio.channels.Channels.access$000(Channels.java:61)
        at java.nio.channels.Channels$1.write(Channels.java:174)
        at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
        at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
        at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:295)
        at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
        at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
        at java.io.BufferedWriter.flush(BufferedWriter.java:254)
        at hudson.util.AtomicFileWriter.flush(AtomicFileWriter.java:97)
        at com.thoughtworks.xstream.core.util.QuickWriter.flush(QuickWriter.java:75)
        ... 102 more
 
                                                            

And then some more exceptions

 
                                                                Mar 07, 2018 5:03:41 PM
org.jenkinsci.plugins.workflow.cps.CpsStepContext completed
WARNING: already completed
CpsStepContext[74:node]:Owner[Language-Tests/JavaScript/2290:Language-Tests/JavaScript
#2290]
java.lang.IllegalStateException: delivered here
        at org.jenkinsci.plugins.workflow.cps.CpsStepContext.completed(CpsStepContext.java:353)
        at org.jenkinsci.plugins.workflow.cps.CpsStepContext.onFailure(CpsStepContext.java:329)
        at org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask$PlaceholderExecutable.run(ExecutorStepExecution.java:705)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:429)
 
                                                            

and finally.

 
                                                                Mar 07, 2018 5:03:41 PM
org.jenkinsci.plugins.workflow.cps.CpsStepContext completed
INFO: earlier failure
java.lang.RuntimeException: Failed to serialize
org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage$Tag#actions
for class org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage$Tag
        at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:256)
        at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224)
        at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
        at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209)
        at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150)
        at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
        at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
        at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)
        at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:82)
        at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37)
        at com.thoughtworks.xstream.XStream.marshal(XStream.java:1026)
        at com.thoughtworks.xstream.XStream.marshal(XStream.java:1015)
        at com.thoughtworks.xstream.XStream.toXML(XStream.java:988)
        at hudson.XmlFile.write(XmlFile.java:181)
        at org.jenkinsci.plugins.workflow.support.PipelineIOUtils.writeByXStream(PipelineIOUtils.java:30)
        at org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage.writeNode(SimpleXStreamFlowNodeStorage.java:182)
        at org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage.saveActions(SimpleXStreamFlowNodeStorage.java:193)
        at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$TimingFlowNodeStorage.saveActions(CpsFlowExecution.java:1702)
        at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.saveActions(CpsFlowExecution.java:1129)
        at org.jenkinsci.plugins.workflow.graph.FlowNode.save(FlowNode.java:446)
        at org.jenkinsci.plugins.workflow.graph.FlowNode.persistSafe(FlowNode.java:452)
        at org.jenkinsci.plugins.workflow.graph.FlowNode.access$100(FlowNode.java:61)
        at org.jenkinsci.plugins.workflow.graph.FlowNode$1.add(FlowNode.java:411)
        at org.jenkinsci.plugins.workflow.graph.FlowNode$1.add(FlowNode.java:401)
        at java.util.AbstractList.add(AbstractList.java:108)
        at hudson.model.Actionable.addAction(Actionable.java:155)
        at org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution.start(ExecutorStepExecution.java:104)
        at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:229)
        at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:153)
        at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:108)
        at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:48)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
        at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:19)
        at semmleNode.call(/mnt/jenkins-home/jobs/Language-Tests/jobs/JavaScript/builds/2290/libs/semmle-utils/vars/semmleNode.groovy:4)
        at semmleJob.call(/mnt/jenkins-home/jobs/Language-Tests/jobs/JavaScript/builds/2290/libs/semmle-utils/vars/semmleJob.groovy:88)
        at ___cps.transform___(Native Method)
        at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57)
        at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
        at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
        at sun.reflect.GeneratedMethodAccessor218.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
        at com.cloudbees.groovy.cps.impl.ClosureBlock.eval(ClosureBlock.java:46)
        at com.cloudbees.groovy.cps.Next.step(Next.java:83)
        at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
        at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
        at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:122)
        at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:261)
        at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
        at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:19)
        at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:35)
        at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:32)
        at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
        at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:32)
        at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:174)
        at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:331)
        at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:82)
        at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:243)
        at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:231)
        at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
        at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
        at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused by: com.thoughtworks.xstream.io.StreamException:  : null
        at com.thoughtworks.xstream.core.util.QuickWriter.flush(QuickWriter.java:77)
        at com.thoughtworks.xstream.core.util.QuickWriter.write(QuickWriter.java:37)
        at com.thoughtworks.xstream.io.xml.PrettyPrintWriter.endNode(PrettyPrintWriter.java:317)
        at com.thoughtworks.xstream.io.WriterWrapper.endNode(WriterWrapper.java:37)
        at com.thoughtworks.xstream.io.path.PathTrackingWriter.endNode(PathTrackingWriter.java:48)
        at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:65)
        at com.thoughtworks.xstream.converters.collections.ArrayConverter.marshal(ArrayConverter.java:45)
        at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
        at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
        at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)
        at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:265)
        at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:252)
        at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224)
        at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
        at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209)
        at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150)
        at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
        at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
        at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)
        at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:82)
        at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37)
        at com.thoughtworks.xstream.XStream.marshal(XStream.java:1026)
        at com.thoughtworks.xstream.XStream.marshal(XStream.java:1015)
        at com.thoughtworks.xstream.XStream.toXML(XStream.java:988)
        at hudson.XmlFile.write(XmlFile.java:181)
        at org.jenkinsci.plugins.workflow.support.PipelineIOUtils.writeByXStream(PipelineIOUtils.java:30)
        at org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage.writeNode(SimpleXStreamFlowNodeStorage.java:182)
        at org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage.saveActions(SimpleXStreamFlowNodeStorage.java:193)
        at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$TimingFlowNodeStorage.saveActions(CpsFlowExecution.java:1702)
        at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.saveActions(CpsFlowExecution.java:1129)
        at org.jenkinsci.plugins.workflow.graph.FlowNode.save(FlowNode.java:446)
        at org.jenkinsci.plugins.workflow.graph.FlowNode.persistSafe(FlowNode.java:452)
        at org.jenkinsci.plugins.workflow.graph.FlowNode.access$100(FlowNode.java:61)
        at org.jenkinsci.plugins.workflow.graph.FlowNode$1.add(FlowNode.java:411)
        at org.jenkinsci.plugins.workflow.graph.FlowNode$1.add(FlowNode.java:401)
        at java.util.AbstractList.add(AbstractList.java:108)
        at hudson.model.Actionable.addAction(Actionable.java:155)
        at org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution.start(ExecutorStepExecution.java:104)
        at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:229)
        at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:153)
        at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:108)
        at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:48)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
        at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:19)
        ... 33 more
Caused by: java.nio.channels.ClosedByInterruptException
        at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
        at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:216)
        at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
        at java.nio.channels.Channels.writeFully(Channels.java:101)
        at java.nio.channels.Channels.access$000(Channels.java:61)
        at java.nio.channels.Channels$1.write(Channels.java:174)
        at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
        at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
        at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:295)
        at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
        at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
        at java.io.BufferedWriter.flush(BufferedWriter.java:254)
        at hudson.util.AtomicFileWriter.flush(AtomicFileWriter.java:97)
        at com.thoughtworks.xstream.core.util.QuickWriter.flush(QuickWriter.java:75)
        ... 77 more 
                                                            

Once the job has hung, we need to kill the job.

Any assistance would be welcomed in trying to find the root cause and solving this problem.

Add Comment

This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)

andrew.bayer@gmail.com (JIRA)

unread,

Mar 8, 2018, 6:52:02 PM3/8/18

to jenkinsc...@googlegroups.com

Andrew Bayer assigned an issue to Sam Van Oort

Sam Van Oort - looks like this might be touching stuff in the storage changes?

Jenkins /

JENKINS-50020

CpsFlowExecution unresponsive and Failed to serialize SimpleXStreamFlowNodeStorage

Change By:	Andrew Bayer
Assignee:	Sam Van Oort

Add Comment

svanoort@cloudbees.com (JIRA)

unread,

Mar 8, 2018, 8:52:02 PM3/8/18

to jenkinsc...@googlegroups.com

Sam Van Oort commented on

JENKINS-50020

Re: CpsFlowExecution unresponsive and Failed to serialize SimpleXStreamFlowNodeStorage

Andrew Bayer Unlikely, probably a serialization error.

Add Comment

andrew.bayer@gmail.com (JIRA)

unread,

Mar 8, 2018, 9:22:02 PM3/8/18

to jenkinsc...@googlegroups.com

Andrew Bayer assigned an issue to Andrew Bayer

Jenkins /

JENKINS-50020

CpsFlowExecution unresponsive and Failed to serialize SimpleXStreamFlowNodeStorage

Change By:	Andrew Bayer
Assignee:	Sam Van Oort Andrew Bayer

Add Comment

andrew.bayer@gmail.com (JIRA)

unread,

Mar 8, 2018, 9:22:02 PM3/8/18

to jenkinsc...@googlegroups.com

Andrew Bayer commented on

JENKINS-50020

Re: CpsFlowExecution unresponsive and Failed to serialize SimpleXStreamFlowNodeStorage

Alrightie, I’ll dig tomorrow.

Add Comment

henti@geekware.co.za (JIRA)

unread,

Mar 9, 2018, 4:49:04 AM3/9/18

to jenkinsc...@googlegroups.com

Henti Smith updated an issue

Jenkins /

JENKINS-50020

CpsFlowExecution unresponsive and Failed to serialize SimpleXStreamFlowNodeStorage

Change By:	Henti Smith

This happens to multiple jobs in our queues, but the logs are related to one job. the job stop processing and the master logs contains the following

{code:java}

Mar 07, 2018 4:51:52 PM org.jenkinsci.plugins.workflow.support.concurrent.Timeout lambda$ping$0
INFO: Running CpsFlowExecution[Owner[Language-Tests/JavaScript/2290:Language-Tests/JavaScript #2290]] unresponsive for 5 sec
-- SNIP --
Mar 07, 2018 4:55:18 PM org.jenkinsci.plugins.workflow.support.concurrent.Timeout lambda$ping$0
INFO: Running CpsFlowExecution[Owner[Language-Tests/JavaScript/2290:Language-Tests/JavaScript #2290]] unresponsive for 3 min 30 sec

{code}

Shortly thereafter we see an exception.

{code:java}

{code}

And then some more exceptions

{code:java}

Mar 07, 2018 5:03:41 PM
org.jenkinsci.plugins.workflow.cps.CpsStepContext completed
WARNING: already completed
CpsStepContext[74:node]:Owner[Language-Tests/JavaScript/2290:Language-Tests/JavaScript
#2290]
java.lang.IllegalStateException: delivered here
at org.jenkinsci.plugins.workflow.cps.CpsStepContext.completed(CpsStepContext.java:353)
at org.jenkinsci.plugins.workflow.cps.CpsStepContext.onFailure(CpsStepContext.java:329)
at org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask$PlaceholderExecutable.run(ExecutorStepExecution.java:705)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)

{code}
and finally.
{code:java}

at semmleNode companyNode .call(/mnt/jenkins-home/jobs/Language-Tests/jobs/JavaScript/builds/2290/libs/ semmle company -utils/vars/ semmleNode companyNode .groovy:4)
at semmleJob companyJob .call(/mnt/jenkins-home/jobs/Language-Tests/jobs/JavaScript/builds/2290/libs/ semmle company -utils/vars/ semmleJob companyJob .groovy:88)

... 77 more{code}

Once the job has hung, we need to kill the job.

Any assistance would be welcomed in trying to find the root cause and solving this problem.

Add Comment

dom@semmle.com (JIRA)

unread,

Mar 9, 2018, 8:21:03 AM3/9/18

to jenkinsc...@googlegroups.com

Dominic Hargreaves commented on

JENKINS-50020

Re: CpsFlowExecution unresponsive and Failed to serialize SimpleXStreamFlowNodeStorage

Thanks for taking a look! It's possibly worth noting that after this happens a lot we also see a completely unresponsive web interface which forces us to kill jenkins and start up again, accompanied by these log messages:

 
                                                                Mar 01, 2018 1:02:56 PM jenkins.metrics.api.Metrics$HealthChecker execute
WARNING: Some health checks are reporting as unhealthy: [thread-deadlock : [Handling POST /ajaxExecutors from 10.42.2.5 : RequestHandlerThread[#8880] View/ajaxExecutors.jelly locked on java.util.concurrent.locks.ReentrantReadWriteLock$No
nfairSync@3acfc25d (owned by Executor #0 for jenkins-linux-build-slaveb4d1b0 : executing PlaceholderExecutable:ExecutorStepExecution.PlaceholderTask{runId=JOBNAME#1247,label=(linux||linux-fallback||linux-scalable),context=Cp
sStepContext[10:node]:Owner[JOBNAME/1247:JOBNAME #1247],cookie=null,auth=null}):
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
at hudson.model.Executor.getAsynchronousExecution(Executor.java:646)
at hudson.model.Executor.isDisplayCell(Executor.java:666)
at hudson.model.Computer.getDisplayExecutors(Computer.java:991)
at sun.reflect.GeneratedMethodAccessor1217.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)
at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)
at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185)
at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75)
at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)
at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74)
at org.apache.commons.jelly.tags.core.CoreTagLibrary$3.run(CoreTagLibrary.java:134)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:150)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
at org.apache.commons.jelly.tags.core.OtherwiseTag.doTag(OtherwiseTag.java:41)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
at org.apache.commons.jelly.tags.core.ChooseTag.doTag(ChooseTag.java:38)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
at org.apache.commons.jelly.tags.core.OtherwiseTag.doTag(OtherwiseTag.java:41)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
at org.apache.commons.jelly.tags.core.ChooseTag.doTag(ChooseTag.java:38)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)
at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63)
at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53)
at org.kohsuke.stapler.jelly.JellyFacet$1.dispatch(JellyFacet.java:97)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:782)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:99)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
, Executor #0 for jenkins-linux-build-slaveb4d1b0 : executing PlaceholderExecutable:ExecutorStepExecution.PlaceholderTask{runId=JOBNAME#1247,label=(linux||linux-fallback||linux-scalable),context=CpsStepContext[10:node]:Owner[JOBNAME/1247:JOBNAME #1247],cookie=null,auth=null} locked on java.util.concurrent.locks.ReentrantLock$NonfairSync@34082b74 (owned by jenkins.util.Timer [#7]):
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
at hudson.model.Queue._withLock(Queue.java:1430)
at hudson.model.Queue.withLock(Queue.java:1293)
at jenkins.model.Nodes.updateNode(Nodes.java:178)
at jenkins.model.Jenkins.updateNode(Jenkins.java:2115)
at hudson.model.Node.save(Node.java:136)
at hudson.model.Node.setTemporaryOfflineCause(Node.java:271)
at hudson.model.Computer.setTemporarilyOffline(Computer.java:714)
at com.microsoft.azure.vmagent.AzureVMAgent.setCleanUpAction(AzureVMAgent.java:326)
at com.microsoft.azure.vmagent.AzureVMCloudOnceRetentionStrategy.done(AzureVMCloudOnceRetentionStrategy.java:94)
at com.microsoft.azure.vmagent.AzureVMCloudOnceRetentionStrategy.done(AzureVMCloudOnceRetentionStrategy.java:69)
at com.microsoft.azure.vmagent.AzureVMCloudOnceRetentionStrategy.taskCompleted(AzureVMCloudOnceRetentionStrategy.java:56)
at hudson.slaves.SlaveComputer.taskCompleted(SlaveComputer.java:343)
at hudson.model.queue.WorkUnitContext.synchronizeEnd(WorkUnitContext.java:140)
at hudson.model.Executor.finish1(Executor.java:477)
at hudson.model.Executor.completedAsynchronous(Executor.java:499)
at jenkins.model.queue.AsynchronousExecution.setExecutor(AsynchronousExecution.java:115)
at hudson.model.Executor.run(Executor.java:434)
, jenkins.util.Timer [#7] locked on java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@3acfc25d (owned by Executor #0 for jenkins-linux-build-slaveb4d1b0 : executing PlaceholderExecutable:ExecutorStepExecution.PlaceholderTask{runId=JOBNAME#1247,label=(linux||linux-fallback||linux-scalable),context=CpsStepContext[10:node]:Owner[JOBNAME/1247:JOBNAME #1247],cookie=null,auth=null}):
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
at hudson.model.Executor.isParking(Executor.java:674)
at hudson.model.Queue.maintain(Queue.java:1473)
at hudson.model.Queue$MaintainTask.doRun(Queue.java:2851)
at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51)
at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
]] 
                                                            

Add Comment

svanoort@cloudbees.com (JIRA)

unread,

Mar 9, 2018, 4:35:02 PM3/9/18

to jenkinsc...@googlegroups.com

Sam Van Oort commented on

JENKINS-50020

Re: CpsFlowExecution unresponsive and Failed to serialize SimpleXStreamFlowNodeStorage

So, there's three defects here:

1. The latter defect is probably root-caused with a Pipeline internal issue – something with dangling tasks that aren't properly killed, and it will probably be resolved once I can reliably reproduce an issue we've been seeing lately and issue a fix.
2. The Queue is having some sort of locking issue with the PlaceholderTask for Pipeline – no clue here, but it shouldn't be able to hang. The actual trigger for this is trying to display the Executors in use which interacts with both the Executor and the Queue.
3. The actual failure to serialize a trivial FlowNode, along with an associated InterruptedException (a timeout?).

Henti Smith Do you have any idea how to regularly reproduce this with a simpler job?

Add Comment

henti@geekware.co.za (JIRA)

unread,

Mar 15, 2018, 7:03:03 AM3/15/18

to jenkinsc...@googlegroups.com

Henti Smith updated an issue

Jenkins /

JENKINS-50020

CpsFlowExecution unresponsive and Failed to serialize SimpleXStreamFlowNodeStorage

Change By:	Henti Smith

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)

at companyNode.call(/mnt/jenkins-home/jobs/Language-Tests/jobs/JavaScript/builds/2290/libs/company-utils/vars/companyNode.groovy:4)
at companyJob.call(/mnt/jenkins-home/jobs/Language-Tests/jobs/JavaScript/builds/2290/libs/company-utils/vars/companyJob.groovy:88)

at ___cps.transform___(Native Method)
at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)

at sun.reflect.GeneratedMethodAccessor218.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)

az

Add Comment

henti@geekware.co.za (JIRA)

unread,

Mar 15, 2018, 7:13:02 AM3/15/18

to jenkinsc...@googlegroups.com

Henti Smith commented on

JENKINS-50020

Re: CpsFlowExecution unresponsive and Failed to serialize SimpleXStreamFlowNodeStorage

Sam Van Oort

Our build is pretty simple.

 
                                                                @Library('semmle-utils') _

semmleJob {
semmleNode(getLabelExpr('linux')) {
stage('Checkout') {
semmleCheckout(gitRepo, gitBranch, gitSha)
}
stage('Test') {
sh "./clean"
sh "./build target/test/javascript-extractor-tests/results.xml"
sh "./build target/test/javascript/results.xml"
sh "./build target/test/common-javascript/results.xml"
junit 'target/test/*/*.xml'
}
}
}
 
                                                            

semmleJob checks to see if the job is mergable or out of date

semmleNode checks to see if the last build on the node failed and if so clean up first otherwise periodic cleanups are triggered.

semmleCheckout dis sane git checkouts with submodules etc based on a sha

I suspect it might be related to the Azure VM agent plugin, as can be seen in Dominic's post as we had a similar deadlock this morning.

Add Comment

pavel+jenkins@avgustinov.com (JIRA)

unread,

Mar 17, 2018, 5:52:03 AM3/17/18

to jenkinsc...@googlegroups.com

Pavel Avgustinov commented on

JENKINS-50020

Re: CpsFlowExecution unresponsive and Failed to serialize SimpleXStreamFlowNodeStorage

2. The Queue is having some sort of locking issue with the PlaceholderTask for Pipeline – no clue here, but it shouldn't be able to hang. The actual trigger for this is trying to display the Executors in use which interacts with both the Executor and the Queue.

I believe PR#3354 fixes/works around the deadlock issue (which is very similar to JENKINS-46248). The serialization issues discussed here are separate.

Add Comment

svanoort@cloudbees.com (JIRA)

unread,

Apr 11, 2018, 7:45:02 PM4/11/18

to jenkinsc...@googlegroups.com

Sam Van Oort commented on

JENKINS-50020

Re: CpsFlowExecution unresponsive and Failed to serialize SimpleXStreamFlowNodeStorage

I suspect the pipeline-internal issue is at least partially addressed in the latest workflow-cps and workflow-job releases, and issues with ErrorAction are handled in the latest workflow-api releases.

Add Comment

svanoort@cloudbees.com (JIRA)

unread,

Apr 30, 2018, 8:45:02 PM4/30/18

to jenkinsc...@googlegroups.com

Sam Van Oort closed an issue as Duplicate

The fixes to ~~JENKINS-50752~~ should handle the serialization issues.

Jenkins /

JENKINS-50020

CpsFlowExecution unresponsive and Failed to serialize SimpleXStreamFlowNodeStorage

Change By:	Sam Van Oort
Status:	Open Closed
Resolution:	Duplicate

Add Comment

philip_douglas@sn.scee.net (JIRA)

unread,

Jul 2, 2018, 7:47:01 AM7/2/18

to jenkinsc...@googlegroups.com

Philip Douglas commented on

JENKINS-50020

Re: CpsFlowExecution unresponsive and Failed to serialize SimpleXStreamFlowNodeStorage

We're still seeing this problem with the following versions:

Jenkins core 2.107.3

workflow-cps 2.53

workflow-job 2.21

Once the CpsFlowExecution unresponsive errors start, they continue until I restart the server. The Web UI becomes unresponsive within minutes of the errors starting (possibly immediately, I don't have sufficiently accurate monitoring to be sure).

Does the core need to be updated as well (there isn't an LTS release with the fix in yet) or should the plugin update be enough?

Add Comment

This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396)

svanoort@cloudbees.com (JIRA)

unread,

Jul 2, 2018, 8:02:02 AM7/2/18

to jenkinsc...@googlegroups.com

Sam Van Oort commented on

JENKINS-50020

Re: CpsFlowExecution unresponsive and Failed to serialize SimpleXStreamFlowNodeStorage

Philip Douglas I'm sorry to hear that. This fix does not require a core update, but since the cause here is resolved I'm assuming your issues represent a different problem generating similar results. So that we can investigate, please open a new ticket up, and include all of the Pipeline plugin versions (one of the fixes here was in workflow-api plugin) and the actual stack trace of the error.

Add Comment

philip_douglas@sn.scee.net (JIRA)

unread,

Jul 4, 2018, 5:35:01 AM7/4/18

to jenkinsc...@googlegroups.com

Philip Douglas commented on

JENKINS-50020

Re: CpsFlowExecution unresponsive and Failed to serialize SimpleXStreamFlowNodeStorage

Ok, thanks. I've created a new ticket here: https://issues.jenkins-ci.org/browse/JENKINS-52362

Add Comment

Reply all

Reply to author

Forward