[JIRA] (JENKINS-55287) Pipeline: Failure to load flow node: FlowNode was not found in storage for head

1,445 views
Skip to first unread message

dnusbaum@cloudbees.com (JIRA)

unread,
Mar 25, 2019, 11:58:03 AM3/25/19
to jenkinsc...@googlegroups.com
Devin Nusbaum updated an issue
 
Jenkins / Bug JENKINS-55287
Pipeline: Failure to load flow node: FlowNode was not found in storage for head
Change By: Devin Nusbaum
Summary: Pipeline Groovy plugin issue cause : Failure to jenkins process exit load flow node: FlowNode was not found in storage for head
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

nils.schmidt.ger@gmail.com (JIRA)

unread,
Apr 15, 2019, 3:55:02 AM4/15/19
to jenkinsc...@googlegroups.com
Nils Schmidt commented on Bug JENKINS-55287
 
Re: Pipeline: Failure to load flow node: FlowNode was not found in storage for head

It seems we are having a similar issue. Maybe the stacktrace helps. In our case the problem occurs roughly 1 out of 20 builds. And it always happens after we were waiting for a particular resource to be freed. 

22:47:50 Trying to acquire lock on [someJar.zip]
22:47:50 Found 0 available resource(s). Waiting for correct amount: 1.
22:47:50 [someJar.zip] is locked, waiting...
Creating placeholder flownodes because failed loading originals.
java.io.IOException: Tried to load head FlowNodes for execution Owner[SomeJob/bugfix #254] but FlowNode was not found in storage for head id:FlowNodeId 1:133
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:678)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:715)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:664)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:526)
	at hudson.model.RunMap.retrieve(RunMap.java:225)
	at hudson.model.RunMap.retrieve(RunMap.java:57)
	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501)
	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483)
	at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381)
	at hudson.model.RunMap.getById(RunMap.java:205)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:901)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:912)
	at org.jenkinsci.plugins.workflow.cps.CpsStepContext.getFlowExecution(CpsStepContext.java:474)
	at org.jenkinsci.plugins.workflow.cps.CpsStepContext.getThreadGroupSynchronously(CpsStepContext.java:243)
	at org.jenkinsci.plugins.workflow.cps.CpsStepContext.getThreadSynchronously(CpsStepContext.java:237)
	at org.jenkinsci.plugins.workflow.cps.CpsStepContext.doGet(CpsStepContext.java:298)
	at org.jenkinsci.plugins.workflow.support.DefaultStepContext.get(DefaultStepContext.java:67)
	at org.jenkins.plugins.lockableresources.LockableResourcesManager.unlockNames(LockableResourcesManager.java:396)
	at org.jenkins.plugins.lockableresources.LockableResourcesManager.unlockNames(LockableResourcesManager.java:404)
	at org.jenkins.plugins.lockableresources.LockableResourcesManager.unlock(LockableResourcesManager.java:347)
	at org.jenkins.plugins.lockableresources.LockableResourcesManager.unlock(LockableResourcesManager.java:335)
	at org.jenkins.plugins.lockableresources.queue.LockRunListener.onCompleted(LockRunListener.java:95)
	at hudson.model.listeners.RunListener.fireCompleted(RunListener.java:209)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.finish(WorkflowRun.java:586)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$900(WorkflowRun.java:133)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun$FailOnLoadListener.lambda$onNewHead$0(WorkflowRun.java:983)
	at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
Finished: FAILURE

andrew.bayer@gmail.com (JIRA)

unread,
May 7, 2019, 2:33:02 PM5/7/19
to jenkinsc...@googlegroups.com
Andrew Bayer assigned an issue to Unassigned
 
Change By: Andrew Bayer
Assignee: vjuranek

rodrigc@FreeBSD.org (JIRA)

unread,
May 7, 2019, 2:39:03 PM5/7/19
to jenkinsc...@googlegroups.com
Craig Rodrigues commented on Bug JENKINS-55287
 
Re: Pipeline: Failure to load flow node: FlowNode was not found in storage for head

I encountered this:

Creating placeholder flownodes because failed loading originals. java.io.IOException: Tried to load head FlowNodes for execution Owner[DEV/Porx-07/82:DEV/Porx-07 #82] but FlowNode was not found in storage for head id:FlowNodeId 1:121 at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:679) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:716) at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:664) at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:526) at hudson.model.RunMap.retrieve(RunMap.java:225) at hudson.model.RunMap.retrieve(RunMap.java:57) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483) at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381) at hudson.model.RunMap.getById(RunMap.java:205) at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:901) at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:912) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178) at jenkins.model.Jenkins.<init>(Jenkins.java:989) at hudson.model.Hudson.<init>(Hudson.java:85) at hudson.model.Hudson.<init>(Hudson.java:81) at hudson.WebAppMain$3.run(WebAppMain.java:233) Finished: FAILURE

rodrigc@FreeBSD.org (JIRA)

unread,
May 7, 2019, 2:40:02 PM5/7/19
to jenkinsc...@googlegroups.com
Craig Rodrigues edited a comment on Bug JENKINS-55287
I encountered this:
{noformat}
Creating placeholder flownodes because failed loading originals. java.io.IOException: Tried to load head FlowNodes for execution Owner[DEV/Porx-07/82:DEV/Porx-07 #82] but FlowNode was not found in storage for head id:FlowNodeId 1:121 at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:679) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:716) at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:664) at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:526) at hudson.model.RunMap.retrieve(RunMap.java:225) at hudson.model.RunMap.retrieve(RunMap.java:57) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483) at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381) at hudson.model.RunMap.getById(RunMap.java:205) at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:901) at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:912) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178) at jenkins.model.Jenkins.<init>(Jenkins.java:989) at hudson.model.Hudson.<init>(Hudson.java:85) at hudson.model.Hudson.<init>(Hudson.java:81) at hudson.WebAppMain$3.run(WebAppMain.java:233) Finished: FAILURE {noformat}
 

rodrigc@FreeBSD.org (JIRA)

unread,
May 7, 2019, 2:42:03 PM5/7/19
to jenkinsc...@googlegroups.com

jsnmcdev@gmail.com (JIRA)

unread,
May 8, 2019, 5:06:03 PM5/8/19
to jenkinsc...@googlegroups.com

We also hit the exact same issue on a PR build today as the one posted by Craig Rodrigues

rodrigc@FreeBSD.org (JIRA)

unread,
May 8, 2019, 7:08:01 PM5/8/19
to jenkinsc...@googlegroups.com

@Jason MCDev I talked to @abayer on the #jenkins IRC channel. He said that he thinks that this problem occurs if we rebuild a pipeline from a specific stage.

During the build of a pipeline, an XML file is generated into the workspace which has flownode information.

@abayer suggested a workaround of deleting the build from the build history to remove the flownode XML. Then the pipeline should be run from the beginning (not a rebuild).

@abayer's suggestion doesn't match what I saw, since I did not rebuild from a stage, yet I saw the problem.

jsnmcdev@gmail.com (JIRA)

unread,
May 13, 2019, 5:10:02 PM5/13/19
to jenkinsc...@googlegroups.com

Craig Rodrigues We are always running full clean builds, so I concur that the suggestion does not fit our use case.  It has really been causing us headaches for sure - random failures in release pipelines.

jsnmcdev@gmail.com (JIRA)

unread,
Jun 6, 2019, 8:21:02 PM6/6/19
to jenkinsc...@googlegroups.com

Any movement here or additional information that I can provide? We are seeing this error multiple times a day.

msanders@punyhuman.com (JIRA)

unread,
Jun 13, 2019, 12:06:05 PM6/13/19
to jenkinsc...@googlegroups.com

Hi there,

I'm also curious for whether there has been any movement with fixing this bug? We are encountering it with nightly scheduled builds, yet later in the day, the same build will succeed.

Jenkins 2.176.1 running on Windows, communicating with other Windows slave agents.

java.io.IOException: Tried to load head FlowNodes for execution Owner[BardsTale4/110:BardsTale4 #110] but FlowNode was not found in storage for head id:FlowNodeId 1:128
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:679)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:716)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:664)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:526)
	at hudson.model.RunMap.retrieve(RunMap.java:225)
	at hudson.model.RunMap.retrieve(RunMap.java:57)
	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501)
	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483)
	at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381)
	at hudson.model.RunMap.getById(RunMap.java:205)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:901)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:912)
	at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65)
	at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57)
	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
	at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178)
	at jenkins.model.Jenkins.<init>(Jenkins.java:995)
	at hudson.model.Hudson.<init>(Hudson.java:85)
	at hudson.model.Hudson.<init>(Hudson.java:81)
	at hudson.WebAppMain$3.run(WebAppMain.java:233)
Finished: FAILURE

dnusbaum@cloudbees.com (JIRA)

unread,
Jul 9, 2019, 4:03:02 PM7/9/19
to jenkinsc...@googlegroups.com

I have also seen this occasionally on ci.jenkins.io, but have no idea about root cause. Does anyone have a minimal Jenkinsfile that can consistently reproduce the issue? If no one has that, then the next best thing would be a build folder for a build that failed with such an error. The relevant information to look for would be whether the flow node mentioned in the exception really was missing or if it is present on disk but could not be found or loaded for some other reason.

marvin.grit@gmail.com (JIRA)

unread,
Aug 29, 2019, 3:46:05 AM8/29/19
to jenkinsc...@googlegroups.com

After multiple stable nightly runs we suddenly got this error in our pipeline as well. Running on Jenkins 2.163.

Creating placeholder flownodes because failed loading originals.
java.io.IOException: Tried to load head FlowNodes for execution Owner[Pipeline/93:Pipeline #93] but FlowNode was not found in storage for head id:FlowNodeId 1:201
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:678)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:715)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:664)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:526)
	at hudson.model.RunMap.retrieve(RunMap.java:225)
	at hudson.model.RunMap.retrieve(RunMap.java:57)
	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501)
	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483)
	at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381)
	at hudson.model.RunMap.getById(RunMap.java:205)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:901)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:912)
	at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65)
	at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57)
	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
	at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178)
	at jenkins.model.Jenkins.<init>(Jenkins.java:1000)
	at hudson.model.Hudson.<init>(Hudson.java:85)
	at hudson.model.Hudson.<init>(Hudson.java:81)
	at hudson.WebAppMain$3.run(WebAppMain.java:233)

In the log of; /log/all  we also see this: 

Jenkins Log
aug 28, 2019 10:29:58 PM INFO jenkins.InitReactorRunner$1 onAttainedPrepared all plugins
aug 28, 2019 10:29:58 PM INFO jenkins.InitReactorRunner$1 onAttainedStarted all plugins
aug 28, 2019 10:30:07 PM INFO jenkins.InitReactorRunner$1 onAttainedAugmented all extensions
aug 28, 2019 10:30:07 PM INFO jenkins.model.Jenkins setBuildsAndWorkspacesDirUsing non default builds directories: C:/Jenkins/${ITEM_FULLNAME}/builds.
aug 28, 2019 10:30:07 PM INFO jenkins.model.Jenkins setBuildsAndWorkspacesDirUsing non default workspaces directories: C:/Jenkins/${ITEM_FULLNAME}/workspace.
aug 28, 2019 10:30:08 PM INFO hudson.plugins.project_inheritance.projects.actions.service.CachingInheritanceListService Service CachingInheritanceListService instantiated
aug 28, 2019 10:30:08 PM INFO hudson.plugins.project_inheritance.projects.actions.InheritanceListActionFactory Extension InheritanceListActionFactory instantiated
aug 28, 2019 10:30:17 PM INFO jenkins.InitReactorRunner$1 onAttainedLoaded all jobs
aug 28, 2019 10:30:17 PM INFO hudson.model.AsyncPeriodicWork$1 runStarted Download metadata
aug 28, 2019 10:30:17 PM INFO hudson.util.Retrier startAttempt #1 to do the action check updates server
aug 28, 2019 10:30:18 PM INFO org.springframework.context.support.AbstractApplicationContext prepareRefreshRefreshing org.springframework.web.context.support.StaticWebApplicationContext@1a8fabb: display name [Root WebApplicationContext]; startup date [Wed Aug 28 22:30:18 CEST 2019]; root of context hierarchy
aug 28, 2019 10:30:18 PM INFO org.springframework.context.support.AbstractApplicationContext obtainFreshBeanFactoryBean factory for application context [org.springframework.web.context.support.StaticWebApplicationContext@1a8fabb]: org.springframework.beans.factory.support.DefaultListableBeanFactory@c17335
aug 28, 2019 10:30:18 PM INFO org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletonsPre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@c17335: defining beans [authenticationManager,activeDirectory]; root of factory hierarchy
aug 28, 2019 10:30:18 PM INFO hudson.plugins.active_directory.ActiveDirectoryAuthenticationProvider Active Directory domain is DC=ksserver,DC=local
aug 28, 2019 10:30:19 PM INFO org.springframework.context.support.AbstractApplicationContext prepareRefreshRefreshing org.springframework.web.context.support.StaticWebApplicationContext@1141fc2: display name [Root WebApplicationContext]; startup date [Wed Aug 28 22:30:19 CEST 2019]; root of context hierarchy
aug 28, 2019 10:30:19 PM INFO org.springframework.context.support.AbstractApplicationContext obtainFreshBeanFactoryBean factory for application context [org.springframework.web.context.support.StaticWebApplicationContext@1141fc2]: org.springframework.beans.factory.support.DefaultListableBeanFactory@13ffd5c
aug 28, 2019 10:30:19 PM INFO org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletonsPre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@13ffd5c: defining beans [filter,legacy]; root of factory hierarchy
aug 28, 2019 10:30:19 PM INFO jenkins.InitReactorRunner$1 onAttainedCompleted initialization
aug 28, 2019 10:30:26 PM INFO hudson.model.UpdateSite updateDataObtained the latest update center data file for UpdateSource default
aug 28, 2019 10:30:31 PM WARNING jenkins.branch.WorkspaceLocatorImpl getWorkspaceRootJENKINS-2111 path sanitization ineffective when using legacy Workspace Root Directory ‘C:/Jenkins/${ITEM_FULLNAME}/workspace’; switch to ‘${JENKINS_HOME}/workspace/${ITEM_FULL_NAME}’ as in JENKINS-8446 / JENKINS-21942

aug 28, 2019 10:30:31 PM WARNING org.jenkinsci.plugins.workflow.cps.CpsFlowExecution onLoadError initializing storage and loading nodes, will try to create placeholders for: CpsFlowExecution[Owner[Pipeline/93:Pipeline #93]]
java.io.IOException: Tried to load head FlowNodes for execution Owner[Pipeline/93:Pipeline #93] but FlowNode was not found in storage for head id:FlowNodeId 1:201
	at <same stacktrace as above>

aug 28, 2019 10:30:31 PM INFO org.jenkinsci.plugins.workflow.cps.CpsFlowExecution createPlaceholderNodesCreating placeholder flownodes for execution: CpsFlowExecution[Owner[Pipeline/93:Pipeline #93]]
aug 28, 2019 10:30:31 PM INFO hudson.model.DownloadService$Downloadable loadObtained the updated data file for hudson.tasks.Maven.MavenInstaller
aug 28, 2019 10:30:31 PM WARNING org.jenkinsci.plugins.workflow.job.WorkflowRun getExecutionFound incomplete build with completed execution - display name: Pipeline #93

 

oxygenxo@gmail.com (JIRA)

unread,
Oct 18, 2019, 10:51:04 AM10/18/19
to jenkinsc...@googlegroups.com

We've faced this issue today. Our 300+ parallel pipeline builds have failed simultaneously due to the error in ticket's description. But I've found the following in Jenkins server's logs in the same time:

[207043.805s][warning][os,thread] Attempt to protect stack guard pages failed (0x00007f6d188c9000-0x00007f6d188cd000).
[207043.805s][warning][os,thread] Attempt to deallocate stack guard pages failed.
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f6d187c8000, 16384, 0) failed; error='Not enough space' (errno=12)
[207043.810s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
[207043.811s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 16384 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /tmp/hs_err_pid6.log

So I guess something is wrong with our settings (out of memory on server with 180GB of RAM? Our Jenkins instance's load isn't that much, we've run the same load on VM with 60GB of RAM, but with 20 GB of swap)
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

yves.schumann@ti8m.ch (JIRA)

unread,
Nov 19, 2019, 3:15:08 AM11/19/19
to jenkinsc...@googlegroups.com

We've spot the same today, all our nightly pipelines break with this error:

04:02:41  04:02:40,990  INFO JettyHandler:43 - Got request: /status.xml. Sender is 10.1.53.1:40440
Creating placeholder flownodes because failed loading originals.
java.io.IOException: Tried to load head FlowNodes for execution Owner[RT-Jobs/RT-Nightly/9:RT-Jobs/RT-Nightly #9] but FlowNode was not found in storage for head id:FlowNodeId 1:526
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:679)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:716)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:662)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:526)
	at hudson.model.RunMap.retrieve(RunMap.java:225)
	at hudson.model.RunMap.retrieve(RunMap.java:57)
	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501)
	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483)
	at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381)
	at hudson.model.RunMap.getById(RunMap.java:205)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:899)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:910)
	at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65)
	at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57)
	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
	at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178)
	at jenkins.model.Jenkins.<init>(Jenkins.java:995)
	at hudson.model.Hudson.<init>(Hudson.java:85)
	at hudson.model.Hudson.<init>(Hudson.java:81)
	at hudson.WebAppMain$3.run(WebAppMain.java:233)
Finished: UNSTABLE

Quite annoying as it follows Murphy's Law: Occurs in the exact worst moment... :-/

oxygenxo@gmail.com (JIRA)

unread,
Nov 23, 2019, 3:26:03 PM11/23/19
to jenkinsc...@googlegroups.com

I think this is happening due to our pipeline durability settings override - we've set it to MAX_PERFORMANCE. Therefore all pipelines die when Jenkins master dies

stoiky@gmail.com (JIRA)

unread,
Jan 6, 2020, 1:40:03 PM1/6/20
to jenkinsc...@googlegroups.com
Mihai Stoichitescu updated an issue
 
Change By: Mihai Stoichitescu
Attachment: plugins_versions_2.190.1.txt

stoiky@gmail.com (JIRA)

unread,
Jan 6, 2020, 1:52:05 PM1/6/20
to jenkinsc...@googlegroups.com
Mihai Stoichitescu commented on Bug JENKINS-55287
 
Re: Pipeline: Failure to load flow node: FlowNode was not found in storage for head

We are also getting this from time to time after updating Jenkins from 2.121.3 to 2.190.1

I've attached a list of our plugins used.

dnusbaum@cloudbees.com (JIRA)

unread,
Jan 17, 2020, 2:39:04 PM1/17/20
to jenkinsc...@googlegroups.com

For everyone experiencing this issue: we'd like to be able to fix this, but we really need more information to do so. Please, whenever you encounter the error in the description of the ticket, zip the build folder ($JENKINS_HOME/jobs/$PATH_TO_JOB/builds/$BUILD_NUMBER/) of the build that failed and upload it here, redacting any sensitive content as necessary, and include any relevant information on frequency of the issue, steps to reproduce (did it happen after Jenkins was restarted normally, or did Jenkins crash), any exceptions or warnings in the Jenkins system logs that seem relevant, etc.

My best hypothesis is that there is a concurrency issue that causes the flow graph to not be saved in in some cases where it should be saved in all durability modes, but this is only noticed in performance-optimized mode because the Pipeline is not saved regularly like in other modes. One thing that we could do to investigate this hypothesis would be to instrument all uses of BulkChange in workflow-job and workflow-cps with optional logging to understand if bulk changes being aborted when a save is requested could be causing the problem.

dnusbaum@cloudbees.com (JIRA)

unread,
Jan 17, 2020, 2:40:13 PM1/17/20
to jenkinsc...@googlegroups.com
Devin Nusbaum updated an issue
 
Change By: Devin Nusbaum
{noformat}
Creating placeholder flownodes because failed loading originals.
java.io.IOException: Tried to load head FlowNodes for execution Owner[Platform Service FBI Test/1605:Platform Service FBI Test #1605] but FlowNode was not found in storage for head id:FlowNodeId 1:17

at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:678)
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:715)
at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:659)
at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:525)

at hudson.model.RunMap.retrieve(RunMap.java:225)
at hudson.model.RunMap.retrieve(RunMap.java:57)
at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:499)
at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:481)
at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:379)
at hudson.model.RunMap.getById(RunMap.java:205)
at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:896)
at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:907)

at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65)
at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178)
at jenkins.model.Jenkins.<init>(Jenkins.java:975)

at hudson.model.Hudson.<init>(Hudson.java:85)
at hudson.model.Hudson.<init>(Hudson.java:81)
at hudson.WebAppMain$3.run(WebAppMain.java:233)
Finished: FAILURE
{noformat}

dnusbaum@cloudbees.com (JIRA)

unread,
Jan 17, 2020, 2:41:23 PM1/17/20
to jenkinsc...@googlegroups.com
Devin Nusbaum updated an issue
Note from maintainer: We'd like to be able to fix this, but we really need more information to do so. Please, whenever you encounter the error in the description of the ticket, zip the build folder ( { {$JENKINS_HOME/jobs/$PATH_TO_JOB/builds/$BUILD_NUMBER/}}) of the build that failed and upload it here, redacting any sensitive content as necessary, and include any relevant information on frequency of the issue, steps to reproduce (did it happen after Jenkins was restarted normally, or did Jenkins crash), any exceptions or warnings in the Jenkins system logs that seem relevant, etc. Thanks!

{ noformat}

dnusbaum@cloudbees.com (JIRA)

unread,
Jan 17, 2020, 3:06:20 PM1/17/20
to jenkinsc...@googlegroups.com

One thing that we could do to investigate this hypothesis would be to instrument all uses of BulkChange in workflow-job and workflow-cps with optional logging to understand if bulk changes being aborted when a save is requested could be causing the problem.

Looking into this some more, I think flow nodes and program.dat are saved through more direct paths that appear to bypass BulkChange, so that might not help. Maybe the implementations of FlowNodeStorage need some synchronization? That seems like a decent place to start.

dnusbaum@cloudbees.com (JIRA)

unread,
Jan 17, 2020, 3:35:07 PM1/17/20
to jenkinsc...@googlegroups.com

I filed https://github.com/jenkinsci/workflow-support-plugin/pull/104 based on my above hypothesis. If anyone is running into this issue consistently, I can create a build of the plugin based on that PR to see if it helps. I have not been able to reproduce the issue on demand, so I don't have a good way to validate whether that PR actually helps or not.

dnusbaum@cloudbees.com (JIRA)

unread,
Jan 27, 2020, 11:00:10 AM1/27/20
to jenkinsc...@googlegroups.com

Here is the SNAPSHOT build of workflow-support 3.5 with my PR in case anyone would like to try it out. If you do try out that build, or even if you don't try it, if you still run into this issue, please upload a build folder of the build that had the issue as described in my note in the issue description here.

dnusbaum@cloudbees.com (JIRA)

unread,
Jan 27, 2020, 11:03:03 AM1/27/20
to jenkinsc...@googlegroups.com
Devin Nusbaum edited a comment on Bug JENKINS-55287
[Here is the SNAPSHOT build of workflow-support 3.5 |https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/workflow/workflow-support/3.5-rc749.b9dda4d47229/workflow-support-3.5-rc749.b9dda4d47229.hpi] is the SNAPSHOT build of workflow-support 3.5 with my PR in case anyone would like to try it out. If you do try out that build, or even if you don't try it, if you still run into this issue, please upload a build folder of the build that had the issue as described in my note in the issue description here.

rodrigc@FreeBSD.org (JIRA)

unread,
Jan 30, 2020, 6:01:03 PM1/30/20
to jenkinsc...@googlegroups.com

Devin Nusbaum If you have tested this a bit, can you merge this and deploy a new workflow-support-plugin?
I keep hitting this at intermittent times, but I can never fully reproduce it.
Unfortunately, this keeps hitting me right in the middle of when I am doing important stuff, and I don't have time to debug this.

henrik.koren@emc.com (JIRA)

unread,
Jan 31, 2020, 5:51:10 AM1/31/20
to jenkinsc...@googlegroups.com
Henrik Koren updated an issue
 
Change By: Henrik Koren
Attachment: 48064.tar.gz

henrik.koren@emc.com (JIRA)

unread,
Jan 31, 2020, 5:51:12 AM1/31/20
to jenkinsc...@googlegroups.com
Henrik Koren commented on Bug JENKINS-55287
 
Re: Pipeline: Failure to load flow node: FlowNode was not found in storage for head

48064.tar.gz I have the same issues 

I have uploaded the entire build from my Master

Creating placeholder flownodes because failed loading originals.
java.io.IOException: Tried to load head FlowNodes for execution Owner[Utils/CreateNodePipeline/48064:Utils/CreateNodePipeline #48064] but FlowNode was not found in storage for head id:FlowNodeId 1:20
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:679)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:716)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:664)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:526)
	at hudson.model.RunMap.retrieve(RunMap.java:225)
	at hudson.model.RunMap.retrieve(RunMap.java:57)
	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501)
	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483)
	at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381)
	at hudson.model.RunMap.getById(RunMap.java:205)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:901)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:912)
	at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65)
	at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57)
	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
	at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178)
	at jenkins.model.Jenkins.<init>(Jenkins.java:999)
	at hudson.model.Hudson.<init>(Hudson.java:85)
	at hudson.model.Hudson.<init>(Hudson.java:81)
	at hudson.WebAppMain$3.run(WebAppMain.java:233)
Finished: FAILURE
Add Comment Add Comment
 

dnusbaum@cloudbees.com (JIRA)

unread,
Feb 5, 2020, 8:30:03 AM2/5/20
to jenkinsc...@googlegroups.com

I got feedback from some users who were able to test the PR that it did not fix the issue. I am waiting for a build folder from those users to confirm that they are seeing the same issue. I think the next step would be to make a new hypothesis as to the issue and submit a PR. If anyone has even a semi-consistent way to reproduce the issue, it would make it much more likely that we could fix it.

Henrik Koren Thanks!

Craig Rodrigues No, see above.

f.modler@gmx.net (JIRA)

unread,
Feb 13, 2020, 10:05:05 AM2/13/20
to jenkinsc...@googlegroups.com

Any news? This is biting us on a regular basis.

Senthilkumar.Palanisamy@in.bosch.com (JIRA)

unread,
Mar 19, 2020, 3:43:03 AM3/19/20
to jenkinsc...@googlegroups.com

Hi,

We are facing this issue frequently is there any workaround or any fix.

Our Jenkins version 2.204.1
This message was sent by Atlassian Jira (v7.13.12#713012-sha1:6e07c38)
Atlassian logo

manuelramonleonjimenez@gmail.com (JIRA)

unread,
Mar 24, 2020, 11:44:18 AM3/24/20
to jenkinsc...@googlegroups.com

I'm facing this issue on https://ci.jenkins.io/blue/organizations/jenkins/Plugins%2Fldap-plugin/detail/PR-40/4/pipeline/ for this PR: https://github.com/jenkinsci/ldap-plugin/pull/40

I guess Olivier Lamy can give you more information Devin Nusbaum. The log shows:

[2020-03-24T12:26:50.886Z] java.io.IOException: Tried to load head FlowNodes for execution Owner[Plugins/ldap-plugin/PR-40/4:Plugins/ldap-plugin/PR-40 #4] but FlowNode was not found in storage for head id:FlowNodeId 1:76
[2020-03-24T12:26:50.886Z] 	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:679)
[2020-03-24T12:26:50.886Z] 	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:716)
[2020-03-24T12:26:50.886Z] 	at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:672)
[2020-03-24T12:26:50.886Z] 	at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:533)
[2020-03-24T12:26:50.886Z] 	at hudson.model.RunMap.retrieve(RunMap.java:225)
[2020-03-24T12:26:50.886Z] 	at hudson.model.RunMap.retrieve(RunMap.java:57)
[2020-03-24T12:26:50.886Z] 	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501)
[2020-03-24T12:26:50.886Z] 	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483)
[2020-03-24T12:26:50.886Z] 	at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381)
[2020-03-24T12:26:50.886Z] 	at hudson.model.RunMap.getById(RunMap.java:205)
[2020-03-24T12:26:50.886Z] 	at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:921)
[2020-03-24T12:26:50.886Z] 	at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:932)
[2020-03-24T12:26:50.886Z] 	at org.jenkinsci.plugins.workflow.cps.CpsStepContext.getExecution(CpsStepContext.java:217)
[2020-03-24T12:26:50.886Z] 	at org.jenkinsci.plugins.workflow.cps.CpsStepContext.getThreadGroupSynchronously(CpsStepContext.java:242)
[2020-03-24T12:26:50.886Z] 	at org.jenkinsci.plugins.workflow.cps.CpsStepContext.access$000(CpsStepContext.java:97)
[2020-03-24T12:26:50.886Z] 	at org.jenkinsci.plugins.workflow.cps.CpsStepContext$1.call(CpsStepContext.java:263)
[2020-03-24T12:26:50.886Z] 	at org.jenkinsci.plugins.workflow.cps.CpsStepContext$1.call(CpsStepContext.java:261)
[2020-03-24T12:26:50.886Z] 	at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
[2020-03-24T12:26:50.886Z] 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[2020-03-24T12:26:50.886Z] 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[2020-03-24T12:26:50.886Z] 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[2020-03-24T12:26:50.886Z] 	at java.base/java.lang.Thread.run(Thread.java:834)
[2020-03-24T12:26:50.886Z] Finished: FAILURE

bitwiseman@gmail.com (JIRA)

unread,
Mar 27, 2020, 7:34:04 PM3/27/20
to jenkinsc...@googlegroups.com
Liam Newman updated an issue
 
Change By: Liam Newman
*IMPORTANT: NOTE FROM A MAINTAINER:*

+STOP!  YOUR STACK TRACE ALONE IS NOT GOING TO HELP US SOLVE THIS!+

(sorry to all caps but we're not going to make progress on this issue with commenters adding insufficient information)

* Note from maintainer: We'd like to be able to fix this, but we really need more information to do so. Please, whenever you encounter the error in the description of the ticket, zip the build folder ({{$JENKINS_HOME/jobs/$PATH_TO_JOB/builds/$BUILD_NUMBER/}}) of the build that failed and upload it here, redacting any sensitive content as necessary, and include any relevant information on frequency of the issue, steps to reproduce (did it happen after Jenkins was restarted normally, or did Jenkins crash), any exceptions or warnings in the Jenkins system logs that seem relevant, etc. Thanks! *

 
 {noformat}

Creating placeholder flownodes because failed loading originals.
java.io.IOException: Tried to load head FlowNodes for execution Owner[Platform Service FBI Test/1605:Platform Service FBI Test #1605] but FlowNode was not found in storage for head id:FlowNodeId 1:17
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:678)
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:715)
at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:659)
at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:525)
at hudson.model.RunMap.retrieve(RunMap.java:225)
at hudson.model.RunMap.retrieve(RunMap.java:57)
at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:499)
at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:481)
at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:379)
at hudson.model.RunMap.getById(RunMap.java:205)
at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:896)
at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:907)

at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65)
at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178)
at jenkins.model.Jenkins.<init>(Jenkins.java:975)

at hudson.model.Hudson.<init>(Hudson.java:85)
at hudson.model.Hudson.<init>(Hudson.java:81)
at hudson.WebAppMain$3.run(WebAppMain.java:233)
Finished: FAILURE
{noformat}

bitwiseman@gmail.com (JIRA)

unread,
Mar 27, 2020, 7:41:03 PM3/27/20
to jenkinsc...@googlegroups.com
Liam Newman updated an issue
*IMPORTANT: NOTE FROM A MAINTAINER:*

+STOP!  YOUR STACK TRACE ALONE IS NOT GOING TO HELP US
SOLVE THIS!+


(sorry to all caps but we're not going to make progress on this issue with commenters adding insufficient information)

*Note from maintainer: We'd like to be able to fix this, but we really need more information to do so. Please, whenever you encounter the error in the description of the ticket, zip the build folder ({{$JENKINS_HOME/jobs/$PATH_TO_JOB/builds/$BUILD_NUMBER/}}) of the build that failed and upload it here, redacting any sensitive content as necessary, and include any relevant information on frequency of the issue, steps to reproduce (did it happen after Jenkins was restarted normally, or did Jenkins crash), any exceptions or warnings in the Jenkins system logs that seem relevant, etc. Thanks!*

bitwiseman@gmail.com (JIRA)

unread,
Mar 27, 2020, 7:44:03 PM3/27/20
to jenkinsc...@googlegroups.com
Liam Newman commented on Bug JENKINS-55287
 
Re: Pipeline: Failure to load flow node: FlowNode was not found in storage for head

IMPORTANT: NOTE FROM A MAINTAINER:

STOP!  YOUR STACK TRACE ALONE IS NOT GOING TO HELP SOLVE THIS!

(sorry to all caps but we're not going to make progress on this issue with commenters adding insufficient information)

We'd like to be able to fix this, but we really need more information to do so. Please, whenever you encounter the error in the description of the ticket, zip the build folder ($JENKINS_HOME/jobs/$PATH_TO_JOB/builds/$BUILD_NUMBER/) of the build that failed and upload it here, redacting any sensitive content as necessary, and include any relevant information on frequency of the issue, steps to reproduce (did it happen after Jenkins was restarted normally, or did Jenkins crash), any exceptions or warnings in the Jenkins system logs that seem relevant, etc. Thanks!

Ramon Leon
**Sorry, I don't think we can get that information from ci.jenkins.io - the infra team is overloaded and we need the zip of the build folder immediately (or very soon) after the error.

 

rodrigc@FreeBSD.org (JIRA)

unread,
Mar 29, 2020, 1:53:03 PM3/29/20
to jenkinsc...@googlegroups.com

Liam Newman I think a lot of pressure is being put on the end user to perform weird steps in order to help debug this problem. This is very hard to do, and since the problem
is intermittent, it is hard to catch.

Why don't you modify the pipeline plugin so at the point that the pipeline tries to throw an exception with message * but FlowNode was not found in storage for head*, the plugin zips up the entire job directory, and puts it in some place like $TEMPDIR ? That way the end user doesn't have to perform a lot of weird steps, but they can simply retrieve a zip file with all the stuff the is needed for the pipeline plugin maintainer to diagnose and root cause the problem?

rodrigc@FreeBSD.org (JIRA)

unread,
Mar 29, 2020, 2:01:07 PM3/29/20
to jenkinsc...@googlegroups.com
Craig Rodrigues edited a comment on Bug JENKINS-55287
[~bitwiseman] I think a lot of pressure is being put on the end user to perform weird steps in order to help debug this problem.  This is very hard to do, and since the problem

is intermittent, it is hard to catch.

Why don't you modify the pipeline plugin so at the point that the pipeline tries to throw an exception with message * but FlowNode was not found in storage for head*, the plugin zips up the entire job directory, and puts it in some place like $TEMPDIR ?  That way the end user doesn't have to perform a lot of weird steps, but they can simply retrieve a zip file with all the stuff the is needed for the pipeline plugin maintainer to diagnose and root cause the problem?

rodrigc@FreeBSD.org (JIRA)

unread,
Mar 29, 2020, 2:02:07 PM3/29/20
to jenkinsc...@googlegroups.com
Craig Rodrigues edited a comment on Bug JENKINS-55287
[~bitwiseman] I think a lot of pressure is being put on the end user to perform weird steps in order to help debug this problem.  This is very hard to do.  Since the problem

is intermittent, it is hard to catch.

Why don't you modify the pipeline plugin so at the point that the pipeline tries to throw an exception with message *but FlowNode was not found in storage for head*, the plugin zips up the entire job directory, and puts it in some place like $TEMPDIR ?
You can release a version of the plugin with this change.
  That way the end user doesn't have to perform a lot of weird steps, but they can simply retrieve a zip file with all the stuff the is needed for the pipeline plugin maintainer to diagnose and root cause the problem?

rodrigc@FreeBSD.org (JIRA)

unread,
Mar 29, 2020, 2:02:07 PM3/29/20
to jenkinsc...@googlegroups.com
Craig Rodrigues edited a comment on Bug JENKINS-55287
[~bitwiseman] I think a lot of pressure is being put on the end user to perform weird steps in order to help debug this problem.  This is very hard to do , and since .  Since the problem

is intermittent, it is hard to catch.

Why don't you modify the pipeline plugin so at the point that the pipeline tries to throw an exception with message *but FlowNode was not found in storage for head*, the plugin zips up the entire job directory, and puts it in some place like $TEMPDIR ?  That way the end user doesn't have to perform a lot of weird steps, but they can simply retrieve a zip file with all the stuff the is needed for the pipeline plugin maintainer to diagnose and root cause the problem?

stevenmchaves@gmail.com (JIRA)

unread,
Mar 30, 2020, 10:44:03 AM3/30/20
to jenkinsc...@googlegroups.com

I don't have rights to zip my Build Job folder.

dnusbaum@cloudbees.com (JIRA)

unread,
Mar 30, 2020, 10:51:13 AM3/30/20
to jenkinsc...@googlegroups.com
Devin Nusbaum updated an issue
 
Change By: Devin Nusbaum
*IMPORTANT: NOTE FROM A MAINTAINER:*

+STOP!  YOUR STACK TRACE ALONE IS NOT GOING TO HELP SOLVE THIS!+


(sorry to all caps but we're not going to make progress on this issue with commenters adding insufficient information)

*Note from maintainer: We'd like to be able to fix this, but we really need more information to do so. Please, whenever you encounter the error in the description of the ticket, zip the build folder ({{$JENKINS_HOME/jobs/$PATH_TO_JOB/builds/$BUILD_NUMBER/}}) of the build that failed and upload it here along with the Jenkins system logs , redacting any sensitive content as necessary, and include any relevant information on frequency of the issue, steps to reproduce (did it happen after Jenkins was restarted normally, or did Jenkins crash), any exceptions or warnings messages in the Jenkins system logs that seem relevant, etc. Thanks!*

f.modler@gmx.net (JIRA)

unread,
Mar 30, 2020, 10:56:05 AM3/30/20
to jenkinsc...@googlegroups.com
Falko Modler commented on Bug JENKINS-55287
 
Re: Pipeline: Failure to load flow node: FlowNode was not found in storage for head

Devin Nusbaum & Liam Newman

I have three zipped build folders (all from around the same time, March 12th) but did not yet have the time ro redact sensitive data. Do you have some hints what and where to look for (regarding sensitive data)?

f.modler@gmx.net (JIRA)

unread,
Mar 30, 2020, 10:57:02 AM3/30/20
to jenkinsc...@googlegroups.com
Falko Modler edited a comment on Bug JENKINS-55287
[~dnusbaum] & [~bitwiseman]

I have three zipped build folders (all from around the same time, March 12th) but did not yet have the time ro to redact sensitive data. Do you have some hints what and where to look for (regarding sensitive data)?

dnusbaum@cloudbees.com (JIRA)

unread,
Mar 30, 2020, 11:59:13 AM3/30/20
to jenkinsc...@googlegroups.com

I was looking into this issue recently and wanted to check whether the code that saves Pipelines using the PERFORMANCE_OPTIMIZED durability level when Jenkins is shutting down (CpsFlowExecution.suspendAll) is running successfully in these cases to make sure that the problem isn't something like the service configuration for Jenkins not giving it enough time to shut down.

For users seeing the problem, please check the Jenkins system logs for the shutdown that occurred before you saw this error. Do you see messages similar to these, in particular for the build that failed with the exception in the description of this ticket?

  • About to try to checkpoint the program for buildCpsFlowExecutionOwner[YourJobName/BuildNumber:YourJobName #BuildNumber]]
  • Trying to save program before shutdown org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$8@RandomHash
  • Finished saving program before shutdown org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$8@RandomHash

Those messages should come in groups of three, in that order, and are logged at INFO level, and there should not be any WARNING messages interspersed between them or surrounding them. There should be a set of those messages for each Pipeline running using the PERFORMANCE_OPTIMIZED durability level when Jenkins shuts down. If you have configured Jenkins to run as a service, I would also check the service logs to see if Jenkins has been shutting down successfully or if there are any warnings/errors when Jenkins shuts down.

Falko Modler It depends on what your builds are doing and what you consider sensitive. The main files to check are build.xml, which contains a textual form of the Pipeline, and log, which contains the build logs. You might want to check for internal hostnames or IP addresses, unmasked passwords or API keys in your build log, etc.

dnusbaum@cloudbees.com (JIRA)

unread,
Mar 30, 2020, 3:52:13 PM3/30/20
to jenkinsc...@googlegroups.com
Devin Nusbaum edited a comment on Bug JENKINS-55287
I was looking into this issue recently and wanted to check whether the code that saves Pipelines using the PERFORMANCE_OPTIMIZED durability level when Jenkins is shutting down ([CpsFlowExecution.suspendAll|https://github.com/jenkinsci/workflow-cps-plugin/blob/e8d4c696277a044a808380bab6c6c7b06ef6cab9/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsFlowExecution.java#L1572]) is running successfully in these cases to make sure that the problem isn't something like the service configuration for Jenkins not giving it enough time to shut down.


For users seeing the problem, please check the Jenkins system logs for the shutdown that occurred before you saw this error. Do you see messages similar to these, in particular for the build that failed with the exception in the description of this ticket?
* About to try to checkpoint the program for buildCpsFlowExecution \ [Owner \ [YourJobName/BuildNumber:YourJobName #BuildNumber] |#BuildNumber ] ]
* Trying to save program before shutdown org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$8@RandomHash
* Finished saving program before shutdown org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$8@RandomHash


Those messages should come in groups of three, in that order, and are logged at INFO level, and there should not be any WARNING messages interspersed between them or surrounding them. There should be a set of those messages for each Pipeline running using the PERFORMANCE_OPTIMIZED durability level when Jenkins shuts down. If you have configured Jenkins to run as a service, I would also check the service logs to see if Jenkins has been shutting down successfully or if there are any warnings/errors when Jenkins shuts down.


[~famod] It depends on what your builds are doing and what you consider sensitive. The main files to check are {{build.xml}}, which contains a textual form of the Pipeline, and {{log}}, which contains the build logs. You might want to check for internal hostnames or IP addresses, unmasked passwords or API keys in your build log, etc.
Reply all
Reply to author
Forward
0 new messages