FileNotFound Exception for program.dat in almost all the jobs of a scheduled pipeline

Shivam Pokhriyal

unread,

Jan 24, 2020, 6:04:11 AM1/24/20

to Jenkins Users

Not a duplicate of https://issues.jenkins-ci.org/browse/JENKINS-53775 since I'm not using a Job DSL plugin.

I have a pipeline that is scheduled to run midnight everyday. But since last couple of weeks, I'm getting program.dat file not found exception very frequently for all the jobs in the pipeline. Unfortunately, I cannot figure out why this is happening. I tried enabling concurrent builds in the pipeline configure option but still no luck.

```

java.io.FileNotFoundException: /var/lib/jenkins/jobs/pipeline-runner/builds/66320/program.dat (No such file or directory)
	at java.io.FileInputStream.open0(Native Method)
	at java.io.FileInputStream.open(FileInputStream.java:195)
	at java.io.FileInputStream.<init>(FileInputStream.java:138)
	at org.jenkinsci.plugins.workflow.support.pickles.serialization.RiverReader.openStreamAt(RiverReader.java:188)
	at org.jenkinsci.plugins.workflow.support.pickles.serialization.RiverReader.restorePickles(RiverReader.java:136)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.loadProgramAsync(CpsFlowExecution.java:774)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:740)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:668)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:529)
	at hudson.model.RunMap.retrieve(RunMap.java:225)
	at hudson.model.RunMap.retrieve(RunMap.java:57)
	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501)
	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483)
	at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381)
	at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:346)
	at jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:276)
	at jenkins.model.lazy.LazyBuildMixIn.getLastBuild(LazyBuildMixIn.java:242)
	at org.jenkinsci.plugins.workflow.job.WorkflowJob.getLastBuild(WorkflowJob.java:240)
	at org.jenkinsci.plugins.workflow.job.WorkflowJob.getLastBuild(WorkflowJob.java:103)
	at hudson.model.Job.isLogUpdated(Job.java:308)
	at org.jenkinsci.plugins.workflow.job.WorkflowJob.getCauseOfBlockage(WorkflowJob.java:304)
	at hudson.model.Queue.getCauseOfBlockageForTask(Queue.java:1226)
	at hudson.model.Queue.getCauseOfBlockageForItem(Queue.java:1189)
	at hudson.model.Queue.maintain(Queue.java:1552)
	at hudson.model.Queue$MaintainTask.doRun(Queue.java:2885)
	at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72)
	at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
Caused: java.io.IOException: Failed to load build state
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$3.onSuccess(CpsFlowExecution.java:855)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$3.onSuccess(CpsFlowExecution.java:853)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$4$1.run(CpsFlowExecution.java:907)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:38)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Finished: FAILURE

```

Shivam Pokhriyal

unread,

Jan 24, 2020, 7:42:47 AM1/24/20

to Jenkins Users

More Details:

I checked the contents of jenkins.log file and found out exceptions there

```

java.io.IOException: java.lang.RuntimeException: Failed to serialize hudson.model.Actionable#actions for class org.jenkinsci.plugins.workflow.job.WorkflowRun

at hudson.XmlFile.write(XmlFile.java:200)

at org.jenkinsci.plugins.workflow.support.PipelineIOUtils.writeByXStream(PipelineIOUtils.java:30)

at org.jenkinsci.plugins.workflow.job.WorkflowRun.save(WorkflowRun.java:1158)

at hudson.BulkChange.commit(BulkChange.java:98)

at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1475)

at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:493)

at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:38)

at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)

at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)

at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

Caused by: java.lang.RuntimeException: Failed to serialize hudson.model.Actionable#actions for class org.jenkinsci.plugins.workflow.job.WorkflowRun

at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:254)

at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:222)

at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)

at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:208)

at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:149)

at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)

at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)

at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)

at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:82)

at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37)

at com.thoughtworks.xstream.XStream.marshal(XStream.java:1026)

at com.thoughtworks.xstream.XStream.marshal(XStream.java:1015)

at com.thoughtworks.xstream.XStream.toXML(XStream.java:988)

at hudson.XmlFile.write(XmlFile.java:193)

... 14 more

Caused by: java.lang.RuntimeException: Failed to serialize org.jenkinsci.plugins.awsdevicefarm.AWSDeviceFarmTestResultAction#log for class org.jenkinsci.plugins.awsdevicefarm.AWSDeviceFarmTestResultAction