| jenkins--failure.log I let the standard pipelines run 20 minutes or so. There wasn't any sign of degraded performance to speak of. Here's the beginning and end of the vmstat output:
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
02/11/2017 01:47:41c
2 0 0 12885760 189508 831916 0 0 15 268 273 144 75 0 24 1 0
(snip)
02/11/2017 02:06:41c
0 0 0 12805028 191464 832472 0 0 0 432 175 204 0 0 99 1 0
That amount of decrease in free memory is pretty small. The UI remains responsive throughout. To contrast, here's a similar timespan, with the declarative jobs running. Bu the time I stopped vmstat recording, the UI had stopped responding.
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
02/11/2017 02:25:33c
1 0 0 13255960 195416 915500 0 0 13 251 251 144 65 0 33 1 0
(snip)
02/11/2017 02:45:33c
4 0 0 11748496 199188 981928 0 0 0 1471 1290 735 93 0 7 0 0
The declarative jobs are consuming more resource, but I think that's expected. The log, which I'll attach as jenkins--failure.log, is interesting. The jobs run for a while. Then I start to see these:
Feb 11, 2017 2:27:34 AM org.jenkinsci.plugins.workflow.job.WorkflowRun finish
INFO: DeclarativePipelines/inlineThing72 #80 completed: SUCCESS
Feb 11, 2017 2:27:35 AM org.jenkinsci.plugins.workflow.job.WorkflowRun finish
INFO: DeclarativePipelines/inlineThing70 #80 completed: SUCCESS
Feb 11, 2017 2:27:39 AM org.jenkinsci.plugins.workflow.cps.CpsFlowExecution cleanUpHeap
WARNING: failed to clean up memory from Owner[DeclarativePipelines/inlineThing63/81:DeclarativePipelines/inlineThing63 #81]
java.lang.NullPointerException
at sun.reflect.UnsafeFieldAccessorImpl.ensureObj(UnsafeFieldAccessorImpl.java:57)
at sun.reflect.UnsafeQualifiedObjectFieldAccessorImpl.get(UnsafeQualifiedObjectFieldAccessorImpl.java:38)
at java.lang.reflect.Field.get(Field.java:393)
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.cleanUpGlobalClassSet(CpsFlowExecution.java:1034)
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.cleanUpLoader(CpsFlowExecution.java:966)
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.cleanUpLoader(CpsFlowExecution.java:968)
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.cleanUpLoader(CpsFlowExecution.java:968)
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.cleanUpLoader(CpsFlowExecution.java:957)
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.cleanUpLoader(CpsFlowExecution.java:957)
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.cleanUpHeap(CpsFlowExecution.java:941)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:363)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:80)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:240)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:228)
at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:63)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Things start to look different shortly after this. There are more NPEs, then a bunch of successful builds. Around line 49101 in jenkins--failure.log, which I'll attach, I see what looks to me like a new error:
Feb 11, 2017 2:28:23 AM org.jenkinsci.plugins.workflow.cps.CpsThreadGroup saveProgramIfPossible
WARNING: program state save failed
java.io.IOException: Failed to persist /var/lib/jenkins/jobs/DeclarativePipelines/jobs/inlineThing96/builds/85/program.dat
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.saveProgram(CpsThreadGroup.java:461)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.saveProgram(CpsThreadGroup.java:427)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.saveProgramIfPossible(CpsThreadGroup.java:415)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:360)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:80)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:240)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:228)
at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:63)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Failed to serialize org.jenkinsci.plugins.pipeline.modeldefinition.agent.DeclarativeAgent#context for class org.jenkinsci.plugins.pipeline.modeldefinition.agent.impl.Any
at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:256)
at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224)
Rather than speculate any further on my own I've attached the log. I put a "comment" in at line 49102, just to help me find things. |
