[JIRA] (JENKINS-42026) Memory leak while running concurrent declarative pipelines

1 view
Skip to first unread message

kshultz@cloudbees.com (JIRA)

unread,
Feb 14, 2017, 10:59:02 AM2/14/17
to jenkinsc...@googlegroups.com
Karl Shultz created an issue
 
Jenkins / Bug JENKINS-42026
Memory leak while running concurrent declarative pipelines
Issue Type: Bug Bug
Assignee: Unassigned
Components: workflow-cps-plugin
Created: 2017/Feb/14 3:58 PM
Environment: openjdk version "1.8.0_45-internal"
Ubuntu 15.04
workflow-cps 2.27-SNAPSHOT (private-1155b889-jenkins)
pipeline-model-definition 1.0.1
core 2.32.2
Priority: Major Major
Reporter: Karl Shultz

Running >100 Pipeline builds can cause OutOfMemoryErrors, leading to Jenkins eventually becoming unresponsive.

Sample log snippet below. After I get this issue opened, I'll recreate to produce cleaner logs.

WARNING: RunListener failed
java.lang.OutOfMemoryError: GC overhead limit exceeded

Feb 14, 2017 2:47:04 PM org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService reportProblem
WARNING: Unexpected exception in CPS VM thread: CpsFlowExecution[Owner[PipelinePipelines/pipeline-testcases---parallel-abuse180/198:PipelinePipelines/pipeline-testcases---parallel-abuse180 #198]]
java.lang.OutOfMemoryError: GC overhead limit exceeded

Feb 14, 2017 2:47:03 PM hudson.model.AsyncPeriodicWork doRun
INFO: com.cloudbees.jenkins.plugin.metrics.views.Alerter thread is still running. Execution aborted.
Feb 14, 2017 2:47:10 PM hudson.model.AsyncPeriodicWork doRun
INFO: com.cloudbees.jenkins.plugin.metrics.views.Alerter thread is still running. Execution aborted.
Feb 14, 2017 2:47:20 PM hudson.model.AsyncPeriodicWork doRun
INFO: com.cloudbees.jenkins.plugin.metrics.views.Alerter thread is still running. Execution aborted.
Feb 14, 2017 2:47:27 PM org.jenkinsci.plugins.workflow.job.WorkflowRun finish
INFO: PipelinePipelines/pipeline-testcases---parallel-abuse24 #198 completed: FAILURE
Feb 14, 2017 2:47:28 PM hudson.model.AsyncPeriodicWork doRun
INFO: com.cloudbees.jenkins.plugin.metrics.views.Alerter thread is still running. Execution aborted.
Feb 14, 2017 2:47:30 PM org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService reportProblem
WARNING: Unexpected exception in CPS VM thread: CpsFlowExecution[Owner[PipelinePipelines/pipeline-testcases---parallel-abuse180/198:PipelinePipelines/pipeline-testcases---parallel-abuse180 #198]]
java.lang.NullPointerException
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:41)
	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:165)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:328)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:80)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:240)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:228)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Feb 14, 2017 2:47:35 PM hudson.triggers.SafeTimerTask run
SEVERE: Timer task hudson.model.Queue$MaintainTask@34463f11 failed
java.lang.OutOfMemoryError: GC overhead limit exceeded

Feb 14, 2017 2:47:46 PM hudson.model.AsyncPeriodicWork doRun
INFO: com.cloudbees.jenkins.plugin.metrics.views.Alerter thread is still running. Execution aborted.
Feb 14, 2017 2:47:55 PM hudson.model.AsyncPeriodicWork doRun
INFO: com.cloudbees.jenkins.plugin.metrics.views.Alerter thread is still running. Execution aborted.
Feb 14, 2017 2:48:00 PM hudson.model.AsyncPeriodicWork doRun
INFO: com.cloudbees.jenkins.plugin.metrics.views.Alerter thread is still running. Execution aborted.
Feb 14, 2017 2:48:06 PM hudson.model.AsyncPeriodicWork doRun
INFO: com.cloudbees.jenkins.plugin.metrics.views.Alerter thread is still running. Execution aborted.
Feb 14, 2017 2:48:14 PM hudson.model.AsyncPeriodicWork doRun
INFO: com.cloudbees.jenkins.plugin.metrics.views.Alerter thread is still running. Execution aborted.
Feb 14, 2017 2:48:21 PM hudson.model.AsyncPeriodicWork doRun
INFO: com.cloudbees.jenkins.plugin.metrics.views.Alerter thread is still running. Execution aborted.
Feb 14, 2017 2:48:24 PM org.jenkinsci.plugins.workflow.flow.FlowExecutionList load
WARNING: null
java.io.IOException: Unable to read /var/lib/jenkins/org.jenkinsci.plugins.workflow.flow.FlowExecutionList.xml
	at hudson.XmlFile.read(XmlFile.java:144)
	at org.jenkinsci.plugins.workflow.flow.FlowExecutionList.load(FlowExecutionList.java:99)
	at org.jenkinsci.plugins.workflow.flow.FlowExecutionList.register(FlowExecutionList.java:112)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:233)
	at hudson.model.ResourceController.execute(ResourceController.java:98)
	at hudson.model.Executor.run(Executor.java:404)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
Atlassian logo

jglick@cloudbees.com (JIRA)

unread,
Feb 14, 2017, 11:15:01 AM2/14/17
to jenkinsc...@googlegroups.com

jglick@cloudbees.com (JIRA)

unread,
Feb 14, 2017, 11:16:01 AM2/14/17
to jenkinsc...@googlegroups.com
Jesse Glick commented on Bug JENKINS-42026
 
Re: Memory leak while running concurrent declarative pipelines

That NPE from SandboxContinuable I have never seen before. I do not think it should lead to a leak, even in this one build—we still cleanUpHeap (at least as of workflow-cps PR 101, which I think you are running)—but still of interest.

Need a heap dump to diagnose (send privately).

kshultz@cloudbees.com (JIRA)

unread,
Feb 14, 2017, 11:28:01 AM2/14/17
to jenkinsc...@googlegroups.com
Reply all
Reply to author
Forward
0 new messages