[JIRA] (JENKINS-61251) Out of memory issues during builds

Issue Type:	Bug
Assignee:	Unassigned
Attachments:	1.png, 2.png, 3.png, 4.png
Components:	core
Created:	2020-02-27 09:23
Environment:	openjdk 11.0.6 2020-01-14 OpenJDK Runtime Environment (build 11.0.6+10-post-Ubuntu-1ubuntu118.04.1) OpenJDK 64-Bit Server VM (build 11.0.6+10-post-Ubuntu-1ubuntu118.04.1, mixed mode, sharing) Jenkins ver. 2.204.2 Plugins: JavaScript GUI Lib: ACE Editor bundle plugin (ace-editor): 1.1 Active Directory plugin (active-directory): 2.16 Analysis Model API Plug-in (analysis-model-api): 7.0.4 AnsiColor (ansicolor): 0.6.2 Ant Plugin (ant): 1.10 OWASP Markup Formatter Plugin (antisamy-markup-formatter): 1.6 Apache HttpComponents Client 4.x API Plugin (apache-httpcomponents-client-4-api): 4.5.10-2.0 Authentication Tokens API Plugin (authentication-tokens): 1.3 Authorize Project (authorize-project): 1.3.0 bouncycastle API Plugin (bouncycastle-api): 2.17 Branch API Plugin (branch-api): 2.5.4 Build Blocker Plugin (build-blocker-plugin): 1.7.3 Build Name and Description Setter (build-name-setter): 2.0.3 Build Timeout (build-timeout): 1.19 Folders Plugin (cloudbees-folder): 6.9 Cobertura Plugin (cobertura): 1.15 Code Coverage API Plugin (code-coverage-api): 1.1.2 Command Agent Launcher Plugin (command-launcher): 1.3 Compact Columns (compact-columns): 1.10 Conditional BuildStep (conditional-buildstep): 1.3.6 Credentials Plugin (credentials): 2.3.0 Credentials Binding Plugin (credentials-binding): 1.20 Display URL API (display-url-api): 2.3.2 Docker Commons Plugin (docker-commons): 1.15 Docker Pipeline (docker-workflow): 1.21 DTKit 2 API. (dtkit-api): 2.1.1-1 Durable Task Plugin (durable-task): 1.33 Email Extension Plugin (email-ext): 2.68 External Monitor Job Type Plugin (external-monitor-job): 1.7 Extra Columns Plugin (extra-columns): 1.21 Forensics API Plugin (forensics-api): 0.6.1 Filesystem Trigger Plug-in (fstrigger): 0.39 Git client plugin (git-client): 3.0.0 GIT server Plugin (git-server): 1.8 JavaScript GUI Lib: Handlebars bundle plugin (handlebars): 1.1.1 Jackson 2 API Plugin (jackson2-api): 2.10.0 Javadoc Plugin (javadoc): 1.5 JAXB plugin (jaxb): 2.3.0.1 Oracle Java SE Development Kit Installer Plugin (jdk-tool): 1.4 JavaScript GUI Lib: jQuery bundles (jQuery and jQuery UI) plugin (jquery-detached): 1.2.1 JSch dependency plugin (jsch): 0.1.55.1 JUnit Plugin (junit): 1.28 LDAP Plugin (ldap): 1.21 Lockable Resources plugin (lockable-resources): 2.7 Mailer Plugin (mailer): 1.29 Matrix Authorization Strategy Plugin (matrix-auth): 2.5 Matrix Project Plugin (matrix-project): 1.14 Maven Integration plugin (maven-plugin): 3.4 Mercurial plugin (mercurial): 2.8 JavaScript GUI Lib: Moment.js bundle plugin (momentjs): 1.1.1 Monitoring (monitoring): 1.81.0 PAM Authentication plugin (pam-auth): 1.6 Parameterized Trigger plugin (parameterized-trigger): 2.35.2 Pipeline: Build Step (pipeline-build-step): 2.9 Pipeline Graph Analysis Plugin (pipeline-graph-analysis): 1.10 Pipeline: Input Step (pipeline-input-step): 2.11 Pipeline: Milestone Step (pipeline-milestone-step): 1.3.1 Pipeline: Model API (pipeline-model-api): 1.4.0 Pipeline: Declarative Agent API (pipeline-model-declarative-agent): 1.1.1 Pipeline: Declarative (pipeline-model-definition): 1.4.0 Pipeline: Declarative Extension Points API (pipeline-model-extensions): 1.4.0 Pipeline: REST API Plugin (pipeline-rest-api): 2.12 Pipeline: Stage Step (pipeline-stage-step): 2.3 Pipeline: Stage Tags Metadata (pipeline-stage-tags-metadata): 1.4.0 Pipeline: Stage View Plugin (pipeline-stage-view): 2.12 Plain Credentials Plugin (plain-credentials): 1.5 Resource Disposer Plugin (resource-disposer): 0.14 Robot Framework plugin (robot): 2.0.0 Role-based Authorization Strategy (role-strategy): 2.15 Run Condition Plugin (run-condition): 1.2 Safe Restart Plugin (saferestart): 0.3 SCM API Plugin (scm-api): 2.6.3 Script Security Plugin (script-security): 1.70 SSH Credentials Plugin (ssh-credentials): 1.18 SSH Slaves plugin (ssh-slaves): 1.31.0 Structs Plugin (structs): 1.20 Timestamper (timestamper): 1.10 Token Macro Plugin (token-macro): 2.10 Trilead API Plugin (trilead-api): 1.0.5 Validating String Parameter Plugin (validating-string-parameter): 2.4 Warnings Next Generation Plugin (warnings-ng): 7.3.0 WMI Windows Agents Plugin (windows-slaves): 1.5 Pipeline (workflow-aggregator): 2.6 Pipeline: API (workflow-api): 2.37 Pipeline: Basic Steps (workflow-basic-steps): 2.18 Pipeline: Groovy (workflow-cps): 2.80 Pipeline: Shared Groovy Libraries (workflow-cps-global-lib): 2.15 Pipeline: Nodes and Processes (workflow-durable-task-step): 2.35 Pipeline: Job (workflow-job): 2.36 Pipeline: Multibranch (workflow-multibranch): 2.21 Pipeline: SCM Step (workflow-scm-step): 2.9 Pipeline: Step API (workflow-step-api): 2.22 Pipeline: Supporting APIs (workflow-support): 3.3 Workspace Cleanup Plugin (ws-cleanup): 0.37 xUnit plugin (xunit): 2.3.6
Labels:	performance
Priority:	Critical
Reporter:	Andrej Staruch

Hi,

we've started to get an Out of memory issues on our Jenkins instance last week. There was no notable change in the code (1 month ago, we've installed warnings-ng for pylint reports and Cobertura), but the problems started last week.

My guess is that the memory should be released, but something is holding it. I was trying to go through a your help page and tried to analyze a Heap Dump. The notable is Java.Util.HashMap class has almost all allocated bytes.

I'm providing a 2 heap dumps, one during a build link (~2.93GB) and one after the Jenkins instance is unresponsible due to the Out Of Memory link (~3.57GB).

They suggest that the problem is org.codehaus.groovy.ast.ClassNode, but to say the truth, I'm not a Java developer so I'm not getting what can be an issue.

Also, I'm providing a /var/log/jenkins/jenkins.log which contains lots of warnings, which may help: link (~30MB)

I've already spent a great of time debugging what is happening, but I can't get my head around.

Could you take a look or point where the problem should be?

Thanks
Andrej

This message was sent by Atlassian Jira (v7.13.12#713012-sha1:6e07c38)

staruch.andrej@gmail.com (JIRA)

unread,

Feb 28, 2020, 3:38:03 AM2/28/20

to jenkinsc...@googlegroups.com

Andrej Staruch updated an issue

Jenkins /

Change By:	Andrej Staruch
Attachment:	fileDescriptors.jpg

staruch.andrej@gmail.com (JIRA)

unread,

Feb 28, 2020, 3:38:04 AM2/28/20

to jenkinsc...@googlegroups.com

Andrej Staruch updated an issue

Jenkins /

Change By:	Andrej Staruch
Attachment:	threadCount.jpg

staruch.andrej@gmail.com (JIRA)

unread,

Feb 28, 2020, 3:39:02 AM2/28/20

to jenkinsc...@googlegroups.com

Andrej Staruch commented on

Re: Out of memory issues during builds

The problem is in Java. Our server's Java vas upgraded from jdk-8 to jdk-11. After downgrading it back to jdk-8, thing went back to normal.

Java 11 has some issues regarding open files I guess. We switched back to jdk-8 around 13:00 and this was the only change. We were then able to run our builds without any hassle. We can on our monitory graphs that probably Jenkins, or some of our other plugin, has a problem with file descriptors?

dbeck@cloudbees.com (JIRA)

unread,

Feb 29, 2020, 8:42:02 AM2/29/20

to jenkinsc...@googlegroups.com

Daniel Beck commented on

Re: Out of memory issues during builds

open files

If you use the "Resource Root URL" feature, that has a known problem with unclosed open files in 2.204.x: ~~JENKINS-61121~~. That will be addressed in LTS next month in 2.222.1.

dbeck@cloudbees.com (JIRA)

unread,

Feb 29, 2020, 8:43:03 AM2/29/20

to jenkinsc...@googlegroups.com

Daniel Beck closed an issue as Cannot Reproduce

Closing for now, first comment indicates it's an obsolete problem.

Jenkins /

Change By:	Daniel Beck
Status:	Open Closed
Resolution:	Cannot Reproduce

staruch.andrej@gmail.com (JIRA)

unread,

Mar 2, 2020, 3:51:09 AM3/2/20

to jenkinsc...@googlegroups.com

Andrej Staruch commented on

Re: Out of memory issues during builds

Just FYI: we don't use "Resource Root URL" feature. So there is some other problem with java11, but I was not able to narrow it down which plugin was doing it.