[JIRA] (JENKINS-57725) Proc.executor inherits ContextClassLoader dynamically

28 views
Skip to first unread message

dnusbaum@cloudbees.com (JIRA)

unread,
May 28, 2019, 3:22:01 PM5/28/19
to jenkinsc...@googlegroups.com
Devin Nusbaum created an issue
 
Jenkins / Bug JENKINS-57725
Proc.executor inherits ContextClassLoader dynamically
Issue Type: Bug Bug
Assignee: Devin Nusbaum
Attachments: 57954701-a859f600-78c1-11e9-823f-6dd0b2e07758.png
Components: core
Created: 2019-05-28 19:21
Priority: Minor Minor
Reporter: Devin Nusbaum

The threads in the thread pool used for Proc.joinWithTimeout inherit their context class loader from the thread that calls Proc.joinWithTimeout, which seems undesirable. While looking into JENKINS-50223, I found a case where one of the threads in the thread pool was holding onto a CleanGroovyClassLoader class loader from workflow-cps (see the attached screenshot), which is definitely undesirable.

We should wrap this thread pool in ClassLoaderSanityThreadFactory to prevent this kind of issue from happening.

Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

dnusbaum@cloudbees.com (JIRA)

unread,
May 28, 2019, 3:22:02 PM5/28/19
to jenkinsc...@googlegroups.com

dnusbaum@cloudbees.com (JIRA)

unread,
May 28, 2019, 3:22:02 PM5/28/19
to jenkinsc...@googlegroups.com
Devin Nusbaum started work on Bug JENKINS-57725
 
Change By: Devin Nusbaum
Status: Open In Progress

o.v.nenashev@gmail.com (JIRA)

unread,
Jun 18, 2019, 8:25:02 AM6/18/19
to jenkinsc...@googlegroups.com

o.v.nenashev@gmail.com (JIRA)

unread,
Jun 18, 2019, 8:25:02 AM6/18/19
to jenkinsc...@googlegroups.com
Change By: Oleg Nenashev
Status: In Review Resolved
Resolution: Fixed
Released As: Jenkins 2.181

ogondza@gmail.com (JIRA)

unread,
Jul 1, 2019, 3:14:02 AM7/1/19
to jenkinsc...@googlegroups.com
Oliver Gondža updated an issue
Change By: Oliver Gondža
Labels: lts 2.176.2 - candidate fixed

2bluesc@gmail.com (JIRA)

unread,
Jul 22, 2019, 6:49:02 PM7/22/19
to jenkinsc...@googlegroups.com
Kyle Manna updated an issue
Change By: Kyle Manna
Attachment: Screenshot from 2019-07-22 15-35-03.png

2bluesc@gmail.com (JIRA)

unread,
Jul 22, 2019, 6:52:01 PM7/22/19
to jenkinsc...@googlegroups.com
Kyle Manna edited a comment on Bug JENKINS-57725
 
Re: Proc.executor inherits ContextClassLoader dynamically
This was mentioned in the [LTS changelog for v2.176.2| [ https://jenkins.io/changelog-stable/] ] , but I'm experiencing something related.  My Jenkins system with 20 executors will creep towards 20k Tasks (processes + threads) before it hits the systemd imposed DefaultTasksMax limit of 19660 on my system.

 

 A java heap dump shows that  4675 of the total 4877 match the expression {{Timer-.*}} and have the Context Class Loader-> {{org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$CleanGroovyClassLoader}}.

!Screenshot from 2019-07-22 15-35-03.png!

 

 Bottom right corner of my Jenkins UI confirms I'm running the latest version (Jenkins ver. 2.176.2)with all plugins updated on OpenJDK Runtime Environment (build 1.8.0_222-b05) with. Ubuntu 18.04

2bluesc@gmail.com (JIRA)

unread,
Jul 22, 2019, 6:52:02 PM7/22/19
to jenkinsc...@googlegroups.com

This was mentioned in the [LTS changelog for v2.176.2|https://jenkins.io/changelog-stable/], but I'm experiencing something related.  My Jenkins system with 20 executors will creep towards 20k Tasks (processes + threads) before it hits the systemd imposed DefaultTasksMax limit of 19660 on my system.

 

 A java heap dump shows that  4675 of the total 4877 match the expression Timer-.* and have the Context Class Loader-> org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$CleanGroovyClassLoader.

 

 Bottom right corner of my Jenkins UI confirms I'm running the latest version (Jenkins ver. 2.176.2)with all plugins updated on OpenJDK Runtime Environment (build 1.8.0_222-b05) with. Ubuntu 18.04

2bluesc@gmail.com (JIRA)

unread,
Jul 22, 2019, 7:15:01 PM7/22/19
to jenkinsc...@googlegroups.com
Kyle Manna updated an issue
Change By: Kyle Manna
Attachment: d119bce8-b89a-4ece-9327-a7fbfc791ff9.png

2bluesc@gmail.com (JIRA)

unread,
Jul 22, 2019, 7:15:02 PM7/22/19
to jenkinsc...@googlegroups.com

Additional plot from netdata tasks section showing the process limit skyrocket before derailing the Jenkins process and making it unstable/undefined:

SEVERE: Timer task hudson.model.Queue$MaintainTask@621715d8 failed
java.lang.OutOfMemoryError: unable to create new native thread

2bluesc@gmail.com (JIRA)

unread,
Jul 22, 2019, 7:16:02 PM7/22/19
to jenkinsc...@googlegroups.com
Kyle Manna edited a comment on Bug JENKINS-57725
This was mentioned in the [LTS changelog for v2.176.2|https://jenkins.io/changelog-stable/], but I'm experiencing something related.  My Jenkins system with 20 executors will creep towards 20k Tasks (processes + threads) before it hits the systemd imposed DefaultTasksMax limit of 19660 on my system.

 

 A java heap dump shows that  4675 of the total 4877 match the expression {{Timer-.*}} and have the Context Class Loader-> {{org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$CleanGroovyClassLoader}}.

!Screenshot from 2019-07-22 15-35-03.png |thumbnail !

 

 Bottom right corner of my Jenkins UI confirms I'm running the latest version (Jenkins ver. 2.176.2)with all plugins updated on OpenJDK Runtime Environment (build 1.8.0_222-b05) with. Ubuntu 18.04

2bluesc@gmail.com (JIRA)

unread,
Jul 22, 2019, 7:17:01 PM7/22/19
to jenkinsc...@googlegroups.com
Kyle Manna edited a comment on Bug JENKINS-57725
Additional plot from netdata tasks section showing the process limit task plot skyrocket before derailing the Jenkins process and making it unstable/undefined when the systemd CGroup imposes the TasksMax limit :

{code:java}

SEVERE: Timer task hudson.model.Queue$MaintainTask@621715d8 failed
java.lang.OutOfMemoryError: unable to create new native thread
{code}

!d119bce8-b89a-4ece-9327-a7fbfc791ff9.png|thumbnail!
< /td>

dnusbaum@cloudbees.com (JIRA)

unread,
Jul 25, 2019, 10:03:02 AM7/25/19
to jenkinsc...@googlegroups.com

Kyle Manna It looks like you have a distinct issue with two aspects: One, something is causing a bunch of timer threads to be created and not cleaned up, and two, those timer threads are inheriting their context class loader dynamically. The first issue is more serious, to investigate it I would search the jenkinsci organization on GitHub for all uses of java.util.Timer, cross-check that with the set of plugins you have installed, and then examine the ways that the plugins you have installed are using java.util.Timer based on the search. My guess is that something is repeatedly calling new Timer rather than storing a single timer somewhere and reusing it (maybe also better to switch to using an ExecutorService if that is the problem). Either way I'd go ahead and open a separate issue for your problem.

2bluesc@gmail.com (JIRA)

unread,
Jul 26, 2019, 2:25:02 PM7/26/19
to jenkinsc...@googlegroups.com

Devin Nusbaum thanks for the response, I've created a new issue @ JENKINS-58684 as you recommended and included all my plugins in the list. All of the plugins are pretty mainstream.

Reply all
Reply to author
Forward
0 new messages