[JIRA] (JENKINS-58528) Channel listeners (SlaveCommandStatistics.Statistics) retained after node removal

18 views
Skip to first unread message

ogondza@gmail.com (JIRA)

unread,
Jul 17, 2019, 9:05:02 AM7/17/19
to jenkinsc...@googlegroups.com
Oliver Gondža created an issue
 
Jenkins / Bug JENKINS-58528
Channel listeners (SlaveCommandStatistics.Statistics) retained after node removal
Issue Type: Bug Bug
Assignee: Oliver Gondža
Components: support-core-plugin
Created: 2019-07-17 13:04
Priority: Major Major
Reporter: Oliver Gondža

OOM situation investigation revealed that {com.cloudbees.jenkins.support.impl.SlaveCommandStatistics#statistics}} is always growing. For instances with dynamically provisioned nodes this is effectively a memory leak. Detection:

 

println "Listeneres " + ExtensionList.lookupSingleton(com.cloudbees.jenkins.support.impl.SlaveCommandStatistics.class).@statistics.size()
println "Computers " + Jenkins.instance.computers.size()

 

Values from some of the instances with varying size/uptime:

Listeneres 1990
Computers 5

---

println "Listeneres " + ExtensionList.lookupSingleton(com.cloudbees.jenkins.support.impl.SlaveCommandStatistics.class).@statistics.size()
println "Computers " + Jenkins.instance.computers.size()

---

println "Listeneres " + ExtensionList.lookupSingleton(com.cloudbees.jenkins.support.impl.SlaveCommandStatistics.class).@statistics.size()
println "Computers " + Jenkins.instance.computers.size()

 

The instance torn down had 30K instances.
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

ogondza@gmail.com (JIRA)

unread,
Jul 17, 2019, 9:07:03 AM7/17/19
to jenkinsc...@googlegroups.com
Oliver Gondža updated an issue
Change By: Oliver Gondža
OOM situation investigation revealed that \{com.cloudbees.jenkins.support.impl.SlaveCommandStatistics#statistics}} is always growing. For instances with dynamically provisioned nodes this is effectively a memory leak. Detection:
 
{noformat}

println "Listeneres " + ExtensionList.lookupSingleton(com.cloudbees.jenkins.support.impl.SlaveCommandStatistics.class).@statistics.size()
println "Computers " + Jenkins.instance.computers.size()

 {noformat}
  
Values from some of the instances with varying size/uptime:
{noformat}
Listeneres 1990
Computers 5
---
println " Listeneres " + ExtensionList.lookupSingleton(com.cloudbees.jenkins.support.impl.SlaveCommandStatistics.class).@statistics.size() 62
println " Computers " + Jenkins.instance.computers.size() 33

---
println " Listeneres " + ExtensionList.lookupSingleton(com.cloudbees.jenkins.support.impl.SlaveCommandStatistics.class).@statistics.size() 16521
println " Computers " + Jenkins.instance.computers.size() 16

{noformat}

 

The instance torn down had 30K instances.

ogondza@gmail.com (JIRA)

unread,
Jul 17, 2019, 9:17:02 AM7/17/19
to jenkinsc...@googlegroups.com
Oliver Gondža commented on Bug JENKINS-58528
 
Re: Channel listeners (SlaveCommandStatistics.Statistics) retained after node removal

Jesse Glick, as an author of the report, would you say this makes sense preserving after computer is gone (vs. only while it is present)? I am considering when to rotate the records - at node removal or keeping it around for a while to ease investigation...

ogondza@gmail.com (JIRA)

unread,
Jul 17, 2019, 9:17:02 AM7/17/19
to jenkinsc...@googlegroups.com
Oliver Gondža updated an issue
Change By: Oliver Gondža
OOM situation investigation revealed that \ { {
com.cloudbees.jenkins.support.impl.SlaveCommandStatistics#statistics}} is always growing. For instances with dynamically provisioned nodes this is effectively a memory leak. Detection:

{noformat}
println "Listeneres " + ExtensionList.lookupSingleton(com.cloudbees.jenkins.support.impl.SlaveCommandStatistics.class).@statistics.size()
println "Computers " + Jenkins.instance.computers.size()
{noformat}
Values from some of the instances with varying size/uptime:
{noformat}
Listeneres 1990
Computers 5
---
Listeneres 62
Computers 33
---
Listeneres 16521
Computers 16

{noformat}
 

The instance torn down had 30K instances.

ogondza@gmail.com (JIRA)

unread,
Jul 17, 2019, 9:39:02 AM7/17/19
to jenkinsc...@googlegroups.com
Oliver Gondža started work on Bug JENKINS-58528
 
Change By: Oliver Gondža
Status: Open In Progress

ogondza@gmail.com (JIRA)

unread,
Jul 17, 2019, 9:39:02 AM7/17/19
to jenkinsc...@googlegroups.com

jglick@cloudbees.com (JIRA)

unread,
Jul 17, 2019, 10:08:01 AM7/17/19
to jenkinsc...@googlegroups.com

Yes we need to preserve statistics after agent removal. I commented in more detail in the PR.

egutierrez@cloudbees.com (JIRA)

unread,
Aug 6, 2019, 5:44:03 AM8/6/19
to jenkinsc...@googlegroups.com
Evaristo Gutierrez resolved as Fixed
 
Change By: Evaristo Gutierrez
Status: In Progress Resolved
Resolution: Fixed
Released As: support-core-2.60
Reply all
Reply to author
Forward
0 new messages