[JIRA] (JENKINS-59404) High CPU usage in MapperBase.importFromFile in Benchmark Plugin

6 views
Skip to first unread message

artalus-mail@yandex.ru (JIRA)

unread,
Sep 17, 2019, 4:54:02 AM9/17/19
to jenkinsc...@googlegroups.com
Artalus S. created an issue
 
Jenkins / Bug JENKINS-59404
High CPU usage in MapperBase.importFromFile in Benchmark Plugin
Issue Type: Bug Bug
Assignee: Daniel Mercier
Attachments: cpu.png, Screenshot_2019-09-17_11-47-53.png
Components: benchmark-plugin
Created: 2019-09-17 08:53
Environment: Jenkins ver. 2.184
Benchmark 1.0.9
Priority: Major Major
Reporter: Artalus S.

Several days ago we've noticed that our Jenkins master had almost all its 12 core under 100% load. Apparently it's been that way for several days straight. Using Monitoring plugin we've figured out there was about 10 threads with tremendous amount of time spent in `java.util.TreeMap.put`. Killing those threads one by one didn't break anything, so we assumed it to be some strange jenkins quirk and went on. AFAIK stack traces weren't informative.

Today I noticed there are 3 threads each eating 100% of core again. This time stack traces are more informative (see attachments). I cannot confirm that those two issues with CPU load are directly correlated, but the current one seems legit enough.

Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

daniel.mercier@autodesk.com (JIRA)

unread,
Sep 17, 2019, 6:26:04 AM9/17/19
to jenkinsc...@googlegroups.com
Daniel Mercier commented on Bug JENKINS-59404
 
Re: High CPU usage in MapperBase.importFromFile in Benchmark Plugin

Hi Artalus,

How big is the history of  builds maintained for jobs?

When accessing the result page, the plugin  loads the  result files for all builds in a multithreaded fashion. At the time, it uses  all available threads minus a few to maintain standard operations. This loading depending on the size of the  history of builds  can reduce CPU availability on the computer until loading is complete. It was implemented to speed load the result page. I could add an option to select the number of threads being used.

I am currently traveling but I can look into implementing the change next week. Otherwise, if you have the time, you can create your own custom branch and submit your proposed change.

Let me know,

artalus-mail@yandex.ru (JIRA)

unread,
Sep 17, 2019, 8:37:01 AM9/17/19
to jenkinsc...@googlegroups.com

Hi Daniel. I am unsure that the problem we are experiencing is related to multithreaded file loading. There are several jobs with up to 1000 builds in history, but the ones that use Benchmark store about 100 builds. It seems to me that there are some thread getting stuck constantly reloading files – either this, or some files causing for (enContent : jsonObject.entrySet()) { ... this.builds.add(build);  in importFromFile() to loop much more times that it should.

daniel.mercier@autodesk.com (JIRA)

unread,
Sep 23, 2019, 3:01:02 PM9/23/19
to jenkinsc...@googlegroups.com

Hi Artalus,

Back to work. There is a good chance that the threads are kept alive because I forgot to close the InputStreamReader. The behavior would match your observations and it is a safe change to add anyway. All the Unit tests are passing so I am pushing the modification to version '1.0.11' .

I apologize for the impact it had on your system.

Let me know if you still observe the CPU overload.

Thank you,

Daniel

Reply all
Reply to author
Forward
0 new messages