[JIRA] (JENKINS-56773) High master CPU usage introduced by Blue Ocean 1.14.0

17 views
Skip to first unread message

ojacques2@gmail.com (JIRA)

unread,
Mar 27, 2019, 5:38:02 AM3/27/19
to jenkinsc...@googlegroups.com
Olivier Jacques created an issue
 
Jenkins / Bug JENKINS-56773
High master CPU usage introduced by Blue Ocean 1.14.0
Issue Type: Bug Bug
Assignee: Unassigned
Attachments: image-2019-03-27-10-31-38-911.png, image-2019-03-27-10-35-23-276.png
Components: blueocean-plugin
Created: 2019-03-27 09:37
Environment: BlueOcean 1.14.0
Jenkins 1.168
Priority: Major Major
Reporter: Olivier Jacques

We went from BO 1.13.2 to BO 1.14.0.
After the upgrade, we have seen a major CPU increase on our Jenkins master, leading to really slow UI and job execution, with threads being locked - see screenshot.

Our master is configured to not run any job ("only build jobs with label expression matching this node").
Using Jenkins monitor, we saw that ~90% of the CPU time was consumed by calls to the rest endpoint /blue/rest/organizations/jenkins/computers - see screenshot.

Looking at the release notes from 1.14.0, it seems that JENKINS-52825 has introduced that endpoint and should be an area to investigate.

We had to rollback BO to 1.13.2 which does not exhibit that behavior.
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

ojacques2@gmail.com (JIRA)

unread,
Mar 27, 2019, 5:42:02 AM3/27/19
to jenkinsc...@googlegroups.com

gmogan@cloudbees.com (JIRA)

unread,
Apr 2, 2019, 4:04:01 PM4/2/19
to jenkinsc...@googlegroups.com

So had a bit of a chat with keith (original executor plugin author) and it was never really designed for more than a proof of concept on how to write a BlueOcean plugin.

We are pretty sure its caused by https://github.com/jenkinsci/blueocean-plugin/blob/e2dd04dcb3015063a2e31c8d37c90ee5c1ef4d98/blueocean-executor-info/src/main/js/ExecutorInfoService.js#L14 (which will grab executor info every time any pipeline does work)

So Plans:

  • I think the plan is going to be to drop making the executor plugin as a requirement, it'll have to be manually installed again (Sorry Craig)
  • Put a throttle on the lookup so it doesn't call nearly as often.

Since there is a work around in place until the next release, and I personally have a few pressing issues to deal with first, i'm going to leave it unassigned, but we'll get it fixed before we do the next release.

ojacques2@gmail.com (JIRA)

unread,
Apr 3, 2019, 5:18:01 AM4/3/19
to jenkinsc...@googlegroups.com

Thanks Gavin Mogan.

gmogan@cloudbees.com (JIRA)

unread,
Apr 8, 2019, 5:22:02 PM4/8/19
to jenkinsc...@googlegroups.com

devyn.goetsch@code42.com (JIRA)

unread,
Apr 16, 2019, 12:08:02 PM4/16/19
to jenkinsc...@googlegroups.com

This impacted our company and we had to do the same rollback.

we went from 1.6.x to 1.14.0, the next work day jenkins became unresponsive under load, and our jvm monitoring told us that the blue ocean rest computers endpoint was using most of the cpu time.

We rolled back to 1.13.2 and disabled the Blue Ocean Executor Info plugin and our Jenkins master is far more responsive and page loads for blue ocean are much lower.

Our builds have 3 stages, the first one is a single stage, the second one has 3 parallel stages, and the third one has 50 parallel stages, 32 of which are computed dynamically and launched with the "node" step.

Please revert the change to make the Blue Ocean Executor Info plugin a required dependency.  I'm sure it would be okay on a smaller system, but for a system like ours it is untenable.

gmogan@cloudbees.com (JIRA)

unread,
Apr 25, 2019, 8:05:06 PM4/25/19
to jenkinsc...@googlegroups.com
Gavin Mogan resolved as Fixed
 
Change By: Gavin Mogan
Status: Open Resolved
Resolution: Fixed
Released As: 1.15.0

gmogan@cloudbees.com (JIRA)

unread,
Apr 25, 2019, 8:05:07 PM4/25/19
to jenkinsc...@googlegroups.com
Gavin Mogan assigned an issue to Gavin Mogan
Change By: Gavin Mogan
Assignee: Gavin Mogan

sfri@widex.com (JIRA)

unread,
Jul 4, 2019, 4:19:03 AM7/4/19
to jenkinsc...@googlegroups.com
Søren Friis updated an issue
Change By: Søren Friis
Attachment: computers_GET.png

sfri@widex.com (JIRA)

unread,
Jul 4, 2019, 4:23:02 AM7/4/19
to jenkinsc...@googlegroups.com
Søren Friis commented on Bug JENKINS-56773
 
Re: High master CPU usage introduced by Blue Ocean 1.14.0

We are still seeing a very high amount of CPU time spend servicing the following endpoint on Blue Ocean 1.17.0

/blue/rest/organizations/jenkins/computers/

computers_GET.png

I tried blocking the endpoint in my browser, and everything seemed to function normally on Blue Ocean pages.
Can the polling of this endpoint be disabled or reduced?

jenkins@gavinmogan.com (JIRA)

unread,
Jul 4, 2019, 11:17:03 AM7/4/19
to jenkinsc...@googlegroups.com

disable the executor plugin, its no longer required

dbeck@cloudbees.com (JIRA)

unread,
Oct 8, 2019, 12:04:04 PM10/8/19
to jenkinsc...@googlegroups.com
Daniel Beck reopened an issue
 

Removing a plugin from the aggregator/meta plugin doesn't remove the broken functionality from any instances that have it installed.

Like ci.jenkins.io which got shit for the last several weeks for its recent stability problems. Imagine my surprise seeing dozens of threads hanging in the endpoint mentioned in this issue, and how this issue has been resolved.

At the very least, admins that generally take care of their instance need to be informed about this with an admin monitor.
Change By: Daniel Beck
Resolution: Fixed
Status: Resolved Reopened
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

mark.earl.waite@gmail.com (JIRA)

unread,
Jan 28, 2020, 7:46:03 AM1/28/20
to jenkinsc...@googlegroups.com
Mark Waite commented on Bug JENKINS-56773
 
Re: High master CPU usage introduced by Blue Ocean 1.14.0

As far as I can tell, https://github.com/jenkinsci/blueocean-plugin/releases/tag/blueocean-parent-1.21.0 includes this change. Thanks Jesse Glick for that pull request! I think this can be marked 'Closed' in release blue ocean 1.21.0 with a code fix to prevent the issue.

Can one of the users that encountered this problem confirm that it is now resolved?

mwos@sumologic.com (JIRA)

unread,
Mar 23, 2020, 10:13:04 AM3/23/20
to jenkinsc...@googlegroups.com

We use Blue Ocean and Blue Ocean Executor Info plugins in version 1.22.0 and still have a problem with 

```

/blue/rest/organizations/jenkins/computers/

```

endpoint. It is called very often and hangs for quite a long time. Not sure why informations about all executors is needed in the first place when opened blue ocean for a given job and a run. Those informations do not seem to be utilised anywhere in the view.
This message was sent by Atlassian Jira (v7.13.12#713012-sha1:6e07c38)
Atlassian logo

mwos@sumologic.com (JIRA)

unread,
Mar 23, 2020, 10:15:03 AM3/23/20
to jenkinsc...@googlegroups.com
Michał Woś edited a comment on Bug JENKINS-56773
We use Blue Ocean and Blue Ocean Executor Info plugins in version 1.22.0 and still have a problem with 

```

/blue/rest/organizations/jenkins/computers/

```

endpoint. It is called very often and hangs for quite a long ~5sec each time. Not sure why informations about all executors is needed in the first place when opened blue ocean for a given job and a run. Those informations do not seem to be utilised anywhere in the view.

mark.earl.waite@gmail.com (JIRA)

unread,
Mar 23, 2020, 10:23:04 AM3/23/20
to jenkinsc...@googlegroups.com

Michał Woś sorry that you've encountered the problem. Per the comment from Gavin Mogan, the e Blue Ocean Executor Info plugin is no longer required. You might be able to resolve the issue on your installation by removing the Blue Ocean Executor Info plugin. It is not installed on ci.jenkins.io (likely because Daniel Beck removed it). I was able to remove it from my Jenkins instance without seeing any harmful side effects.

I agree with Daniel that it would be much better to have an administrative monitor or a Jenkins Health Advisor warning that Blue Ocean Executor Info is not required and may cause performance issues.

jenkins@gavinmogan.com (JIRA)

unread,
Apr 20, 2020, 2:55:03 PM4/20/20
to jenkinsc...@googlegroups.com
Gavin Mogan assigned an issue to Unassigned
 
Change By: Gavin Mogan
Assignee: Gavin Mogan
Reply all
Reply to author
Forward
0 new messages