Jira (PUP-10225) Puppetserver 6 performance regression

27 views
Skip to first unread message

Iain Buclaw (JIRA)

unread,
Jan 8, 2020, 3:57:03 AM1/8/20
to puppe...@googlegroups.com
Iain Buclaw created an issue
 
Puppet / Bug PUP-10225
Puppetserver 6 performance regression
Issue Type: Bug Bug
Affects Versions: PUP 6.7.2
Assignee: Unassigned
Attachments: puppetserver-contextswitches.png, puppetserver-cpu.png, puppetserver-load.png
Created: 2020/01/08 12:56 AM
Fix Versions: PUP 5.y
Priority: Normal Normal
Reporter: Iain Buclaw

Puppet Version: 6.11.1
Puppet Server Version: 6.7.2
OS Name/Version: Ubuntu/16.04

High load/cpu usage on puppetserver in version 6.

Desired Behavior: Same as puppet 5.

Actual Behavior: 6x more context switches, 6x higher load average for the same amount of traffic/requests as when running puppetserver 5.

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
Atlassian logo

Iain Buclaw (JIRA)

unread,
Jan 8, 2020, 3:58:03 AM1/8/20
to puppe...@googlegroups.com
Iain Buclaw updated an issue
Change By: Iain Buclaw
*Puppet Version: 6.11.1*
*Puppet Server Version: 6.7.2*
*OS Name/Version: Ubuntu/16.04*


High load/cpu usage on puppetserver in version 6.

*Desired Behavior:* Same as puppet 5.

*Actual Behavior:* 6x more context switches, 6x higher load average for the same amount of traffic/requests as when running puppetserver 5.

Attaching host graphs that show before/after for the upgrade from puppetserver 5.3.10 to 6.7.2

!puppetserver-load.png|thumbnail!

!puppetserver-contextswitches.png|thumbnail!

!puppetserver-cpu.png|thumbnail!

Iain Buclaw (JIRA)

unread,
Jan 8, 2020, 3:59:03 AM1/8/20
to puppe...@googlegroups.com
Iain Buclaw updated an issue
Change By: Iain Buclaw
Method Found: Needs Assessment Other (please specify in description)

Josh Cooper (JIRA)

unread,
Jan 8, 2020, 7:17:02 PM1/8/20
to puppe...@googlegroups.com

Iain Buclaw (JIRA)

unread,
Jan 9, 2020, 3:18:03 AM1/9/20
to puppe...@googlegroups.com
Iain Buclaw updated an issue
Change By: Iain Buclaw
Attachment: puppet-agent-runtimes.png

Iain Buclaw (JIRA)

unread,
Jan 9, 2020, 3:18:03 AM1/9/20
to puppe...@googlegroups.com
Iain Buclaw commented on Bug PUP-10225
 
Re: Puppetserver 6 performance regression

If it's of any use, there are currently 222 servers being managed by this puppetserver, the agents are running once every 30 minutes. PuppetDB is present, as well Foreman.

Looking at top, it's only the puppetserver process that is consuming high CPU resources.

It's hard to tell, however puppet agent run times have increased between 25-50% across the board, for instance on one node it went from a pretty consistent 9 seconds config retrieval; 25 seconds runtime, to an erratic 8-14 seconds config retrieval, 28-50 seconds runtime. However that started occurring only after the agents were updated from version 5.5.17 to 6.11.1.

Iain Buclaw (JIRA)

unread,
Jan 9, 2020, 7:29:03 AM1/9/20
to puppe...@googlegroups.com
Iain Buclaw updated an issue
Change By: Iain Buclaw
Attachment: puppet5-breakdown.png

Iain Buclaw (JIRA)

unread,
Jan 9, 2020, 7:29:03 AM1/9/20
to puppe...@googlegroups.com
Iain Buclaw updated an issue
Change By: Iain Buclaw
Attachment: puppet6-breakdown.png

Iain Buclaw (JIRA)

unread,
Jan 13, 2020, 9:32:04 AM1/13/20
to puppe...@googlegroups.com
Iain Buclaw commented on Bug PUP-10225

After three days uptime, puppetserver appears to stall and doesn't take any more new connections.

 

Noticed by `puppet-agent -t` processes that had been running for over an hour on all servers.

Iain Buclaw (JIRA)

unread,
Jan 29, 2020, 11:17:04 AM1/29/20
to puppe...@googlegroups.com
Iain Buclaw commented on Bug PUP-10225

Mitigated the problem by changing the following:

 

  • /etc/default/puppetserver:

 

-JAVA_ARGS="-Xms512M -Xmx5g"
+JAVA_ARGS="-Xms2048M -Xmx3g -XX:ReservedCodeCacheSize=1G

  • /etc/puppetlabs/puppetserver/conf.d/puppetserver.conf

-    max-active-instances: 6
+    max-active-instances: 3

Also independently increased puppetdb max memory as well:

  • /etc/default/puppetdb

-JAVA_ARGS="-Xmx384m"
+JAVA_ARGS="-Xmx1G"

 

 

Henrik Lindberg (JIRA)

unread,
Jan 29, 2020, 12:37:04 PM1/29/20
to puppe...@googlegroups.com

There are other reports about performance issues - I just added a link to one such issue that has just been fixed. Others are in the pipeline.
There has been some extensive firefighting on performance issues lately and what you report are hopefully caused by what has been found and fixed. Checkout release notes as new releases are rolled out...

Nick Walker (JIRA)

unread,
Feb 5, 2020, 1:02:03 PM2/5/20
to puppe...@googlegroups.com
Nick Walker commented on Bug PUP-10225

Iain Buclaw is everything stable after you changed your java args? If so shall we close this out or do you have any new information to share?

Iain Buclaw (JIRA)

unread,
Feb 6, 2020, 3:55:04 AM2/6/20
to puppe...@googlegroups.com
Iain Buclaw commented on Bug PUP-10225

Nick Walker - everything is stable, is there a recommended tunings page where this can be put in?

Nick Walker (JIRA)

unread,
Feb 6, 2020, 10:33:04 AM2/6/20
to puppe...@googlegroups.com

Maggie Dreyer (Jira)

unread,
Nov 9, 2020, 5:14:03 PM11/9/20
to puppe...@googlegroups.com

The important bit for this was likely configuring codecahce. That is included in the tuning page you linked. I'm going to resolve this.

This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)
Atlassian logo
Reply all
Reply to author
Forward
0 new messages