Puppetmaster performance degradation after update to 3.8.2

205 views
Skip to first unread message

Adam Winberg

unread,
Sep 16, 2015, 6:48:16 AM9/16/15
to puppet...@googlegroups.com
Upgraded from 3.7.3 to 3.8.2 and have after that had very slow compilation and file serving from the puppet master. We're running our puppetmaster with apache/passenger, ca. 900 hosts. Complete puppet runs earlier took 10-15s and after update take 30-300s. Very unstable performance. 

Even running with just a few clients the runtimes are really slow. Decreasing the 'PassengerMaxPoolSize' makes the runs faster, but then it cant cope with an increased load. 

I have also tried with the jvm puppetserver with roughly the same performance (or lack thereof). I noticed the same pattern, if i set 'max-active-instances' in puppetserver.conf to 2, I get acceptable performance with a handful of clients, but with increased load it doesnt cope. If I raise the 'max-active-instances' the performance is even worse. 

This is running on an vmware server, 8 cpu and 12GB RAM. 

Cant find anything in the release notes that explain this performance drops, does anyone recognize the problem?

I have now downgraded to 3.7.3 and my puppet runs are as fast as ever again. 

Any help appreciated. 

Regards
Adam 


Josh Cooper

unread,
Sep 16, 2015, 7:02:44 PM9/16/15
to puppet...@googlegroups.com
Hi Adam,

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAFCL3HvvEC4HA4xD655sAPMBMznVdsqcQCYp5FHn%3D_SQ8TG2qg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

I would take a look at the `--profile` option as described in https://puppetlabs.com/blog/tune-puppet-performance-profiler, making sure to update config.ru as needed.

Josh

--
Josh Cooper
Developer, Puppet Labs

PuppetConf 2015 is right around the corner! Join us October 5-9 in Portland, OR. Register now.

Adam Winberg

unread,
Sep 17, 2015, 1:03:50 AM9/17/15
to puppet...@googlegroups.com
I actually tried the --profile option but never got it to actually record any 'PROFILE' log data. The logdest and debug flags works as expected, but no 'PROFILE' entries in the logfile.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CA%2Bu97umgvfrNbJFeSRk8Rvnxkys-MAaSHEQKCeKqETy1OrXG1w%40mail.gmail.com.

Adam Winberg

unread,
Oct 13, 2015, 9:34:30 AM10/13/15
to Puppet Users
ok, so I have been tinkering a bit more with this and I my problems start when I upgrade to puppet 3.7.5. So 3.7.4 works fine, but 3.7.5 and 3.8.x is showing some really terrible performance for me.

Running puppetserver (jvm) with 3.7.4, 9 jruby instances and 12G Xmx I get really good performance, catalog compilation times ranging from 1-3s. Upgrade to 3.7.5, restart puppetserver, and the compilation takes forever (40-90s) which in turn causes the server to get overloaded with puppet agents which causes a lot of failed runs. There only error messages in the logs are
"java.nio.channels.WritePendingException: null"
which probably is some timeout error caused by the excessive load on the server.

I can't see anything in the release notes for 3.7.5 that would explain this, but someone out there is sure to have a better eye for that (I hope).

I also added this to https://tickets.puppetlabs.com/browse/SERVER-819, since it seems to be a similar problem even though I do not use puppetdb. I'm hoping that someone out there can think of any reason for my problems.

Regards
Adam



On Thursday, September 17, 2015 at 7:03:50 AM UTC+2, Adam Winberg wrote:
I actually tried the --profile option but never got it to actually record any 'PROFILE' log data. The logdest and debug flags works as expected, but no 'PROFILE' entries in the logfile.
On Thu, Sep 17, 2015 at 1:02 AM, Josh Cooper <jo...@puppetlabs.com> wrote:
Hi Adam,

On Wed, Sep 16, 2015 at 3:47 AM, Adam Winberg <adam.w...@gmail.com> wrote:
Upgraded from 3.7.3 to 3.8.2 and have after that had very slow compilation and file serving from the puppet master. We're running our puppetmaster with apache/passenger, ca. 900 hosts. Complete puppet runs earlier took 10-15s and after update take 30-300s. Very unstable performance. 

Even running with just a few clients the runtimes are really slow. Decreasing the 'PassengerMaxPoolSize' makes the runs faster, but then it cant cope with an increased load. 

I have also tried with the jvm puppetserver with roughly the same performance (or lack thereof). I noticed the same pattern, if i set 'max-active-instances' in puppetserver.conf to 2, I get acceptable performance with a handful of clients, but with increased load it doesnt cope. If I raise the 'max-active-instances' the performance is even worse. 

This is running on an vmware server, 8 cpu and 12GB RAM. 

Cant find anything in the release notes that explain this performance drops, does anyone recognize the problem?

I have now downgraded to 3.7.3 and my puppet runs are as fast as ever again. 

Any help appreciated. 

Regards
Adam 


--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.

I would take a look at the `--profile` option as described in https://puppetlabs.com/blog/tune-puppet-performance-profiler, making sure to update config.ru as needed.

Josh

--
Josh Cooper
Developer, Puppet Labs

PuppetConf 2015 is right around the corner! Join us October 5-9 in Portland, OR. Register now.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.

Adam Winberg

unread,
Oct 15, 2015, 3:33:47 PM10/15/15
to puppet...@googlegroups.com
for posterity - a new puppetlabs ticket was created for this, https://tickets.puppetlabs.com/browse/PUP-5380, where the root cause of the problem is pinpointed. 

Hi Adam,

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

I would take a look at the `--profile` option as described in https://puppetlabs.com/blog/tune-puppet-performance-profiler, making sure to update config.ru as needed.

Josh

--
Josh Cooper
Developer, Puppet Labs

PuppetConf 2015 is right around the corner! Join us October 5-9 in Portland, OR. Register now.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/95f54835-40bf-4b9e-9ece-e9c0187876be%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages