Catalog compilation performance issues in Puppet 3

[Cesar]

Hi all,

we have some performance issues with Puppet 3.3 catalog compilation (we are using Passenger to host Puppet master processes).

It looks like a single Puppet master process is being used for the catalog compilation, and importing the catalogs is taking ~3 minutes.

Apr 19 10:21:14 ieatlms3841 puppet-master[6700]: Handling request: GET /production/node/ieatrcxb3742
Apr 19 10:21:15 ieatlms3841 puppet-master[6700]: Using cached facts for ieatrcxb3742
Apr 19 10:21:15 ieatlms3841 puppet-master[6700]: Caching node for ieatrcxb3742
Apr 19 10:21:15 ieatlms3841 puppet-master[6700]: node supports formats: pson b64_zlib_yaml yaml raw
Apr 19 10:21:15 ieatlms3841 puppet-master[6700]: PROFILE [69862410013880] 1.1 Rendered result in Puppet::Network::Format[pson]: took 0.0014 seconds
Apr 19 10:21:15 ieatlms3841 puppet-master[6700]: PROFILE [69862410013880] 1.2 Sent response: took 0.0000 seconds
Apr 19 10:21:15 ieatlms3841 puppet-master[6700]: PROFILE [69862410013880] 1 Processed request GET /production/node/ieatrcxb3742: took 0.0159 seconds
...

Apr 19 10:21:20 ieatlms3841 puppet-master[6700]: PROFILE [69862429450080] 1.1 Setup server facts for compiling: took 0.0172 seconds
Apr 19 10:21:20 ieatlms3841 puppet-master[6700]: 'replace facts' command for ieatrcxb3742 submitted to PuppetDB with UUID 9a23c445-5203-4987-9faf-a187ac7dd0b3
Apr 19 10:21:20 ieatlms3841 puppet-master[6700]: PROFILE [69862429450080] 1.2 Found facts: took 0.1696 seconds
Apr 19 10:21:20 ieatlms3841 puppet-master[6700]: Using cached facts for ieatrcxb3742
Apr 19 10:21:20 ieatlms3841 puppet-master[6700]: Caching node for ieatrcxb3742
Apr 19 10:21:20 ieatlms3841 puppet-master[6700]: PROFILE [69862429450080] 1.3 Found node information: took 0.0133 seconds
Apr 19 10:21:20 ieatlms3841 puppet-master[6700]: importing '/etc/puppet/manifests/plugins/ieatlms3841.pp' in environment production
Apr 19 10:21:27 ieatlms3841 puppet-master[6700]: importing '/etc/puppet/manifests/plugins/ieatrcxb3739.pp' in environment production
Apr 19 10:21:42 ieatlms3841 puppet-master[6700]: importing '/etc/puppet/manifests/plugins/ieatrcxb3740.pp' in environment production
Apr 19 10:21:51 ieatlms3841 puppet-master[6700]: importing '/etc/puppet/manifests/plugins/ieatrcxb3742.pp' in environment production
Apr 19 10:21:56 ieatlms3841 puppet-master[6700]: importing '/etc/puppet/manifests/plugins/ieatrcxb3743.pp' in environment production
Apr 19 10:22:05 ieatlms3841 puppet-master[6700]: importing '/etc/puppet/manifests/plugins/ieatrcxb3744.pp' in environment production
Apr 19 10:22:11 ieatlms3841 puppet-master[6700]: importing '/etc/puppet/manifests/plugins/ieatrcxb3755.pp' in environment production
...

Apr 19 10:24:03 ieatlms3841 puppet-master[6700]: importing '/etc/puppet/manifests/plugins/ieatrcxb5894.pp' in environment production
Apr 19 10:24:06 ieatlms3841 puppet-master[6700]: importing '/etc/puppet/manifests/plugins/ieatrcxb6430.pp' in environment production
Apr 19 10:24:14 ieatlms3841 puppet-master[6700]: PROFILE [69862429450080] 1.4.1 Compile: Set node parameters: took 0.0025 seconds
Apr 19 10:24:14 ieatlms3841 puppet-master[6700]: PROFILE [69862429450080] 1.4.2 Compile: Created settings scope: took 0.0295 seconds
Apr 19 10:24:14 ieatlms3841 puppet-master[6700]: PROFILE [69862429450080] 1.4.3 Compile: Evaluated main: took 0.0083 seconds

...

Apr 19 10:24:29 ieatlms3841 puppet-master[6700]: (Firewall[1018 vmmonitord ipv6]) Adding default for require
Apr 19 10:24:29 ieatlms3841 puppet-master[6700]: PROFILE [69862429450080] 1.4.7 Compile: Finished catalog: took 0.1885 seconds
Apr 19 10:24:31 ieatlms3841 puppet-master[6700]: PROFILE [69862429450080] 1.4 Compiled catalog for ieatrcxb3742 in environment production: took 191.6076 seconds
Apr 19 10:24:31 ieatlms3841 puppet-master[6700]: Compiled catalog for ieatrcxb3742 in environment production in 191.61 seconds
Apr 19 10:24:31 ieatlms3841 puppet-master[6700]: Caching catalog for ieatrcxb3742

I was wondering whether there is any way to reduce catalog compilation? Could the catalog compilation be split across different Puppet master processes or something like that?

Thanks,

Cesar.

This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail.

[James Pryor]

Cesar,

For general catalog compilation time improvements, you might be best served by using Puppet Server ( https://docs.puppet.com/puppetserver/1.2/services_master_puppetserver.html )  instead of passenger.

In 2014, my team & I performed our own sort of benchmark tests of the puppet master server. From https://indico.cern.ch/event/320819/contributions/742968/ download the PDF
https://indico.cern.ch/event/320819/contributions/742968/attachments/619020/851687/Configuration_Management_Change_Management-James-Pryor.pdf

and please see pages 32 - 37 that show the load on the single puppet master that was running Apache + passenger.

Approximately six months later 2015, another test was done on the same hardware with the same software, but with the replacement of Apache & Passenger with Puppet Server 1.x. From https://indico.cern.ch/event/346931/contributions/817788/ download the PDF https://indico.cern.ch/event/346931/contributions/817788/attachments/684633/940417/Configuration_Management_at_the_RACF.pdf
and check pages 3 - 8 and see that compilation times were effectively cut in half.

So in summary, in an apples-to-apples comparison on the same hardware, Puppet Server 1.x is close to twice as fast as Apache & Passenger.

Now there are some differences between Puppet Server & Passenger which is documented here:
https://docs.puppet.com/puppetserver/1.2/puppetserver_vs_passenger.html

Now all that said, Puppet 3.x has been End of Life-ed as of 2017-01-01 ( https://groups.google.com/forum/#!topic/puppet-announce/tSe_4KTS7t0 ). You should consider working toward a migration path from Puppet 3.3 to the latest Puppet 3.8.x , and then look toward testing with Puppet 4.x, and either land on Puppet 4.x or maybe even wait for Puppet 5.x (which is hinted at coming in 2017 https://groups.google.com/d/msg/puppet-dev/-H1pHJM6NLE/II3X8Cw6AwAJ).

https://docs.puppet.com/puppet/4.10/index.html#upgrading-from-puppet-3
https://docs.puppet.com/puppet/4.10/upgrade_major_pre.html

Regards,

James

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/c04da5d1-bb08-46a1-ab95-ffe3e52069a2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Cesar]

Hi James,

thanks for your thorough answer.

We definitely have in mind that we should upgrade Puppet to 4.X to get all the benefits of Puppet Server, but until that happens we were wondering whether there was a simpler way to boost performance on the Puppet master until we can perform an upgrade to Puppet 4.x

Thanks,

Cesar.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

[Ramin K]

On 4/20/17 3:32 AM, Cesar wrote:
>
> I was wondering whether there is any way to reduce catalog compilation?
> Could the catalog compilation be split across different Puppet master
> processes or something like that?

- upgrade to 3.8.x, definitely performances gains to be had.

- Passenger 5/5.1 and tune for the CPU/workload,
https://ask.puppet.com/question/13433/how-should-i-tune-passenger-to-run-puppet/

- Run Ruby 2.1 on the master. Performance in Ruby 1.8.7 was terrible.
Expect a 30-50% bump for each version you jump from 1.8.7, 1.9.3, 2.0,
and 2.1.

Ruby version is the performance limiter in Puppet 3.x rather than Puppet
3 vs 4 or the app server at least in puppet-server 1.x. It's unfortunate
that *every* benchmark I've seen comparing Puppet 3/Passenger vs Puppet
4/JVM was done with Ruby 1.8.7 including the benchmarks Cern did. IME
running modern Ruby gets you very very close to Puppet server
performance. It also helps you migrate to Puppet 4 if you have custom
functions, types/providers, etc which aren't Ruby 1.9+ ready.

Ramin

[Cesar]

Hi Ramin,

thanks for those suggestions! 

I didn't know Ruby was so important there. We do use 1.8.7 (and had some issues while developing providers and report processors because it is such and old version), as it was the officially supported version for Red Hat 6. I think we have a big margin there for improvement, so I guess that will be our first move before upgrading to Puppet Server.

Cesar.