Open source puppet performance monitoring/tuning?

[dkoleary]

I think I've taken my open source puppet server to the limits; but, I'd like to be able to confirm that and find out what resources are needed.

Short version of the problem: When we first started, puppet agent runs would take 5-10 seconds. Now, while we still have a limited number of modules, we're looking at 1400+ systems under management. Most runs still take in the 5-10 second range but a growing number are taking 400+ seconds to complete.

Short version of the question: How do I know what needs to be tuned or examined?

An example: the first paragraph of https://docs.puppet.com/pe/latest/config_puppetserver.html says "Tuning JRuby on Puppet server". What would be the symptom that suggests JRuby needs tuning?

An ISBN for a book would be great. I find all sorts of books on learning puppet but not too many on tuning it.

Any hints or suggestions will be greatly appreciated.

Doug O'Leary

[Chuck Anderson]

On Thu, Jan 12, 2017 at 12:46:11PM -0800, dkoleary wrote:
> Short version of the problem: When we first started, puppet agent runs would take 5-10 seconds. Now, while we still have a limited number of modules, we're looking at 1400+ systems under management. Most runs still take in the 5-10 second range but a growing number are taking 400+ seconds to complete.
>
> Short version of the question: How do I know what needs to be tuned or examined?

Do you use Augeas? I found that limiting Augeas with lens, incl, and
context improves performance drastically:

augeas { "ipaddr-$dev":
lens => 'Shellvars.lns',
incl => "/etc/sysconfig/network-scripts/ifcfg-${dev}",
context => "/files/etc/sysconfig/network-scripts/ifcfg-${dev}"

[Garrett Honeycutt]

Hi Doug,

The times for agent runs typically do not correspond to any Puppet
related tuning. The places that the agent comes into contact with the
master during a puppet run, once they have received the catalog is file
serving, filebucketing changed files and sending a report.

Metrics to look for are how long does it take your puppet master to
compile a catalog and are any of your agents timing out when requesting
catalogs.

Try checking out last_run_summary.yaml for figuring out which resources
take the most time during the agent run. Typically I see poorly
configured name servers and requests outside of the network as ways in
which the agent will take longer to run.

This thread might also help.

https://ask.puppet.com/question/3/where-can-i-find-information-about-sizing-for-puppet-servers/

Best regards,
-g

--
Garrett Honeycutt
@learnpuppet
Puppet Training with LearnPuppet.com
Mobile: +1.206.414.8658

[dkoleary]

Hey;

thanks for the reply.  I do use augeas sparsely.  Turns out that's not the issue but your suggestion is very intriguing.  I'm going to be trying that out shortly.

Thanks

Doug

[dkoleary]

Hi, Garret;

Your suggestion to review last_run_summary.yaml results in one huge step in the right direction.  

# grep file: /opt/puppetlabs/puppet/cache/state/last_run_summary.yaml

  file: 121.817228425

So, how do I go about finding out which file resource(s) are causing the problem?

Thanks a lot for the tip.  Appreciate it immensely.

Doug O'Leary

[R.I.Pienaar]

----- Original Message -----
> From: "dkoleary" <dkol...@olearycomputers.com>
> To: "puppet-users" <puppet...@googlegroups.com>
> Sent: Friday, 13 January, 2017 19:43:20
> Subject: Re: [Puppet Users] Open source puppet performance monitoring/tuning?

> Hi, Garret;
>
> Your suggestion to review last_run_summary.yaml results in one huge step in
> the right direction.
>
> # grep file: /opt/puppetlabs/puppet/cache/state/last_run_summary.yaml
> file: 121.817228425
>
> So, how do I go about finding out which file resource(s) are causing the
> problem?

You can use https://github.com/ripienaar/puppet-reportprint to report on slow
things

See the sample report.

I think it'll work with Puppet 4, it's been a while, open an issue if it doesnt

[dkoleary]

Thanks; That looks like a pretty cool too; unfortunately, I can't seem to get it to run.  

Centos 6.8 fully patched:

$ ./report_print.rb 

./report_print.rb:17:in `require': no such file to load -- puppet (LoadError)

        from ./report_print.rb:17

Some googling showed a 'gem install' command but that results in:

# gem install puppet

ERROR:  Error installing puppet:

        puppet requires Ruby version >= 1.9.3.

Any tips?

Thanks again.

Doug O'Leary

[R.I.Pienaar]

----- Original Message -----
> From: "dkoleary" <dkol...@olearycomputers.com>
> To: "puppet-users" <puppet...@googlegroups.com>
> Sent: Friday, 13 January, 2017 20:52:30
> Subject: Re: [Puppet Users] Open source puppet performance monitoring/tuning?

> Thanks; That looks like a pretty cool too; unfortunately, I can't seem to
> get it to run.
>
> Centos 6.8 fully patched:
>
> $ ./report_print.rb
> ./report_print.rb:17:in `require': no such file to load -- puppet
> (LoadError)
> from ./report_print.rb:17

if you're on puppet 4 you'll need to do:

/opt/puppetlabs/puppet/bin/ruby report_print.rb

basically you have to use the ruby that puppet uses

[dkoleary]

Excellent!  Thank you very much.  That will come in handy.

Doug

[Garrett Honeycutt]

Thank you RI! This is really helpful and it works great with Puppet v4.