Puppet/Nagios/PuppetDB slow performance

[Jonathan Gazeley]

Hi all,

For months now I've been using Puppet with PuppetDB backend to manage my
Nagios configs. We now have 1200+ services being checked and including
servicedependency, host, hostdependency, command and other Nagios
resources being managed, that's about 3000 resources in total.
Unfortunately this means the monitoring server takes about 8 minutes to
perform a Puppet run.

I'm not exactly sure what the bottleneck is - whether it's doing 3000
calls to PuppetDB (although the load on the PuppetDB server and its
Postgres database are low), or whether it is parsing nagios_service.cfg
that causes the slowdown. It's about 500kb and 11,000 lines.

I know it is possible to squirt the Nagios config into several smaller
files when the resources are collected on the monitoring server, but I
gather this also means that Nagios resources cannot be purged if the
default file locations are not used. The ability to purge old configs
automatically is essential to our environment.

So does anyone have any suggestions on how to improve the performance of
Puppet when handling large Nagios configs?

Cheers,
Jonathan

[Ken Barber]

Its probably the nagios resource, its well known for not scaling when
the files contain a lot of entries. That would explain why your
PuppetDB is showing low load - as its probably not the bottleneck.

The most performant work-around I've seen for this is that you can use
the nagios_* resources to export, but then use custom collectors that
query PuppetDB (using Dalen's puppetdbquery for example:
https://github.com/dalen/puppet-puppetdbquery) that generates custom
templates for your nagios_*.cfg files. If nagios_service.cfg is the
main bottleneck, you can focus on this one to start with.

This way you continue to use the model of the nagios_* resources, but
at collection time you improve performance.

ken.

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users...@googlegroups.com.
> To post to this group, send email to puppet...@googlegroups.com.
> Visit this group at http://groups.google.com/group/puppet-users?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Alexander Bien]

On 24.05.2013 16:58, Ken Barber wrote:
> Its probably the nagios resource, its well known for not scaling when
> the files contain a lot of entries. That would explain why your
> PuppetDB is showing low load - as its probably not the bottleneck.

I second that. We have mysql storeconfigs, about 5000 Services, 2,5 MB
nagios_* files and >50k lines of config. The puppet run takes aprox 45
minutes. I have heard people on irc say that its not uncommon.

- alex

[David Schmitt]

You can cut quite a bit off that time by using templates and files (one
per service/host) and purging on the base directory.

Regards, David

[Jonathan Gazeley]

Hi David,

Sorry to reply to a stale thread. Please could you explain a bit more on
your solution? Presumably I need to pas some extra arguments to my
spaceship operator that collects the Nagios resources, but how is the
directory-level purging done?

Thanks,
Jonathan

[David Schmitt]

I'm doing it this way:

file {
"/etc/nagios/conf.d":
ensure => directory,
source => "puppet:///modules/nagios/empty",
ignore => [ '.gitkeep' ],
recurse => true,
force => true,
purge => true;
}

File<<| tag == 'nagios::service' |>>

define nagios::service(...) {
@@file {
"/etc/nagios/conf.d/${host}_service_${name}":
content => template("nagios/service.erb"),
tag => 'nagios::service',
...;
}

This way you collect a minimal number of simple resources. This still is
dog slow when you have a couple of thousand services, but it sure beats
the heck out of parsing the complete nagios config for each nagios_*
resource.



Regards, David

[Chris Handy]

If you are looking to implement it via the puppetdbquery module that Ken was referring to earlier, here is some code to do so with.  I recently implemented this approach with my own setup and it worked out great.  About 2500 nagios_service resources went from taking 20-30 minutes to about 2.  Only issue i ran into was with puppet dashboard when the size of the report log became too big to save for one run.  I had to alter the mysql DB columns from text to mediumtext and now its all good.

https://gist.github.com/richardc/4714701