Generating monitoring from PuppetDB without exported resources

[Luke Bigum]

Hello,

Is anybody doing (or know of someone doing) any advanced parsing of Puppet resources from PuppetDB, perhaps for the purpose of generating config for centralised monitoring, or, even acceptance/integration tests?  The traditional way is to use Exported Resources, but I've been toying with the idea of bypassing that and building config straight off data in PuppetDB.  I'm looking for people who may be doing this, tried something similar, or anyone interested in bouncing ideas around.

Cheers,

-Luke

[Tim Skirvin]

Luke Bigum <Luke....@lmax.com> writes:

> Is anybody doing (or know of someone doing) any advanced parsing of Puppet
> resources from PuppetDB, perhaps for the purpose of generating config for
> centralised monitoring, or, even acceptance/integration tests?

*wave* What are you trying to do? We populate our monitoring
(check_mk) with puppetdb data, primarily based off of system role. It
works wonderfully.

- Tim Skirvin (tski...@fnal.gov)
--
HPC Systems Administrator / Developer http://www.linkedin.com/in/tskirvin
USCMS-T1 Collaboration Fermilab ECF

[Angel L. Mateo]

El 15/03/18 a las 18:44, Luke Bigum escribió:

Maybe you could use pql to query puppetdb and manually create
monitoring configuration objects.

But I don't see the point for this complexity.

--
Angel L. Mateo Martínez
Sección de Telemática
Área de Tecnologías de la Información
y las Comunicaciones Aplicadas (ATICA)
http://www.um.es/atica
Tfo: 868889150
Fax: 868888337

[Luke Bigum]

I guess I'm not 100% on what I'm trying to do yet, nor am I sure it's a good idea or too complicated... Which is why I'm asking what other people do :-)

I already bypass exporting and realising resources for our Nagios service checks.  This was a performance enhancement - we've got 10s of 1000s of Nagios checks per server, and realising all those resources into Ruby objects was really slow (this is back before PuppetServer).  Instead we have a template making a PuppetDB API call, getting back a blob of JSON and parsing that into Nagios Service definitions.  It is querying Defined Type resources from Puppet though so it's pretty easy to parse into Nagios:

  url4 = 'http://puppet:8081/pdb/query/v4/resources/Nagios::Config::Service'

However that requires that we add a Nagios::Config::Service resource into a Puppet catalog somewhere in order to get a check.  Some part of me thinks this is a bit wasteful... Here's a simple contrived example: if I was monitoring PuppetLabs Apache::Vhosts, I would have two resources in a catalog:

  apache::vhost { 'foo.com': }

  nagios::config::service { 'https_check_foo.com': }

Why do I need the second resource if all the information I need is already in the first resource?  Could I not just parse the PuppetDB data looking for Apache::Vhost resources directly?  That would mean I wouldn't have to have a Profile of my own code to add my own monitoring resource.  If I had something that could do that and generate Nagios config, perhaps it wouldn't be too hard to extend it to generate boilerplate Goss or ServerSpec config for acceptance testing, the same way Puppet Retrospec does for unit tests...  The monitoring config would be somewhat decoupled from my Puppet runs, I could change the way checks are defined without a Puppet Agent catalog compilation needing to occur.

There are two big disadvantages I can see. If the interface of Apache::Vhost changes, the generated monitoring breaks with it.  The second is that any complicated monitoring that requires an extra package or script to be installed on a machine is going to be defined in Puppet anyway, so moving the check definitions out of Puppet in order to avoid wasteful code doesn't make much sense any more.  I think I've just talked myself out of it :-)

Thoughts?