inspect resources that are already added to a manifest

[Luke Bigum]

Hi all,

Does anyone know of a way to inspect resources that are already parsed in a node's manifest during catalog compilation? This would certainly need some serious Ruby Fu.

As an example, imagine I have a number of arbitrary files defined by multiple classes and it is impossible to get an all encompassing list of these files:

file { 'woof': }

file { 'cows': }

file { 'meow': }

...

$all_files = inline_template(...)

I would like to be able to gather those file names into a Puppet variable - this would be parse order dependent. It would be fantastic if it could handle exported resources that have just been collected as well.

Thanks,

-Luke

[jcbollinger]

On Saturday, December 22, 2012 12:20:10 PM UTC-6, Luke Bigum wrote:

Hi all,

Does anyone know of a way to inspect resources that are already parsed in a node's manifest during catalog compilation? This would certainly need some serious Ruby Fu.

This is a bad idea.  If your the Puppet circuits in your brain didn't trip over "inspect", they certainly should have sounded the alarm over "serious Ruby Fu".  You are fighting against the tool.

 

As an example, imagine I have a number of arbitrary files defined by multiple classes and it is impossible to get an all encompassing list of these files:

file { 'woof': }

file { 'cows': }

file { 'meow': }

...

$all_files = inline_template(...)

I would like to be able to gather those file names into a Puppet variable - this would be parse order dependent. It would be fantastic if it could handle exported resources that have just been collected as well.

And "parse-order dependent"?  Of course it is.  You need a Puppet-bogometer.

So what configuration objective are you actually trying to accomplish here?  There is likely a more robust, less Rubyriffic way to accomplish it.


John

[Matthias Viehweger]

Hi John!

On Wed, Jan 02, 2013 at 07:51:37AM -0800, jcbollinger wrote:
> You need a Puppet-bogometer.

I need one, too! Where can I order it?

Cheers,
Matthias Viehweger
--
Serververwaltung und Softwareentwicklung

https://www.heute-kaufen.de
Prinzessinnenstraße 20 - 10969 Berlin

[Luke Bigum]

Ohh don't worry, John, my bogometer was going off like crazy, the needle almost broke ;-)

I'm taking shortcuts in my spare time with a tool that's probably 70% right for the job. It's for monitoring - I really like the idea of a Puppet module to describe or advertise how to monitor itself, it keeps them very self contained.

Just a bit more on this - I generally see three categories of monitoring tools. Ones that are configured separately from your CRM and end up being a source of truth on their own are in my mind the worst. The next level up are ones either defined from or derived from your CRM. The best are auto-discovery, but they cost an absolute fortune. I'm trying to move my team from the first one to the second one with as little "new tools" as possible, which is where the "70% right for the job" comment comes from.

I'm using exported resources to describe how modules are monitored. The problem is that exported resources are not the equivalent of raw information passing. So when I want to start doing trickier things like group and analyse what is collected, exported resources don't cut it because it's not what they are designed for.

Specifically what I was trying to do was collect exported resources of the same type and group them on the monitoring server. There is no predefined list of service names anywhere (unless you parse the node definitions) so that's why I wanted to go from resource collection to Array of Strings. A colleague has managed to reduce my 300 lines to 50 though so the need for craziness is reduced somewhat. We still need to do the "Export a File" trick and run a script on the monitoring server to build the complex configuration that exported resources are not designed to handle.

The next iteration of this work might be to scrap resource collection in favour of querying PuppetDB directly to figure out what to monitor, but that's a lot more work than I'm prepared to do at this stage. Maybe in a few months... ;-)

-Luke

[jcbollinger]

On Thursday, January 3, 2013 4:08:54 AM UTC-6, Luke Bigum wrote:

I'm taking shortcuts in my spare time with a tool that's probably 70% right for the job. It's for monitoring - I really like the idea of a Puppet module to describe or advertise how to monitor itself, it keeps them very self contained.

Just a bit more on this - I generally see three categories of monitoring tools. Ones that are configured separately from your CRM and end up being a source of truth on their own are in my mind the worst. The next level up are ones either defined from or derived from your CRM. The best are auto-discovery, but they cost an absolute fortune. I'm trying to move my team from the first one to the second one with as little "new tools" as possible, which is where the "70% right for the job" comment comes from.

I'm using exported resources to describe how modules are monitored. The problem is that exported resources are not the equivalent of raw information passing. So when I want to start doing trickier things like group and analyse what is collected, exported resources don't cut it because it's not what they are designed for.

Specifically what I was trying to do was collect exported resources of the same type and group them on the monitoring server. There is no predefined list of service names anywhere (unless you parse the node definitions) so that's why I wanted to go from resource collection to Array of Strings. A colleague has managed to reduce my 300 lines to 50 though so the need for craziness is reduced somewhat. We still need to do the "Export a File" trick and run a script on the monitoring server to build the complex configuration that exported resources are not designed to handle.

The next iteration of this work might be to scrap resource collection in favour of querying PuppetDB directly to figure out what to monitor, but that's a lot more work than I'm prepared to do at this stage. Maybe in a few months... ;-)

Querying PuppetDB in fact sounds like a good approach.  One alternative I see would be to extend the main Puppet engine.  Possibly that could take the form of a new Puppet face (which, now that I think of it, might just end up a front end to the DB query).

You could also inspect the master's catalog cache instead of hooking directly into compilation.  That would give you all the information you want, without parse-order issues.  You could monitor the cache directory for changes to make the system nearly as immediate as what you you're doing now.

Or if you have a list of known systems to monitor, then you can generate catalogs for them at any time via the "puppet catalog" face.


John

[Peter Brown]

You have made me curious.

I am going to make some wild assumptions and probably leap to some incorrect conclusions.

Querying PupptDB seems like the wrong way to achieve this in my opinion.

I put my monitoring (and firewall rules incidentally) into each class on a node and import into my monitoring server (nagios currently but intend on switching to icinga soon) them based on tags.

I was using global vars but are currently rewriting my modules to use heira to set the monitoring server and a few other settings.

This is working nicely for me.

I had a few different ideas along the way but my current iteration is getting close to awesome.

It also gives me fine grained control over whether a node gets sms alerts or escalation and such.

Is this the kind of thing you are attempting to achieve?

I can provide extra complexity to how it works if you need it.

The next iteration of this work might be to scrap resource collection in favour of querying PuppetDB directly to figure out what to monitor, but that's a lot more work than I'm prepared to do at this stage. Maybe in a few months... ;-)

-Luke

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/OLpl0Bx1q5kJ.

To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

[jcbollinger]

On Sunday, January 6, 2013 9:23:17 PM UTC-6, Pete wrote:

I put my monitoring (and firewall rules incidentally) into each class on a node and import into my monitoring server (nagios currently but intend on switching to icinga soon) them based on tags.

I was using global vars but are currently rewriting my modules to use heira to set the monitoring server and a few other settings.

This is working nicely for me.

I had a few different ideas along the way but my current iteration is getting close to awesome.

It also gives me fine grained control over whether a node gets sms alerts or escalation and such.

Something along those lines would be my preferred way to do it if I were willing and able to modify the classes involved.  My inference from Luke's comments was that he wanted to avoid doings so, and perhaps that he was specifically looking for a solution that did not rely on the cooperation of the modules involved.  That would be an eminently reasonable objective where third-party modules are in the picture, but not an unreasonable one even where all modules were built in-house.


John

[Peter Brown]

I wrote most of the modules in my setup myself and that makes it easier.

I am attempting to be tricky with my monitoring modules and make them extensible so adding extra monitoring classes should be pretty simple.

I am also intending on releasing them when I am happy with them.

 

John

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/L5WxTz43T8sJ.