testing and exported (nagios) resources

[Jason Antman]

Hello,

I have 3 puppet stacks (master, puppetdb, enc) - dev, test/qa and prod.
Dev is used for initial development and testing of code (including
puppet), which is then promoted to test and then prod.

I'd like to start using the nagios types to configure monitoring, via
exported resources (yes I'm aware of the issues with the builtins, but
they'll have to do for now). I only have one Nagios server, and I'd like
to reliably monitor at least some stuff on the dev and test puppet
nodes. So... setting up all 3 puppet stacks to export resources that are
realized somehow on the Nagios server isn't a possibility, as bad
manifests/modules could affect the monitoring of one of the dev or test
hosts.

What's the safe way to "freeze" exported resources, or prevent them from
being changed? The best that I can come up with so far is to have the
nagios server connected to the production puppetmaster, and when I want
to update the (exported resource) monitoring configuration for one of
the dev or test nodes, have to do a one-time run on each node in
question against the prod puppet master.

Any other thoughts or theories?

Thanks,
Jason Antman

[Felix Frank]

Hi,

I must be missing an essential piece here.

All three of your puppet stack nodes must be present in each instance,
no? The production master manages all three masters, normally. To change
monitoring of either of them, you update the production manifests.

Of course, if you implement some new monitoring feature on the dev
master, you must have that node run puppet against its local dev master
to export resources, then the nagios server also against the dev master
to import them. But that is just the usual dev workflow, I assume.

So I suspect things aren't so simple for you. I just don't see in what
manner.

Thanks in advance for any clarification,
Felix

[Jason Antman]

On 12/04/2013 08:22 AM, Felix Frank wrote:
> Hi,
>
> I must be missing an essential piece here.
>
> All three of your puppet stack nodes must be present in each instance,
> no? The production master manages all three masters, normally. To change
> monitoring of either of them, you update the production manifests.

What do you mean by "present in each instance"? Each stack is
self-contained - it has its own ENC, PuppetDB, and Master. Each stack's
master manages itself and its stack, out of a "production" environment
(git master). The logic behind this is that if the production stack
suffers an outage (i.e. hardware failure) the ENC data is imported to
the test stack, and nodes are seamlessly moved over. Yes, I'm aware of
the tradeoff that certain things aren't constrained by environment, and
bad code in one environment on the dev or test masters could bring down
that stack.

I'm not just worried about monitoring the puppet stack itself, I'm also
worried about monitoring the nodes. I.e. the dev stack has a bunch of
VMs that exist to test code, and they need to be monitored in Nagios.

>
> Of course, if you implement some new monitoring feature on the dev
> master, you must have that node run puppet against its local dev master
> to export resources, then the nagios server also against the dev master
> to import them. But that is just the usual dev workflow, I assume.

Yeah, that's understood. But what about the production monitoring? I'd
need to run all of the nodes in the environment against the production
master to actually export the nagios configs to the nagios server... or
else I'd need (what I'm asking about) some way of exporting the Nagios
configs from the dev and test masters to the Nagios server, but only for
one environment (broken in puppet... exported resources don't work this
way) or only when manually requested...

[Felix Frank]

Ah, so this is the meat of your problem. I think I get it now.

Tough call. Seeing as running multiple puppetdbs in parallel is not
really an expected scenario, there is quite possibly no way to share the
required resources between them.

Are nagios resources being actively removed from your nagios server when
they have not been exported to the puppet master being queried? I would
have thought that they just cease to be in management and left alone by
puppet.

[Jeff Bachtel]

Three thoughts. The first would be to have node definitions for monitor on both dev and test environments, doing a minimal amount of work to generate nagios host definitions in a disjoint directory that you include in your nagios config.

So:

/var/lib/{puppet,puppet-dev,puppet-test}

/etc/{puppet,puppet-dev,puppet-test}

/etc/nagios/object/{dev,test}

To further isolate your nagios box from harm, dev and test environment runs can be done from an unprivileged user and puppet agent runs can be tied to host/service additions/removals in dev/test. I know this idea directly violates you saying "So... setting up all 3 puppet stacks to export resources that are realized somehow on the Nagios server isn't a possibility, as bad manifests/modules could affect the monitoring of one of the dev or test hosts." but it seems the least-harm least-gross way to do this.

The other way that came to mind was an aperiodic dump/insert of relevant postgresql tables relating to exported resources from dev/test into the production postgresql puppetdb. This would require investigating the schema in use, and cleanup could get tricky.

The third way that came to mind was to use the inventory service http://docs.puppetlabs.com/guides/inventory_service.html to loop over hostnames, GET'ing yaml from dev/test and PUT'ing it onto the production server. I don't know how deletions would be handled, there, or even what you'd want your failure mode to be.

Jeff

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/529D363C.4030202%40jasonantman.com.
For more options, visit https://groups.google.com/groups/opt_out.