Client updates

[Douglas Garstang]

I probably waste quite a bit of time each week restarting the puppet client, and logging systems, and tailing the messages file, to see if my puppet changes worked. Is there a better way? How do people normally do this?

Sure, I can use puppetrun or mcollective to cause puppet to do a config run on the client, but I still need to log in and look at the messages file. The reports that the clients generate, and which are available on the server contain, I believe yield enough information to determine if there was any syntax errors or not. I guess they contain more, but, even though they are yaml files, they contain ruby objects, which non ruby parsers won't read. Would have been nice if it was standard yaml.

Anyway.... ideas?

Doug.

[Brian Cully]

On May 25, 2011, at 1:12 PM, Douglas Garstang wrote:

> I probably waste quite a bit of time each week restarting the puppet client, and logging systems, and tailing the messages file, to see if my puppet changes worked. Is there a better way? How do people normally do this?

I use monit to keep the puppet agent running in case it craps out, and puppet dashboard to keep an eye on all my nodes.

-bjc

[Douglas Garstang]

Looking for a non GUI option.

Doug.

[Brian Gupta]

Basically the ways I know of:

1) Don't run puppet as a daemon, but run it out of cron every X mins.
2) Setup a cronjob that checks if puppet is running and restart it if not.
3) Setup a nagios job that checks to see if puppet is running
4) Presuming you are managing your puppet code in some sort of version
control system, you can "publish" the latest timestamp of your latest
commit, and also push that out via puppet, and setup a nagios check
that compares them and if they ever are behind by more than X mins you
can crit on it. You could take this a step further and have nagios
automatically restart puppet if they are behind, but if you think it
might be a syntax error, I would probably skip that. (As an aside you
should do syntax checking before committing. I added the commands to
do so below)
5) Foreman (or dashboard) can monitor puppet runs, and flag any broken runs.

Check syntax:
puppet --noop --parseonly memcached.pp

Check erb syntax:
erb -x -T '-' memcached-sessions.erb |ruby -c

We use a combination of 2,3,4 and 5.

Cheers,
Brian

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>

[Nathan Clemons]

Mcollective with mc-puppet should work if you patch it to not daemonize (you'll need to adjust the timeouts as well, of course). Errors running will bubble up to the list of nodes that failed the run.

--

Nathan Clemons

http://www.livemocha.com

The worlds largest online language learning community

[R.I.Pienaar]

----- Original Message -----
> Basically the ways I know of:
>
> 1) Don't run puppet as a daemon, but run it out of cron every X mins.
> 2) Setup a cronjob that checks if puppet is running and restart it if
> not.
> 3) Setup a nagios job that checks to see if puppet is running
> 4) Presuming you are managing your puppet code in some sort of
> version
> control system, you can "publish" the latest timestamp of your latest
> commit, and also push that out via puppet, and setup a nagios check
> that compares them and if they ever are behind by more than X mins
> you
> can crit on it. You could take this a step further and have nagios
> automatically restart puppet if they are behind, but if you think it
> might be a syntax error, I would probably skip that. (As an aside you
> should do syntax checking before committing. I added the commands to
> do so below)
> 5) Foreman (or dashboard) can monitor puppet runs, and flag any
> broken runs.

6) use the last run summary

# cat /var/lib/puppet/state/last_run_summary.yaml
---
time:
group: 0.000852
last_run: 1306343446
class: 0.002389
yumrepo: 0.037513
service: 2.300486
schedule: 0.002291
cron: 0.001797
config_retrieval: 18.9993901252747
total: 37.9802451252747
package: 0.371819
filebucket: 0.00032
file: 15.664177
exec: 0.593368
user: 0.005843
resources:
total: 384
events:
total: 0
changes:

[Douglas Garstang]

On Wed, May 25, 2011 at 10:22 AM, Brian Gupta <brian...@brandorr.com> wrote:

Basically the ways I know of:

1) Don't run puppet as a daemon, but run it out of cron every X mins.
2) Setup a cronjob that checks if puppet is running and restart it if not.
3) Setup a nagios job that checks to see if puppet is running
4) Presuming you are managing your puppet code in some sort of version
control system, you can "publish" the latest timestamp of your latest
commit, and also push that out via puppet, and setup a nagios check
that compares them and if they ever are behind by more than X mins you
can crit on it. You could take this a step further and have nagios
automatically restart puppet if they are behind, but if you think it
might be a syntax error, I would probably skip that. (As an aside you
should do syntax checking before committing. I added the commands to
do so below)
5) Foreman (or dashboard) can monitor puppet runs, and flag any broken runs.

Check syntax:
puppet --noop --parseonly memcached.pp

Check erb syntax:
erb -x -T '-' memcached-sessions.erb |ruby -c

We use a combination of 2,3,4 and 5.

I must have phrased my question wrong, because it seems to have been misinterpreted. We're pretty much doing everything you suggested, with the exception of Foreman, which I don't like.

I just need a way to confirm that the changes I made to the config were executed successfully. I also need to know if other changes snuck in somehow, and everything has gone south for one reason or another. The only way I can think of to do this, is to actually force puppet to do a run, and then eyeball /var/log/messages immediately.

Doug.

 

[Douglas Garstang]

On Wed, May 25, 2011 at 10:26 AM, Nathan Clemons <nat...@livemocha.com> wrote:

Mcollective with mc-puppet should work if you patch it to not daemonize (you'll need to adjust the timeouts as well, of course). Errors running will bubble up to the list of nodes that failed the run.

Failed.... both for syntax errors AND dependency errors? Don't know what you mean by 'bubble up'.

Doug.

 

[Kinzel, David]

>-----Original Message-----
>From: puppet...@googlegroups.com
>[mailto:puppet...@googlegroups.com] On Behalf Of Brian Cully
>Sent: Wednesday, May 25, 2011 11:16 AM
>To: puppet...@googlegroups.com
>Subject: Re: [Puppet Users] Client updates
>
>On May 25, 2011, at 1:12 PM, Douglas Garstang wrote:
>
>> I probably waste quite a bit of time each week restarting
>the puppet client, and logging systems, and tailing the
>messages file, to see if my puppet changes worked. Is there a
>better way? How do people normally do this?

We log to a central syslog server, and send out a daily email of any
changes/failures.

>
> I use monit to keep the puppet agent running in case it
>craps out, and puppet dashboard to keep an eye on all my nodes.

We also use storedconfigs and can just query the compiled time to see if
any clients stopped reporting.

>
>-bjc

>
>--
>You received this message because you are subscribed to the
>Google Groups "Puppet Users" group.
>To post to this group, send email to puppet...@googlegroups.com.
>To unsubscribe from this group, send email to
>puppet-users...@googlegroups.com.
>For more options, visit this group at
>http://groups.google.com/group/puppet-users?hl=en.
>
>

This email communication and any files transmitted with it may contain confidential and or proprietary information and is provided for the use of the intended recipient only. Any review, retransmission or dissemination of this information by anyone other than the intended recipient is prohibited. If you receive this email in error, please contact the sender and delete this communication and any copies immediately. Thank you.
http://www.encana.com

[Douglas Garstang]

We're using puppet 2.6.4, and we don't have that file...

[fre] root@prov01 ~:# ls -l /var/lib/puppet/state

total 56

-rw-r--r-- 1 root root   112 Apr 17 21:18 classes.txt

drwxr-xr-x 2 root root  4096 Apr 10 17:14 graphs

-rw-rw---- 1 root root 47421 May 25 17:25 state.yaml

[fre] root@prov01 ~:# 

Doug.

 

[Nathan Clemons]

Mcollective hides successes and only shows you failures, keeping the signal to noise ratio very high.

If you run puppet without daemonizing, anything that causes the config to not be applied successfully is going to show up as an error.

--

Nathan Clemons

http://www.livemocha.com

The worlds largest online language learning community

[Douglas Garstang]

On Wed, May 25, 2011 at 10:35 AM, Nathan Clemons <nat...@livemocha.com> wrote:

Mcollective hides successes and only shows you failures, keeping the signal to noise ratio very high.

If you run puppet without daemonizing, anything that causes the config to not be applied successfully is going to show up as an error.

Yes, but then, when there is an error, the daemon stops running. Assuming you had nagios checking for that, you've still got to wait for the nagios alert to find out, and in the mean time, the guy who is on call this week, who gets the texts is going to be even more pissed than he already is for getting alerts at 3am. 

Doug.

 

[Brian Gupta]

What do you mean by "executed successfully"? Assuming I understand
your question correctly, if it is untested code, and it passes a
syntax checker, and doesn't generate errors, you pretty much need to
spot check at least one one system that the code in question does what
you intended. Many folks do this in a dev environment, before pushing
it to their puppetmaster. I don't really see a way around this that
wouldn't end up being a lot more work than it's worth.

[Nathan Clemons]

It depends on how you use it. In our environment, we don't leave the Puppet daemon running. We only do on-demand runs... that way changes only happen when we push them out.

Your mileage obviously may vary, but if you're looking for tight control, that's something to consider.

--

Nathan Clemons

http://www.livemocha.com

The worlds largest online language learning community

[Douglas Garstang]

Executed successfully =

a) No syntax errors

b) No dependency errors

c) No other weird stuff

Just because there are no dependency errors on one system, doesn't mean that there won't be dependency errors on another system. No two systems are ever identical, so spot checks won't cover all bases.

We're a startup with limited resources, with the typical total lack of resources. We don't have the luxury of a dev environment. We have 4 data centres, different rules for each, and one is PCI compliant, which is akin to having a root canal done every time you touch it. Even if we did, that doesn't preclude making sure everything ran ok. It's still needed.

Doug.

[Disconnect]

Foreman can email error reports automatically. (Without having to use the GUI..) IIRC you can also make it email actions (changes) - if not, it shouldn't be that hard to add.

We use nagios to check puppet's state file for age - if it is too old, puppet hasn't run. Between those two, afaict, we're covering basically everything. And once a day, foreman also emails a status summary ('interesting' hosts - changes, failures, outdated/no report, etc)

Puppet tries to run, finds a problem: Foreman emails the report.
Puppet runs and makes changes, no errors: Do you want email? It'll show on the 'active hosts' page, and in the nightly summary email.
Puppet doesn't run: Nagios alerts on that host

And FWIW I'm the only sysadmin (also at a startup w/ no dev environment for puppet) and its really not impossible to get identical servers. (In fact, our first answer to "wtf is wrong with server21-foo" is "meh, whatever, trash the vm and recreate it".. Well.. thats first after "lets make sure it didn't get hacked" of course..) My workload went -way- down once puppet was working properly (complete manifests) and even more when I got foreman's vm deployments going.

On Wed, May 25, 2011 at 1:48 PM, Douglas Garstang <doug.g...@gmail.com> wrote:

Executed successfully =

a) No syntax errors

b) No dependency errors

c) No other weird stuff

Just because there are no dependency errors on one system, doesn't mean that there won't be dependency errors on another system. No two systems are ever identical, so spot checks won't cover all bases.

We're a startup with limited resources, with the typical total lack of resources. We don't have the luxury of a dev environment. We have 4 data centres, different rules for each, and one is PCI compliant, which is akin to having a root canal done every time you touch it. Even if we did, that doesn't preclude making sure everything ran ok. It's still needed.

Doug.

[Brian Gupta]

Douglas,

I have to concur with disconnect, now that we are using Foreman, many
of the things we setup in the past to catch these things are now
redundant. I don't know why you "don't like" Foreman, but I have to
say along with our initial decision to use puppet, and managing our
configs with version control, adopting Foreman stands up there as one
of our big infrastructure management wins. (And we aren't even
using Foreman's provisioning capabilities).

May I ask why you don't like Foreman? (We were for a long time
hesitant to use Foreman, as it wasn't an "official" project, but we
had reached a point where we needed a new ENC, and a technical
evaluation of both Foreman and Dashboard lead us to the realization
that functionality-wise it was a lot closer to what we needed. (And I
suspect your needs may be similar to ours.)

Cheers,
Brian

[Doug Warner]

We use the tagmail report to catch reports tagged "Err" and email that to us.
This in addition to "last run" nagios checks seem to cover the bases for us.
We run puppet as a daemon.

-Doug

[Douglas Garstang]

On Wed, May 25, 2011 at 1:19 PM, Brian Gupta <brian...@brandorr.com> wrote:

Douglas,

I have to concur with disconnect, now that we are using Foreman, many
of the things we setup in the past to catch these things are now
redundant. I don't know why you "don't like" Foreman, but I have to
say along with our initial decision to use puppet, and managing our
configs with version control, adopting Foreman stands up there as one
of our big infrastructure management wins. (And we aren't even
using Foreman's provisioning capabilities).

May I ask why you don't like Foreman? (We were for a long time
hesitant to use Foreman, as it wasn't an "official" project, but we
had reached a point where we needed a new ENC, and a technical
evaluation of both Foreman and Dashboard lead us to the realization
that functionality-wise it was a lot closer to what we needed. (And I
suspect your needs may be similar to ours.)

Well, for one, I can't import my modules. I thought I'd take another shot at it, and the documentation says to run this to import your modules:

rake puppet:import:puppet_classes RAILS_ENV=production

Our puppet isn't in a standard location, it's in /etc/puppet/common, so, the above commands barfs. How can I change the default ?

Doug.

 

[Mark Stanislav]

Hmm, I don't think that's the issue. Where are you running that command?

Try...

cd /usr/share/foreman && rake puppet:import:puppet_classes RAILS_ENV=production

YMMV depending on where foreman is; works for me that way.

-Mark

[Jake - USPS]

Maybe if you edit /foreman-location/config/settings.yaml and
uncomment :modulepath and set it to your modulepath and then restart
the webserver? This is just an educated guess as I use a standard
location, but that seems like a good start. Otherwise, if you have
more questions about foreman you can check http://groups.google.com/group/foreman-users

Regards,
Jake

On May 26, 12:14 pm, Douglas Garstang <doug.garst...@gmail.com> wrote:

[Douglas Garstang]

On Thu, May 26, 2011 at 10:19 AM, Jake - USPS <jacob.m...@usps.gov> wrote:

Maybe if you edit /foreman-location/config/settings.yaml and
uncomment :modulepath and set it to your modulepath and then restart
the webserver?  This is just an educated guess as I use a standard
location, but that seems like a good start.  Otherwise, if you have
more questions about foreman you can check http://groups.google.com/group/foreman-users

Thanks, I was able to import the modules.

However, it really isn't clear from the documentation how you Foreman can act as a front-end to external nodes without managing them. I don't want Foreman to manage external nodes. It has no revision control, which is a deal breaker in a production environment. I therefore still want to have our external node files in the same place, have those as the source of truth, and have Foreman read those. 

There's other weird things going on with the GUI too... like when adding a new host manually, the drop downs are all empty.

Doug.

[Disconnect]

Most of that is a documentation gap - you need to go under settings (in the new version - far-right dropdown in new and old) and populate the various forms there. Things like distributions, host groups, etc.. The #foreman channel on freenode or the mailing list are both really good resources.