Re: [Puppet Users] Puppet spuriously removing classes from classes.txt

[Josh Cooper]

Hi Matt,

On Fri, Oct 19, 2012 at 8:55 AM, Matt Carroll <oho...@gmail.com> wrote:
> I've been asked to look at a problem on an overseas rig whereby certain bits
> of config were going awry. This was down to the fact that they were querying
> the mcollective registration database and feeding back in to the configs,
> using queries based on class data. The mcollective + mongodb setup is
> working fine, but it would appear that the classes.txt file which
> mcollective reads to get configuration management classes is, on certain
> servers, spuriously being emptied of all classes other than 'settings'
>
> Seeing as the settings class is being left in, I can only assume that it
> must be puppet doing the modification. Has anyone ever seen behaviour like
> this before? It's truly bizarre, and even stranger is that after this has
> happened, sometimes the classes populate again. Every time I've run puppet
> manually with puppetd --test or run it --noop, it's run just fine and
> without noop it always seems to fix the problem, but later the node will
> show up as having only one class again and even later fix itself, so it's
> next to impossible to pin down.
>
> This is 2.6.7 on Ubuntu running against a centralised puppetmaster.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/puppet-users/-/zi9NP9ysFkcJ.
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.

There was this issue (http://projects.puppetlabs.com/issues/4518),
which was fixed in early 2.6. However, I think RI has said he's seen
this more recently. You may want to ask on the mcollective-users list
if you don't get a reply here.

Josh

--
Josh Cooper
Developer, Puppet Labs

[Matt Carroll]

Hmm, I'm not sure that this is the same issue... the issue details the classes.txt only containing a newline wheras I'm experiencing a problem whereby the file sometimes only contains the default class 'settings'. It appears as if the node is compiling spuriously against a different environment with an empty default class every now and again.

I'm going to look in to the possibility Of upgrading the rig, as it may fix the problem, and seeing as it's quite an old version it's not really too necessary to know exactly what was causing the behaviour.

[R.I.Pienaar]

depending on version it also writes resources.txt, so you can determine
if when classes.txt is empty if it really was not managing any resources.

Being someone who cares deeply for classes.txt, I've not observed this
behavior

[jcbollinger]

On Friday, October 19, 2012 10:55:03 AM UTC-5, Matt Carroll wrote:

I've been asked to look at a problem on an overseas rig whereby certain bits of config were going awry. This was down to the fact that they were querying the mcollective registration database and feeding back in to the configs, using queries based on class data. The mcollective + mongodb setup is working fine, but it would appear that the classes.txt file which mcollective reads to get configuration management classes is, on certain servers, spuriously being emptied of all classes other than 'settings'

Seeing as the settings class is being left in, I can only assume that it must be puppet doing the modification. Has anyone ever seen behaviour like this before? It's truly bizarre, and even stranger is that after this has happened, sometimes the classes populate again. Every time I've run puppet manually  with puppetd --test or run it --noop, it's run just fine and without noop it always seems to fix the problem, but later the node will show up as having only one class again and even later fix itself, so it's next to impossible to pin down.

This is 2.6.7 on Ubuntu running against a centralised puppetmaster.

Are you certain the behavior is buggy?  The fact that Puppet rights itself makes me think that Puppet itself is not the problem.

What if there is a cron job (or a human admin) that periodically does something like "puppetd --onetime --no-daemonize --tags settings"?  Or what if the affected node's manifests use schedules in a way that occasionally excludes all classes except 'settings'?  The former, at least, should show up in the system log.

You might also get some insight by wrapping puppetd in a script that makes a timestamped backup of classes.txt after each run.  Done right, that ought to establish whether Puppet is the one writing the unexpected content, and if so then the timestamps when the bad files appear might tell you something useful.  Perhaps you could dredge the same information out of your system logs, but the timestamped backups might be easier to analyze (and they would pinpoint where in your logs to look for additional information).


John

[Matt Carroll]

I'm not sure whether it's buggy behaviour or not, but by comparing the daemon.log and an inotify left running on the file I can confirm that it's puppet that's making the changes. Whether or not it's someone running puppet against a stupid environment or not remains to be seen, but I still need to find out what is going on...

Like I say, this is not directly my infrastructure or I'd have a better idea.