Crontab overwritten by Puppet

[Kmbu]

Hello folks..

I suddenly found the crontab on one of my puppet clients overwritten. It had many entries, including one created by Puppet. Suddenly during one run it was overwritten, keeping only the Puppet entry and removing everything else. Nothing changed between the previous runs and the one that re-created the crontab in terms of configuration. I'm running Puppet 2.7.6.

Fri Jul 06 22:00:16 +0200 2012 Puppet (notice): Reopening log files
Fri Jul 06 22:03:58 +0200 2012 Puppet (notice): Finished catalog run in 5.34 seconds
Fri Jul 06 22:10:15 +0200 2012 Puppet (notice): Reopening log files
Fri Jul 06 22:11:18 +0200 2012 Puppet (notice): Finished catalog run in 5.30 seconds
Fri Jul 06 22:20:15 +0200 2012 Puppet (notice): Reopening log files
Fri Jul 06 22:23:05 +0200 2012 Puppet (notice): Finished catalog run in 5.81 seconds
Fri Jul 06 22:30:15 +0200 2012 Puppet (notice): Reopening log files
Fri Jul 06 22:35:04 +0200 2012 Puppet (notice): Finished catalog run in 5.92 seconds
Fri Jul 06 22:40:15 +0200 2012 Puppet (notice): Reopening log files
Fri Jul 06 22:44:35 +0200 2012 /Stage[main]/Cre-base/Cron[puppet-run]/ensure (notice): created
Fri Jul 06 22:44:39 +0200 2012 Puppet (notice): Finished catalog run in 5.54 seconds

Why would one run suddenly do this?

Regards,

[Felix Frank]

Hi,

On 07/09/2012 03:48 PM, Kmbu wrote:
> Fri Jul 06 22:44:35 +0200 2012
> /Stage[main]/Cre-base/Cron[puppet-run]/ensure (notice): created
> Fri Jul 06 22:44:39 +0200 2012 Puppet (notice): Finished catalog run in
> 5.54 seconds
>
> Why would one run suddenly do this?

Are you absolutely certain that nothing unrelated deleted the whole
crontab in the meantime?

Only scenario I can really think of here.

Regards,
Felix

[Kmbu]

Hi,

Thanks for supporting. We've been running this environment of around 1000 servers for at least 5 years and have never seen a crontab suddenly disappear before. We've only had Puppet in place for a month or so.

Regards,

[Romeo Theriault]

On Tue, Jul 10, 2012 at 10:41 PM, Kmbu <yum...@hotmail.com> wrote:
> Hi,
>
> Thanks for supporting. We've been running this environment of around 1000
> servers for at least 5 years and have never seen a crontab suddenly
> disappear before. We've only had Puppet in place for a month or so.
>
> Regards,

Unfortunately, I've seen the same issue occur. I posted to the list
about it a while back and had some others say they'd seen similiar.
I've only seen this happen on Solaris so far but seeing it happen once
was enough for me to pull the plug on using puppet to manage users
crontab files on all our boxes, solaris and rhel.

On RHEL I've worked around the issue by dropping crontab files in
/etc/cron.d/ but on solaris I have no viable work-around at the
moment.

Others have mentioned that I should just manage the whole crontab
file in puppet... but this isn't really an option for me at the
moment.

On solaris at least the issue seemed to be related to facter hanging
(due to the picld daemon hanging).

Not that this really solves your problem but if your on a *nix that
supports it you might want to look at converting to dropping cron
files in /etc/cron.d/ or similar.

--
Romeo

[Kmbu]

Thanks for the corroboration. At least I know I'm not crazy.

This needs to be reported as a bug. I'm not sure how to go about it, though.

It seems you know more about this than I do. Do you think you could report it?

On Wednesday, 11 July 2012 22:35:21 UTC+2, Romeo Theriault wrote:

[Stefan Schulte]

It looks like prefetching (executing �crontab -l root� to get current
cronentries) failed in some way. As a result puppet's in-memory
representation of the crontab is empty. When puppet now evaluates your
Cron['puppet-run'] resource, it detects it to be out of sync (is absent,
should be present), so the in-memory representation now contains only
your 'puppet-run' cronjob. If the new crontab is written back to disk,
you will loose every other cronentry.

Normally I would expect an error message if prefetching failed:

Could not prefetch cron provider

but that doesn't seem to be the case here. On the other hand there is an
outstanding bug about failures beeing silently ignored on solaris [1]

So a failure when running �crontab -l root� (do you manage crontabs of
other users as well? You may hit [2] in that case) would at least
explain the log output but the interesting question now is:
Can you think of a situation *why* the command failed?

[1] http://projects.puppetlabs.com/issues/14283
[2] http://projects.puppetlabs.com/issues/5752

[Kmbu]

Hi,

Thanks for the explanation. It's definitely not [2]. Only one user's crontab is being managed via Puppet, and it's not root. However, there are other jobs there which disappear when this happens. I really need a fix for this! If "crontab -l" fails, I would rather that Puppet just ignored the cron job configuration. This should be a paramter or something.

Now I have to come in to the office every morning wondering if any of my cron jobs have disappeared..

Regards,

It looks like prefetching (executing �crontab -l root� to get current

cronentries) failed in some way. As a result puppet's in-memory
representation of the crontab is empty. When puppet now evaluates your
Cron['puppet-run'] resource, it detects it to be out of sync (is absent,
should be present), so the in-memory representation now contains only
your 'puppet-run' cronjob. If the new crontab is written back to disk,
you will loose every other cronentry.

Normally I would expect an error message if prefetching failed:

    Could not prefetch cron provider

but that doesn't seem to be the case here. On the other hand there is an
outstanding bug about failures beeing silently ignored on solaris [1]

So a failure when running �crontab -l root� (do you manage crontabs of