Lock file /var/lib/puppet/state/puppetdlock

[Keith Edmunds]

I'm just starting a roll out of Puppet and I'm seeing a problem on maybe
25% of client nodes. The symptoms are that the clients stop updating. In
the Puppetmaster log, I'm seeing things like:

Feb 9 20:10:23 vs4 puppetmasterd[17942]: Compiled catalog for xxxx in
0.05 seconds
Feb 9 20:40:41 vs4 puppetmasterd[17942]: Compiled catalog for xxxx in
0.05 seconds
Feb 9 21:11:16 vs4 puppetmasterd[17942]: Compiled catalog for xxxx in
1.83 seconds
Feb 9 21:41:37 vs4 puppetmasterd[17942]: Compiled catalog for xxxx in
0.91 seconds

These are all for the same client; everything appears normal until 21:41,
then no more checks from the client (it's now 10:17 on Feb 10).

On the client, I tried running puppetd manually:

# puppetd -t
notice: Lock file /var/lib/puppet/state/puppetdlock exists; skipping
catalog run

A look at the lock file:

# ls -l /var/lib/puppet/state/puppetdlock
-rw-r--r-- 1 root root 5 2009-02-09 22:11 /var/lib/puppet/state/puppetdlock

...shows that it was probably created at the next run after the last one
logged on the Puppetmaster (above).

Looking at the lock file:

# echo $(cat /var/lib/puppet/state/puppetdlock)
32400
# ps -fp 32400
UID PID PPID C STIME TTY TIME CMD
root 32400 1 0 Feb06 ? 00:01:41 ruby /usr/sbin/puppetd -w 5

...shows that the puppetd is still running.

Why would the lock file be created and not subsequently deleted?

If it helps, it is likely that the Puppetmaster was very busy at that
time, but even so I would expect the client to deal with that graciously.

Maybe related, maybe not: I can't stop puppetd in the usual way:

# /etc/init.d/puppet stop
Stopping puppet configuration management tool.
# ps -fp 32400
UID PID PPID C STIME TTY TIME CMD
root 32400 1 0 Feb06 ? 00:01:41 ruby /usr/sbin/puppetd -w 5

If I 'kill -9' the puppetd process, remove the /var/run/puppetd.pid file
and remove the lock file, I can restart puppetd and it runs OK for a
while, but eventually the puppetdlock file causes this problem again.

Versions: 0.24.5-3, the Debian Lenny package compiled for Debian Etch.

Grateful for any suggestions / pointers / etc.

Thanks,
Keith

[Ohad Levy]

maybe not related but similar problem... Does puppet changes your time
(ntp)? If the time changes while puppet is running it could lead for a
very long time that puppet is waiting....

[Keith Edmunds]

> Does puppet changes your time
> (ntp)? If the time changes while puppet is running it could lead for a
> very long time that puppet is waiting....

No, although all servers are running NTP. I don't think that's the
problem: the problem is the stale puppetdlock.

Thanks for the suggestion -
Keith

[Luke Kanies]

I haven't seen this for absolute ages, so whatever used to cause it is
unlikely to be the problem now.

Is there maybe a signal being sent to puppetd? If not, any chance you
can get a stack trace from the clients? You'd have to leave a
client running in the foreground with stdout redirected.

Otherwise.... I've no idea, because you're the first to run into it
that I know of.

Note that you should be able to run 'puppetd --enable' to remove that
stale lock file.

--
One of the keys to happiness is a bad memory. -- Rita Mae Brown
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com

[Paul Lathrop]

On Mon, Feb 16, 2009 at 7:39 PM, Luke Kanies <lu...@madstop.com> wrote:
> Otherwise.... I've no idea, because you're the first to run into it
> that I know of.
>
> Note that you should be able to run 'puppetd --enable' to remove that
> stale lock file.

I periodically run into this issue as well; it only happens on our Xen
VMs. Not sure if that helps.

I just have a cron job that runs puppetd --enable on these systems.

--Paul

[Jeff Adams]

We have had this happening occasionally (approx. 1 out of ~120 VMs,
every other week or so) on our OpenVZ VMs as well.

We're running Debian Etch, and over the past week upgraded to 0.24.5-3,
so I'm keeping an eye on things to see if that resolved it for us.

- Jeff

[Keith Edmunds]

On Mon, 16 Feb 2009 21:39:55 -0600, lu...@madstop.com said:

> Note that you should be able to run 'puppetd --enable' to remove that
> stale lock file.

Old thread revisited. Some clients occasionally don't clear the lockfile
in the subject. A couple of people have suggested, as above, that "puppetd
--enable" should clear it. It doesn't:

# ls -l /var/lib/puppet/state/puppetdlock
-rw-r--r-- 1 root root 5 2009-03-20 14:50 /var/lib/puppet/state/puppetdlock
# puppetd --enable
# ls -l /var/lib/puppet/state/puppetdlock
-rw-r--r-- 1 root root 5 2009-03-20 14:50 /var/lib/puppet/state/puppetdlock
#

My only recourse now would seem to be a cron job that checks the age of
the puppetdlock and restarts the puppet client if the lockfile is older
than a given age.

This is with:

# puppetd --version
0.24.5
#

Any ideas as to why this is happening?

Thanks,
Keith