Agents still running but no runs

[Julien Cornuwel]

Hi,

Recently, I've had some nodes show as 'unresponsive' in dashboard.
'puppet kick' from the puppetmaster fails with exit code 3 (already running).
Same thing if I run puppet locally.
On the servers, the puppet process is running but the logs show no
activity since the last run.
A simple restart solves the problem but it comes back later.

I found a bug report but it was specific to a kernel version. Here,
the problematic nodes have a variety of kernel and OS versions :
- SLES 11, puppet 2.7.14, kernel 2.6.32.12
- OpenSUSE 12.1, puppet 2.7.14, kernel 3.1.0
- OpenSUSE 11.1, puppet 2.7.6, kernel 2.6.27.7
- Debian 5, puppet 2.7.6, kernel 2.6.26

Puppetmaster is running on Debian 6, puppet 2.7.18, kernel 2.6.32-5-amd64.
I've been changing stuff in Puppet this week, so I suppose it's
related. But I can't figure out what.
I added the 'listen' directive to be able to use 'puppet kick', which
works fine. And I configured Puppet to use Dashboard as an ENC, which
works fine too.

Below is my puppet.conf (same on the master and the nodes), in case I
made an obvious mistake.
If you guys have any idea what could cause this...

Regards,


### /etc/puppet/puppet.conf
[main]
logdir = /var/log/puppet
rundir = /var/run/puppet
ssldir = $confdir/ssl
pluginsync = true
server = puppet

[agent]
classfile = $vardir/classes.txt
localconfig = $vardir/localconfig
report = true
listen = true

[master]
ssl_client_header = SSL_CLIENT_S_DN
ssl_client_verify_header = SSL_CLIENT_VERIFY
reports = store, https
reporturl = https://dashboard:443/reports/upload
facts_terminus = inventory_active_record
dbadapter = mysql
dbname = inventory
dbuser = ************
dbpassword = ************
dbserver = localhost
node_terminus = exec
external_nodes = /usr/bin/env
PUPPET_DASHBOARD_URL=https://dashboard:443
/usr/share/puppet-dashboard/bin/external_node