All Hosts Shown as Unresponsive. Worker Restart Fails. Clients Seem Happy?

[Dave Mankoff]

[On my dev server] For the past 3 days, my puppet master has shown all of the hosts as unresponsive. However, when I log into one of the hosts and run "puppet agent --test" everything seems to run without complaint. Meanwhile the master shows an ever increasing number of pending tasks.

When I try to restart the dashboard workers, I get the following:

# /etc/init.d/pe-puppet-dashboard-workers restart

 * Restarting PE Puppet Dashboard Workers

ERROR: there is already one or more instance(s) of the program running

ERROR: there is already one or more instance(s) of the program running

 * PE Puppet Dashboard Worker (pid ) is not running

                                                                                                                  [fail]

Rebooting the computer does not seem to help. For that matter, I think this problem started after a recent system update and reboot. It is running Ubuntu 12.04. I can not seem to find anything meaningful in the logs, though perhaps I am looking in the wrong places. Any advice? Things worked fine just a few days ago, and I have not touched Puppet.

[Mike Beauchamp]

I was able to fix this by going into the /etc/init.d/pe-puppet-dashboard-workers and editing the CPU line in the start section.

Previously it had {CPU:-2}, and I changed that to 4.   Instantly cleared the queue, and nodes showed up.  

[Gary Larizza]

I think there might be a bug buried here.  I hit this before a demo this morning - the problem was that the pid files at /opt/puppet/share/puppet-dashboard/tmp/pids/delayed_job.{0,1}.pid  were present but there was no PID in them (and the background workers weren't running).  Removing the pidfiles allowed the service to start, but we need to discover why these files are being created empty.  My master was on Ubuntu 12.04 and PE 2.7.0

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

To post to this group, send email to puppet...@googlegroups.com.

Visit this group at http://groups.google.com/group/puppet-users?hl=en.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Gary Larizza

Professional Services Engineer

[Aaron]

So, I just ran into this after my VM ran out of space.  I removed all of the reports from /var/lib/puppet/reports# to free up space and then restarted the server.  After the restart, I could run the following

$ sudo service apache2 start

* Starting web server apache2 [ OK ]

$ sudo service puppetmaster start

* Starting puppet master [ OK ]

$ sudo service puppet-dashboard-workers start

* Starting Puppet Dashboard Workers

but had the same errors as OP.  

ERROR: there is already one or more instance(s) of the program running

ERROR: there is already one or more instance(s) of the program running

 * Puppet Dashboard Worker (pid ) is not running

Puppet dashboard in Open Source was showing up when I went to the website, but my 3 dev nodes were unresponsive and it just kept building up pending tasks.

I followed the steps Gary did by removing the pid and also deleted the .log files located in /usr/share/puppet-dashboard/tmp/pids.

puppet --version 3.8.3

puppet dashboard 1.2.23