invoke-rc.d errors - what happened to this system?

[JonY]

Debian 6 machine running 3.7.1. Attempts to run 'puppet agent --test' give a stream of errors like this:

invoke-rc.d: WARNING: 'invoke-rc.d dnsmasq start' called

invoke-rc.d: during shutdown sequence.

invoke-rc.d: enabling safe mode: initscript policy layer disabled.

If I do a 'service dnsmasq start' after this it works ok with no error messages. 

'runlevel' reports '3 6'.

The system is apparently bolluxed somehow but I'm not sure what's wrong. Where should I look?

[JonY]

Looking at the debug log it appears that puppet is calling '/etc/init.d/dnsmasq status' before this message appears.. but when I enter the same command subsequently I'm told ''running'. and nothing else.

Similar messages appear for puppet, ifplugd, apache2.. every service that I monitor with puppet. 

[jcbollinger]

I have little experience with Debian, but your system apparently thinks it is in the middle of restarting (the current runlevel is 6).  That dovetails with the messages you receive about services being started while a shutdown is in progress.

I cannot explain why you get diagnostics during the Puppet run, but not from the command line.  My best guess would be that Puppet is using a different method to attempt to start services than the 'service' command.  This would be a function of the 'provider' selected to manage Service resources on the affected machine; which you can find at the beginning of the agent's log when you run with --debug messages enabled.

I think the first direction to investigate, however, is why the machine thinks it's shutting down.  I predict that the agent will stop emitting such the diagnostics you observed when you get the runlevel fixed, which you probably can achieve most easily by restarting the machine.  Do verify that the runlevel is correct immediately after system start, and do monitor whether an agent run actually causes that runlevel issue.  It may be, however, that the issue arose from a one-off event.


John