Monitoring services

[Yasha Zislin]

Hello,

I have multiple instances of a service running on linux server. This service has status, restart, start and stop init scripts.
One of the instances keeps dying for unknown reasons (probably network related).
I have puppet configured to monitor the service but it doesnt consider it to be down when one of the instances dies.

Is there a way to configure puppet to initiate restart when one of the instances dies?

Here is my current config

class myApp::service {
  service { "myService":
    ensure     => running,
    hasstatus  => true,
    hasrestart => true,
    enable     => true,
  }
}

Thanks.

[Nikola Petrov]

You better use another tool for this that will "provision" your process.
There are many examples for this:

http://smarden.org/runit/
http://upstart.ubuntu.com/cookbook/
http://mmonit.com/monit/

maybe more. I am currently using upstart and it is maybe one of the
easiest to setup and integrates with the puppet service type

--
Nikola

> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/5de0eace-4117-4ac1-8cd2-23e6efc17237%40googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Jason Antman]

Yasha,

What distribution are you running? Is there any chance that you've
specified somewhere a non-default provider for the Service type?

I'm confused... this doesn't seem to be a Puppet issue to me. You
include "hasstatus => true"... so Puppet should restart the service if
your init script returns non-0. Are you sure your init script is
returning non-zero to trigger the restart?

How are you configuring the multiple instances? The myApp::service class
that you show below can only be applied once on a given node.

-jantman

[Yasha Zislin]

Jantman,

The problem is that init status script does return 0 when puppet checks the service. Since I have one of the two instances of service running, init script returns 0 and that's not cool.
Multiple instances just run with different parameters like interface. So when you look at ps auxf, you would see two entries. When you run init status script, it returns two PIDs.

After I posted this question, I got an idea.
I've tweaked my init script so when one instance dies, return value is non-zero. Or in fact it is 3 which means that service is dead.

Now the problem is that puppet doesnt restart it for some reason. It states that service status has changed from "stopped" to "running" but the service restart command does not get initiated.

P.S. Thanks nikolavp for response but I need to get this working with puppet.

[Jason Antman]

Yasha,

Interesting. When it says the status has changed from stopped to running, it should be calling the restart command. Can you post the puppet agent --debug output somewhere, either attached or pastebin/etc.? And perhaps the init script itself?

So you're using one init script for multiple instances. Have you considered using separate init scripts (templated by Puppet?) and separate Service resources for each instance (i.e. "service1", "service2", or something like that)? Controlling multiple instances via one init script is possible, and I've seen it done at a lot of places, but it usually adds to confusion, and increases the possibility of problems (like this one).

-Jason

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/cd959b13-bc2f-44c1-8cf5-30f0521cd723%40googlegroups.com.

[Neil]

A common way to do this is to symlink from the real init script to service-instancename
The real script can check it was called as a link; extract the I stance name and source a configuration file of that name.
Would a example be useful?

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/5274615A.4070601%40jasonantman.com.

[Puppet List]

A common way to do this is to symlink from the real init script to service-instancename
The real script can check it was called as a link; extract the I stance name and source a configuration file of that name.
Would a example be useful?

[Yasha Zislin]

Jason,

So when both instances die of this service, puppet does restart it. When one instance dies, like I said earlier, it gives that message "stopped to running" and nothing happens.

I've ran debug and in addition to typical stopped to running message, I got "Unscheduling refresh on Service [myService]".
Not sure what that means.

Also, does puppet only look at the status code when determining when to restart? and if hasrestart is set to true, it just tries to run service myService restart?

Thanks

[Yasha Zislin]

I got it fixed. :)

If anybody is interested, here is the problem/solution.

So it turns out that puppet actually calls "start" init script instead of restart (even though I've specified "hasrestart=true"). If one of the instances is still running, start script will say that service is already running. It was not smart enough to figure out that only one is running.

I can share details about my script. Multiple instances are run for different NICs. Actual ETHX are specified in a variable (ie array) like "eth0 eth1".
So to figure out if only one instance is running or more, simple check is performed. See below (this code is on start init script):

PIDS=`/sbin/pidof $BASE`
PIDS=($PIDS)
INT=($LOG_INTERFACE)
if [ "${#PIDS[@]}" -eq "${#INT[@]}" ]
then
        echo -n $"$BASE: already running"
        echo ""
        exit $RETVAL
fi
killproc /usr/local/bin/$BASE
------------------------------

After that just continue with regular start script.


Done. This worked with Puppet 3.3.1.

Thanks all for motivating me to solve this problem :)