How to get an "alert" when a process dies
nicc777
Even though I use Linux a lot, I have not been doing a lot of hard
core Linux sysadmin stuff for some time, hence the question here :-)
Is there a way to trigger an event (like running a script) when a
process dies? Perhaps even a script to restart a process if it detects
that it's dead?
I have been using cron for monitoring processes up to now, but the
monitoring interval is 1 minute apart. I have a project (streaming of
nature) that should not go down, but the problem is that sometimes it
dies and it takes up to a minute before the traditional monitoring
script will restart it.
Any ideas?
Thanks - Nico.
Nico Kadel-Garcia
Nagios.
Nico, too
Sebastian "lunar" Wiesner
Suffers from the very problem, the OP was trying to avoid: A quite great
delay between the death of the specific process and the monitoring tool
running the check and thus noticing the dead process. Iirc, nagios uses
even higher intervals than one minute between checks.
I guess, the OP wants something like inotify for processes: Passive
checking, yielding an _immediate_ notification about state changes, without
the unavoidable latency of active checks as nagios performs them.
--
Freedom is always the freedom of dissenters.
(Rosa Luxemburg)
Centurion
Totally untested, but it should get your creative juices flowing:
#!/bin/bash
PROC=someprocess # The process to check
WAIT=10 # Seconds to wait between checks
while [ 1 ]
do
ps -ef|grep $PROC &>/dev/null
if [ $? -ne 0 ]; then
# grep didn't find "$PROC"...dead?
/etc/init.d/$PROC restart &>/dev/null
# Send some notification saying what happened.
# Maybe re-run the process check??
fi
# Now sleep and re-check the process after a specified delay
sleep $WAIT
done
Now all you need to do, is start that, fork it to the background and voila.
Add whatever notification method you want in the "if...fi" block. This
script will run until you (or something else) kills it by virtue of
the "while" loop.
If you really wanted to get fancy and check a few processes, change "PROC"
into an array and iterate through it, or use a "for PROC in proc1 proc2
etc" type construct. There are many ways to extend this, but whatever you
need to do can be based on this to provide sub-minute scheduling.
Cheers,
James
--
It's lucky you're going so slowly, because you're going in the wrong
direction.
nicc777
Exactly - Time delays is what I try to get away from. Even seconds can
mean potential customers leaving your service because there's no
service :-(
Thanks for explaining my problem more clearly !
nicc777
Thanks a lot - I came to the same conclusion using another scripting
language, but I still sit with a 1 second delay - which I assume is
sufficient for my current scenario.
Still - I would like to know if there is a real time notification
mechanism.
Does any body know if it's possible to hook something into the Kernel
(custom module) maybe? My thinking is maybe something that can "catch"
signals on the kernel level. Thanks again :-)
Reinhard Tartler
> Is there a way to trigger an event (like running a script) when a
> process dies? Perhaps even a script to restart a process if it detects
> that it's dead?
>
> I have been using cron for monitoring processes up to now, but the
> monitoring interval is 1 minute apart. I have a project (streaming of
> nature) that should not go down, but the problem is that sometimes it
> dies and it takes up to a minute before the traditional monitoring
> script will restart it.
In the end you are looking for some sort of process supervisor.
Ralf Fassel
| Is there a way to trigger an event (like running a script) when a
| process dies? Perhaps even a script to restart a process if it detects
| that it's dead?
#!/bin/sh
while true ; do
run_your_process
status=$?
case $status in
0) : probably ok? ; break ;;
*) echo "process has exited with status $status, re-run"
esac
donex
?
R'
Nico Kadel-Garcia
OK. You need a monitor task that checks much more frequently. A cron script
could assure that the *MONITOR* is up and running, but the monitor could be a
simple shell script with a built-in 'restart if not working' setup.
Robert Nichols
nicc777 <nic...@gmail.com> wrote:
:Hi all,
Can't you just run your process from a simple script that loops,
restarting your process whenever it terminates:
while :
do
myprocess [args ... ]
done
Alternatively, if this process is one that you want to start when the
system boots then you could run it directly from /etc/inittab, using
"respawn" in the action field. If you do that, you'll probably want
to invoke your process via 'su' or else it will be running with root
privileges.
--
Bob Nichols AT comcast.net I am "RNichols42"
Nico Kadel-Garcia
Put a pause in there, of say a second, your script is going to hoover up
whatever CPU is available for itself.
nicc777
> * nicc777 <nicc...@gmail.com>
Cool - not what I expected, but it does work perfectly :-)
I think I was making the problem to complicated.
Thanks a mil
nicc777
Robert Nichols
Nico Kadel-Garcia <nka...@gmail.com> wrote:
Curious. I've got a couple of scripts that basically look like that,
and the shell obediently waits, consuming no CPU whatever. It will use
CPU time only if 'myprocess' keeps terminating immediately. And, if
that process is as vital as the OP suggests, having a shell sitting in a
loop trying to restart it isn't a big problem, though flooding some log
file with error messages could indeed be nasty.
Anyway, my suggestion wasn't really intended as a complete solution.
For bulletproofing, you'd want to check whether the program you are
trying to invoke exists and is executable, and limit the restart
rate if it keeps dying immediately. Something like:
#!/bin/bash
PROG=/bin/true
N=0
while :
do
if [ ! -x $PROG ]; then
echo "$PROG is not executable" >&2
exit 1
fi
if [ $N -lt $SECONDS ]; then
N=$SECONDS
elif [ $(($N - 5)) -ge $SECONDS ]; then
echo "$PROG restarting too frequently -- sleeping" >&2
sleep 60
fi
N=$(($N+1))
$PROG [args ... ]
done
nicc777
<SEE_SIGNAT...@localhost.localdomain.invalid> wrote:
> In article <486902DF.3040...@gmail.com>,> :> In article <a9bba346-32a6-452e-a73a-c5980867a...@34g2000hsh.googlegroups.com>,
Ahh - this is excellent as well. Thanks a mil!
pedroart...@gmail.com
> Is there a way to trigger an event (like running a script) when a
> process dies? Perhaps even a script to restart a process if it detects
> that it's dead?
you want any kind of exit code? You may try logger tool...
ex:
updatedb || logger -t updatedb -- Process Terminated &
You will get a message at your log's (usualy /var/log/*) or you may
specify a socket to send packets to (see man logger). And you may also
change the logging priority (see man again).
nicc777
Thanks - it's already in my script :-)
Douglas O'Neal
In /etc/inittab:
yp:2345:respawn:/your/bin/script
Don't re-invent the wheel when you don't have to.
Doug
Fred Weigel
> Hi all,
>
> Even though I use Linux a lot, I have not been doing a lot of hard
> core Linux sysadmin stuff for some time, hence the question here :-)
>
> Is there a way to trigger an event (like running a script) when a
> process dies? Perhaps even a script to restart a process if it detects
> that it's dead?
For your purposes, start the program from inittab. The init process will
relaunch if it dies.
dye
If you know the pid of your process, won't the "wait" shell
command do what you want, without any sleep/polling?
--Ken
--
Ken R. Dye an optimist is a guy |
Chicago, Illinois that has never had |
www.geocities.com/MotorCity/Track/8746 much experience |
dye1146 at sbcglobal dot net archy |
Chris Davies
>process dies? Perhaps even a script to restart a process if it detects
>that it's dead?
Drop it into inittab. Job done.
Chris
Florian Diesch
Wrap it with something like
while [ ! -e /var/run/yourprogram.stop ]; do
yourprogram
sleep 1
done
This restarts the program as long as no file
/var/run/yourprogram.stop exists.
Florian
--
<http://www.florian-diesch.de/>
-----------------------------------------------------------------------
** Hi! I'm a signature virus! Copy me into your signature, please! **
-----------------------------------------------------------------------