Even though I use Linux a lot, I have not been doing a lot of hard
core Linux sysadmin stuff for some time, hence the question here :-)
Is there a way to trigger an event (like running a script) when a
process dies? Perhaps even a script to restart a process if it detects
that it's dead?
I have been using cron for monitoring processes up to now, but the
monitoring interval is 1 minute apart. I have a project (streaming of
nature) that should not go down, but the problem is that sometimes it
dies and it takes up to a minute before the traditional monitoring
script will restart it.
Any ideas?
Thanks - Nico.
Nagios.
Nico, too
Suffers from the very problem, the OP was trying to avoid: A quite great
delay between the death of the specific process and the monitoring tool
running the check and thus noticing the dead process. Iirc, nagios uses
even higher intervals than one minute between checks.
I guess, the OP wants something like inotify for processes: Passive
checking, yielding an _immediate_ notification about state changes, without
the unavoidable latency of active checks as nagios performs them.
--
Freedom is always the freedom of dissenters.
(Rosa Luxemburg)
Totally untested, but it should get your creative juices flowing:
#!/bin/bash
PROC=someprocess # The process to check
WAIT=10 # Seconds to wait between checks
while [ 1 ]
do
ps -ef|grep $PROC &>/dev/null
if [ $? -ne 0 ]; then
# grep didn't find "$PROC"...dead?
/etc/init.d/$PROC restart &>/dev/null
# Send some notification saying what happened.
# Maybe re-run the process check??
fi
# Now sleep and re-check the process after a specified delay
sleep $WAIT
done
Now all you need to do, is start that, fork it to the background and voila.
Add whatever notification method you want in the "if...fi" block. This
script will run until you (or something else) kills it by virtue of
the "while" loop.
If you really wanted to get fancy and check a few processes, change "PROC"
into an array and iterate through it, or use a "for PROC in proc1 proc2
etc" type construct. There are many ways to extend this, but whatever you
need to do can be based on this to provide sub-minute scheduling.
Cheers,
James
--
It's lucky you're going so slowly, because you're going in the wrong
direction.
Exactly - Time delays is what I try to get away from. Even seconds can
mean potential customers leaving your service because there's no
service :-(
Thanks for explaining my problem more clearly !
Thanks a lot - I came to the same conclusion using another scripting
language, but I still sit with a 1 second delay - which I assume is
sufficient for my current scenario.
Still - I would like to know if there is a real time notification
mechanism.
Does any body know if it's possible to hook something into the Kernel
(custom module) maybe? My thinking is maybe something that can "catch"
signals on the kernel level. Thanks again :-)
> Is there a way to trigger an event (like running a script) when a
> process dies? Perhaps even a script to restart a process if it detects
> that it's dead?
>
> I have been using cron for monitoring processes up to now, but the
> monitoring interval is 1 minute apart. I have a project (streaming of
> nature) that should not go down, but the problem is that sometimes it
> dies and it takes up to a minute before the traditional monitoring
> script will restart it.
In the end you are looking for some sort of process supervisor.
#!/bin/sh
while true ; do
run_your_process
status=$?
case $status in
0) : probably ok? ; break ;;
*) echo "process has exited with status $status, re-run"
esac
donex
?
R'
OK. You need a monitor task that checks much more frequently. A cron script
could assure that the *MONITOR* is up and running, but the monitor could be a
simple shell script with a built-in 'restart if not working' setup.
Can't you just run your process from a simple script that loops,
restarting your process whenever it terminates:
while :
do
myprocess [args ... ]
done
Alternatively, if this process is one that you want to start when the
system boots then you could run it directly from /etc/inittab, using
"respawn" in the action field. If you do that, you'll probably want
to invoke your process via 'su' or else it will be running with root
privileges.
--
Bob Nichols AT comcast.net I am "RNichols42"
Put a pause in there, of say a second, your script is going to hoover up
whatever CPU is available for itself.
Cool - not what I expected, but it does work perfectly :-)
I think I was making the problem to complicated.
Thanks a mil
Curious. I've got a couple of scripts that basically look like that,
and the shell obediently waits, consuming no CPU whatever. It will use
CPU time only if 'myprocess' keeps terminating immediately. And, if
that process is as vital as the OP suggests, having a shell sitting in a
loop trying to restart it isn't a big problem, though flooding some log
file with error messages could indeed be nasty.
Anyway, my suggestion wasn't really intended as a complete solution.
For bulletproofing, you'd want to check whether the program you are
trying to invoke exists and is executable, and limit the restart
rate if it keeps dying immediately. Something like:
#!/bin/bash
PROG=/bin/true
N=0
while :
do
if [ ! -x $PROG ]; then
echo "$PROG is not executable" >&2
exit 1
fi
if [ $N -lt $SECONDS ]; then
N=$SECONDS
elif [ $(($N - 5)) -ge $SECONDS ]; then
echo "$PROG restarting too frequently -- sleeping" >&2
sleep 60
fi
N=$(($N+1))
$PROG [args ... ]
done
Ahh - this is excellent as well. Thanks a mil!
you want any kind of exit code? You may try logger tool...
ex:
updatedb || logger -t updatedb -- Process Terminated &
You will get a message at your log's (usualy /var/log/*) or you may
specify a socket to send packets to (see man logger). And you may also
change the logging priority (see man again).
Thanks - it's already in my script :-)
In /etc/inittab:
yp:2345:respawn:/your/bin/script
Don't re-invent the wheel when you don't have to.
Doug
For your purposes, start the program from inittab. The init process will
relaunch if it dies.
If you know the pid of your process, won't the "wait" shell
command do what you want, without any sleep/polling?
--Ken
--
Ken R. Dye an optimist is a guy |
Chicago, Illinois that has never had |
www.geocities.com/MotorCity/Track/8746 much experience |
dye1146 at sbcglobal dot net archy |
Drop it into inittab. Job done.
Chris
Wrap it with something like
while [ ! -e /var/run/yourprogram.stop ]; do
yourprogram
sleep 1
done
This restarts the program as long as no file
/var/run/yourprogram.stop exists.
Florian
--
<http://www.florian-diesch.de/>
-----------------------------------------------------------------------
** Hi! I'm a signature virus! Copy me into your signature, please! **
-----------------------------------------------------------------------