Feature requests: trap external signals (KILL,TERM,INT,etc)
Jez Hancock
First of all many thanks for a great script, have just spent the last
few days working on getting grok to block comment trackback spammers,
ftp/ssh auth brute attacks, smtp auth attacks and so.
All works very well but have got stuck on log rotation...
I have grok watching a number of logfiles - the following entry in
grok.conf is typical:
-snip-
file "/path/to/spamblock.log" {
type "spamblock rejects" {
match = "REJECTED: [No API-created comments|Trackback URL
invalid|Filtered by Akismet\.com].*, IP %IP%.*";
reaction = {
my $type = "spamblocklog";
my $ip = meta2string("%IP%", $v);
my $cmd = "pfctl -t $type -T add $ip";
my $outfile = "/etc/pf/tables/$type.pf";
`$cmd && echo "$ip" >> $outfile`;
};
};
};
-snip-
So here this works well - grok watches for any matching patterns and
if it finds any it will first act to block the IP at the firewall
using pfctl and then go on to add the IP to a pf table file in /etc/pf/
tables.
However the problem starts when the /path/to/spamblock.log file is
rotated (which is daily) - grok continues to watch the old logfile
instead of the new logfile.
To get around this I hacked together a simple script to restart grok
just after midnight every day. In doing so though I found that - as
mentioned in the last post to this list - when grok is terminated
externally (for example using something like 'kill <pid>' where <pid>
is the pid of grok) the child processes that grok has spawned do not
die and continue to run. These processes are those 'tail -f0' and
'sh' starter scripts.
This problem is the same with the FreeBSD rc script installed into /
usr/local/etc/rc.d. Executing:
/usr/local/etc/rc.d/grok stop
will kill the main grok process, but fails to kill off any child
processes that grok has spawned (to be fair though it shouldn't really
be the job of the rc script to kill off any of grok's children).
So my feature request would be to have grok trap and handle external
signal requests. For example, when a kill signal is sent on the
commandline like:
kill 1234
(where 1234 is the pid of grok)
would it please be possible to have grok trap that signal and as part
of it's shutdown process go on to in turn kill any child processes
that it's spawned.
I tried doing this myself in the grok script using something like:
$SIG{KILL} = \®exhack::hackteardown();
but this failed woefully - my perl is just bearly bordering on
intermediate so ... :(
My hackaround at present is a perl script I run daily at 1201 which
grabs each pid with '/usr/local/bin/grok' in the commandline. It then
checks to see if each pid is a session leader or not and if it is, it
kills any child procs using:
/bin/kill -- -<pid>
(basically kills any process with the PGID of <pid>).
It also kills any procs that aren't session leaders (this is either
the process that forks or that is forked, didn't work out which!).
All of this is very hacked up and prone to dodginess! Have only just
done this tonight and it appears to work, but would be a lot easier to
just have grok trap the kill signal it receives and then act on it
internally to kill off any child procs.
Hope this is clear.
Here's the perl sub from my rotate script, should give the gist of
what it does:
-snip-
sub grok_stop {
# Stop grok:
$pids=`$pgrep -f "[p]erl.*$grok"`;
@pids=split("\n", $pids);
if(@pids == 0){
die("No running procs found, exiting\n");
}
foreach(@pids){
# find out if this is a session leader
$ps=`ps -o "state=" $_`;
if($ps=~/s/){
# this is a sess leader, kill all proces with
this pgid:
$cmd="$kill -- -$_";
`$cmd`;
} else {
$cmd="$kill $_";
`$cmd`;
}
}
}
-snip-
An even better feature for grok to have included would be one which
handles log rotation without the need to have grok restart at all -
I'm no IPC guru but would this be possible for example by trapping INT
interrupt signals (which newsyslog can send out when it rotates
logfiles) and then rereading the grok config file to start watching
the new logfile?
Once again though, many thanks for a great script!
Regards.
--
Jez Hancock
- System Administrator / PHP Developer
http://munk.me.uk/
http://freebsd.munk.me.uk/ - FreeBSD Admin Weblog
http://tf-b4rt.berlios.de/ - Torrentflux-b4rt
http://ipfwstats.sf.net/ - IPFW peruser traffic logging
Jez Hancock
> All works very well but have got stuck on log rotation...
Turns out there's a very simple way of handling log rotation - use the
'-0F' (large F instead of small f) tail switch and you don't need to
handle log rotation at all. Far too simple! Now grok will just run
and run ad infinitum, automatically detecting when files it's watching
have rotated. Thanks to Mel on this post from the freebsd-questions
mailing list:
http://groups.google.com/group/muc.lists.freebsd.questions/msg/102b062f0f2ffdfe
However there's still the issue of grok trapping external TERM/KILL
signals and the like so that any child procs it's spawned will be
dispatched with due diligence - would be good to have grok shut down
cleanly if it's in the background. This problem becomes less urgent
for me personally though now - with the use of the -0F switch I can
just leave grok running ad infinitum and not worry about restarting
grok every time a file is rotated.
Cheers.
--
Jez Hancock
- System Administrator / PHP Developer
http://freebsd.munk.me.uk/ - A FreeBSD Admin Weblog
http://ipfwstats.sf.net/ - ipfw peruser traffic logging
Jordan Sissel
>
> On Tue, Apr 15, 2008 at 11:42 PM, Jez Hancock <jez.h...@gmail.com> wrote:
> > All works very well but have got stuck on log rotation...
>
> Turns out there's a very simple way of handling log rotation - use the
> '-0F' (large F instead of small f) tail switch and you don't need to
> handle log rotation at all. Far too simple! Now grok will just run
> and run ad infinitum, automatically detecting when files it's watching
> have rotated. Thanks to Mel on this post from the freebsd-questions
> mailing list:
>
Good catch. Potentially the workaround for this (without hacking in
perl) woudl be to swap
file "foo" {. ....
with
exec "tail -0F foo" { ...
> http://groups.google.com/group/muc.lists.freebsd.questions/msg/102b062f0f2ffdfe
>
> However there's still the issue of grok trapping external TERM/KILL
> signals and the like so that any child procs it's spawned will be
> dispatched with due diligence - would be good to have grok shut down
> cleanly if it's in the background. This problem becomes less urgent
> for me personally though now - with the use of the -0F switch I can
> just leave grok running ad infinitum and not worry about restarting
> grok every time a file is rotated.
>
> Cheers.
>
I though the process death bug was fixed. I'll work on a patch.
-Jordan
Jez Hancock
I noticed the 'hack' and wasn't sure about it - left it well alone!
The only thing I feel is missing for me personally is the signal
handling. Mainly the two following signals:
KILL TERM (and other signals that are meant to reign death on processes!)
As above really, just to be able to send a signal ala 'kill -9 <pid>'
where <pid> is the grok 'parent' process.
Perhaps to facilitate this it might be worth having a '-p' switch so
you can state on the commandline where you want a pid file placing
which could contain the parent process id.
grok could then check for stale PID file first and if already running,
exit gracefully; otherwise continue to write the pid to the pidfile.
Having a pidfile would make it easier to send signals like:
KILL etc - to kill the process off and any child procs.
HUP - to have grok reread in it's configuration file
I'd just started on writing the PID file stuff, but realized quickly I
don't quite understand the reasons behind the hackinit method in the
'regexhack' package (I was wondering why on earth processes were
forking when I'd used exit liberally to debug... only to realize that
the regexhack::hackinit() call actually forks, hence the grok procs in
bg!).
So I did have a go... if it helps the pseudocode was like this (I
stopped coding when realized I didn't understand the hackinit code!):
if (exists($opts->{"b"})) {
# First check if we want to write to PID file:
if (exists($opts->{"p"})) {
my $pidfile = $opts->{"p"};
# Check for stale PID file first:
if( defined($pidfile) && -f $pidfile ){
open(PID, "< $pidfile") or die("Could not open $pidfile.\n");
my $pid = <PID>;
close PID;
chomp $pid;
# Send a CHLD sig to $pid. If PID is active running proc, return is 1;
# otherwise return is 0:
my $status = kill("SIGTERM", $pid);
if($status == 1){
# die gracefully, explain already running:
} elsif($status == 0) {
# Not already running, stale PID file, unlink:
}
}
# Continue to write PID file with parent pid $$:
}
I'll add a patch file anyway just in case.
Apart from the signal handling grok's been working very nicely on this
server for the last week or so now - basically using it to detect
trackback spam, brute force auth attacks, http XSS abuse and so on and
add offending IPs to respective pf tables.
Seems to be making a difference - I think blocking the hosts
completely makes a difference, before I was just blocking HTTP access
if a zombie attacked trackback comments, but since blocking them
outright on any port I've noticed a lot of DNS requests denied to
those IPs so hopefully perhaps they'll report back to their botnet
master that my host is dead (well I live in hope anyway :- ).
Once again, many thanks.
Will attach patch file now (FWIW!)