duplicate message removal

[Robin Becker]

We have a bunch of FreeBSD 6.x servers which we administer remotely. As
part of that we get the normal root job mails emailed to a mailing list
which the admins(mostly me) can inspect at leisure and also use for
historical purposes.

Trouble is many of the emails get huge because of repeated messages
typically stuff like

xxx.yyy.com login failures:
Feb 22 20:07:54 app3 sshd[56886]: reverse mapping checking getaddrinfo
for 216-194-26-66.ny.ny.metconnect.net failed - POSSIBLE BREAKIN ATTEMPT!

etc etc

All these servers are running denyhosts, but we still see lots of these
messages.

I was wondering if there's any simple compression script which notices
the repetitions (apart from timestamp) and can remove the many
duplicates etc etc.
--
Robin Becker
_______________________________________________
freebsd-...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questi...@freebsd.org"

[Mel]

On Saturday 23 February 2008 11:32:41 Robin Becker wrote:

> Trouble is many of the emails get huge because of repeated messages
> typically stuff like
>
> xxx.yyy.com login failures:
> Feb 22 20:07:54 app3 sshd[56886]: reverse mapping checking getaddrinfo
> for 216-194-26-66.ny.ny.metconnect.net failed - POSSIBLE BREAKIN ATTEMPT!
>
> etc etc
>
> All these servers are running denyhosts, but we still see lots of these
> messages.
>
> I was wondering if there's any simple compression script which notices
> the repetitions (apart from timestamp) and can remove the many
> duplicates etc etc.

Well, there's two things you can do:
Attack the problem:
Install sysutils/grok on the machines and start it with following config:

file "/var/log/auth.log" {
type "ssh-illegal-user" {
match = "Invalid user %USERNAME% from %IP%";
threshold = 5; # 5 hits ...
key = "%IP%"; # from a single ip ...
interval = 60; # in 1 minutes
reaction = "/sbin/pfctl -t sshscan -Tadd %IP%";
};

type "ssh-scan-possible" {
match = "Did not receive identification string from %IP%";
threshold = 3;
interval = 60;
reaction = "/sbin/pfctl -t sshscan -Tadd %IP%";
};
};

If you replace:
my $TAIL = "tail -0f";
with
my $TAIL = "/usr/bin/tail -0F"; # capital F

in grok, you will never have to look at it again. Else you will have to
restart it whenever newsyslog(8) rotates the auth.log.
Note that this uses pf, and assumes the table <sshscan> is blocking table.
I'm sure it can be done with other firewalls as well.

Attack the report:
daily_status_security_loginfail_enable="NO" >>/etc/periodic.conf

Then write your own and put it in /usr/local/etc/periodic/security/. I've
written something similar with php for mail rejects, that groups sender or
hostname and error message together, but any scripting language that you're
comfy with should work.

--
Mel