Alarm event when monitoring log files

[Emmanuel Gadaix]

Hello POEers.

I am using Wheel::FollowTail to monitor a set of 24 log files that indicates
the status of some links connected to a system (for the curious: SS7 links
connected to an SMSC, the system that sends SMS messages to and from your
mobile phones). The idea was to get an alert when one of the links goes
down, as well as an alert when it comes up again.

Please allow me to post the relevant sections of the code:

sub InitLogs
{
my ($xdx, $sys, $name, $log, $rec);
$poe_records{_start} = \&begin_watchers;
for ($xdx = 1; $xdx <= 24; $xdx++)
{
$name = "link$xdx";
$log = "$PATH1/xdxmanmx$xdx.log";
$smsc_logs{$name} = $log;
$rec = $name . "_record";
$poe_records{$rec} = \&xdx_got_record;
}

$poe_records{log_reset} = \&generic_log_reset;
$poe_records{log_error} = \&generic_log_error;
}

sub begin_watchers
{
my $heap = $_[HEAP];
while ( my ($service, $log_file) = each %smsc_logs )
{
my $log_watcher = POE::Wheel::FollowTail->new
( Filename => $log_file,
InputEvent => $service . "_record",
ResetEvent => "log_reset",
ErrorEvent => "log_error",
);
$heap->{services}->{ $log_watcher->ID } = $service;
$heap->{watchers}->{ $log_watcher->ID } = $log_watcher;
}
}

sub xdx_got_record
# Handle log events
{
my $record = $_[10];
my $wheel_id = $_[11];
my $heap = $_[3];
my $service = $heap->{services}->{$wheel_id};

if ($record =~ /eactivated/) { &SendAlert("$service deactivated"); }
if ($record =~ /onnection is ready/) { &SendAlert("$service
activated"); }
}

&InitLogs;
POE::Session->create(inline_states => {%poe_records});
$poe_kernel->run();

Pretty simple. All log files get the same treatement. A simple regex match
then sends an alert whenever we have a link status change that appears in
one of the 24 log files.
It works great. POE is a fantastic tool, I can't even contemplate the
headache of doing that the traditional way.

Now, here is my problem. Sometimes a link goes down for a few seconds and
comes up again. And sometimes this up/down for a few seconds situation
happens continuously for a few hours, e.g. in case of transmission network
instability. So my little POE code generates A LOT of alerts during those
times, which are received by 5 support staff through their phones, often
waking them up in the middle of the night for a problem that isn't really
one.

So I have been asked the following: " would it be possible to send such
alerts only when the time between 'deactivated' and 'connection is ready' is
more than 60 seconds? "

At first I answered cockily "of course, anything can be done". Then I
started looking into how to do this, and my headache started. The subrouting
that handles log events (xdx_got_record) has no idea on when the next event
will come up so it is impossible to measure the time there.

I have been reading a bit the POE docs to find how this could be done but I
must admit I am still confused.

Before I dig further I would love to hear about your comments on this,
whether or not this is possible at all and what would be possible ways to
implement this. Any hint would be really appreciated.

Best regards
Emmanuel

[Emmanuel Gadaix]

douglasstvnsn wrote:

In the handler sub xdx_got_record, store the value and timestamp of the
previous deactivate or activate. Then, when you get a new one , compare and
suppress or send.

Thanks but this will not work.

I need to send "link deactivated" alarm when it has been deactivated (e.g.
no "link active" message received) for more than a minute.

So imagine the scenario whereby the log messages are:

"link deactivated" at 17:00:00
"link active" at 19:00:00

I need to send the alarm at 17:01, not at 19:00

And in the scenario like:

"link deactivated" at 17:00:00
"link active" at 17:00:30

Then no alarm are sent (it was only deactivated 30 seconds).

[Martijn van Beers]

So, you have to send the alarm at 17:01.

$alarm_id = $kernel->delay_set('sendalert', 60, $service); # at 17:00
$heap->{scheduled_alerts}->{$service} = $alarm_id;

and cancel it at 17:00:30

$alarm_id = delete $heap->{scheduled_alerts}->{$service};
$kernel->alarm_remove($alarm_id);

plus a new sendalert event handler that sends the alert, and clears the
scheduled_alerts hash for $service.

An unrelated note: use ARG0 and ARG1 instead of 10 and 11; it's much
clearer that way, and they get exported for you by default.

Martijn

[Lars Thegler]

On Wed, Sep 10, 2008 at 4:21 PM, Emmanuel Gadaix <gad...@gmail.com> wrote:
> So imagine the scenario whereby the log messages are:
>
> "link deactivated" at 17:00:00
> "link active" at 19:00:00
>
> I need to send the alarm at 17:01, not at 19:00
>
>
> And in the scenario like:
>
> "link deactivated" at 17:00:00
> "link active" at 17:00:30
>
> Then no alarm are sent (it was only deactivated 30 seconds).
>

You need to use timers events (delayed messages).

/Lars