Perl Script for parsing log files.

[Asad]

Hi All ,

Greetings !

I have completed O'Reily first book Learning Perl . Started writing
small perl programs . However I basically do log file analysis , hence was
thinking is there any perl code around in any of the book or if anyone
already developed for the following requirement :

Nowadays I manually read the log files to read the errors which are either
fatal or warning or ignorable . The purpose is I want to develop a web page
as in I upload this log file and it searches for those keywords (fatal or
warning or ignorable) and if found those keywords display the 10line before
it and after it .

Please share your thoughts. Unable to start .

Thanks,
--
Asad Hasan
+91 9945666911

[David Precious]

At what point are you stuck?

Learning Perl should cover the basics you're describing - opening a
file, looping over it line by line, and checking if a line matches a
regular expression should all be quite easy for you.

(I would recommend working this out as a standalone script you run
locally to start with, and worry about the handling file uploads etc
afterwards, rather than trying to tackle it all at once.)

When you're looping over the file, you can easily keep the last 5 lines
in a "buffer" (use an array, push() a line to it each time, and if it
contains > 5 lines, shift() the oldest off the front); when you find a
line that matches your pattern, print those five lines, then the line
you're processing, then the next 5.

What you're talking about sounds a lot like "grep -C5 filename",
though - possible wheel re-invention :)

Cheers

Dave P


--
David Precious ("bigpresh") <dav...@preshweb.co.uk>
http://www.preshweb.co.uk/ www.preshweb.co.uk/twitter
www.preshweb.co.uk/linkedin www.preshweb.co.uk/facebook
www.preshweb.co.uk/cpan www.preshweb.co.uk/github

[timothy adigun]

Hi,

I agree with David, using a "cache" works very well.

But if your log files are not so large ( that I don't know :) ), you can
consider reading the file into an array ( of course, there are several
modules that does it for you, if you want ), then loop through the array,
once you get the line you wanted using regex, then take a range 10 lines
from and 10 lines to.

For Example:
[code]
use warnings;
use strict;
use File::Slurp qw(read_file);
use constant LIMIT => 3; ## Number of lines wanted

my $file = $ARGV[0];

my @lines = read_file($file);

for ( 0 .. $#lines ) {
if ( $lines[$_] =~ m{^\s+?\b(it|mary|to)\b}i ) {
print join " " => '*', @lines[ $_ - LIMIT .. $_ + LIMIT ], $/;
}
}
[/code]

ON the CLI:
Usage: perl_script.pl Mary_has_a_little_lamp_file.txt

You have to install the module `File::Slurp` if you don't have it.
You might also consider using the module `Tie::File` this comes with your
Perl installation.




--
Tim

[Asad]

Hi All ,

          Thanks to you all for the inputs . I have created the following script :

use warnings;

use strict;

use File::Slurp qw(read_file);

use constant LIMIT => 3;    ## Number of lines wanted

use constant UPPERLIMIT => 9;

open (OUTFILE, '>','d:\perl\OUTPUT.TXT');

#my $file = $ARGV[0];

my $filename = 'LOGFILE';

open(FH, '<', $filename) or die $!;

my @lines = read_file($filename);

for ( 0 .. $#lines ) {

    if ( $lines[$_] =~ m{\b(VERIFICATION_FAILED)\b}i ) {

    print "$_\n";

         print $lines[$_];  

       print OUTFILE join " " => '*', @lines[ $_ - LIMIT .. $_ + UPPERLIMIT ], $/;

    }

elsif ($lines[$_] =~ m{\bfatal\b}i)

{

        print OUTFILE join " " => '*', @lines[ $_ - LIMIT .. $_ + UPPERLIMIT ], $/;

}

elsif ($lines[$_] =~ m{\b'Exception Severity: 1'\b})

{

        print OUTFILE join " " => '*', @lines[ $_ - LIMIT .. $_ + UPPERLIMIT ], $/;

}

elsif ($lines[$_] =~ m{\bstop\b}i)

{

        print OUTFILE join " " => '*', @lines[ $_ - LIMIT .. $_ + UPPERLIMIT ], $/;

}

elsif ($lines[$_] =~ m{\bfail\b}i)

{

        print OUTFILE join " " => '*', @lines[ $_ - LIMIT .. $_ + UPPERLIMIT ], $/;

}

else 

{

print " No Errors found \n";

}

Please review if the above code seems ok .

Also want to know how to get this to UI(html page) interface for example a html page which has a browse button to select the log file and it displays the output parsing through the logic any module which this does or what you suggest.

Any pointers for UI interface or any book I can refer would be helpful.

Thanks,

[Ed Davis]

Hi

My first contribution to the list!

Probably the simplest way to do this is (once you have apache2 installed and working) to create a script that looks a bit like this:

!#/usr/bin/perl

use strict;

use warnings;

sub mylogparser {

Your script

}

mylogparser ();

print "Content-type: text/html\n\n"; #VERY IMPORTANT!!

print <<WEBPAGE;  #Start multiline variable which is just printed as HTML

<all of your HTML goes here including reference to CSS etc and use $variables for dynamic content e.g your log data>

WEBPAGE #note no semi colon

Your script will run on load so if you want to refresh it, maybe use a java script to do that. For pointers consider CGI but this will do what you are after.  Use a WYSIWYG editor to make it pretty and keep a copy to reuse :)

Hth.

[Shlomi Fish]

Hi Asad,

some comments on your script:

On Wed, 8 May 2013 11:34:14 +0530

Asad <asad.ha...@gmail.com> wrote:

> Hi All ,
>

> Thanks to you all for the inputs . I have created the following
> script :
>
> use warnings;
> use strict;
> use File::Slurp qw(read_file);
> use constant LIMIT => 3; ## Number of lines wanted
> use constant UPPERLIMIT => 9;
>
> open (OUTFILE, '>','d:\perl\OUTPUT.TXT');

1. Don't use bareword filehandles - use lexical ones instead.

2. Use autodie, or the "or die" thing:

http://perl-begin.org/tutorials/bad-elements/#open-function-style

>
> #my $file = $ARGV[0];
>
> my $filename = 'LOGFILE';
>
> open(FH, '<', $filename) or die $!;

1. Don't use bareword filehandles.

2. Including a meaningful message in the "die $!".

>
> my @lines = read_file($filename);
>
> for ( 0 .. $#lines ) {

Don't iterate using $_:

http://perl-begin.org/tutorials/bad-elements/#overuse_dollar_underscore

Use an explicit, lexical, iteration variable.

In recent perls, you can also use keys(@lines).

> if ( $lines[$_] =~ m{\b(VERIFICATION_FAILED)\b}i ) {

Use «(?: ... )» (for clustering) instead of «( ... )» (for capturing and in
this case, it is completely not-needed.

> print "$_\n";
> print $lines[$_];
> print OUTFILE join " " => '*', @lines[ $_ - LIMIT .. $_ + UPPERLIMIT
> ], $/;

MAke sure the upper and lower limits to the range are bounded.

> }
> elsif ($lines[$_] =~ m{\bfatal\b}i)
> {
> print OUTFILE join " " => '*', @lines[ $_ - LIMIT .. $_ +
> UPPERLIMIT ], $/;
> }
> elsif ($lines[$_] =~ m{\b'Exception Severity: 1'\b})
> {
> print OUTFILE join " " => '*', @lines[ $_ - LIMIT .. $_ +
> UPPERLIMIT ], $/;
> }
> elsif ($lines[$_] =~ m{\bstop\b}i)
> {
> print OUTFILE join " " => '*', @lines[ $_ - LIMIT .. $_ +
> UPPERLIMIT ], $/;
> }

1. You're using « $lines[$_] » too much - you should assign it to a variable.

2. There's a lot of duplicate code in the «print OUTFILE join " " => '*',

@lines[ $_ - LIMIT .. $_ + UPPERLIMIT ], $/;»

> elsif ($lines[$_] =~ m{\bfail\b}i)
> {
> print OUTFILE join " " => '*', @lines[ $_ - LIMIT .. $_ +
> UPPERLIMIT ], $/;
> }
> else
> {
> print " No Errors found \n";
> }
>
> Please review if the above code seems ok .

Hope it helped.

>
> Also want to know how to get this to UI(html page) interface for example a
> html page which has a browse button to select the log file and it displays
> the output parsing through the logic any module which this does or what you
> suggest.

You should write a module for that, and then use Dancer or whatever. See:

1. http://perl-begin.org/topics/modules-and-packages/

2. http://perl-begin.org/uses/web/

>
> Any pointers for UI interface or any book I can refer would be helpful.
>

The books are likely going to be out-of-date.

Regards,

Shlomi Fish

--
-----------------------------------------------------------------
Shlomi Fish http://www.shlomifish.org/
First stop for Perl beginners - http://perl-begin.org/

Jack: Hi Sophie!
Sophie: Don’t “Hi, Sophie!” me.
Jack: Don’t “Don’t ‘Hi, Sophie!’ me” me!

Please reply to list if it's a mailing list post - http://shlom.in/reply .