Log File Trimming

[Matt]

I have a very large log file. Once a day I would like to parse it and
delete all the old stuff over a month old.

It has entries like this in it.

Thu Oct 16 01:31:38 2003
Start port 5
End port 10
blah

Thu Oct 16 01:45:08 2003
Start port 99
End Port 3
blah

It just keeps appending to the end of the file. I would like to delete all
entries older then 1 month. That would mean deleting everything located
above that date. I think it would be a fairly simple task of searching for
the first date younger then the cut off date. Then the hard part of
deleting everything above that line then commiting to disk. How would you
do that?

Matt

[Uri Guttman]

>>>>> "M" == Matt <nospam...@yahoo.com> writes:

M> It just keeps appending to the end of the file. I would like to
M> delete all entries older then 1 month. That would mean deleting
M> everything located above that date. I think it would be a fairly
M> simple task of searching for the first date younger then the cut
M> off date. Then the hard part of deleting everything above that
M> line then commiting to disk. How would you do that?

you have an interesting problem. the trouble is that your log
generator/server is still writing to the file at the seek point it knows
about. so it may leave a major gap in the file if you delete the earlier
chunk. or your server could be reopening the log file each time which
solves the first problem but you then have a race condition regarding
the removal of the early entries. but you could then copy the older
entries to a new file and rename it to the old file which is
atomic. this only works if the server reopens the file. many servers
have a way to trigger them to rotate a log file i.e. close the current
one, rename it (with some date or cycle suffix) and open a new empty log
file.

so it all comes down to how the server is managing and writing to the
log file. we would need more info on that before a proper solution can
be discussed.

uri

--
Uri Guttman ------ u...@stemsystems.com -------- http://www.stemsystems.com
--Perl Consulting, Stem Development, Systems Architecture, Design and Coding-
Search or Offer Perl Jobs ---------------------------- http://jobs.perl.org

[Tore Aursand]

On Tue, 27 Jan 2004 23:04:21 -0600, Matt wrote:
> I have a very large log file. Once a day I would like to parse it and
> delete all the old stuff over a month old.

> [...]

Here's how I would have solved it:

1. Process each line in the log file, looking for a date. [1]
2. If date is inside the wanted range, one month old or newer [2],
set a 'write' flag.
3. If the 'write' flag is set, write the line to a new file.
4. When done reading through the log file, delete the it and
rename the new file to the name as the log file.

[1] Date::Parse
[2] Date::Calc

--
Tore Aursand <to...@aursand.no>
"I know not with what weapons World War 3 will be fought, but World War
4 will be fought with sticks and stones." -- Albert Einstein