TSE: Merging of log files

[knud van eeden]

Hello,

Has anybody already already created in TSE / is using interesting utilities / can give advise about how to 

 merge log files?

Thanks in advance

with friendly greetings

Knud van Eeden

[knud van eeden]

I have now written a program in TSE which does the

following:

1. Given a list of log files in different formats

(e.g. Apache, TomCat, ...).

E.g.

logfile1

logfile2

logfile3

...

2. You want those log files sorted and merged in chronological order, so that you can e.g. easy the cause and effect (e.g. between some client and its server)

3. First you create once a TSE regular expression to extract the

date/time

YYYY-MM-DD hh:mm:ss

from that type of log file.

4. That regular expression varies per log file in general, because each log files has usually its own regular expression, because the date/time format varies.

5. Then you search all those given log files, using their specific regular expression, and you extract

5.1 the date/time

5.2 the linenumber where that date/time occurs in that file

5.3 the filename where that date/time occurs

5.4 the file type (e.g. Apache, TomCat, ...)

Thus creating lines looking like this:

<date/time>; <linenumber>; <filename>; <filetype>

6. You make a very big list of all such lines e.g. in a buffer or temporary file

7. Then you convert all the date/time values in that big file and convert these to a standard format (which you put in front of each of those lines for sorting purposes).

A standard format is typically

YYYY-MM-DD hh:mm:ss

So you get lines like this:

YYYY-MM-DD hh:mm:ss <date/time>; <linenumber>; <filename>; <filetype>

8. Then you sort those standard formats, which is possible out of the box when that format is used.

9. After sorting removing that standard format in front of the lines, because not necessary anymore after sorting.

(So remove the YYYY-MM-DD hh:mm:ss in front).

10. Then you go through each of these sorted lines from top to bottom, and because you know for each line the date/time, line number, filename

you can go to each of those filenames in TSE, goto that line number, search for that date/time.

When found you highlight the block between that date/time and the next searched and found date/time in the file. Then you copy  that block and paste the result e.g. in a buffer.

This can be one line, but also multiple lines.

11. The final result is a merge of all those files, record per record, chronologically sorted.

12. You see thus what happens at the same time in all those different files chunked together which will allow in general to better

see cause and effect (e.g. something happens always in the client and a moment later there occurs some reaction in the other server file).

Note: I use now also the program LogViewPlus as another method.

https://www.logviewplus.com

Note: Here are some more links: https://www.google.com/search?q=how+to+merge+log+files