Printing all contents EXCEPT matching pattern

[rhor]

Hello All!

I have a large log file where most of the content is something I do not need, you can see the example of the log below:


-------------------------------------------------------------------------------
01-03-2013 12:02:16 ANR0984I Process 6105 for MIGRATION started in the
BACKGROUND at 12:02:16. (PROCESS: 6105)
01-03-2013 12:02:16 ANR1000I Migration process 6105 started for storage pool
HYPERVFILE automatically, highMig=90, lowMig=0,
duration=None. (PROCESS: 6105)
01-03-2013 12:02:16 ANR0984I Process 6106 for MIGRATION started in the
BACKGROUND at 12:02:16. (PROCESS: 6106)
01-03-2013 12:02:16 ANR1000I Migration process 6106 started for storage pool
FILESRVRECL automatically, highMig=0, lowMig=0,
duration=None. (PROCESS: 6106)
01-03-2013 12:02:16 ANR1100I Migration started for volume
D:\TSM\SERVER1\FILEPOOLS\HYPERVFILE\000364E2.BFS, storage
pool HYPERVFILE (process number 6105). (PROCESS: 6105)
01-03-2013 12:02:16 ANR1102I Removable volume D:\TSM\SERVER1\FILEPOOLS\HYPERV-
FILE\000364E2.BFS is required for migration. (PROCESS:
6105)
01-03-2013 12:02:16 ANR1102I Removable volume D:\TSM\SERVER1\FILEPOOLS\HYPERV-
FILE\000364E3.BFS is required for migration. (PROCESS:
6105)
01-03-2013 12:02:16 ANR1102I Removable volume D:\TSM\SERVER1\FILEPOOLS\HYPERV-
FILE\000364E4.BFS is required for migration. (PROCESS:
6105)
01-03-2013 12:02:16 ANR1102I Removable volume D:\TSM\SERVER1\FILEPOOLS\HYPERV-
FILE\000364E5.BFS is required for migration. (PROCESS:
6105)
01-03-2013 12:02:16 ANR1102I Removable volume D:\TSM\SERVER1\FILEPOOLS\HYPERV-
FILE\000364E6.BFS is required for migration. (PROCESS:
6105)
01-03-2013 12:02:16 ANR1102I Removable volume D:\TSM\SERVER1\FILEPOOLS\HYPERV-
FILE\000364E7.BFS is required for migration. (PROCESS:
6105)
01-03-2013 12:02:16 ANR1102I Removable volume D:\TSM\SERVER1\FILEPOOLS\HYPERV-
FILE\000364E8.BFS is required for migration. (PROCESS:
6105)
01-03-2013 12:02:16 ANR1100I Migration started for volume
D:\TSM\SERVER1\FILEPOOLS\RECLAIMPOOLS\00036876.BFS,
storage pool FILESRVRECL (process number 6106). (PROCESS:
6106)
01-03-2013 12:02:16 ANR1102I Removable volume D:\TSM\SERVER1\FILEPOOLS\RECLAI-
MPOOLS\00036876.BFS is required for migration. (PROCESS:
6106)
01-03-2013 12:02:16 ANR1101I Migration ended for volume
D:\TSM\SERVER1\FILEPOOLS\HYPERVFILE\000364E2.BFS.
(PROCESS: 6105)
01-03-2013 12:02:16 ANR1100I Migration started for volume
D:\TSM\SERVER1\FILEPOOLS\HYPERVFILE\000364E7.BFS, storage
pool HYPERVFILE (process number 6105). (PROCESS: 6105)
01-03-2013 12:02:16 ANR1102I Removable volume D:\TSM\SERVER1\FILEPOOLS\HYPERV-
FILE\000364E2.BFS is required for migration. (PROCESS:
6105)

-------------------------------------------------------------------------------

There are many repetitions of messages I do not need and which are being pain in the arse when trying to read it.
I would like cut some of them out, so what I need to do is to DO NOT print matching regexp (whole line) and 2 following lines.
It's easy if I want to do the opposite - print matching pattern and some additional lines, but that's not my goal.

I tried to do this by matching pattern and following lines or with patterns like the one below:

awk "$3 ~/ANR110[0-2]I/, $NF ~/[0-9+]\)/ {print}"
(This one gives me exactly the opposite of what I want to do!)

I've tried to deny it with "!" but with no luck.
Unfortunately I have no idea how to invert it, so I anyone could help would appreciate that.

Ideally I would like to write awk that would cut out matching line based on event number (like "ANR1102I") and following lines until the end of the record.
It would be perfect if I could combine more than one pattern.

P.S. I'm trying to do this under Windows version of awk if it is of any importance...

[Janis Papanagnou]

Your sample data seems to indicate that you want not only 2 but maybe
occasionally 3 more lines skipped, depending on the record length. If
I understood your task correctly, I think this awk program should do

/^[0-9]/ {skip=($3~/ANR110[0-2]I/)}
!skip

BTW; on WinDOS you should put that program in a file and call it with
option -f as in

awk -f yourawkscript yourdata

Janis

[robert....@gmail.com]

Janis,

Thank you for your fast response. Unfortunatelly I couldn't get back to you faster. I ran your script with my data and it seems to do exactly what I was trying to accomplish! Thank you very much for that.
I need to look more through it to confirm this but it looks very promising. I will get back as soon as I would be sure it solved my problem, in the meantime could you please explain how exactly does it work (i never saw skip function before...), how can it properly cut exact number of lines to get rid of whole record and no more or less?

Robert

[Janis Papanagnou]

On 23.03.2013 23:48, robert....@gmail.com wrote:
> W dniu sobota, 23 marca 2013 04:14:39 UTC+1 użytkownik Janis Papanagnou napisał:

[...]

>>
>> Your sample data seems to indicate that you want not only 2 but maybe
>> occasionally 3 more lines skipped, depending on the record length. If
>> I understood your task correctly, I think this awk program should do
>>
>> /^[0-9]/ {skip=($3~/ANR110[0-2]I/)}
>> !skip
>>
>>

[...]
> [...] I will get back as

> soon as I would be sure it solved my problem, in the meantime could you
> please explain how exactly does it work (i never saw skip function
> before...),

'skip' is no function, it's a variable.

> how can it properly cut exact number of lines to get rid of
> whole record and no more or less?

Whether to skip or not is determined on lines starting with a digit in
column 1; on each such lines the field #3 is compared to the pattern and
the boolean value is assigned to the variable skip. Then all lines that
are not to skip will be printed; print is the dafault action if only a
condition is defined (as in line 2).

Janis

>
> Robert
>>
[...]

[Robert Figura]

On Sat, 23 Mar 2013 04:14:39 +0100
Janis Papanagnou <janis_pa...@hotmail.com> wrote:

> > 01-03-2013 12:02:16 ANR1100I Migration started for volume
> > D:\TSM\SERVER1\FILEPOOLS\HYPERVFILE\000364E7.BFS, storage
> > pool HYPERVFILE (process number 6105). (PROCESS: 6105)
> > 01-03-2013 12:02:16 ANR1102I Removable volume D:\TSM\SERVER1\FILEPOOLS\HYPERV-
> > FILE\000364E2.BFS is required for migration. (PROCESS:
> > 6105)

> /^[0-9]/ {skip=($3~/ANR110[0-2]I/)}
> !skip

Auw. Nice pattern :)
Thanks Janis!

Kind Regards,
Robert Figura

--
http://spuerwerk.dyndns.org/~rfigura/

[robert....@gmail.com]

W dniu niedziela, 24 marca 2013 01:26:54 UTC+1 użytkownik Janis Papanagnou napisał:

Janis,

Now i understand - thank you for your help and explanation. This is great and makes my work a lot easier - it works exactly as expected.
Once again thank you and have a great day!

Cheers,
Robert Horowski