Question on ignoring lines with import

[Kick Megatron]

Hello, I have a feed which has two lines per IP address reported - one with CONNECT and one with DISCONNECT.
I would like to import only one of the lines.

Besides I could do a grep prior to Megatron import, is there a possibility to omit lines with a specific pattern (i.e. ignore all lines with DISCONNECT for instance)?

Thanks,
Kick

[Megatron Developer]

Short answer:

filter.preLineProcessor.classNames.0=se.sitic.megatron.filter.RegExpLineFilter
filter.regExpLineFilter.excludeRegExp=DISCONNECT

Long answer: You have three options:

1. RegExpLineFilter: Filter lines using a reg-exp. One example is a Shadowserver config [1] where we filter out lines contaning "spam" or "dnschanger". Another example is [2], where lines ending with "logfile turned over" are filtered out. Search for RegExpLineFilter in the "job-type" directory for more examples.

2. AttributeFilter: Filter log records using the parsed attribute and a reg-exp. Is more precise than a RegExpLineFilter because it's operates on one single attribute and not the whole line. See [3] for a good example.

3. OsCommandProcessor: Use the grep-approach but call grep from Megatron. See [4] for an example with sed.

[1] https://github.com/cert-se/megatron-java/blob/master/conf/job-type/shadowserver-drone2.properties
[2] https://github.com/cert-se/megatron-java/blob/master/conf/job-type/web-apache.properties
[3] https://github.com/cert-se/megatron-java/blob/master/conf/job-type/urlquery.properties
[4] https://github.com/cert-se/megatron-java/blob/master/conf/job-type/stopforumspam.properties

  /Tor

[Megatron Developer]

[Hmm, Google Groups removes empty lines. Re-post with better formatting(?)] 

 

Short answer:

filter.preLineProcessor.classNames.0=se.sitic.megatron.filter.RegExpLineFilter
filter.regExpLineFilter.excludeRegExp=DISCONNECT

Long answer: You have three options:

1. RegExpLineFilter: Filter lines using a reg-exp. One example is a Shadowserver config [1] where we filter out lines contaning "spam" or "dnschanger". Another example is [2], where lines ending with "logfile turned over" are filtered out. Search for RegExpLineFilter in the "job-type" directory for more examples.

2. AttributeFilter: Filter log records using the parsed attribute and a reg-exp. Is more precise than a RegExpLineFilter because it's operates on one single attribute and not the whole line. See [3] for a good example.

3. OsCommandProcessor: Use the grep-approach but call grep from Megatron. See [4] for an example with sed.

[1] https://github.com/cert-se/megatron-java/blob/master/conf/job-type/shadowserver-drone2.properties
[2] https://github.com/cert-se/megatron-java/blob/master/conf/job-type/web-apache.properties
[3] https://github.com/cert-se/megatron-java/blob/master/conf/job-type/urlquery.properties
[4] https://github.com/cert-se/megatron-java/blob/master/conf/job-type/stopforumspam.properties

  /Tor

 

On Friday, November 1, 2013 1:23:21 PM UTC+1, kick_megatron wrote: