A way to replace/substitute the value of a parser.item?

[Ole Christian Hansen]

Hi.

I'm working on the Shadowserver Quote of The Day-service.
The hostname can either be a hostname, but also "null" (yes,
encapsulated) or blank.
Which could have been nice and sweet, other than that my mysql-db
translate the null value to an actual string (it seems).

So if hostname field is set to the value null, and then do a
select * from log_entry where hostname is null and job_id = xxx
Will produce nothing.

Which in itself is not a problem, but I would reckon this will
lead the HostnameDecorator not to trigger any action?

Question is:
1)Is it possible to remove/substitute strings from items. And on
specific variables/parser.items?

2)Is it possible to use to removeEnclosingCharsFromValue on specific
variables/parser.items?

Best regards
Ole Christian
/NorCERT

[Tor Johnson]

> I'm working on the Shadowserver Quote of The Day-service.

I'm working on the following Shadowserver reports:

* dns_openresolver
* microsoft_sinkhole
* scan_snmp
* scan_netbios
* scan_chargen
* scan_qotd

I hope to release the configs next week. The config for "Quote of the Day (QOTD)"-report
is attached. It's not tested at all.

> 1)Is it possible to remove/substitute strings from items. And on
> specific variables/parser.items?

Yes, use a rewriter. Example:

# Hostname may be "null". Rewrite to "".
parser.rewriters.0=hostname:(?i)(?:^null$)|(.*)-->$1

> 2)Is it possible to use to removeEnclosingCharsFromValue on specific
> variables/parser.items?

No, not on a specific variable. In this case it's works to remove double
quotes from all variable values:

# Remove enclosing "", e.g. from hostname.
parser.removeEnclosingCharsFromValue="

> The hostname can either be a hostname, but also "null" (yes,
> encapsulated) or blank.

Shadowserver use different values for "no value", for example:

* ""
* [blank]
* "null"
* 0

The config must then override the parser.item for the variable:

# ASN and port may be an integer or ""
parser.item.asn=\d*|""
parser.item.port=\d*|""|

/Tor

[Ole Christian Hansen]

> I'm working on the following Shadowserver reports:
>
> * dns_openresolver
> * microsoft_sinkhole
> * scan_snmp
> * scan_netbios
> * scan_chargen
> * scan_qotd

Actually I finished the last four today, except for the null-problem.

> # Hostname may be "null". Rewrite to "".
> parser.rewriters.0=hostname:(?i)(?:^null$)|(.*)-->$1

Hmm. That didn't seem to do the trick.
I suspect I work on an older Megatron version. The one I use
is from about August last year. Was the rewriters implemented back then?

[Ole Christian Hansen]

> Hmm. That didn't seem to do the trick.
> I suspect I work on an older Megatron version. The one I use
> is from about August last year. Was the rewriters implemented back then?

That seems to be it. I can see from the release notes this is new from
the 1.0.11 version.

Any quick fix for this, I mean get a missing class, and add the function
call in the parser, or would there be lot more dependencies?
Othwerwise this seems to be a job created for the practical extraction
and reporting languange ;)

[Tor Johnson]

> I suspect I work on an older Megatron version. The one I use
> is from about August last year. Was the rewriters implemented back then?

True, AttributeValueRewriter have not been released and is only available in
the GitHub-repo. You need to build Megatron using the Ant-script to use it.

> Any quick fix for this, I mean get a missing class, and add the function
> call in the parser, or would there be lot more dependencies?

One workaround is to not parse the hostname-field and let HostnameDecorator
do a reverse DNS lookup and add the hostname (if available).

First add HostnameDecorator:

# IP, ASN and country code exists already in file, but hostname is
# re-evaluated due to "null" strings in the field.
decorator.classNames.0=se.sitic.megatron.decorator.HostnameDecorator

Then skip parsing of $hostname by replacing it with "(?:".*?"|""|)" (without quotes).

Replace:

# Skips the following fields: protocol, tag, quote, region, and city.
parser.lineRegExp=^"$logTimestamp","$ipAddress",(?:".*?"|""|),$port,$hostname,(?:".*?"|""|),(?:".*?"|""|),$asn,"$countryCode",(?:".*?"|""|),(?:".*?"|""|)

to the following:

# Skips the following fields: protocol, hostname (re-evaluated), tag, quote, region, and city.
parser.lineRegExp=^"$logTimestamp","$ipAddress",(?:".*?"|""|),$port,(?:".*?"|""|),(?:".*?"|""|),(?:".*?"|""|),$asn,"$countryCode",(?:".*?"|""|),(?:".*?"|""|)

> Othwerwise this seems to be a job created for the practical extraction
> and reporting languange ;)

Nooooo, please don't mention the P-language again on this list :)

/Tor

[Tor Johnson]

Config with both alternatives, with or without rewriter, is attached.

I discovered a bug in AttributeValueRewriter. Fix have been pushed.

/Tor

[Ole Christian Hansen]

>
> First add HostnameDecorator:
> decorator.classNames.0=se.sitic.megatron.decorator.HostnameDecorator

> parser.lineRegExp=^"$logTimestamp","$ipAddress",(?:".*?"|""|),$port,(?:".*?"|""|),(?:".*?"|""|),(?:".*?"|""|),$asn,"$countryCode",(?:".*?"|""|),(?:".*?"|""|)

Nice one. Will save it for later use.
Just had a chat with one of the Shadowserver guys. They will filter out
these null's, so hopefully that will be the final solution

>
>> Othwerwise this seems to be a job created for the practical extraction
>> and reporting languange ;)>
> Nooooo, please don't mention the P-language again on this list :)

Can't give you any guarantees there ;)

/Ole Christian