Mapping on AS, IP address not in DB, org_id set to NULL

[Kick Megatron]

HI,
In my DB I have set most of our IP address ranges and claim our main 'AS' on one Organisation.
When an events comes in for an IP address not in the DB, but matches on ‘AS', the org_id is not set to the ‘AS' holder.


mysql> select count(*) from log_entry where org_id is NULL;
+----------+
| count(*) |
+----------+
| 233 |
+—————+

And no emails with /home/u_megatron/megatron-python/script/mega-sender.py for these IP addresses.

Something I should adjust / take into account?

Thanks,
Kick

[Tor Johnson]

> In my DB I have set most of our IP address ranges and claim our main 'AS' on one Organisation.

Sounds like a good setup.

> When an events comes in for an IP address not in the DB, but matches on ‘AS', the org_id is not set to the ‘AS' holder.

Then something has went wrong. Please check the following:

* Is AS added to the log row? Check "log_entry.asn" and "log_entry.asn2" in the database, or do a
file export (--nod-db --export).

* How is AS added? Sometimes it's parsed from the log file [1], and sometimes it's added by the
AsnGeoIpDecorator [2], [3].

* Is the AS matched? Is OrganizationOrCountryCodeFilter used and "filter.organizationFilter.matchAsn=true"?
See [2] (no matching) and [3] (will match AS).

[1] https://github.com/cert-se/megatron-java/blob/master/conf/job-type/shadowserver-drone2.properties
[2] https://github.com/cert-se/megatron-java/blob/master/conf/job-type/ip-flowing-fast.properties
[3] https://github.com/cert-se/megatron-java/blob/master/conf/job-type/compromised-accounts.properties

/Tor

[go...@acode.se]

Hi Kick!


I had a look in our database and we have a lot of records where the log-entry has no organisation.
That happens when there is no organisation with a matching domain name, ASN or IP-address.

I ran this query agains our DB and got no hits at all, which is correct:
select count(*) from log_entry le, asn where le.org_id = NULL and le.asn is not NULL and le.asn = asn.asn;

I find it strange that your log-entries are not bound to an organisation where the ASN is defined.

Please let us know you find any more clues that can help us trouble shoot this.

/Göran

[Kick Megatron]

Hi Goran / Tor, thanks for your response.

Goran, your query will (I believe) always give ‘0’ back due to the ‘=‘ usage. 

Please verify: select * from log_entry le join asn on (le.asn = asn.asn) where le.org_id is NULL and le.asn is not NULL;

I think I found the issue. I missed:

# Add AS number and name as additional items
decorator.asnGeoIpDecorator.useAsnInLogEntry=false
decorator.asnGeoIpDecorator.addAsName=true

Now being added - will monitor closely (and test when I have some time over).

Thanks,

Kick

--
You received this message because you are subscribed to the Google Groups "Megatron Hacking" group.
To unsubscribe from this group and stop receiving emails from it, send an email to megatron-hacki...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[go...@acode.se]

Sorry, the equal sign was a mistake, it should be:
select count(*) from log_entry le, asn where le.org_id is NULL and le.asn is not NULL and le.asn = asn.asn;
Which also always returns zero.

Good that you found the problem.

/Göran

[Kick Megatron]

Hello, to start the year nice - need to come back on an issue described below - which I seem to not have fixed completely … 

I have the following situation:

- One of my Organisations I have only set with an AS number (all others I have their IP address ranges set).

- The parsing is matching the ASN number (shown with —export, example below), but misses the country code (not sure if relevant)

- The Organisation is not matched - therefore no email address seen in the below —export output. 

- The ASN is not written to the DB

- When I add the IP address range to the organisation / AS in the DB, it matches completely - adds the country code and the organisation.

What am I missing (?)

Example data used:

Src IP address: 192.168.1.2

ASN: 65455

—

2014-01-01 00:34:37 UTC 192.168.1.2                                                                                                          asn="65455", asn2="", Category="openresolvers", Comment="", DstIP="      ", DstPort="    ", Malware="", OldCategory="openresolvers", SrcPort="   "       0       2014-01-02 10:29:39 UTC                                 2014-01-01 00:34:37     192.168.1.2  65455    openresolvers           GB              2014-01-02 02:37:03     openresolvers   ripencc GB      2684829

---

Note: the above is one line.

—

filter.preLineProcessor.classNames.0=se.sitic.megatron.filter.LineNumberFilter

filter.lineNumberFilter.excludeIntervals=1-1

# Skip file if same as previous file

general.fileAlreadyProcessedAction=skip

fileProcessor.classNames.0=se.sitic.megatron.fileprocessor.DiffProcessor

# Filter: Entries are decorated *before* filter.

#filter.preStorage.classNames.0=se.sitic.megatron.filter.OrganizationOrCountryCodeFilter

filter.organizationFilter.matchIpAddress=true

#filter.organizationFilter.matchHostname=true

filter.organizationFilter.matchAsn=true

#filter.countryCodeFilter.includeCountryCodes=

# Add AS number and name as additional items

decorator.asnGeoIpDecorator.useAsnInLogEntry=false

decorator.asnGeoIpDecorator.addAsName=false

—

Thanks - and happy new year to you all.

Kick

[Tor Johnson]

> - The parsing is matching the ASN number (shown with —export, example below), but misses the country code (not sure if relevant)

Do you parse ASN och country code (i.e. is "$asn" and "$countryCode" included in "parser.lineRegExp")?

You have two choices:

1. Parse ASN from the log line (see e.g. [1]).

2. Skip ASN field but parse IP and lookup ASN from IP by using AsnGeoIpDecorator (or AsnDecorator if AS
info is imported to the database). See [2]. AsnGeoIpDecorator is included in CombinedDecorator by default.

> - The ASN is not written to the DB

This is the problem. There is no ASN to match. Either ASN is not parsed correctly or AsnGeoIpDecorator
is not working.

If you send me the configuration file + sample data off list I can take a look.

[1] https://github.com/cert-se/megatron-java/blob/master/conf/job-type/brobot2.properties
[2] https://github.com/cert-se/megatron-java/blob/master/conf/job-type/ip-flowing-fast.properties

/Tor

[Kick Megatron]

Hi Tor, I do not “trust” the provided ASN or Country code provided in the lines. The Source IP address is my leading info.
The ASN is therefore retrieved from the IP address (with the decorator) - and the org should be found via the ASN (or country code I now understand).

I seem to have fixed the issue with the following changes.

Added:
===
# Adds ASN + CC
decorator.classNames.0=se.sitic.megatron.decorator.AsnGeoIpDecorator
decorator.classNames.1=se.sitic.megatron.decorator.CountryCodeDecorator

Commented out the following two lines

# Add AS number and name as additional items

#decorator.asnGeoIpDecorator.useAsnInLogEntry=false
#decorator.asnGeoIpDecorator.addAsName=false


The removed config lines seems to be around "decorator.asnGeoIpDecorator.useAsnInLogEntry=false”.
I probably miss the understanding of what this line is accomplishing. I am not parsing the ASN from the logline, but use the decorator. By stating false - I am unable to map the Organization.

With the above mentioned changed - I now see the Org being found for IP addresses not known in the DB (and therefore mapping must be done on ASN or CC).


I will sent you an email off-list for verification.


Thanks,
Kick

[Tor Johnson]

> Hi Tor, I do not “trust” the provided ASN or Country code provided in the lines. The Source IP address is my leading info.

Good thinking! For some sources I trust the provided ASN and CC.

> # Adds ASN + CC
> decorator.classNames.0=se.sitic.megatron.decorator.AsnGeoIpDecorator
> decorator.classNames.1=se.sitic.megatron.decorator.CountryCodeDecorator

This is not necessary if you use the default settings. The original "megatron-globals.properties" [1]
contains the following:

# List of class name for decorators to use.
decorator.classNames.0=se.sitic.megatron.decorator.CombinedDecorator

[...]

# -- CombinedDecorator
# List of class name for CombinedDecorator to use.
decorator.combinedDecorator.classNames.0=se.sitic.megatron.decorator.IpAddressDecorator
decorator.combinedDecorator.classNames.1=se.sitic.megatron.decorator.AsnGeoIpDecorator
decorator.combinedDecorator.classNames.2=se.sitic.megatron.decorator.HostnameDecorator
decorator.combinedDecorator.classNames.3=se.sitic.megatron.decorator.CountryCodeFromHostnameDecorator
decorator.combinedDecorator.classNames.4=se.sitic.megatron.decorator.CountryCodeDecorator

> Commented out the following two lines
> # Add AS number and name as additional items
> #decorator.asnGeoIpDecorator.useAsnInLogEntry=false
> #decorator.asnGeoIpDecorator.addAsName=false

I guess this solved your problem.

[1] https://github.com/cert-se/megatron-java/blob/master/conf/dev/megatron-globals.properties

/Tor