Is it possible to disable DNS lookups completely for a job-type ?

[Kick Megatron]

Hi, I have a parser for compromised accounts, where I use a field as hostname to be able to map the actual organisation.
The hostname however do not have a DNS entry, it is purely used to map the organisation. And this is filling the megatron.log with the errors like below.

I did test with the inclusion of "dnsJava.useDnsJava=false” - but does not make a difference.

Any advise?

Thanks,
Kick


Example:

Organization 1
…
Domain names : DummyDomainForOrg1.com


2014-04-03 09:20:11.947 WARN - DNS lookup failed for hostname: DummyDomainForOrg1.com
java.net.UnknownHostException: unknown host
at org.xbill.DNS.Address.lookupHostName(Address.java:239)
at org.xbill.DNS.Address.getByName(Address.java:264)
at se.sitic.megatron.util.IpAddressUtil.dnsLookup(IpAddressUtil.java:319)
at se.sitic.megatron.decorator.IpAddressDecorator.execute(IpAddressDecorator.java:36)
at se.sitic.megatron.decorator.CombinedDecorator.execute(CombinedDecorator.java:67)
at se.sitic.megatron.decorator.DecoratorManager.executeDecorators(DecoratorManager.java:52)
at se.sitic.megatron.core.JobManager.processLine(JobManager.java:470)
at se.sitic.megatron.core.JobManager.execute(JobManager.java:140)
at se.sitic.megatron.core.JobScheduler.processFile(JobScheduler.java:145)
at Megatron.processCommands(Megatron.java:248)
at Megatron.main(Megatron.java:160)

[Tor Johnson]

Yes, it's possible to disable DNS lookups completely for a job-type.
Just remove IpAddressDecorator (hostname --> IP) and HostnameDecorator
(IP --> hostname) from the list of decorators:

decorator.classNames.0=se.sitic.megatron.decorator.AsnGeoIpDecorator
decorator.classNames.1=se.sitic.megatron.decorator.CountryCodeFromHostnameDecorator
decorator.classNames.2=se.sitic.megatron.decorator.CountryCodeDecorator

Default is to use the the CombinedDecorarator, which use the following
decorators:

# List of class name for CombinedDecorator to use.
decorator.combinedDecorator.classNames.0=se.sitic.megatron.decorator.IpAddressDecorator
decorator.combinedDecorator.classNames.1=se.sitic.megatron.decorator.AsnGeoIpDecorator
decorator.combinedDecorator.classNames.2=se.sitic.megatron.decorator.HostnameDecorator
decorator.combinedDecorator.classNames.3=se.sitic.megatron.decorator.CountryCodeFromHostnameDecorator
decorator.combinedDecorator.classNames.4=se.sitic.megatron.decorator.CountryCodeDecorator

In Megatron-lingo a "decorator" adds (or modify) data for a log record, e.g. DNS name from
an IP address.

> I did test with the inclusion of "dnsJava.useDnsJava=false" - but does not make a difference.

Nope, this just switch class library for the DNS lookups. Please turn this property on because
dnsjava <http://www.dnsjava.org/> is much faster than the implementation in the JDK.

/Tor

On 2014-04-03 11:32, Kick Megatron wrote:
> Hi, I have a parser for compromised accounts, where I use a field as hostname to be able to map the actual organisation.
> The hostname however do not have a DNS entry, it is purely used to map the organisation. And this is filling the megatron.log with the errors like below.
>
> I did test with the inclusion of "dnsJava.useDnsJava=false" - but does not make a difference.
>
> Any advise?
>
> Thanks,
> Kick
>
>
> Example:
>
> Organization 1

> ...

[Kick Megatron]

Hi Tor, thanks.

The decorators are set in the megatron.properties file. Tested with removal and it works.
How can you overwrite those values in a specific job-file - or should decorators actually be set only in job-types and not in the megatron.proporties file?

Thanks,
Kick

> --
> You received this message because you are subscribed to the Google Groups "Megatron Hacking" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to megatron-hacki...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

[Tor Johnson]

> The decorators are set in the megatron.properties file. Tested with removal and it works.

Hold your horses! Changes in "megatron-globals.properties" will affect *all* job-types.

Just add the changes to "decorator.classNames.*" in the specific job-type. Examples:

https://github.com/cert-se/megatron-java/blob/master/conf/job-type/brobot.properties
https://github.com/cert-se/megatron-java/blob/master/conf/job-type/ip-flowing.properties

Almost every property is defined globally in "megatron-globals.properties" and
can be overridden in the specific job-type configuration (conf/job-type/*).

/Tor

[Kick Megatron]

Thanks Tor, I have just declared the decorators as empty in the job-type.
Ps. I oversaw the impact on the megatron.properties file, was just for testing. But uncommenting the declarations in the job-type would make no sense and there was no true/false possibilties (I could find).


Regards,
Kick