New release: Megatron v1.0.11

[Tor Johnson]

Ahoy List!

I'm happy to announce that a new version of Megatron is available:
<https://www.cert.se/megatron/megatron-pub-1.0.11.tar.gz>
<https://github.com/cert-se/megatron-java/releases>

The new version contains the following new features and improvements:

* New: Organization report is target for organizations with a huge volume of
abuse cases, e.g. ISPs and web hotels. Instead of getting one email per data
source, they will receive one report every 24 hours that is machine parsable.

* New: Log entries in mail body are now also included as an attachment in a
standardized format that is machine parsable.

* New: MultithreadedDnsProcessor makes DNS lookups and reverse lookups in
several threads to increase performance. See the following configurations:
ip-flowing and ip-flowing-verbose.

* New: --whois takes a list of IPs, hostnames, or URLs and print the following
information to stdout for each item: IP, AS, CC, hostname, AS name, and
organization. Items may be specified in a file.

* New: Filter log entries by occurrence (OccurrenceFilter), e.g. "include first
20 matches of the same IP address" or "include log entries with more than 10
occurrences of the same URL".

* New: Added rewriters for both parser and export (parser.rewriters,
export.rewriters). Example: can rewrite URLs from "http" to "hxxp",
or mask IP addresses (10.20.x.x).

* New: Writes file export to stdout instead of to a file (--stdout).

* New: Added property (rss.job.enabled) to turn on/off writing to job RSS file.

* New: Support for Windows epoch ("Windows NT time format").
Set "parser.timestampFormat" to "windowsEpoch".

* Improved: If quarantine is turned off (mail.ipQuarantinePeriod=0) no SQL
queries for quarantine checking are executed. Configuration for each job in
--job-list is loaded.

* Improved: Several new job types (configurations) have been added or updated.

For a complete list of changes, see the release notes:
<https://github.com/cert-se/megatron-java/blob/master/doc/release-notes.txt>


How to Upgrade an Existing installation?

* Copy new jar-file: "dist/sitic-megatron.jar" to "lib/"

* Diff "conf/megatron-globals.properties". The following properties are changed:
- New: rss.job.enabled
- New: parser.rewriters
- New: export.rewriters
- New: filter.occurrenceFilter.attributeNames
- New: filter.occurrenceFilter.excludeIntervals
- New: filter.occurrenceFilter.includeIntervals
- New: filter.occurrenceFilter.fileSorted
- New: fileProcessor.multithreadedDnsProcessor.noOfThreads
- New: fileProcessor.multithreadedDnsProcessor.reverseDnsLookup
- New: fileProcessor.multithreadedDnsProcessor.regExpIp
- New: fileProcessor.multithreadedDnsProcessor.regExpHostname
- New: mail.attachmentHeaderFile
- New: mail.attachmentRowFile
- New: mail.attachmentFooterFile
- New: mail.attachmentName
- New: export.jobTypeNameMapper
- New: report.organization.noOfHours
- New: report.organization.jobTypes
- New: report.organization.recipients

* Diff the "conf/job-type" directory. The following job types are changed:
- New: ikyon
- New: brobot
- New: brobot2
- New: ddos-amplification
- New: ip-flowing-fast
- New: ip-flowing-verbose
- New: megatron-whois-ip
- New: megatron-whois-hostname
- New: shadowserver-openresolver
- New: shadowserver-qotd
- New: shadowserver-chargen
- New: shadowserver-sinkhole-http-drone
- New: shadowserver-netbios
- New: shadowserver-snmp
- New: shadowserver-microsoft-sinkhole
- Improved: shadowserver-sinkhole-http-drone
- Improved: ip-flowing

* "megatron.sh" have been modified.

* Create the following indices (if not present):
CREATE INDEX mj_started ON mail_job (started);
CREATE INDEX mj_finished ON mail_job (finished);
CREATE INDEX mjlem_job_id ON mail_job_log_entry_mapping (mail_job_id);
CREATE INDEX mjlem_log_id ON mail_job_log_entry_mapping (log_entry_id);

* Verify that everything works:
- megatron.sh --version
- megatron.sh --whois 1.2.3.4


I have also updated the FAQ:
<https://github.com/cert-se/megatron-java/wiki/Megatron-FAQ>

Best regards,

/Tor