Anybody familiar with parsing .net 64bit timestamp

[Kick Megatron]

Hello, I am working on a parser for logfiles having Microsoft .Net 64 bit timestamps

Timestamp examples:

130379098763733422

130379098763889429

Which are representing: “Essentially represents the number of 100-nanosecond intervals since 00:00 01/01/1601.”

And I have a reference to: http://msdn.microsoft.com/en-us/library/system.datetime.tofiletimeutc.aspx

Anybody has this working already to parse in Megatron (or has an idea) ?

Thanks,

Kick

[Tor Johnson]

Windows epoch (starting at year 1601) is not supported in Megatron.

Unix epoch is supported by specifying "epochInSec" or "epochInMs" for the
property "parser.timestampFormat" in "megatron-globalsproperties" [1]

Possible solutions:

* Implement a script that converts the timestamp before parsing it in
Megatron. The script can be called from Megatron by using
"fileProcessor.osCommandProcessor.command". See [2] for an example
with sed.

* Add support for "windowsEpochinNs" in LogEntryMapper [3]. See line
107 how "epochInSec" is implemented.

I wrote a simple test-program which includes your timestamps:

public static void main(String[] args) {
long[] windowsEpochVals = { 130379098763733422L, 130379098763889429L };
// diff between Windows epoch and the Java epoch (in milliseconds)
// see https://svn.apache.org/repos/asf/activemq/activemq-dotnet/Apache.NMS/trunk/src/main/csharp/Util/DateUtils.cs
long epochDiff = 1164447360000L;
for (int i = 0; i < windowsEpochVals.length; i++) {
long windowsEpoch = windowsEpochVals[i];
Date date = new Date((windowsEpoch / 1000000L) + epochDiff);
System.out.println(windowsEpoch + " --> " + date);
}
}

Output:
130379098763733422 --> Wed Jan 12 11:00:58 CET 2011
130379098763889429 --> Wed Jan 12 11:00:58 CET 2011

Is this correct?

Since Java timestamp is in milliseconds and Windows is in nanoseconds the
value will be truncated.

[1] https://github.com/cert-se/megatron-java/blob/master/conf/dev/megatron-globals.properties
[2] https://github.com/cert-se/megatron-java/blob/master/conf/job-type/stopforumspam.properties
[3] https://github.com/cert-se/megatron-java/blob/master/src/se/sitic/megatron/parser/LogEntryMapper.java

/Tor

[Kick Megatron]

HI Tor / Goran, thanks so far.

For clarity, my 2 example lines should result in:

130379098763733422

Epoch: 1393436276

Wed, 26 Feb 2014 17:37:56 GMT

130379098763889429

Epoch: 1393436276 (seems to be same Epoch I see now as above …)

Another example to force different epoch:

130379691525956712

Epoch: 1393495552

Thu, 27 Feb 2014 10:05:52 GMT

The way I got the above info is using multiple sources.

My colleague created quickly the below perl which have been used to verify the above examples - and they match:

#!/usr/bin/perl

use POSIX qw(tzset);

$ENV{TZ} = 'UTC';

tzset;

my $div = "10000000";

my $sub = "11644473600";

my $time = $ARGV[0];

my $epoch = ( $time / $div ) - $sub;

print scalar localtime($epoch) . "\n";

Tor, I am in no java programming ninja position to identify the error in your code :)

Thinking about best way to use in or before Megatron processing.

Regards,

Kick

[Tor Johnson]

> For clarity, my 2 example lines should result in:

Thanks for the test cases. I didn't get that is was 100 ns intervals.
Here is the correct code:

public static void main(String[] args) {

long[] windowsEpochVals = { 130379098763733422L, 130379098763889429L, 130379691525956712L };
// diff between Windows epoch and the Java epoch (in seconds)
// see https://svn.apache.org/repos/asf/activemq/activemq-dotnet/Apache.NMS/trunk/src/main/csharp/Util/DateUtils.cs
long epochDiff = 11644473600L;

for (int i = 0; i < windowsEpochVals.length; i++) {
long windowsEpoch = windowsEpochVals[i];

long unixEpochInSec = (windowsEpoch / 10000000L) - epochDiff;
Date date = new Date(unixEpochInSec * 1000L);
System.out.println(windowsEpoch + " --> " + date + " (" + unixEpochInSec + ")");
}
}

Output:

130379098763733422 --> Wed Feb 26 18:37:56 CET 2014 (1393436276)
130379098763889429 --> Wed Feb 26 18:37:56 CET 2014 (1393436276)
130379691525956712 --> Thu Feb 27 11:05:52 CET 2014 (1393495552)

> Thinking about best way to use in or before Megatron processing.

Give it a try to call the Perl-script from Megatron, e.g.:

fileProcessor.classNames.0=se.sitic.megatron.fileprocessor.OsCommandProcessor
fileProcessor.osCommandProcessor.command=perl convert-timestamps.pl $inputFile

/Tor