in_tail incorrect time field parsing in json format

[Oleg Hordiychuck]

I log some location events in node.js using bunyan into a file with utc 8601 time like this:

{"name":"sectors.js","hostname":"oleh-ThinkPad-Edge-E440","pid":9891,"level":30,"mac":"f0:e7:7e:e1:72:b9","latLng":{"lat":0.013039102105423124,"lng":-0.05976372262298313},"sector":"office","msg":"","time":"2014-12-05T15:09:14.485Z","v":0}

And read this data using fluentd in_tail plugin. But my output has incorrect time:

1970-01-01 03:33:34 +0300 debug.rtls: {"name":"sectors.js","hostname":"oleh-ThinkPad-Edge-E440","pid":9891,"level":30,"mac":"f0:e7:7e:e1:72:b9","latLng":{"lat":0.013300138930075528,"lng":-0.06091011599436664},"sector":"office","msg":"","v":0}

Is it bug?

[Kiyoto Tamura]

In your tail plugin config, add the following line.

time_format %Y-%m-%dT%H:%M:%S

The reason is: unless "time_format" is specified, the JSON parser assumes that the time field is already epoch time and just calls Object#to_t on it, which means "2014-12-05T15:09:14.485Z".to_i, which is nil (hence you get Unix epoch time 0).

Kiyoto

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Check out Fluentd, the open source data collector to unify log management.

[karthik]

Hi,
Even I faced the same issue. I was getting an Unix epoch time 0 ( 1970-01-01) from my in_tail plugin. But after adding  time_format %Y-%m-%dT%H:%M:%S, My issue was resolved. Although I see a performance issue now by adding time_format to my in_tail plugin. Number of lines of logs read/written per second has reduced drastically. Before adding time_format, fluentd used to write 15k logs per second but after adding time_format it is reading at max ~8k logs per second. Please let me know how I can handle this performance degradation.

Thanks,

Karthik

[karthik]

Can anyone please put me in a right direction ?

Thanks