Miliseconds are removed from time when data collected using fluentd

[Sar82]

I am using fluentd as data collector. But milliseconds from time is getting removed. Can you please suggest a solution to keep milliseconds in generated output.

Time is getting parsed only when I specify logstash_format true. But because of this indices are getting created as 'Logstash**' even if I specify index name. If I don't specify 'logstash_format true ' then except time field everything is getting parsed.

Here is my fluentd configuration file,

type tail 
tag json.nwdata 
path /etc/td-agent/access.log 
pos_file /etc/td-agent/access.log.pos 
format json
time_format %Y/%m/%d %H:%M:%S
read_from_head true 
time_key time 
include_time_key true

type elasticsearch
logstash_format true 
time_key time 
flush_interval 10s # for testing 
index_name jsonlog 
type_name jsonlogtype

[Mr. Fiber]

Set "keep_time_key true" in in_tail may help.

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sar82]

It is still not parsing time field. 

[Mr. Fiber]

What do you mean?

Stored events in Elasticsearch doesn't have correct @timestamp field?

On Fri, Aug 21, 2015 at 2:31 PM, Sar82 <sarik...@gmail.com> wrote:

It is still not parsing time field. 

--

[Sar82]

Generated JSON does not contain time field. It just ignores it.

[Robert Swain]

I have the same problem. I saw that sub-second accuracy for timestamps had been implemented in the 0.14.0.pre.1 release.

I am using the fluentd docker log driver, fluentd with fluent-plugin-elasticsearch and elasticsearch. When I look at the logs in Kibana, all the log messages only have second accuracy. This means the log messages are all out of order within one second and makes it unusable for debugging purposes.

Is it possible with 0.14.0.pre.1 or the latest 0.12.x and fluent-plugin-elasticsearch to have records with at least millisecond accuracy? If so, can you describe exactly how?

Thanks,

Rob

[Mr. Fiber]

elasticsearch plugin has time_key option.

See README: https://github.com/uken/fluent-plugin-elasticsearch#time_key

[Robert Swain]

OK. But that's not a complete documentation of the solution.

So then I guess I will need to investigate if the docker log driver and forward input have at least millisecond accuracy. If they do then I can use that. If not then I probably have to use some ruby code to create a timestamp with millisecond or better accuracy and write it to a new key, then use that key as @timestamp in the elasticsearch plugin. This should work for 0.12.x I expect, but will it be necessary in 0.14.x or will there then be a better solution?

[Robert Swain]

Looking at the output of a stdout filter immediately after the forward input with 0.12.x it is clear that it is only providing second accuracy. I have not yet tested with 0.14.x

Looking at the Ruby Time class information, it seems that when requiring the time module, Time.now.utc.iso8601(3) (http://devblog.avdi.org/2009/10/25/iso8601-dates-in-ruby/) should provide a well-formatted timestamp for use with elasticsearch.

I tried this:

<source>

  type forward

  port 24224

  bind 0.0.0.0

</source>

<filter **>

  @type record_transformer

  enable_ruby

  <record>

    timestamp ${ require 'time'; Time.now.utc.iso8601(3) }

  </record>

</filter>

<match **>

  @type             stdout

</match>

And observed this:

2016-01-14 13:10:35 +0000 docker.e50ab96f58d7: {"container_id":"e50ab96f58d7fc5c6a7f54a009baae186b719d7af70efb0da2ea2d36fe1ac467","container_name":"/tiny_northcutt","source":"stdout","log":"Hello","timestamp":"2016-01-14T13:10:35.275Z"}

So I guess the record_transformer part and then the time_key option in the elasticsearch plugin is a pretty simple solution for now.

How should it be done with 0.14.x?

[Mr. Fiber]

> require 'time'

Fluentd requires time module in the core so `require 'time'` is not needed in the configuration.

> How should it be done with 0.14.x?

It depends on logger. v0.14's event time is nano-second resolution.

fluent-logger-golang need to support nano or milli second resolution in the event.