Fluentd doesn't show logs with milliseconds, only unix epoch time.

[gabriel...@integritas.net]

Hi Guys,

I am new to EFK stack and I am trying to see fluentd logs with milliseconds but it's been to hard to make it work. It's almost 3 days trying. I see it works with some plugins such as fluent-plugin-elasticsearch-timestamp-check, however as I am using fluentd v0.14 I think I don't need this plugin as it should support milliseconds.

I am sending logs to stdout to troubleshoot. And I can see that 1 line get's milliseconds. See below, I highlighted it. 

2017-11-10 18:24:12 +0000 [info]: #0 starting fluentd worker pid=49 ppid=45 worker=0

2017-11-10 18:24:12 +0000 [info]: #0 listening port port=24224 bind="0.0.0.0"

2017-11-10 18:24:12 +0000 [info]: #0 fluentd worker is now running worker=0

2017-11-10 18:24:12.211649495 +0000 fluent.info: {"worker":0,"message":"fluentd worker is now running worker=0","time":1510338252.21165}

2017-11-10 18:24:11.000000000 +0000 docker.075226310e66ef7d08e4d5b3eb4ad430abcd5959f355dd62b14ab2543f618f7e: {"source":"stdout","log":"NodeJS Application waithing TCP connection tcp://172.17.0.3:5140...","container_id":"075226310e66ef7d08e4d5b3eb4ad430abcd5959f355dd62b14ab2543f618f7e","container_name":"/node_app","time":1510338251.0}

2017-11-10 18:24:14.000000000 +0000 docker.075226310e66ef7d08e4d5b3eb4ad430abcd5959f355dd62b14ab2543f618f7e: {"container_name":"/node_app","source":"stdout","log":"Start NodeJS Application","container_id":"075226310e66ef7d08e4d5b3eb4ad430abcd5959f355dd62b14ab2543f618f7e","time":1510338254.0}

But the rest of the logs itself all has a bunch of zeros ( 000000000 )

And the time in the logs it's a unix epoch time with no milliseconds.

Here is my fluent.conf

<source>

  @type forward

  port 24224

  bind 0.0.0.0

</source>

<filter docker.**>

  @type record_transformer

  enable_ruby

  <record>

    time ${ Time.at(record["time"].to_i * 1000).strftime("%Y-%m-%dT%H:%M:%S.%L%z") }

  </record>

</filter>

<match *.**>

   @type stdout

time_format %Y-%m-%dT%H:%M:%S.%L%z

time_key time

keep_time_key true

</match>

I was trying to use record_transform to somehow convert it. But nothing seems to work.

Can you please help me?

Thanks

[Mr. Fiber]

> But the rest of the logs itself all has a bunch of zeros

Maybe, you hit current fluentd logging driver limitaion.

https://docs.fluentd.org/v0.12/articles/docker-logging

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.