Datetime conversion in Fluentd for use in Elastic Search

[Marco Pas]

I am looking for a good approach on how to convert a "time_stamp" field in a record to a date that Elastic Search will understand.

Currently i have a hack approach that is probably not the best and i am looking for improvements in that.

My log entries have a structure like;

2016-06-07 19:33:23.596  INFO 45469 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet Engine

So when the log data appears i parse the log and get a record with a "time_stamp" in it being..

time_stamp = 2016-06-07 19:33:23.596

converting
from -> 2016-06-07 19:33:23.596
to > 2016-06-07T19:33:23.596+00:00

To convert the date into a timestamp that Elastic Search understands i am using a record transformer with Ruby enabled. 

<filter springboot.**>
  @type record_transformer
  enable_ruby
  <record>
    es_time_stamp ${ require 'date'; DateTime.parse(__send__('time_stamp')).iso8601(3) }
  </record>
</filter>

This is a hack but works, it keeps the milliseconds etc.. Which are important.

I assume the fluent-plugin-parser is a better option but i seem to hit a wall when trying to convert the date into the es_time_stamp field.
When using the plugin-parser i keep getting warning that the pattern cannot be matched and to me it is not clear how i could use the plugin-parser to convert the date to a value that i actually can use..

<filter springboot.**>
  @type parser
  format /^(?<time>.*?)\s{2,}/
  time_format %Y-%m-%dT%H:%M:%S.%N
  key_name message
</filter>

Any better suggestion or help on how i can do the date conversion?