Elasticsearch not able to identify the time field defined in Fluentd

[Saurabh Vartak]

Issue: I have the logs getting logged into my log file and each log entry has the time information. The time field in the log is not getting recognized in Elasticssearch and hence I am not able to select this time information while creating my Kibana indexes.

Environment:

Ubuntu 18.04

Fluentd 1.12.3 

td-agent 4 (curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-bionic-td-agent4.sh | sh)

elasticsearch-7.12.1

kibana-7.12.1

td-agent.conf:

<source>

  @type tail

  path /var/log/sau.log

  pos_file /var/log/td-agent/sau.pos

  read_from_head true

  tag log

  <parse>

    @type regexp

    expression /^\[(?<logtime>[^\]]*)\] (?<name>[^ ]*) (?<title>[^ ]*) (?<id>\d*)$/

    time_key logtime

    keep_time_key true

    time_format %Y-%m-%d %H:%M:%S %z

    types id:integer

  </parse>

</source>

<match *.**>

  @type elasticsearch

  host localhost

  port 9200

  index_name sauindx1

</match>

log entry sample: [2013-02-28 12:00:00 +0900] sau engineer 1

log entry simulation command: echo "[2013-02-28 12:00:00 +0900] sau engineer 1" | tee -a /var/log/sau.log

Official doc referred: https://docs.fluentd.org/parser/regexp#example

log entry in stdout through fluentd:

2013-02-28 03:00:00.000000000 +0000 log: {"logtime":"2013-02-28 12:00:00 +0900","name":"sau","title":"engineer","id":1}

As shown in the above line, when I do an stdout print from Fluentd, I get the above line which clearly indicates that the time field defined by me is recognized properly. 

However when I divert the logs to Elasticsearch and then create the Kibana index, I am not able to get the time field recognized. 

Is there anything I am doing wrong here? Needed some guidance on this please.

Reference links:

Elasticsearch installation -- https://www.elastic.co/guide/en/elasticsearch/reference/7.12/deb.html#install-deb

Kibana installation -- https://www.elastic.co/guide/en/kibana/7.12/deb.html#install-deb

Fluentd installation -- https://docs.fluentd.org/installation/install-by-deb#step-1-install-from-apt-repository

Regex expression -- https://docs.fluentd.org/parser/regexp#example

[Dmitriy Doroshenko]

Hi,

ES by default accept %Y-%m-%dT%H:%M:%S.%L %z or iso8601 time format.

So try to use record_modifier to change your logtime key value.

Dne čtvrtek 13. května 2021 v 21:11:09 UTC+2 uživatel saurabh...@gmail.com napsal:

[Saurabh Vartak]

Hi Dmitriy,

Firstly thanks a lot for helping me with this query. I had tried the record_modifier approach but somehow I am not able to get it working. I am receiving error when I try to do it. Below is the section that I had introduced. Can you please guide me where I may be going wrong?

<filter *.**>
  @type record_transformer
  enable_ruby
  <record>
    newtime ${DateTime.parse(__send__('logtime')).iso8601(3)}
  </record>
</filter>

Am I doing anything wrong?

Regards,

Saurabh

--
You received this message because you are subscribed to a topic in the Google Groups "Fluentd Google Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/fluentd/w4LwHFy_NJQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to fluentd+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fluentd/fe7f57a2-1e6b-416e-a27d-89df30ef5f0fn%40googlegroups.com.

[Dmitriy Doroshenko]

You could try

<filter **>
    @type record_transformer
    enable_ruby
    <record>
        timestamp ${logtime.iso8601}
    </record>
    remove_keys logtime
</filter>

Dne pátek 14. května 2021 v 10:39:42 UTC+2 uživatel saurabh...@gmail.com napsal:

[Saurabh Vartak]

Hi Dmitriy,

I was finally able to find one way. I created the below record_transformer to convert to ISO8601 format and it worked for me. Thank you very much for the pointer. Please do let me know if there is a better way to do it.

<filter *.**>
  @type record_transformer
  enable_ruby

  auto_typecast true
  <record>
    datetime_received ${require 'time'; Time.parse(record["logtime"].to_s + " +0530").iso8601.to_s}
  </record>
</filter>

[Dmitriy Doroshenko]

Saurabh

could you try

<filter **>

  @type record_transformer

  enable_ruby true

  <record>

    timestamp ${Time.at(record['logtime']).iso8601}

  </record>

<filter>

it works for me

Dne pátek 14. května 2021 v 14:01:36 UTC+2 uživatel saurabh...@gmail.com napsal: