Unable to get time_key setting to work from JSON timestamp

[c...@rca.io]

The issue I'm seeing is that time timestamp in the search is not matching the timestamp in the log.

Here's my config:

<source>

  @type tail

  path /logs/mysql/audit.log

  pos_file /var/log/td-agent/audit.log.pos

  tag mysql.audit

  format json

</source>

<match mysql.audit>

  type elasticsearch

  host isolinear-eg.corp.apple.com

  port 9200

  index_name mysql_audit

  include_tag_key true

  logstash_format true

  logstash_prefix mysql_audit

  time_key audit_record.timestamp

  time_format %Y-%m-%dT%H:%M:%S %Z

  flush_interval 10s # for testing

</match>

And here is the result in elastic search:

{

  "_index": "mysql_audit-2017.04.18",

  "_type": "fluentd",

  "_id": "AVuBrutMy6H0rNsJZZHy",

  "_score": null,

  "_source": {

    "audit_record": {

      "name": "Connect",

      "record": "447474053_2017-04-11T22:30:21",

      "timestamp": "2017-04-18T15:29:01 UTC",

      "connection_id": "21450",

      "status": 0,

      "user": "solver",

      "priv_user": "solver",

      "os_login": "",

      "proxy_user": "",

      "host": "",

      "ip": "10.108.251.201",

      "db": "solver"

    },

    "tag": "mysql.audit",

    "@timestamp": "2017-04-18T10:29:02-05:00"

  },

  "fields": {

    "@timestamp": [

      1492529342000

    ]

  },

  "sort": [

    1492529342000

  ]

}

What I'm trying to do is get the record timestamp to match the timestamp in the log (in audit_record.timestamp).

Thanks!

[Mr. Fiber]

> time_key audit_record.timestamp

This refers record["audit_record.timestamp"] key not record]"audit_record"]["timestamp"].

This is why tmestamp uses current time.

You can specify nested timestamp in es side:

http://stackoverflow.com/questions/28572914/elasticsearch-correct-way-to-specify-nested-field-for-timestamp

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.