How to parse JSON from Docker

[Tom Wu]

Enter code here...
Hi all,

I know that you can use parse or flatten hash plugin to parse a while JSON message, how about this format from log driver fluentd? 

2016-01-28 20:04:23 +0000 docker.4643317973d5: {"container_id":"4643317973d56010ecb5990680a40e59be523f04a456686fa565430574e03245","container_name":"/clever_wilson","source":"stdout","log":"{\"key\":\"value\"}","@log_name":"docker.4643317973d5"}

Elasticsearch plugin output looks like, the log field is being treated as string, what I want is to have key being indexed from log body.

{

  "_index": "logstash-2016.01.28",

  "_type": "fluentd",

  "_id": "AVKJ2JABH3dkBxm_sj8h",

  "_score": null,

  "_source": {

    "container_name": "/suspicious_mccarthy",

    "source": "stdout",

    "log": "\"test\":{\"key\":\"value\"}",

    "container_id": "f4a85be832aef70ae4f06258ab7394683a93fdfab704556c68cc976df09d531e",

    "@log_name": "docker.f4a85be832ae",

    "@timestamp": "2016-01-28T20:06:46+00:00"

  },

  "fields": {

    "@timestamp": [

      1454011606000

    ]

  },

  "sort": [

    1454011606000

  ]

}

[Tom Wu]

I think it's more problem of docker where it's log output escaping all double quotes, making it as string

{"log":"{\"key\":\"value\"}\n","stream":"stdout","time":"2016-01-28T21:33:00.642966184Z"}

What's best practice parsing JSON out of Docker json log? Should it be fixed at the source? or replace \" with " in fluentd filter or parser?

 

[Mr. Fiber]

This plugin may help.

https://github.com/gmr/fluent-plugin-json-in-json

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tom Wu]

This is a working solution, https://github.com/docker/docker/issues/17830

[Jakub Veverka]

This was one of the top links when I was looking for solution to this problem.
There is fine solution to solve Docker weird log format in fluentd docs - http://www.fluentd.org/guides/recipes/docker-logging