Fluentd + Docker = confusion

[Bas Tichelaar]

I'm fairly new to Fluentd, but I'm trying to setup a uniform logging layer using the new Docker 1.8 Fluentd logging driver. I got it working using the Docker documentation, but I'm a bit stuck on how to proceed now. My design is as follows: I'm running a Fluentd docker container, and I'm starting new Docker containers using the --log-driver fluentd option. The Fluentd container forwards logs to a central Fluentd collector, which sends the logs to Elasticsearch. Right now, the entries in Elasticsearch look like this for example:

{
  "_index": "log-2015.08.26",
  "_type": "fluentd",
  "_id": "AU9qKb-a9AxG_97j6yht",
  "_score": 1,
  "_source": {
    "source": "stdout",
    "log": "2015-08-26 13:15:09,634 ERROR [CassandraDriverActor] - Error during connecting to Cassandra: The requested compression is not available (some compression require a JAR to be found in the classpath)",
    "container_id": "192948bb6c47fd56402141686604ef7489ae583faea920201aabb72ba30c32c4",
    "container_name": "/mesos-20150826-041101-3860383754-5050-1-S1.765a7a8a-6a1b-4f39-9573-36bd3fd3cd61",
    "tag": "docker.test",
    "@timestamp": "2015-08-26T13:15:09+00:00"
  }
}

This is not very useful though. I need to get the timestamp from the actual logline (preferably in sub-second format) and use that for the @timestamp field, and the log needs to be cleaned. Furthermore, multiline logs (stacktraces) need to be joined. But as far as I can see, there is no option to use a Format or Parser plugin. Any help is really appreciated!

[Mr. Fiber]

Hmm...

If you want to parse "log" field and set parsed result into existing record, use fluent-plugin-parser.

>  Furthermore, multiline logs (stacktraces) need to be joined. But as far as I can see, there is no option to use a Format or Parser plugin.

This is hard because docker logging driver sends multiline log into one line logs.

Other logging drivers too.

So writing stateful filter or output is better to join multiple log contents.

Or output logs to file and using in_tail is another way.

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Satoshi Tagomori]

Hi,

As far as i know, log attributes which docker logging driver submits are flat in the top level of fluentd records.

So there are any reason for why your logs from docker (and its timestamp) are under "_source" in data

stored in your Elasticsearch.

Moris.

--

田籠 聡 (TAGOMORI Satoshi)
Mail: tago...@gmail.com

[qingwei wei]

Hi,

I am facing the similar problem. Is there any example on stateful filter that i can refer to?

Cw

[Eduardo Silva]

I am trying to understand the issue, do you want to accomplish this ?

 "source": "stdout",
    "log": "ERROR [CassandraDriverActor] - Error during connecting to Cassandra: The requested compression is not available (some compression require a JAR to be found in the classpath)",

    "container_id": "192948bb6c47fd56402141686604ef7489ae583faea920201aabb72ba30c32c4",
    "container_name": "/mesos-20150826-041101-3860383754-5050-1-S1.765a7a8a-6a1b-4f39-9573-36bd3fd3cd61",
    "tag": "docker.test",

    "timestamp": "2015-08-26 13:15:09,634"

regards,

--

Eduardo Silva

Open Source, Treasure Data

http://www.treasuredata.com/opensource

 

[qingwei wei]

I am referring to the multiline issue for docker logs.

[Dheeraj Uppalapati]

Hi,

I am trying to find a solution to the multiline docker logs issue as well.

Any pointers how to address this ? There was a mention of stateful filter, any example code we can refer to ?

Thanks,

Dheeraj

[Mr. Fiber]

> There was a mention of stateful filter, any example code we can refer to ?

https://github.com/fujiwara/fluent-plugin-suppress/blob/master/lib/fluent/plugin/filter_suppress.rb

How about this? suppress filter stores one states for checking logs across event streams.

[Dheeraj Uppalapati]

Great! Thanks!