So I am using logspout with a gelf plugin (vincit/logspout-gelf) to output gelf to fluentd (td-agent). Fluentd is configured with a gelf input plugin (MerlinDMC/fluent-plugin-input-gelf). The aim is to capture docker container logs and send them to a single forwarder with one egress point from the EC2 host. The forwarder already hoovers up all the EC2 hosts syslogs happily and adds the correct date/time to each event.
When I start fluentd with this config I get my logs forwarded to Graylog but the time format is wrong (epoch + uptime):
<source>
type gelf
protocol_type udp
port 12202
tag stuff
</source>
A logged event looks like this:
1970-01-01 01:33:38 +0100 stuff: {"version":"1.1","host":"server1","short_message":"2018-02-09T16:07:25.546Z [access-log] ::ffff:1.2.3.4 - \"GET /find HTTP/1.1\" 200 5567 \"-\" \"ELB-HealthChecker/2.0\"1234","level":3,"image_id":"sha256:12345","image_name":"hello-world","container_id":"12345","container_name":"hello_world-task","command":"node bin/hello_world.js"}
Adding a little formatting adds the correct date but loses all the field indexing:
format /^(?<time>[^ ]* [^ ]) (?<message>.)$/
The output now has the correct date but the only field indexed is of course message:
2018-02-09 16:14:45 +0000 stuff: {"message":"::ffff:1.2.3.4 - \\\"GET /find HTTP/1.1\\\" 200 5567 \\\"-\\\" \\\"ELB-HealthChecker/2.0\\\"1234\",\"timestamp\":\"2018-02-09T16:14:45.990867153Z\",\"level\":3,\"image_id\":\"sha256:12345\",\"image_name\":\"hello-world\",\"container_id\":\"12345\",\"container_name\":\"hello_world-task\",\"command\":\"node bin/hello_world.js\"}"}
Is that JSON that is being ingested by fluentd (I guess it is a gelf dict)? How do I get the date corrected and all the fields indexed? I went with "GELF out > GELF in" assuming that I wouldn't have to spend days writing custom formats?
--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.