Parsing nginx with the docker fluentd log driver

[zero bytes]

- fluentd or td-agent version: fluentd-0.12.31

- Environment information, e.g. OS.: efk stack in docker containers (https://github.com/kazu69/docker-efk)

- Your configuration:
<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>
<filter nginx.docker.**>
@type parser
format /^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)
(?: +(?<path>[^\"]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agen
t>[^\"]*)")?$/
time_format %Y-%m-%dT%H:%M:%S.%L%Z
key_name log
reserve_data true
</filter>
<match nginx.docker.*>
 @type elasticsearch
  host 172.17.0.1
  port 9200
  logstash_format true
   flush_interval 5s
</match>

- Your problem explanation. If you have an error logs, write it together.

I'm unable to extract or parse data from the log row in my table in Kibana 5.

The official fluentd nginx regex:
format /^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$/
time_format %d/%b/%Y:%H:%M:%S %z

My error log:
2017/01/07 20:19:00 [error] 7#7: *1 open() "/usr/share/nginx/html/88888" failed (2: No such file or directory), client: 192.168.28.1, server: localhost, request: "GET /88888 HTTP/1.1", host: "192.168.28.133:8080"

When I generate an error with nginx, fluentd reports:
2017-01-07 20:48:45 +0000 [warn]: pattern not match with data '192.168.2
8.1 - - [07/Jan/2017:20:48:45 +0000] "GET /88888 HTTP/1.1" 404 169 "-" "Mozilla/5.0 (Windows N
T 10.0; Win64; x64; rv:50.0) Gecko/20100101 Firefox/50.0" "-"'

[Bobby M.]

your error looks like an access log.  Error & Access logs are two different files in nginx.  Are you forwarding your error logs to fluentd as well?

[zero bytes]

Thank you for you reply!

Yes I'm trying to parse both error and access logs and my config has changed quite a bit now:

<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>

<match nginx.docker.*>
@type rewrite_tag_filter
rewriterule1 source stdout stdout.${tag}
rewriterule2 source stderr stderr.${tag}
</match>

<filter stderr.nginx.docker.*>
  @type parser
  format /^(?<time>[^ ]+ [^ ]+) \[(?<log_level>.*)\] (?<pid>\d*).(?<tid>[^:]*): [^ ](?<categorie>)[^ ](?<path>)[^ ](?<message>.*)$/
  key_name log
</filter>

<filter stdout.nginx.docker.*>
  @type parser
  format nging
  key_name log

</filter>

<match *.nginx.docker.*>

 @type elasticsearch
  host 172.17.0.1
  port 9200
  logstash_format true
   flush_interval 5s
</match>

I've discovered this error happens when the regex isn't valid. Thus I concluded that docker changes the logs to the point that the default format regex for nginx doesn't work.

Now I'm trying to write my own regex for the access log (stdout) and errors (stderr) which is working for stderr.
I'm really not good at regex at all so if you maybe could help me that would be great? :)



Op maandag 9 januari 2017 19:40:44 UTC+1 schreef Bobby M.:

[zero bytes]

Image



Op maandag 9 januari 2017 22:13:37 UTC+1 schreef zero bytes:

<w:LsdException Locked="false" Priority="9" SemiHidden="true" UnhideWhenUsed

[Mr. Fiber]

You can debug your regexp by using fluentd-ui:

http://docs.fluentd.org/articles/fluentd-ui#intail-setting

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.