nginx logs not parsing

[Viq-Md]

Hi,

We use Elastic Search, Logstash and Kibana (ELK) to use for logging from Kubernetes

Currently fluentd is not parsing the logs based on the nginx or php-fpm format. This results in a giant log message with no real searchable value. If properly parsed, we should be able to search on HTTP status, user agent, referring IP, etc.

WHEN I search Kibana for `status:400`
I SHOULD see a list of logs from nginx that have a 400 HTTP response status
SO that I can more easily create Kibana dashboards and debug my applications

Also, we have 6 namespaces and out of them i was told to create a filter for a namespace "kube-system" not to send logs to ElasticSearch.

i have attached the td-agent.conf, Dockerfile and gitlab-ci.yml

This is what i had updated in the td-agent.conf

```

<source>

    @type tail

    format nginx

    tag nginx.access

    path /var/log/nginx/access.log

</source>

<source>

    @type tail

    tag nginx.error

    path /var/log/nginx/error.log

    format multiline

    format_firstline /^\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2} \[\w+\] (?<pid>\d+).(?<tid>\d+): /

    format1 /^(?<time>\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2}) \[(?<log_level>\w+)\] (?<pid>\d+).(?<tid>\d+): (?<message>.*)/

    multiline_flush_interval 3s

</source>

```

Thanks,

Viq

[Mr. Fiber]

> Currently fluentd is not parsing the logs based on the nginx or php-fpm format. This results in a giant log message with no real searchable value. If properly parsed, we should be able to search on HTTP status, user agent, referring IP, etc.

You can check your regexp with your logs in several ways: https://docs.fluentd.org/v0.12/articles/faq#howtos

Does `format nginx` have a problem with /var/log/nginx/access.log?

Do you change log format of nginx?

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Viq-Md]

No there is no problem with /var/log/nginx/access.log

No i didn't change log format of nginx

Can you give me an example of nginx log parsing please

Thanks

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.

[Mr. Fiber]

So the problem is /var/log/nginx/error.log and your multiline regexp, right?

Did you check your regexp is correct via tools mentioned in above link?

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.

[Viq-Md]

Hey,

You said : So the problem is /var/log/nginx/error.log and your multiline regexp, right?

I check the below details where "path /var/log/nginx/access.log" doesn't exist. Where can i find that path.

I had logged into each pods of namespaces and checked that path but i don't see nginx/access.log in /var/log

And in regards to your second comment: Did you check your regexp is correct via tools mentioned in above link?

I checked the link but couldn't find the particular regexp created by me. Can you help me create that.

<source>

    @type tail

    format nginx

    tag nginx.access

    path /var/log/nginx/access.log

</source>

<source>

    @type tail

    tag nginx.error

    path /var/log/nginx/error.log

    format multiline

    format_firstline /^\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2} \[\w+\] (?<pid>\d+).(?<tid>\d+): /

    format1 /^(?<time>\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2}) \[(?<log_level>\w+)\] (?<pid>\d+).(?<tid>\d+): (?<message>.*)/

    multiline_flush_interval 3s

</source>

[Mr. Fiber]

> Can you help me create that.

You need to fill forms of fluentd-ui or fluentular with your regexp and your actual log.

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.