Fluentd not parsing Nginx Error Logs

[prateek gera]

Hi All,

  I have configured format of regex for nginx error logs as:

format /^(?<time>\S+ \S+) \[(?<log_level>[^\]]+)\](?<pid>\d+).(?<tid>[^:]+): (?<message>[^.]+)(?:, client: (?<client>[^,]+))(?:, server:(?<server>[^,]+))(?:, request: (?<request>[^,]+))(?:, upstream: (?<upstream>[^,]+))(?:, host: (?<host>[^,]+))/

   

  and my error logs are as below:

2016/04/27 12:57:02 [error] 9176#0: *1 FastCGI sent in stderr: "PHP message: array (

)" while reading response header from upstream, client: X.X.X.X, server: , request: "POST /wp-json/example-api/v1/service_viewPayment?UId=432 HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.example.com"

but it always showing pattern not matched in td-agent log.Any help really appreciated.

Regards

Prateek Gera

[Mr. Fiber]

> format /^(?<time>\S+ \S+) \[(?<log_level>[^\]]+)\](?<pid>\d+).(?<tid>[^:]+): (?<message>[^.]+)(?:, client: (?<client>[^,]+))(?:, server:(?<server>[^,]+))(?:, request: (?<request>[^,]+))(?:, upstream: (?<upstream>[^,]+))(?:, host: (?<host>[^,]+))/

This pattern seems to be wrriten in your hand, right?

If so, you can use fluentd-ui's interactive editor to debug it.

Please try: http://docs.fluentd.org/articles/fluentd-ui#intail-setting

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[prateek gera]

Hi Masahiro,

   I tried with fluendt-ui editor but still same error:

2016-04-28 10:41:53 +0530 [warn]: pattern not match: "2016/04/28 10:41:53

Prateek Gera

[prateek gera]

Hi Masahiro,

   I tried with fluendt-ui editor but still same error:

2016-04-28 10:41:53 +0530 [warn]: pattern not match: "2016/04/28 10:41:53

Prateek Gera

[Mr. Fiber]

>  I tried with fluendt-ui editor but still same error:

Could you paste the screenshot of regexp testing page?

[prateek gera]

This  is my configuration in td-agent.conf file:

<source>
  type tail
  path /usr/local/nginx/logs/error.log
  tag nginx.error
  format /^(?<time>\S+ \S+) \[(?<log_level>[^\]]+)\](?<pid>\d+).(?<tid>[^:]+): (?<message>[^.]+)(?:, client: (?<client>[^,]+))(?:, server:(?<server>[^,]+))(?:, request: (?<request>[^,]+))(?:, upstream: (?<upstream>[^,]+))(?:, host: (?<host>[^,]+))/
  pos_file /tmp/fluentd--1461820242.pos
</source>

Also find attached screenshots.

[prateek gera]

Its matching till message field in fluentd-ui editor.

[prateek gera]

Please find attached screenshot.

[Mr. Fiber]

Is your log multiline?

[prateek gera]

yes logs are in multiline because message containing more than one line.

[prateek gera]

There are some logs which contains more than one line in message field and some are in single line.

[Mr. Fiber]

See Nginx Error Log section: http://docs.fluentd.org/articles/common-log-formats

Also: http://docs.fluentd.org/articles/parser-plugin-overview

[prateek gera]

Its great logs are being parsed with this configuration but I want to split client,request,host and upstream fields from message field.

[Mr. Fiber]

There are two approaches:

- Parse message field using parsr filter after in_tail

- Modify multiline format for your case

[prateek gera]

Parse message like for access logs of nginx :

format /^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$/

     We can use similar format for error logs correct me if I am wrong.