Nginx format for both access and error log

[enqush bataa]

Hi everyone. 

I'm using fluent-agent-hydra to send nginx access and error logs to EFK stack. It's successfully sending data and output goes to elasticsearch however I cannot parse the log properly. 

My nginx access log example: 

219.85.178.198 - - [05/Jan/2016:11:03:36 +0800] "GET /api/v1/blabla?page=1&mode=all&condition=latest HTTP/1.1" 200 800 "https://blabla.cc/wiki" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36"

error log:

2016/01/05 10:54:35 [error] 20863#0: *101644 open() "/home/blabla/blabla_statics/uploads/1102_1.jpg" failed (2: No such file or directory), client: 118.163.71.51, server: blabla.cc, request: "GET /uploads/1102_1.jpg HTTP/1.1", host: "miir.cc", referrer: "http://blabla.local/discovery/list/1102"

td-agent conf:

<source>

 @type forward

 port 24224

 format /^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] [^ ]* "(?<method>\S+)(?: +(?<path>[^\"]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$/

 time_format %d/%b/%Y:%H:%M:%S %z/

</source>

the problem is it cannot parse correctly. Also another question that I have is how can I also parse error logs? do I need to add another <source> field on my td-agent conf file?

Thanks.

[Mr. Fiber]

Because in_forward doesn't have format parameter.

See article: http://docs.fluentd.org/articles/in_forward

You can use fluent-plugin-parser's filter to parse it.

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[enod]

Thanks so much. I've used filter of this plugin and it works like charm :). 

The type of log messages are still remains with type of 'fluentd'. Is it possible to distinguish 2 log as access and error? 

[enod]

type_name access

works! thanks again.

[swapnil...@housing.com]

Hi enod,

How did you manage to parse the logs correctly ?

Do you parse it on client side with hydra or on fluentd server ?

I am stuck, please help.