Ignoring specific Apache access log entries

[Eddie B]

How do I tell 

<source>

  type tail

  format apache2

  path /var/log/httpd/access_log

  pos_file /var/log/td-agent/apache2.access_log.pos

  tag s3.apache.access

</source>

to ignore lines that contain a certain pattern? For example, lines contaning this type of entry:

{"host":"localhost","user":null,"method":"OPTIONS","path":"*","code":200,"size":null,"referer":null,"agent":"Apache (internal dummy connection)"}

[Kiyoto Tamura]

Hi Eddie,

You might find the rewrite-tag-filter plugin useful. https://github.com/y-ken/fluent-plugin-rewrite-tag-filter

kiyoto

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Eddie B]

Thank you Kiyoto!

Does this look right to you?

<source>

  type tail

  format apache2

  path /var/log/httpd/access_log

  pos_file /var/log/td-agent/apache2.access_log.pos

  tag s3.apache.access

</source>

<match s3.apache.access>

  type rewrite_tag_filter

  capitalize_regex_backreference yes

  rewriterule1 agent ^Apache \(internal dummy connection\)$  clear

</match>

<match clear>

  type null

</match>

[Y.Kentaro (y-ken@github)]

Hi Eddie,

It is a sample config to ignore specific user-agent and rewrite tag for another records.

<source>

  type tail

  format apache2

  path /var/log/httpd/access_log

  pos_file /var/log/td-agent/apache2.access_log.pos

  tag s3.apache.access

</source>

<match s3.apache.access>

  type rewrite_tag_filter

  rewriterule1 agent ^Apache \(internal dummy connection\)$  clear

  rewriterule2 agent ^.+$  apache.access

</match>

<match apache.access>

  # changing this block what you want to do.

  type stdout

</match>

<match clear>

  type null

</match>

I would be happy if you solve this issue.

Thank you.

-- 

Kentaro Yoshida

twitter: @yoshi_ken

GitHub: https://github.com/y-ken

2013年10月17日木曜日 6時09分46秒 UTC+9 Eddie B:

[Naotoshi Seo]

Hi Eddie, 

You may use fluent-plugin-grep https://github.com/sonots/fluent-plugin-grep which is a sister of fluent-plugin-grepcounter which I introduced to you before.  

A sample configuration is as followings:

<source>

  type tail

  format apache2

  path /var/log/httpd/access_log

  pos_file /var/log/td-agent/apache2.access_log.pos

  tag s3.apache.access

</source>

<match s3.apache.access>

  type grep

  input_key agent

  exclude ^Apache \(internal dummy connection\)$

  tag apache.access

</match>

<match apache.access>

  # changing this block what you want to do.

  type stdout

</match>

Of course, fluent-plugin-rewrite-tag-filter also works fine. Use whichever you like. 

Regards,

Naotoshi a.k.a. sonots