ignoring unmatched log entries

[Markus Velten]

Hi,

I have a log format definition, where I explicitly want to ignore log entries with specific values.

This works fine, except that all UNMATCHED entries will land in the fluentd.log ... which is sub-optimal, as the log files are really big and thus the fluentd.log will explode.

Is there an easy way to omit the "[warn]: pattern not match:" messages in the log?

[Satoshi Tagomori]

You can use 'type null' to omit all messages which doesn't match any sections.

At the end of your configuration file:

<match **>

  type null

</match>

2013年10月24日木曜日 0時19分07秒 UTC+9 Markus Velten:

[Markus Velten]

Hi Satoshi,

thanks for your answer, but this didn't work for me. 

I found a workaround by giving "-qq" to the daemon startup parameters but that's also not really desired as the log is completely silent now.

[Masahiro Nakagawa]

> but this didn't work for me. 

What does this sentence mean?

Do you want to omit only 'pattern not match:' log?

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Markus Velten]

Hi,

yes - something like this. 

I have log lines which have different types in it (an integer). And I only want to import log lines which contain a specific value. 

I did it with the logformat regex. What I now encounter is, that all log lines which don't contain the specific value are logged as additional "[warn]" message to the fluentd.log. As the amount of log lines containing not that specific value are 70-80%, the fluentd log will simply explode with these warn messages.

        format /^(?<requestTime>[^\t]*)\t[0-9]+\t(?<appid>[^\t]*)\t(?<requestedHost>[^\t]*)\t(?<path>[^\t]*)\t([^\t]*)\t([^\t]*)\t(?<method>\S+)\t([^\t]*)\t([^\t]*)\t([^\t]*)\t([^\t]*)\t([^\t]*)\t([^\t]*)\t(?<uaid>(?!999-|502-).+[\t]*)\t(?<agent>[^\t]*)\t(?<jsessionid>[^\t]*)\t(?<remoteIP>[^\t]*)\t(?<referrer>[^\t]*)\t(?<carrierid>[^\t]*)\t(?<requestType>5|6|9[^\t]*)\t([^\t]*)\t(?<code>[^\t]*)\t(?<uuid>[^\t]*)\t(?<size>[^\t]*).*$/

As you can see, I'm just interested in log lines with requestType 5, 6 and 9 and uaid not starting with "999-" or "502-" ... basically this works, but ALL other lines produce a "warn" message in the fluentd log (> 70%). 

And that's what I want to prevent. 

I found the switch "-qq" in the startup to omit all logs but ERRORs but actually I'm not sure if this is what I want. At least it works.

Thanks in advance.

Markus

[Kiyoto Tamura]

Hi Markus,

I suggest an alternative approach (I will sketch the high-level process in bullet points followed by an example config).

1. Using in_tail, get all the messages, including the ones that you don't care to import, into fluentd.

2. Using the out_grep plugin (https://github.com/sonots/fluent-plugin-grep) to filter out the messages you don't need. The caveat here is the out_grep plugin can grep by only one field, so in your case, you would have to run it through twice.

3. Now you only have the data you want to import. So send them to the appropriate data destination.

Here is an example config. I put this together in haste without testing. So, try it at your own risk.

<source>

  type tail

  path <your path>

  format /^(?<requestTime>[^\t]*)\t[0-

9]+\t(?<appid>[^\t]*)\t(?<requestedHost>[^\t]*)\t(?<path>[^\t]*)\t([^\t]*)\t([^\t]*)\t(?<method>\S+)\t([^\t]*)\t([^\t]*)\t([^\t]*)\t([^\t]*)\t([^\t]*)\t([^\t]*)\t(?<uaid>.+[\t]*)\t(?<agent>[^\t]*)\t(?<jsessionid>[^\t]*)\t(?<remoteIP>[^\t]*)\t(?<referrer>[^\t]*)\t(?<carrierid>[^\t]*)\t(?<requestType>\d+[^\t]*)\t([^\t]*)\t(?<code>[^\t]*)\t(?<uuid>[^\t]*)\t(?<size>[^\t]*).*$/

  pos_file <your position file>

  tag mylog

</source>

<match mylog>

  type grep

  input_key requestType

  regexp [^569] # only keep 5, 6, or 9 request types

  add_tag_prefix filtered_request_type

</match>

<match filtered_request_type.mylog>

  type grep

  input_key uaid

  regexp ^(999-|502-) # filter out the ones starting with 999- or 502-

  add_tag-prefix filtered_uaid

</match>

<match filtered_uaid.filtered_requet_type.mylog>

  type <your output plugin>
  ...

</match>

Let us know if you have more questions.

kiyoto