grep filter question: add_tag_prefix is not used?

[Alicia Smith]

Hello,

I'm new to this community, and am beginning to use fluentd more and more.

I'm currently trying to filter some logs using the grep plugin based on information here:

http://docs.fluentd.org/v0.12/articles/filter_grep

However, when I start fluentd the output in the td-agent log is:

2017-03-20 18:21:27 +0000 [warn]: parameter 'add_tag_prefix' in
<filter network>
  @type grep
  exclude1 "message (^/|INFO)"
  add_tag_prefix filtered
</filter> is not used.

and I get no logs.

The source block is as follows:
<source>
  type tail
  format syslog
  path /var/log/syslog/network/*.log
  pos_file /var/log/td-agent/fluent-network.pos
  tag network
 #keep_time_key true
  refresh_interval 5
</source>

I was hoping someone could help me understand why this is not used?

Thank you,

Alicia

[Alicia Smith]

Maybe this option is for 0.12 only?

[Eduardo Silva]

Hi Alicia, 

I took your configuration example and I did pretty much the same test case but using a JSON message:

1. Configuration file for td-agent/Fluentd:

<filter network>

  @type grep

  exclude1 "message ^/|INFO"

</filter>

<source>

  @type tail

  format json

  path /home/edsiper/coding/fluentd/test.log

  tag network

  keep_time_key true

  refresh_interval 2

  read_from_head true

</source>

<match **>

  @type stdout

</match>

2. test.log content

{"message": "|INFO must be excluded"}

{"message": "keep this message"}

3. Running the service:

2017-03-20 13:04:46 -0600 [info]: plugin/in_tail.rb:575:initialize: following tail of /home/edsiper/coding/fluentd/test.log

2017-03-20 13:04:46 -0600 network: {"message":"keep this message"}

the add_tag_prefix is not a valid configuration key for the grep filter, instead what you can use in addition to your previous 'grep filter' is the rewrite_tag_filter plugin:

  http://docs.fluentd.org/v0.12/articles/out_rewrite_tag_filter

As an example, you can append the following rule before the final match:

<match network>

  @type rewrite_tag_filter

  rewriterule1 message \.* test.${tag}

</match>

so your output will have a different tag:

2017-03-20 13:13:29 -0600 test.network: {"message":"keep this message"}

note: pay attention to the read_from_head option for testing purposes, otherwise tail plugin will continue reading after it last position.

Please let me know if that works :)

best, 

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Eduardo Silva

Open Source, Treasure Data

http://www.treasuredata.com/opensource

 

[Alicia Smith]

Hello Eduardo,

Thank you much for the example and about read_from_head, this will help my testing!
I'll run with this and see how it goes!

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Mr. Fiber]

Filter doesn't have tag rewrite feature unlike Output.

http://docs.fluentd.org/v0.12/articles/filter-plugin-overview#overview

filter_grep article seems to not mention add_tag_prefix.

I'm not sure why you use add_tag_prefix in filter_grep configuration...

Masahiro

--

You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.

[Mr. Fiber]

An, did you see old article which uses out_grep in the web?