fluent-plugin-Kafka dynamic topics

[Larry Chen]

I am using Fluentd to collect Json data and forward to Kafka. Is it possible for Fluentd to fetch certain field from the Json and use it as Kafka topics so that the data can be forward to different Kafka topics?

[Mr. Fiber]

Hi,

From the source code, fluent-plugin-kafka decides target topic by 3 ways.

https://github.com/htgc/fluent-plugin-kafka/blob/2435e59164e358e9dd760b5d8b07047f2b6d37b7/lib/fluent/plugin/out_kafka_buffered.rb#L162

1. If you record has 'topic' field, kafka plugin uses it.

2. default_topic parameter

3. tag of event

So in v0.12, we have several approaches.

- Set 'topic' field by using record_transformer or record_modifier filter

- Use rewrite_tag_filter to extract field value to tag

- Send a patch to fluent-plugin-kafka to make 'topic' field configurable

Masahiro

On Fri, May 27, 2016 at 3:19 AM, Larry Chen <yiwei...@pearson.com> wrote:

I am using Fluentd to collect Json data and forward to Kafka. Is it possible for Fluentd to fetch certain field from the Json and use it as Kafka topics so that the data can be forward to different Kafka topics?

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Larry Chen]

Thanks a lot! It is really helpful and it works. I appreciate it. I am using the record_transformer to set the topics.

Yiwei