Can the parser plugin emit multiple events?

[Raghu Udiyar]

Hello

Each input event has events embedded in the json structure. So this needs to be parsed and emitted as two or more events. I'm exploring the possibility of using a custom Parser plugin for this, but it only allows to return the time and record. 

Will returning a list of time, records work? As per the parser call, it doesn't look like so : parser.parse(text) { |time, record| ... }

What would be the right approach here, other than writing a custom input plugin?

Thanks,

Raghu

[Mr. Fiber]

>  Will returning a list of time, records work? As per the parser call, it doesn't look like so : parser.parse(text) { |time, record| ... }

Don't return array. Call yield more than once.

This is why we changed parser API before:

http://docs.fluentd.org/v0.12/articles/plugin-development#parser-api

https://github.com/fluent/fluentd/pull/352

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Raghu Udiyar]

On Friday, 26 May 2017 20:20:05 UTC+5:30, repeatedly wrote:

>  Will returning a list of time, records work? As per the parser call, it doesn't look like so : parser.parse(text) { |time, record| ... }

Don't return array. Call yield more than once.

This is why we changed parser API before:

http://docs.fluentd.org/v0.12/articles/plugin-development#parser-api

https://github.com/fluent/fluentd/pull/352

Thanks Masahiro, but we just realised the parser plugin will not work for the input plugin we are using (in_kafka). I'll have to write a filter plugin. As per the docs for this I'll have to define "filter" method. But since I need to emit two or more events, I'm guessing I'll have to override "filter_stream" method. Is my understanding correct here?

Thanks,

Raghu

 

[Mr. Fiber]

>  we just realised the parser plugin will not work for the input plugin we are using (in_kafka).

Yes because in_kafka doesn't use parser plugin for now.

https://github.com/fluent/fluent-plugin-kafka/blob/c2b30e5dd441e81b934775fd2174d7f4bc5d86a7/lib/fluent/plugin/in_kafka.rb#L112

> I'm guessing I'll have to override "filter_stream" method. Is my understanding correct here?

Correct.

--