New to fluentd and unable to parse the custom plugin.

[Hariprasad S]

Hi All,

Im very much new to the fluentd here. I was asked to do a POC by taking the log file from the system via fluentd and parse the logvalues using custom parser and pass those data to Kafka.

Other team  in US have written the ruby file  parser_abc.rb  which does the parsing. 

So, when i tried to run the fluentd command as below:

/usr/local/rvm/gems/ruby-2.3.1/bin/fluentd  -c /etc/td-agent/td-agent.conf -vv

Below is the error thrown as such.

2016-08-19 17:13:00 +0530 [error]: fluent/supervisor.rb:529:rescue in main_process: config error file="/etc/td-agent/td-agent.conf" error="Unknown parser plugin 'abc'. Run 'gem search -rd fluent-plugin' to find plugins"

  2016-08-19 17:13:00 +0530 [debug]: command/fluentd.rb:271:<top (required)>: /usr/local/rvm/gems/ruby-2.3.1/gems/fluentd-0.14.1/lib/fluent/registry.rb:46:in `lookup'

  2016-08-19 17:13:00 +0530 [debug]: command/fluentd.rb:271:<top (required)>: /usr/local/rvm/gems/ruby-2.3.1/gems/fluentd-0.14.1/lib/fluent/plugin.rb:146:in `new_impl'

  2016-08-19 17:13:00 +0530 [debug]: command/fluentd.rb:271:<top (required)>: /usr/local/rvm/gems/ruby-2.3.1/gems/fluentd-0.14.1/lib/fluent/plugin.rb:123:in `new_parser'

  2016-08-19 17:13:00 +0530 [debug]: command/fluentd.rb:271:<top (required)>: /usr/local/rvm/gems/ruby-2.3.1/gems/fluentd-0.14.1/lib/fluent/plugin/in_tail.rb:106:in `configure_parser'

  2016-08-19 17:13:00 +0530 [debug]: command/fluentd.rb:271:<top (required)>: /usr/local/rvm/gems/ruby-2.3.1/gems/fluentd-0.14.1/lib/fluent/plugin/in_tail.rb:93:in `configure'

  2016-08-19 17:13:00 +0530 [debug]: command/fluentd.rb:271:<top (required)>: /usr/local/rvm/gems/ruby-2.3.1/gems/fluentd-0.14.1/lib/fluent/root_agent.rb:234:in `add_source'

td-agent.conf contains the below details:

<source>

 @type  tail

  path /var/log/ppslog.1

  tag abc

  format abc

</source>

<match *.**>

 @type stdout

</match>

kindly tell me how to load the parser_abc.rb file in the td-agent.conf files and execute it .

and what will the output which will comes in stdout.

Thanks

Hari

[Mr. Fiber]

Hi Hari,

http://docs.fluentd.org/articles/plugin-management

Check above official article.

This article explains how to tell your plugin to fluentd.

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Hariprasad S]

Thanks for the reply. But does the Ruby files should be treated as Plugins in this case for custom parsers?

Kindly clarify.

You received this message because you are subscribed to a topic in the Google Groups "Fluentd Google Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/fluentd/NwHfog3ilBI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to fluentd+unsubscribe@googlegroups.com.

[Mr. Fiber]

What does 'treated as Plugins' mean?

If it means file extension, file name should be parser_abc.rb.