trouble installing plugins in td-agent

[Brian Fallik]

Hi,

I'm having trouble installing the fluent-plugin-mail [1] plugin in td-agent [2]. I'm following the instructions and attempting to install the gem using either of the fluent-gem binaries included in td-agent:
  [root@node0 ~]# rpm -ql td-agent | grep fluent-gem
  /usr/lib64/fluent/ruby/bin/fluent-gem
  /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.39/bin/fluent-gem

Oddly (or maybe intentionally) these commands are not identical [3].

No matter which I use to import the plugin, when I try to start td-agent with a `type mail' output match, it fails to find it:
  2013-10-28 19:36:07 +0000 [info]: adding match pattern="email.test" type="mail"
  2013-10-28 19:36:07 +0000 [error]: config error file="/etc/td-agent/td-agent.conf" error="Unknown output plugin 'mail'. Run 'gem search -rd fluent-plugin' to find plugins"
  2013-10-28 19:36:07 +0000 [info]: process finished code=256
  2013-10-28 19:36:07 +0000 [warn]: process died within 1 second. exit.

This is on Centos 6.4.

Thanks,
brian


[1] - https://github.com/u-ichi/fluent-plugin-mail
[2] - td-agent-1.1.17-0.x86_64
[3] -
  [root@node0 ~]# /usr/lib64/fluent/ruby/bin/fluent-gem env
  RubyGems Environment:
    - RUBYGEMS VERSION: 1.8.23
    - RUBY VERSION: 1.9.3 (2012-04-20 patchlevel 194) [x86_64-linux]
    - INSTALLATION DIRECTORY: /usr/local/rvm/gems/ruby-1.9.3-p327
    - RUBY EXECUTABLE: /usr/lib64/fluent/ruby/bin/ruby
    - EXECUTABLE DIRECTORY: /usr/local/rvm/gems/ruby-1.9.3-p327/bin
    - RUBYGEMS PLATFORMS:
      - ruby
      - x86_64-linux
    - GEM PATHS:
       - /usr/local/rvm/gems/ruby-1.9.3-p327
       - /usr/local/rvm/gems/ruby-1.9.3-p327@global
    - GEM CONFIGURATION:
       - :update_sources => true
       - :verbose => true
       - :benchmark => false
       - :backtrace => false
       - :bulk_threshold => 1000
    - REMOTE SOURCES:
       - http://rubygems.org/
  [root@node0 ~]# /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.39/bin/fluent-gem env
  RubyGems Environment:
    - RUBYGEMS VERSION: 1.8.24
    - RUBY VERSION: 1.9.3 (2012-11-10 patchlevel 327) [x86_64-linux]
    - INSTALLATION DIRECTORY: /usr/local/rvm/gems/ruby-1.9.3-p327
    - RUBY EXECUTABLE: /usr/local/rvm/rubies/ruby-1.9.3-p327/bin/ruby
    - EXECUTABLE DIRECTORY: /usr/local/rvm/gems/ruby-1.9.3-p327/bin
    - RUBYGEMS PLATFORMS:
      - ruby
      - x86_64-linux
    - GEM PATHS:
       - /usr/local/rvm/gems/ruby-1.9.3-p327
       - /usr/local/rvm/gems/ruby-1.9.3-p327@global
    - GEM CONFIGURATION:
       - :update_sources => true
       - :verbose => true
       - :benchmark => false
       - :backtrace => false
       - :bulk_threshold => 1000
    - REMOTE SOURCES:
       - http://rubygems.org/

[Kiyoto Tamura]

Brian,

Thanks for reporting the problem. Can you see if fluent-plugin-mail exists in /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/? (ex:  ls /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/ | grep fluent-plugin-mail)

kiyoto

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Brian Fallik]

Kiyoto,

Thanks for the quick reply.

No, the plugin is not installed in that location:
  [root@node0 ~]# find  /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/ | grep fluent-plugin-mail
  [root@node0 ~]#

It is installed at /usr/local/rvm/gems/ruby-1.9.3-p327/gems/fluent-plugin-mail-0.0.4, but that seems system-wide and not fluentd specific. It seems like either fluent-gem isn't installing to the right place or td-agent isn't looking in the right place, or both.

Let me know if I can provide any more details.

brian

--
You received this message because you are subscribed to a topic in the Google Groups "Fluentd Google Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/fluentd/ekODAaJuS9M/unsubscribe.
To unsubscribe from this group and all its topics, send an email to fluentd+u...@googlegroups.com.

[Kiyoto Tamura]

Brian,

Just to humor me, can you try running the following and see if td-agent still fails to detect the mail plugin?

$ /usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-mail

kiiyoto

[Brian Fallik]

Sure:
  [root@node0 ~]#  /usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-mail
  Successfully installed fluent-plugin-mail-0.0.4
  1 gem installed
  Installing ri documentation for fluent-plugin-mail-0.0.4...
  Installing RDoc documentation for fluent-plugin-mail-0.0.4...
  [root@node0 ~]# td-agent
  2013-10-28 23:30:14 +0000 [info]: starting fluentd-0.10.39
  ...
  2013-10-28 23:30:15 +0000 [info]: adding match pattern="email.test" type="mail"
  2013-10-28 23:30:15 +0000 [error]: config error file="/etc/td-agent/td-agent.conf" error="Unknown output plugin 'mail'. Run 'gem search -rd fluent-plugin' to find plugins"
  2013-10-28 23:30:15 +0000 [info]: process finished code=256
  2013-10-28 23:30:15 +0000 [warn]: process died within 1 second. exit.

brian

[Kiyoto Tamura]

Brian,

I will look into this further. For now, you can also place the actually output plugin file (https://github.com/u-ichi/fluent-plugin-mail/blob/master/lib/fluent/plugin/out_mail.rb) in /etc/td-agent/plugin and td-agent should be able to pick it up.

Thanks,

kiyoto

[Masahiro Nakagawa]

Hi Brian,

Could you disable rvm temporally?

I didn't debug yet but rvm rewrites environemnt variable.

It sometimes causes similar problem on some Ruby middleware.

Masahiro

[Brian Fallik]

Masahiro,

I'm not sure how to disable rvm.

In any case, manually copying the plugin into /etc/td-agent/plugin is a temporary workaround while I try to build out the email integration. Once we need to deploy this for real it'd really be great if fluent-gem and td-agent worked properly.

brian

--
You received this message because you are subscribed to a topic in the Google Groups "Fluentd Google Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/fluentd/ekODAaJuS9M/unsubscribe.
To unsubscribe from this group and all its topics, send an email to fluentd+u...@googlegroups.com.

[Brian Fallik]

This doesn't really solve the problem but I did find another workaround:
  $ env GEM_HOME=/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/ GEM_PATH=/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/ fluent-gem install fluent-plugin-mail

installs the plugin into the right directory and td-agent starts up fine.

I'm not sure why fluent-gem isn't guessing the right values on its own. I'm also not sure if the approach I found is safe or if we can easily integrate this into our build outs.

brian

[Masahiro Nakagawa]

> $ env GEM_HOME=/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/ GEM_PATH=/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/ fluent-gem install fluent-plugin-mail

I didn't know this workaround.

> I'm not sure why fluent-gem isn't guessing the right values on its own. I'm also not sure if the approach I found is safe or if we can easily integrate this into our build outs.

RVM replaces some built-in commands with own script to change / setup environment variable.

RVM is the outside of fluent-gem, so fluent-gem can't fully avoid RVM erosion....

This is why many people switched from RVM to rbenv.