Elasticsearch plugin error

[Jay Wilson]

Thoughts on why I am getting the following? According to ruby, the plugins are installed.

from the td-agent.conf file

<match *.**>
  type elasticsearch
  include_tag_key true
  tag_key _jnpr
  flush_interval 1s
</match>
"/etc/td-agent/td-agent.conf" 114 lines, 2221 characters written

root@Ubuntu-1364-001:/var/log/fluentd# /etc/init.d/td-agent configtest
2013-12-11 15:28:16 -0800 [error]: Dry run failed: Unknown output plugin 'elasticsearch'. Run 'gem search -rd fluent-plugin' to find plugins

root@Ubuntu-1364-001:/var/log/fluentd# gem install fluent-plugin-elasticsearch
Successfully installed fluent-plugin-elasticsearch-0.2.0
1 gem installed

root@Ubuntu-1364-001:/var/log/fluentd# /etc/init.d/td-agent configtest
2013-12-11 15:28:55 -0800 [error]: Dry run failed: Unknown output plugin 'elasticsearch'. Run 'gem search -rd fluent-plugin' to find plugins

root@Ubuntu-1364-001:/var/log/fluentd# gem install fluent-plugin-elasticsearch-ruby
Fetching: fluent-plugin-elasticsearch-ruby-0.0.3.gem (100%)
Successfully installed fluent-plugin-elasticsearch-ruby-0.0.3
Installing ri documentation for fluent-plugin-elasticsearch-ruby-0.0.3
1 gem installed

root@Ubuntu-1364-001:/var/log/fluentd# /etc/init.d/td-agent configtest
2013-12-11 15:29:28 -0800 [error]: Dry run failed: Unknown output plugin 'elasticsearch'. Run 'gem search -rd fluent-plugin' to find plugins
root@Ubuntu-1364-001:/var/log/fluentd#

[Masahiro Nakagawa]

See this FAQ: http://docs.fluentd.org/articles/faq#i-installed-td-agent-and-want-to-add-custom-plugins-how-do-i-do-it

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Fang Lee]

I got the problem two days ago, finally I got the solution also.

It's all about ruby and gem env path, try the following command to install the elasticsearch plugin:

GEM_HOME="/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/"

GEM_PATH="/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/"

/usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-elasticsearch

You probably want to change the path to fit your linux system.

Here is the original page, if you can read Japanese:

http://qiita.com/y_matsuwitter/items/901065962edb7ea8c6ec

I hope fluentd team can resovle this troublesome path issue.

在 2013年12月12日星期四UTC+8上午7时47分41秒，Jay Wilson写道：

[Masahiro Nakagawa]

It seems different problem.

Jay uses gem command in system ruby, but td-agent doesn't depend on system ruby.

So he installed a plugin in different environment.

> I hope fluentd team can resovle this troublesome path issue.

We don't have a solution. Please see this comment.

https://groups.google.com/d/msg/fluentd/ekODAaJuS9M/GiHscqhC3Q8J

If RVM expert can resolve this issue, patches are welcome :)

Masahiro

--

[Jay Wilson]

Thank you.  The missing paths GEM_HOME and GEM_PATH were the issue.

[Fang Lee]

It's weird, It still doesn't work even if I use the fluentd build-in fluent-gem.

But it does work after I export the GEM_PATH and GEM_HOME.

I mean the GEM_PATH and GEM_HOME is required and necessary.

在 2013年12月12日星期四UTC+8上午10时35分33秒，repeatedly写道：

[Masahiro Nakagawa]

> It's weird, It still doesn't work even if I use the fluentd build-in fluent-gem.

Without RVM?

[kelly...@greenwavereality.com]

How did you locate the correct path?

I'm having the same problem and it absolutely cripple my project! I used rvm to install latest Ruby since Ubuntu is default Ruby is broken.

Any additional pointers?

[Kiyoto Tamura]

Hi Kelly

I assume you installed Fluentd via "gem install fluentd". Then, "gem install <plugin name>" should work.

If your machine is missing GEM_HOME/GEM_PATH env variables, chances are good that this is not a Fluentd-specific issue (although Fluentd would be impacted) but rvm issue. You might want to let them know.

- If you installed td-agent, then please see this FAQ item: http://docs.fluentd.org/articles/faq#i-installed-td-agent-and-want-to-add-custom-plugins-how-do-i-do-it

Let us know if you have more questions.

--

[kelly...@greenwavereality.com]

That's funny. Because everything works except td-agent.

[Kiyoto Tamura]

>That's funny. Because everything works except td-agent.

td-agent or fluentd? Can you run the following commands and share the outputs?

- which td-agent

- which fluentd

- which ruby

- which gem

Also, if you can run the exact plugin installation command you've been trying to run alongside its error message, that would be hugely helpful.

Thanks!

Kiyoto

[Kelly Black]

Hi Kiyoto,

Sorry I've had so much trouble I've set up a completely fresh instance, and I'm having the same problem.

I've narrowed it down to 'gem install fluent-gem' not finishing correctly, I think it might have to do with a newer version of Ruby I'm not certain.

Please let me try a complete fresh install and take some notes, then I'll post back to you what I find.

Fluentd seems to install correctly, it really is about the gems at this point, and the plugins.

More concise info to come, just wanted to update you, thanks for being patient with me.

Kelly

--
You received this message because you are subscribed to a topic in the Google Groups "Fluentd Google Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/fluentd/CgB75oPotTw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to fluentd+u...@googlegroups.com.

[Kiyoto Tamura]

Hi Kelly,

Thank you for the information. I think our docs may have misled you somewhat.

- There is no gem called "fluent-gem". If you want to install fluentd plugins (e.g., fluent-plugin-mongo, which is a MongoDB plugin), you would run

gem install fluent-plugin-mongo

- That begs the question, "what the hell is fluent-gem then?" fluent-gem is like usual ruby gem, but specifically packaged for Fluentd. This is a non-issue if you install fluentd via "gem install fluentd" because that means you already have "gem" installed on your machine. However, we have users who don't have ruby/gem installed natively and would rather use td-agent, and for those users, fluent-gem comes in handy (This is what http://docs.fluentd.org/articles/faq#i-installed-td-agent-and-want-to-add-custom-plugins-how-do-i-do-it refers to).

So, it looks like you should be able to download whatever plugin you need via "gem install <plugin name>", e.g. "gem install fluent-plugin-elasticsearch".

Let me know if you continue to run into issues.

Thanks!

Kiyoto

[tarun kumar]

- which td-agent

/usr/sbin/td-agent

-which fluentd

/usr/local/bin/fluentd

-which ruby

/usr/bin/ruby

-which gem

/usr/bin/gem

export GEM_HOME=/usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/

export GEM_PATH=/usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/

 2. placing the actually output plugin file (https://github.com/u-ichi/fluent-plugin-mail/blob/master/lib/fluent/plugin/out_mail.rb) in /etc/td-agent/plugin

has not worked too.

3. Also the workAround 

~#  env GEM_HOME=/usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/ GEM_PATH=/usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/fluent-gem install fluent-plugin-elasticsearch   hasnt worked for me.

4. it says 1 gem installed on 

 /usr/lib/fluentd/ruby/bin/gem install fluent-plugin-elasticsearch

Note : for me it is lib and not lib64 on my machine. i donot have lib64.

but on restarting /etc/init.d/td-agent start

 [error]: fluent/supervisor.rb:205:rescue in main_process: config error file="/etc/td-agent/td-agent.conf" error="Unknown output plugin 'elasticsearch'. Run 'gem search -rd fluent-plugin' to find plugins"