Problem with plugin elasticsearch deployment

[Julien Syx]

Hello,

I'm trying to use the puppet module to deploy the td-agent.

Actually it's okay the agent is well deployed. Then I want to install elasticsearch.

When I try to install it with gem, it needs ruby packages/lib/headers to install it.

So I tought about deploying my plugin putting the elasticsearch rb file in /etc/td-agent/plugin/ :

====================

root@myserver-1:/etc/td-agent/plugin# ls

out_elasticsearch.rb

====================

root@myserver-1:/etc/td-agent/plugin# /etc/init.d/td-agent restart

/usr/lib/fluent/ruby/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require': cannot load such file -- patron (LoadError)

from /usr/lib/fluent/ruby/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'

from /etc/td-agent/plugin/out_elasticsearch.rb:3:in `<top (required)>'

from /usr/lib/fluent/ruby/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'

from /usr/lib/fluent/ruby/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'

from /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.45/lib/fluent/plugin.rb:60:in `block in load_plugin_dir'

from /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.45/lib/fluent/plugin.rb:58:in `each'

from /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.45/lib/fluent/plugin.rb:58:in `load_plugin_dir'

from /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.45/lib/fluent/engine.rb:127:in `load_plugin_dir'

from /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.45/lib/fluent/supervisor.rb:335:in `block in init_engine'

from /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.45/lib/fluent/supervisor.rb:332:in `each'

from /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.45/lib/fluent/supervisor.rb:332:in `init_engine'

from /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.45/lib/fluent/supervisor.rb:106:in `dry_run'

from /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.45/lib/fluent/supervisor.rb:83:in `start'

from /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.45/lib/fluent/command/fluentd.rb:168:in `<top (required)>'

from /usr/lib/fluent/ruby/lib/ruby/1.9.1/rubygems/custom_require.rb:55:in `require'

from /usr/lib/fluent/ruby/lib/ruby/1.9.1/rubygems/custom_require.rb:55:in `require'

from /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.45/bin/fluentd:6:in `<top (required)>'

from /usr/lib/fluent/ruby/bin/fluentd:23:in `load'

from /usr/lib/fluent/ruby/bin/fluentd:23:in `<top (required)>'

from /usr/sbin/td-agent:7:in `load'

from /usr/sbin/td-agent:7:in `<main>'

I don't understand : It seems td-agent come with the patron gem but seems to have problem to load.

I have nothing interesting in the log as stacktrace.

How can I deploy easily plugins without installing all ruby libraries etc ?

====================

root@myserver-1:/etc/td-agent/plugin# /usr/lib/fluent/ruby/bin/fluent-gem install fluent-plugin-elasticsearch

Building native extensions.  This could take a while...

ERROR:  Error installing fluent-plugin-elasticsearch:

ERROR: Failed to build gem native extension.

        /usr/lib/fluent/ruby/bin/ruby extconf.rb

checking for curl-config... yes

checking for rb_thread_blocking_region()... *** extconf.rb failed ***

Could not create Makefile due to some reason, probably lack of

necessary libraries and/or headers.  Check the mkmf.log file for more

details.  You may need configuration options.

Provided configuration options:

--with-opt-dir

--without-opt-dir

--with-opt-include

--without-opt-include=${opt-dir}/include

--with-opt-lib

--without-opt-lib=${opt-dir}/lib

--with-make-prog

--without-make-prog

--srcdir=.

--curdir

--ruby=/usr/lib/fluent/ruby/bin/ruby

--with-curl-dir

--without-curl-dir

--with-curl-include

--without-curl-include=${curl-dir}/include

--with-curl-lib

--without-curl-lib=${curl-dir}/lib

/usr/lib/fluent/ruby/lib/ruby/1.9.1/mkmf.rb:381:in `try_do': The compiler failed to generate an executable file. (RuntimeError)

You have to install development tools first.

from /usr/lib/fluent/ruby/lib/ruby/1.9.1/mkmf.rb:461:in `try_link0'

from /usr/lib/fluent/ruby/lib/ruby/1.9.1/mkmf.rb:476:in `try_link'

from /usr/lib/fluent/ruby/lib/ruby/1.9.1/mkmf.rb:619:in `try_func'

from /usr/lib/fluent/ruby/lib/ruby/1.9.1/mkmf.rb:894:in `block in have_func'

from /usr/lib/fluent/ruby/lib/ruby/1.9.1/mkmf.rb:790:in `block in checking_for'

from /usr/lib/fluent/ruby/lib/ruby/1.9.1/mkmf.rb:284:in `block (2 levels) in postpone'

from /usr/lib/fluent/ruby/lib/ruby/1.9.1/mkmf.rb:254:in `open'

from /usr/lib/fluent/ruby/lib/ruby/1.9.1/mkmf.rb:284:in `block in postpone'

from /usr/lib/fluent/ruby/lib/ruby/1.9.1/mkmf.rb:254:in `open'

from /usr/lib/fluent/ruby/lib/ruby/1.9.1/mkmf.rb:280:in `postpone'

from /usr/lib/fluent/ruby/lib/ruby/1.9.1/mkmf.rb:789:in `checking_for'

from /usr/lib/fluent/ruby/lib/ruby/1.9.1/mkmf.rb:893:in `have_func'

from extconf.rb:47:in `<main>'

Gem files will remain installed in /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/patron-0.4.18 for inspection.

Results logged to /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/patron-0.4.18/ext/patron/gem_make.out

====================

root@myserver-1:/etc/td-agent/plugin# cat /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/patron-0.4.18/ext/patron/mkmf.log

find_executable: checking for curl-config... -------------------- yes

--------------------

"gcc -o conftest -I/usr/lib/fluent/ruby/include/ruby-1.9.1/x86_64-linux -I/usr/lib/fluent/ruby/include/ruby-1.9.1/ruby/backward -I/usr/lib/fluent/ruby/include/ruby-1.9.1 -I.    -g -O2  -pedantic -Wall conftest.c  -L. -L/usr/lib/fluent/ruby/lib -Wl,-R/usr/lib/fluent/ruby/lib -L. -Wl,-Bsymbolic-functions -rdynamic -Wl,-export-dynamic     -Wl,-R -Wl,/usr/lib/fluent/ruby/lib -L/usr/lib/fluent/ruby/lib -lruby-static  -lpthread -lrt -ldl -lcrypt -lm   -lc -L/usr/lib/x86_64-linux-gnu -lcurl"

checked program was:

/* begin */

1: #include "ruby.h"

2: 

3: int main(int argc, char **argv)

4: {

5:   return 0;

6: }

/* end */

====================

root@myserver-1:/etc/td-agent/plugin# dpkg -l | grep libcurl

ii  libcurl3:amd64                     7.26.0-1+wheezy9           amd64        easy-to-use client-side URL transfer library (OpenSSL flavour)

ii  libcurl3-gnutls:amd64              7.26.0-1+wheezy8           amd64        easy-to-use client-side URL transfer library (GnuTLS flavour)

ii  libcurl4-openssl-dev               7.26.0-1+wheezy9           amd64        development files and documentation for libcurl (OpenSSL flavour)

ii  python-pycurl                      7.19.0-5                   amd64        Python bindings to libcurl

Don't hesite to ask me if I can give you more info.

The goal is to easily deploy the package everywhere with plugins without installing all ruby environment.

Thanks !

[Masahiro Nakagawa]

Hi Julien,

Hm, some users use fluent-plugin-elasticsearch with td-agent on Ubuntu/Debian.

So "/usr/lib/fluent/ruby/bin/fluent-gem install fluent-plugin-elasticsearch" should be succeeded.

Could you show me your environment?

- OS version

- puppet sample (Sorry, I'm not familiar with Puppet so I want sample files to reproduce)

If this is not known issue with puppet and td-agent,

I try to reproduce the problem.

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Julien Syx]

Hello,

Thanks for your answer.

I fixed the problem installing ruby-dev.

I was thinking that td-agent package came with his own environment and gem should be able to install without modifying the system.

Don't know if I ask in the right place.

Anyway, the problem is fixed, but if I can avoid it ...

[Masahiro Nakagawa]

Hm.. that's wired.

I tried "sudo /usr/lib/fluent/ruby/bin/fluent-gem install fluent-plugin-elasticsearch" without ruby-devel and

it worked.

td-agent includes Ruby headers and libs so building gem should work correctly.

I'm not sure why your environment doesn't refer td-agent's Ruby headers.

Need time to investigate it...

[LinuxMan]

Hi,

I am having the same problem. I ran the following:

/usr/lib/fluent/ruby/bin/fluent-gem install fluent-plugin-elasticsearch 

and got the following error:

ERROR:  Error installing fluent-plugin-elasticsearch:

        ERROR: Failed to build gem native extension.

I would appreciate any help.

Thanks 

[LinuxMan]

On Tuesday, May 13, 2014 8:56:31 AM UTC-6, LinuxMan wrote:

Hi,

I am having the same problem. I ran the following:

/usr/lib/fluent/ruby/bin/fluent-gem install fluent-plugin-elasticsearch 

and got the following error:

ERROR:  Error installing fluent-plugin-elasticsearch:

        ERROR: Failed to build gem native extension.

    Also, when I try to restart the td-agent I get the following error

    2014-05-13 08:57:15 -0600 [error]: Dry run failed: Unknown output plugin 'elasticsearch'. Run 'gem search -rd fluent-plugin' to find plugins

[LinuxMan]

I needed to install ruby-dev and apt-get install libcurl4-gnutls-dev and I was finally able to get the  gem install fluent-plugin-elasticsearch to work!!!

[gzc]

Clean install of Ubuntu14.04 LTS required both the following in order to install fluent elasticsearch plugin

sudo apt-get install ruby-dev libmruby-dev

and

sudo apt-get install libcurl4-gnutls-dev

Then this command worked  sudo /usr/lib/fluent/ruby/bin/fluent-gem  install fluent-plugin-elasticsearch

G

[Masahiro Nakagawa]

> sudo apt-get install ruby-dev libmruby-dev

libmruby-dev is for mruby. Is it needed?

--

[Appychip]

In my case `build-essential` was also needed to be installed as I was using ubuntu-16.04. I got it working by following setup instruction on Appychip

sudo apt-get install make libcurl4-gnutls-dev --yes
sudo apt-get install build-essential