Error starting td-agent

1,437 views
Skip to first unread message

Lucas Gazire

unread,
Aug 24, 2015, 1:14:28 PM8/24/15
to Fluentd Google Group
Hi all,

After installing some plugins through the td-agent-gems i can't start my agent anymore the following message appears:

Starting td-agent: /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:135:in `require': cannot load such file -- fluent/command/fluentd (LoadError)
        from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:135:in `rescue in require'
        from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:144:in `require'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.15/bin/fluentd:6:in `<top (required)>'
        from /opt/td-agent/embedded/bin/fluentd:23:in `load'
        from /opt/td-agent/embedded/bin/fluentd:23:in `<top (required)>'
        from /usr/sbin/td-agent:7:in `load'
        from /usr/sbin/td-agent:7:in `<main>'
                                                           [FAILED]


Any sugestions? I installed every dependecies to use Parser.

Thanks in advance, Lucas.

Mr. Fiber

unread,
Aug 24, 2015, 2:06:33 PM8/24/15
to Fluentd Google Group
That's weird.
Could you show me your command history and OS environment?

I first heard this case so I want to reproduce on my environment.


Masahiro


--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Lucas Gazire

unread,
Aug 24, 2015, 3:23:24 PM8/24/15
to Fluentd Google Group
Sure, sorry for the long command line, offline enviroment...

/usr/sbin/td-agent-gem install /cs/json-1.4.3.gem 
/usr/sbin/td-agent-gem install /cs/thread_safe-0.3.5.gem
/usr/sbin/td-agent-gem install /cs/string-scrub-0.0.5.gem       
/usr/sbin/td-agent-gem install /cs/sigdump-0.2.3.gem 
/usr/sbin/td-agent-gem install /cs/multi_json-1.11.2.gem 
/usr/sbin/td-agent-gem install /cs/multipart-post-2.0.0.gem 
/usr/sbin/td-agent-gem install /cs/http_parser.rb-0.6.0.gem 
/usr/sbin/td-agent-gem install /cs/faraday-0.9.1.gem 
/usr/sbin/td-agent-gem install /cs/excon-0.45.4.gem  
/usr/sbin/td-agent-gem install /cs/cool.io-1.3.1.gem 
/usr/sbin/td-agent-gem install /cs/yajl-ruby-1.2.1.gem 
/usr/sbin/td-agent-gem install /cs/elasticsearch-api-1.0.12.gem 
/usr/sbin/td-agent-gem install /cs/elasticsearch-1.0.12.gem 
/usr/sbin/td-agent-gem install /cs/elasticsearch-transport-1.0.12.gem 
/usr/sbin/td-agent-gem install /cs/fluent-plugin-parser-0.5.0.gem 
/usr/sbin/td-agent-gem install /cs/fluentd-0.12.15.gem 
/usr/sbin/td-agent-gem install /cs/fluent-plugin-elasticsearch-1.0.0.gem 

 /etc/init.d/td-agent start

RedHat Enterprise Linux, RHEL 5.8 (x86_64)

Best regards,
Lucas

Mr. Fiber

unread,
Aug 26, 2015, 9:42:29 AM8/26/15
to Fluentd Google Group
Hmm... it didn't reproduce on my CentOS 6 box with td-agent 2.2.1.
Could you check when this problem happens on your history?
From your log, fluent/command/fluentd is removed from td-agent's fluentd but it should exist.



Lucas Gazire

unread,
Aug 26, 2015, 10:00:25 AM8/26/15
to Fluentd Google Group
How should i check this in my history? All commands runned successfully and i couldn't find where the log file's located.

Thanks for your help.

Lucas Gazire

unread,
Aug 26, 2015, 10:04:39 AM8/26/15
to Fluentd Google Group
Well, i build another VM and when i tried to installed the json-1.4.3.gem i got the following error:

/usr/sbin/td-agent-gem install /cs/json-1.4.3.gem 
WARN: Unresolved specs during Gem::Specification.reset:
      json (>= 1.4.3)
WARN: Clearing out unresolved specs.
Please report a bug if this causes problems.
Building native extensions.  This could take a while...
ERROR:  Error installing /cs/json-1.4.3.gem:
        ERROR: Failed to build gem native extension.

    /opt/td-agent/embedded/bin/ruby extconf.rb
checking for ruby/re.h... yes
checking for ruby/encoding.h... yes
creating Makefile

make "DESTDIR=" clean

make "DESTDIR="
compiling generator.c
generator.c: In function 'fbuffer_inc_capa':
generator.c:342: error: invalid lvalue in assignment
make: *** [generator.o] Error 1

make failed, exit code 2

Gem files will remain installed in /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/json-1.4.3 for inspection.
Results logged to /opt/td-agent/embedded/lib/ruby/gems/2.1.0/extensions/x86_64-linux/2.1.0/json-1.4.3/gem_make.out



On Wednesday, August 26, 2015 at 10:42:29 AM UTC-3, repeatedly wrote:

Mr. Fiber

unread,
Aug 26, 2015, 10:16:36 AM8/26/15
to Fluentd Google Group
> Well, i build another VM and when i tried to installed the json-1.4.3.gem i got the following error:

Yes because json 1.4.3 is toooooo old version, 1.4.3 is released at 2010, so
I think it couldn't compile with newer gcc.

Mr. Fiber

unread,
Aug 26, 2015, 10:24:23 AM8/26/15
to Fluentd Google Group
JFYI.

td-agent includes json 1.8.x so if you install older version than 1.8.x,
fluentd doesn't use older versions.

Lucas Gazire

unread,
Aug 26, 2015, 10:26:00 AM8/26/15
to Fluentd Google Group
So i installed 1.8.3 and yajl and now i got the error:


Starting td-agent: /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:135:in `require': cannot load such file -- yajl (LoadError)
        from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:135:in `rescue in require'
        from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:144:in `require'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/load.rb:9:in `<top (required)>'
        from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:73:in `require'
        from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:73:in `require'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/supervisor.rb:17:in `<top (required)>'
        from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:73:in `require'
        from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:73:in `require'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/command/fluentd.rb:18:in `<top (required)>'
        from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:73:in `require'
        from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:73:in `require'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/bin/fluentd:6:in `<top (required)>'
        from /opt/td-agent/embedded/bin/fluentd:23:in `load'
        from /opt/td-agent/embedded/bin/fluentd:23:in `<top (required)>'
        from /usr/sbin/td-agent:7:in `load'
        from /usr/sbin/td-agent:7:in `<main>'


Mr. Fiber

unread,
Aug 26, 2015, 10:33:55 AM8/26/15
to Fluentd Google Group
What the permission of yajl and your manual installed libraries?
These libraries are accessed by td-agent user?

Lucas Gazire

unread,
Aug 26, 2015, 10:38:09 AM8/26/15
to Fluentd Google Group
He keep giving me this warning even with these dependencies installed and all the permissions in the directory setted.

WARN: Unresolved specs during Gem::Specification.reset:
      json (>= 1.4.3)
      cool.io (< 2.0.0, >= 1.2.2)

Lucas Gazire

unread,
Aug 26, 2015, 10:39:55 AM8/26/15
to Fluentd Google Group
And what is weird is that:

 /usr/sbin/td-agent-gem install /cs/json-1.8.3.gem 
WARN: Unresolved specs during Gem::Specification.reset:
      json (>= 1.4.3)
      cool.io (< 2.0.0, >= 1.2.2)
WARN: Clearing out unresolved specs.


/usr/sbin/td-agent-gem install /cs/cool.io-1.3.1.gem 
WARN: Unresolved specs during Gem::Specification.reset:
      json (>= 1.4.3)
      cool.io (< 2.0.0, >= 1.2.2)

Lucas Gazire

unread,
Aug 26, 2015, 11:18:15 AM8/26/15
to Fluentd Google Group
Is there a way to remove all this installed gems?

Mr. Fiber

unread,
Aug 26, 2015, 11:29:13 AM8/26/15
to Fluentd Google Group
This warning is known-issue and no problem.
Hmm... my environment still can't reproduce this problem.

Your command doesn't have 'sudo' so do you run command as a root user?

Mr. Fiber

unread,
Aug 26, 2015, 11:30:31 AM8/26/15
to Fluentd Google Group
Use "/usr/sbin/td-agent uninstall GEM-NAME".

Lucas Gazire

unread,
Aug 26, 2015, 11:34:59 AM8/26/15
to Fluentd Google Group
Yes, root user, it's my test enviroment so i logged as root.

Lucas Gazire

unread,
Aug 26, 2015, 11:35:59 AM8/26/15
to Fluentd Google Group
I will uninstall, my main idea is to integrate with log4net i will try another approach.

Lucas Gazire

unread,
Aug 26, 2015, 12:24:12 PM8/26/15
to Fluentd Google Group
Removed, reinstalled and now i got a new error (yay!)

 /etc/init.d/td-agent start
Starting td-agent: /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require': cannot load such file -- /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-parser-0.5.0/lib/fluent/plugin/out_parser (LoadError)
        from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/plugin.rb:148:in `block in try_load_plugin'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/plugin.rb:146:in `each'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/plugin.rb:146:in `try_load_plugin'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/plugin.rb:106:in `new_impl'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/plugin.rb:57:in `new_output'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/agent.rb:125:in `add_match'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/agent.rb:60:in `block in configure'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/agent.rb:54:in `each'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/agent.rb:54:in `configure'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/root_agent.rb:82:in `configure'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/engine.rb:93:in `configure'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/engine.rb:77:in `run_configure'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/supervisor.rb:457:in `run_configure'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/supervisor.rb:141:in `block in start'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/supervisor.rb:306:in `call'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/supervisor.rb:306:in `main_process'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/supervisor.rb:281:in `block in supervise'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/supervisor.rb:280:in `fork'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/supervisor.rb:280:in `supervise'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/supervisor.rb:137:in `start'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/command/fluentd.rb:167:in `<top (required)>'
        from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:73:in `require'
        from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:73:in `require'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/bin/fluentd:6:in `<top (required)>'
        from /opt/td-agent/embedded/bin/fluentd:23:in `load'
        from /opt/td-agent/embedded/bin/fluentd:23:in `<top (required)>'
        from /usr/sbin/td-agent:7:in `load'
        from /usr/sbin/td-agent:7:in `<main>'

Mr. Fiber

unread,
Aug 27, 2015, 5:10:14 AM8/27/15
to Fluentd Google Group
Hmm... I can't reproduce this problem with root user.

LOCALHOST /home/repeatedly% sudo su
# ...
[root@localhost repeatedly]# history
curl -L https://td-toolbelt.herokuapp.com/sh/install-redhat-td-agent2.sh | sh
ls
ls /opt/td-agent/
ls -al /opt/td-agent/
/etc/init.d/td-agent restart
cat /var/log/td-agent/td-agent.log
/usr/sbin/td-agent-gem install yajl-ruby --no-document
/etc/init.d/td-agent restart
cat /var/log/td-agent/td-agent.log
history
[root@localhost repeatedly]#  uname -a
Linux localhost.localdomain 2.6.32-431.11.2.el6.x86_64 #1 SMP Tue Mar 25 19:59:55 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
[root@localhost repeatedly]#

Does your environment have special setting?
Other users also install 3rd party plugin but they don't hit this problem.


Mr. Fiber

unread,
Aug 27, 2015, 5:10:52 AM8/27/15
to Fluentd Google Group
Could you show me result of "ls -al /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-parser-0.5.0/lib/fluent/plugin/" ?

Lucas Gazire

unread,
Aug 27, 2015, 7:45:00 AM8/27/15
to Fluentd Google Group
Here is the result:

[root@brlu2500653 cs]# ls -al /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-parser-0.5.0/lib/fluent/plugin/
total 10
drwxrwxrwx 2 root root 1024 Aug 24 11:25 .
drwxrwxrwx 3 root root 1024 Aug 24 11:25 ..
-rw-r----- 1 root root 2815 Aug 24 11:25 out_deparser.rb
-rw-r----- 1 root root 4336 Aug 24 11:25 out_parser.rb

YAMANO Yuji

unread,
Aug 27, 2015, 8:12:19 AM8/27/15
to flu...@googlegroups.com, lucas...@gmail.com
I guess the problem is missing read permission.

On Thu, 27 Aug 2015 04:45:00 -0700 (PDT), Lucas Gazire <lucas...@gmail.com> wrote:

> [root@brlu2500653 cs]# ls -al
> /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-parser-0.5.0/lib/fluent/plugin/
> total 10
> drwxrwxrwx 2 root root 1024 Aug 24 11:25 .
> drwxrwxrwx 3 root root 1024 Aug 24 11:25 ..
> -rw-r----- 1 root root 2815 Aug 24 11:25 out_deparser.rb
> -rw-r----- 1 root root 4336 Aug 24 11:25 out_parser.rb

>>> /etc/init.d/td-agent start
>>> Starting td-agent:
>>> /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in
>>> `require': cannot load such file --
>>> /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-parser-0.5.0/lib/fluent/plugin/out_parser

--
YAMANO Yuji
OGIS-RI

Lucas Gazire

unread,
Aug 27, 2015, 8:47:41 AM8/27/15
to Fluentd Google Group, lucas...@gmail.com
I created another enviroment, and this is working fine.

Changed the permissions in the one that is not working and now i got the error of a plugin that is installed: 
 
Starting td-agent: /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require': cannot load such file -- /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.0.0/lib/fluent/plugin/out_elasticsearch (LoadError)
Reply all
Reply to author
Forward
0 new messages