Error Installing Kafka Output Plugin for FluentD

[Yogesh Sharma]

Hi Group,

I am trying to install Kafka Output Plugin as mentioned on "http://www.fluentd.org/plugins/all" OR "https://github.com/htgc/fluent-plugin-kafka"

Step Performed:

#git clone https://github.com/htgc/fluent-plugin-kafka.git

#cd fluent-plugin-kafka

"gem name 'fluent-plugin-kafka' is already present in fluent-plugin-kafka.gemspec"

#bundle

After running bundle, I am getting below error:

 [root@vagrant-centos64 fluent-plugin-kafka]# bundle

Don't run Bundler as root. Bundler can ask for sudo if it is needed, and installing your bundle as root will break this application for all

non-root users on this machine.

Fetching gem metadata from https://rubygems.org/.........

Fetching version metadata from https://rubygems.org/..

Resolving dependencies...

Gem::Installer::ExtensionBuildError: ERROR: Failed to build gem native extension.

/usr/bin/ruby extconf.rb

checking for main() in -lc... yes

checking for HAVE_RB_IO_T in rubyio.h... no

creating Makefile

make

gcc -I. -I. -I/usr/lib64/ruby/1.8/x86_64-linux -I.  -fPIC -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -fno-strict-aliasing  -fPIC   -c iobuffer.c

iobuffer.c:8:21: error: ruby/io.h: No such file or directory

iobuffer.c: In function ‘IO_Buffer_read_from’:

iobuffer.c:382: error: ‘OpenFile’ undeclared (first use in this function)

iobuffer.c:382: error: (Each undeclared identifier is reported only once

iobuffer.c:382: error: for each function it appears in.)

iobuffer.c:382: error: ‘fptr’ undeclared (first use in this function)

iobuffer.c:386: warning: implicit declaration of function ‘GetOpenFile’

iobuffer.c:387: warning: implicit declaration of function ‘rb_io_set_nonblock’

iobuffer.c: In function ‘IO_Buffer_write_to’:

iobuffer.c:408: error: ‘OpenFile’ undeclared (first use in this function)

iobuffer.c:408: error: ‘fptr’ undeclared (first use in this function)

iobuffer.c: In function ‘buffer_read_frame’:

iobuffer.c:634: warning: pointer targets in initialization differ in signedness

iobuffer.c:634: warning: pointer targets in initialization differ in signedness

iobuffer.c:630: warning: unused variable ‘tmp’

make: *** [iobuffer.o] Error 1

Gem files will remain installed in /usr/lib/ruby/gems/1.8/gems/cool.io-1.3.0 for inspection.

Results logged to /usr/lib/ruby/gems/1.8/gems/cool.io-1.3.0/ext/iobuffer/gem_make.out

An error occurred while installing cool.io (1.3.0), and Bundler cannot continue.

Make sure that `gem install cool.io -v '1.3.0'` succeeds before bundling.

Tried installing cool.io manually:

[root@vagrant-centos64 fluent-plugin-kafka]# gem install cool.io -v '1.3.0'

Building native extensions.  This could take a while...

ERROR:  Error installing cool.io:

ERROR: Failed to build gem native extension.

/usr/bin/ruby extconf.rb

checking for main() in -lc... yes

checking for HAVE_RB_IO_T in rubyio.h... no

creating Makefile

make

gcc -I. -I. -I/usr/lib64/ruby/1.8/x86_64-linux -I.  -fPIC -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -fno-strict-aliasing  -fPIC   -c iobuffer.c

iobuffer.c:8:21: error: ruby/io.h: No such file or directory

iobuffer.c: In function ‘IO_Buffer_read_from’:

iobuffer.c:382: error: ‘OpenFile’ undeclared (first use in this function)

iobuffer.c:382: error: (Each undeclared identifier is reported only once

iobuffer.c:382: error: for each function it appears in.)

iobuffer.c:382: error: ‘fptr’ undeclared (first use in this function)

iobuffer.c:386: warning: implicit declaration of function ‘GetOpenFile’

iobuffer.c:387: warning: implicit declaration of function ‘rb_io_set_nonblock’

iobuffer.c: In function ‘IO_Buffer_write_to’:

iobuffer.c:408: error: ‘OpenFile’ undeclared (first use in this function)

iobuffer.c:408: error: ‘fptr’ undeclared (first use in this function)

iobuffer.c: In function ‘buffer_read_frame’:

iobuffer.c:634: warning: pointer targets in initialization differ in signedness

iobuffer.c:634: warning: pointer targets in initialization differ in signedness

iobuffer.c:630: warning: unused variable ‘tmp’

make: *** [iobuffer.o] Error 1

Gem files will remain installed in /usr/lib/ruby/gems/1.8/gems/cool.io-1.3.0 for inspection.

Results logged to /usr/lib/ruby/gems/1.8/gems/cool.io-1.3.0/ext/iobuffer/gem_make.out

Logs:

[root@vagrant-centos64 fluent-plugin-kafka]# tail -1000f /usr/lib/ruby/gems/1.8/gems/cool.io-1.3.0/ext/iobuffer/gem_make.out

/usr/bin/ruby extconf.rb

checking for main() in -lc... yes

checking for HAVE_RB_IO_T in rubyio.h... no

creating Makefile

make

gcc -I. -I. -I/usr/lib64/ruby/1.8/x86_64-linux -I.  -fPIC -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -fno-strict-aliasing  -fPIC   -c iobuffer.c

iobuffer.c:8:21: error: ruby/io.h: No such file or directory

iobuffer.c: In function ‘IO_Buffer_read_from’:

iobuffer.c:382: error: ‘OpenFile’ undeclared (first use in this function)

iobuffer.c:382: error: (Each undeclared identifier is reported only once

iobuffer.c:382: error: for each function it appears in.)

iobuffer.c:382: error: ‘fptr’ undeclared (first use in this function)

iobuffer.c:386: warning: implicit declaration of function ‘GetOpenFile’

iobuffer.c:387: warning: implicit declaration of function ‘rb_io_set_nonblock’

iobuffer.c: In function ‘IO_Buffer_write_to’:

iobuffer.c:408: error: ‘OpenFile’ undeclared (first use in this function)

iobuffer.c:408: error: ‘fptr’ undeclared (first use in this function)

iobuffer.c: In function ‘buffer_read_frame’:

iobuffer.c:634: warning: pointer targets in initialization differ in signedness

iobuffer.c:634: warning: pointer targets in initialization differ in signedness

iobuffer.c:630: warning: unused variable ‘tmp’

make: *** [iobuffer.o] Error 1

same error when we try using "gem install"

Please suggest .

Thanks,

Yogesh Sharma

[Mr. Fiber]

Hi Yogesh,

You seem to use Ruby 1.8.

Fluentd doesn't support Ruby 1.8.

Fluentd works on Ruby 1.9.3 or later.

I recommend to use Ruby 2.1 or later.

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Yogesh Sharma]

Thanks Masahiro.

gem installed worked for kafka plugins

[root@vagrant-centos64 fluent-plugin-kafka]# gem install fluent-plugin-kafka

Fetching: fluent-plugin-kafka-0.0.11.gem (100%)

Successfully installed fluent-plugin-kafka-0.0.11

Parsing documentation for fluent-plugin-kafka-0.0.11

Installing ri documentation for fluent-plugin-kafka-0.0.11

Done installing documentation for fluent-plugin-kafka after 0 seconds

1 gem installed

However td-agent is still not able to recongnize, Do I need to add/include it some where.

[root@vagrant-centos64 fluent-plugin-kafka]# tail -10f /var/log/td-agent/td-agent.log

2015-04-14 09:12:43 +0000 [error]: dry run failed: Unknown output plugin 'kafka'. Run 'gem search -rd fluent-plugin' to find plugins

2015-04-14 09:29:53 +0000 [error]: dry run failed: Unknown output plugin 'kafka'. Run 'gem search -rd fluent-plugin' to find plugins

Best Regards,
__________________________________________

Yogesh Sharma

Email: yks...@gmail.com | Web: www.initd.in

RHCE, VCE-CIA, RackSpace Cloud U

--
You received this message because you are subscribed to a topic in the Google Groups "Fluentd Google Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/fluentd/NjjU6MvqvzY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to fluentd+u...@googlegroups.com.

[Yogesh Sharma]

Tried with td-agent-gem as well.

[root@vagrant-centos64 plugin]# /usr/sbin/td-agent-gem install fluent-plugin-kafka

WARN: Unresolved specs during Gem::Specification.reset:

      json (>= 1.4.3)

WARN: Clearing out unresolved specs.

Please report a bug if this causes problems.

ERROR:  While executing gem ... (Gem::RemoteFetcher::UnknownHostError)

    no such name (https://api.rubygems.org/quick/Marshal.4.8/poseidon-0.0.5.gemspec.rz)

[root@vagrant-centos64 plugin]#

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Fluentd Google Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/fluentd/NjjU6MvqvzY/unsubscribe.

To unsubscribe from this group and all its topics, send an email to fluentd+unsubscribe@googlegroups.com.

[Mr. Fiber]

> ERROR:  While executing gem ... (Gem::RemoteFetcher::UnknownHostError)

It seems your vagrant box can't access to rubygems.org.
Can your network access to rubygems.org?

I can install it:

% sudo /usr/sbin/td-agent-gem install fluent-plugin-kafka --no-rdoc

WARN: Unresolved specs during Gem::Specification.reset:
      json (>= 1.4.3)
WARN: Clearing out unresolved specs.
Please report a bug if this causes problems.

Fetching: poseidon-0.0.5.gem (100%)
Successfully installed poseidon-0.0.5
Fetching: zookeeper-1.4.10.gem (100%)

Building native extensions.  This could take a while...

Successfully installed zookeeper-1.4.10

Fetching: fluent-plugin-kafka-0.0.11.gem (100%)
Successfully installed fluent-plugin-kafka-0.0.11
Parsing documentation for fluent-plugin-kafka-0.0.11
Installing ri documentation for fluent-plugin-kafka-0.0.11

Parsing documentation for poseidon-0.0.5
Installing ri documentation for poseidon-0.0.5
Parsing documentation for zookeeper-1.4.10
Installing ri documentation for zookeeper-1.4.10
Done installing documentation for fluent-plugin-kafka, poseidon, zookeeper after 1 seconds
3 gems installed

[Yogesh Sharma]

Thanks Mr. Fiber.

I have fixed the kafka-plugins issue. However it is not able to send the data to Kafka topics and no error is being logged.

I am actually very new to td-agent and we are comparing the performace of TD-Agent over other Log Mgmt System, However stuck with initial phase where we need to ship the logs to kafka cluster.

When I am adding a line in logs, pattern apache with type stdout is working which means td-agent is fine, however type=kafka not sending the logs.

#echo "test11" >> /var/log/httpd-access.log

td-agent logs:

2015-04-14 18:56:26 +0530 [info]: adding match pattern="debug.**" type="stdout"

2015-04-14 18:56:26 +0530 [info]: adding match pattern="debug.**" type="stdout"

2015-04-14 18:56:26 +0530 [info]: adding match pattern="apache.**" type="stdout"

2015-04-14 18:56:26 +0530 [info]: adding match pattern="apache.**" type="kafka"

2015-04-14 18:56:26 +0530 [info]: brokers has been set directly: ["pocvertica01:9092"]

2015-04-14 18:56:26 +0530 [info]: adding source type="tail"

2015-04-14 18:56:26 +0530 [info]: adding source type="http"

2015-04-14 18:56:26 +0530 [info]: adding source type="debug_agent"

2015-04-14 18:56:26 +0530 [info]: initialized producer kafka

2015-04-14 18:56:26 +0530 [info]: following tail of /var/log/httpd-access.log

2015-04-14 18:56:26 +0530 [info]: listening dRuby uri="druby://127.0.0.1:24230" object="Engine"

2015-04-14 18:56:32 +0530 apache.access: {"message":"test11"}

2015-04-14 18:56:40 +0530 apache.access: {"message":"test11"}

2015-04-14 19:01:05 +0530 apache.access: {"message":"test1"}

there is no ouput in Kafka Consumer console.

#/bin/kafka-console-consumer.sh --zookeeper 20.0.1.58:2181 --topic mob_api_server_logs_direct_logstash

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".

SLF4J: Defaulting to no-operation (NOP) logger implementation

SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

ALSO while installing snappy for kafka compression , it is failing, for error marked bold , I have already installed ruby-devel (OS CentOS 6.6)

# /usr/sbin/td-agent-gem install snappy --no-rdoc

WARN: Unresolved specs during Gem::Specification.reset:

      json (>= 1.4.3)

WARN: Clearing out unresolved specs.

Please report a bug if this causes problems.

Building native extensions.  This could take a while...

ERROR:  Error installing snappy:

ERROR: Failed to build gem native extension.

    /opt/td-agent/embedded/bin/ruby extconf.rb

checking for main() in -lsnappy... no

./autogen.sh: line 5: libtoolize: command not found

cp /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/snappy-0.0.11/ext/../vendor/snappy/config.h /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/snappy-0.0.11/ext/../vendor/snappy/snappy-c.cc /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/snappy-0.0.11/ext/../vendor/snappy/snappy-c.h /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/snappy-0.0.11/ext/../vendor/snappy/snappy-internal.h /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/snappy-0.0.11/ext/../vendor/snappy/snappy-sinksource.cc /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/snappy-0.0.11/ext/../vendor/snappy/snappy-sinksource.h /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/snappy-0.0.11/ext/../vendor/snappy/snappy-stubs-internal.cc /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/snappy-0.0.11/ext/../vendor/snappy/snappy-stubs-internal.h /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/snappy-0.0.11/ext/../vendor/snappy/snappy-stubs-public.h /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/snappy-0.0.11/ext/../vendor/snappy/snappy.cc /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/snappy-0.0.11/ext/../vendor/snappy/snappy.h /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/snappy-0.0.11/ext

*** extconf.rb failed ***

Could not create Makefile due to some reason, probably lack of necessary

libraries and/or headers.  Check the mkmf.log file for more details.  You may

need configuration options.

Provided configuration options:

--with-opt-dir

--with-opt-include

--without-opt-include=${opt-dir}/include

--with-opt-lib

--without-opt-lib=${opt-dir}/lib

--with-make-prog

--without-make-prog

--srcdir=.

--curdir

--ruby=/opt/td-agent/embedded/bin/ruby

--with-snappylib

--without-snappylib

/opt/td-agent/embedded/lib/ruby/2.1.0/fileutils.rb:1401:in `initialize': No such file or directory @ rb_sysopen - /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/snappy-0.0.11/ext/../vendor/snappy/config.h (Errno::ENOENT)

from /opt/td-agent/embedded/lib/ruby/2.1.0/fileutils.rb:1401:in `open'

from /opt/td-agent/embedded/lib/ruby/2.1.0/fileutils.rb:1401:in `copy_file'

from /opt/td-agent/embedded/lib/ruby/2.1.0/fileutils.rb:483:in `copy_file'

from /opt/td-agent/embedded/lib/ruby/2.1.0/fileutils.rb:400:in `block in cp'

from /opt/td-agent/embedded/lib/ruby/2.1.0/fileutils.rb:1579:in `block in fu_each_src_dest'

from /opt/td-agent/embedded/lib/ruby/2.1.0/fileutils.rb:1588:in `block in fu_each_src_dest0'

from /opt/td-agent/embedded/lib/ruby/2.1.0/fileutils.rb:1586:in `each'

from /opt/td-agent/embedded/lib/ruby/2.1.0/fileutils.rb:1586:in `fu_each_src_dest0'

from /opt/td-agent/embedded/lib/ruby/2.1.0/fileutils.rb:1577:in `fu_each_src_dest'

from /opt/td-agent/embedded/lib/ruby/2.1.0/fileutils.rb:399:in `cp'

from extconf.rb:27:in `<main>'

extconf failed, exit code 1

Gem files will remain installed in /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/snappy-0.0.11 for inspection.

Results logged to /opt/td-agent/embedded/lib/ruby/gems/2.1.0/extensions/x86_64-linux/2.1.0/snappy-0.0.11/gem_make.out

Thanks much for your time here.

Best Regards,
__________________________________________

Yogesh Sharma

Email: yks...@gmail.com | Web: www.initd.in

RHCE, VCE-CIA, RackSpace Cloud U

[Mr. Fiber]

Could you paste your configuration here?

[Yogesh Sharma]

Hi,

Please find the config:

<source>

  type tail

  path /var/log/httpd-access.log

  pos_file /var/log/td-agent/httpd-access.log.pos

  tag apache.access

  read_from_head true

  format /^(?<message>.*)$/

</source>

<match apache.**>

    type stdout

</match>

<match apache.**>

        type                kafka

        brokers             pocvertica01:9092

        default_topic       mob_api_server_logs_direct_logstash

        output_data_type    json

#        compression_codec   snappy

</match>

Best Regards,
__________________________________________

Yogesh Sharma

Email: yks...@gmail.com | Web: www.initd.in

RHCE, VCE-CIA, RackSpace Cloud U

[Mr. Fiber]

I see. It doesn't work.

See this document: http://docs.fluentd.org/articles/config-file#match-order

Forgot to add ML address...

[Mr. Fiber]

> ALSO while installing snappy for kafka compression , it is failing, for error marked bold , I have already installed ruby-devel (OS CentOS 6.6)

> opt/td-agent/embedded/bin/ruby extconf.rb

> checking for main() in -lsnappy... no

> ./autogen.sh: line 5: libtoolize: command not found

Did you install snappy library?

[Yogesh Sharma]

Thanks Mr. Fiber.

Kafka Plugins issue is resovled. we are able to send to kafka topics. However , We need to understand the format of pos_file in input type tail

My Log File:

[root@poc_fluentd plugin]# echo "testing_fluntd" >> /var/log/httpd-access.log

Its offset and inode:

[root@poc_fluentd plugin]# ls -li /var/log/httpd-access.log

41 -rw-r--r-- 1 root root 15 Apr 16 11:58 /var/log/httpd-access.log

Fluentd/td-agent pos_file output. It is not matching the offset of the log file. May I please know how to read it if I have to know how much file is being processed.

[root@poc_fluentd plugin]# cat /var/log/td-agent/httpd-access.log.pos

/var/log/httpd-access.log 000000000000000f 0000000000000029

Best Regards,
__________________________________________

Yogesh Sharma

Email: yks...@gmail.com | Web: www.initd.in

RHCE, VCE-CIA, RackSpace Cloud U

[Yogesh Sharma]

It seems to be Hex value.

Best Regards,
__________________________________________

Yogesh Sharma

Email: yks...@gmail.com | Web: www.initd.in

RHCE, VCE-CIA, RackSpace Cloud U

[Yogesh Sharma]

++FluentD

Best Regards,
__________________________________________

Yogesh Sharma

Email: yks...@gmail.com | Web: www.initd.in

RHCE, VCE-CIA, RackSpace Cloud U

On Tue, Apr 21, 2015 at 5:28 PM, Yogesh Sharma <yks...@gmail.com> wrote:

Hi,

Need one suggestion. Below is the td-agent config.  I have used * but it is taking only one file, there are two more files present but fluentd is not picking up those. Am I doing anything wrong. Please suggest.

<source>

    type tail

    path /apps/logs/server.log.2015*

    pos_file /var/log/td-agent/httpd-access.log.pos

    tag mobapi.serverlog

    read_from_head true

    format /^(?<message>.*)$/

  </source>

  <filter mobapi.serverlog>

    type record_transformer

    <record>

      host poc_fluentd

      file mob_api_server_log

    </record>

  </filter>

  <match mobapi.serverlog>

    type kafka

    brokers 20.0.0.4:9092

    default_topic MOB_API_LOGS_FLUENTD

    output_data_type json

    output_include_time true

  </match>

Log:

2015-04-21 17:22:24 +0530 [info]: adding filter pattern="mobapi.serverlog" type="record_transformer"

2015-04-21 17:22:24 +0530 [info]: adding match pattern="mobapi.serverlog" type="kafka"

2015-04-21 17:22:24 +0530 [info]: brokers has been set directly: ["20.0.0.4:9092"]

2015-04-21 17:22:24 +0530 [info]: adding source type="tail"

2015-04-21 17:22:24 +0530 [info]: initialized producer kafka

2015-04-21 17:22:24 +0530 [info]: following tail of /apps/logs/server.log.2015-03-19

Best Regards,
__________________________________________

Yogesh Sharma

Email: yks...@gmail.com | Web: www.initd.in

RHCE, VCE-CIA, RackSpace Cloud U

---------- Forwarded message ----------
From: Mr. Fiber <repea...@gmail.com>
Date: Thu, Apr 16, 2015 at 1:01 PM
Subject: Re: Error Installing Kafka Output Plugin for FluentD
To: Yogesh Sharma <yks...@gmail.com>

Ahhh. Yeah, you are correct ;)

On Thu, Apr 16, 2015 at 4:29 PM, Yogesh Sharma <yks...@gmail.com> wrote:

Yes, I got it and replied. Seems like you did not noticed :)

Best Regards,
__________________________________________

Yogesh Sharma

Email: yks...@gmail.com | Web: www.initd.in

RHCE, VCE-CIA, RackSpace Cloud U

On Thu, Apr 16, 2015 at 12:57 PM, Mr. Fiber <repea...@gmail.com> wrote:

> It is not matching the offset of the log file

What do you mean? 15 is f in hex number.

in_tail uses hex number for offset and inode.

https://github.com/fluent/fluentd/blob/b5f65ac1868f5322a4b2a54e2fc5d99fad91efe6/lib/fluent/plugin/in_tail.rb#L646

[Zheng Gao]

Hi Sharma,

I met same problem with you which has similar error message when start td-agent service, could you tell me how you fix this issue?

root@wleppt-156:/etc/td-agent# tail -5 /var/log/td-agent/td-agent.log  

2015-08-12 17:16:59 +0800 [error]: dry run failed: Unknown output plugin 'kafka'. Run 'gem search -rd fluent-plugin' to find plugins

2015-08-12 17:29:01 +0800 [error]: dry run failed: Unknown output plugin 'kafka'. Run 'gem search -rd fluent-plugin' to find plugins

2015-08-12 17:35:39 +0800 [error]: dry run failed: Unknown output plugin 'kafka_buffered'. Run 'gem search -rd fluent-plugin' to find plugins

2015-08-12 17:46:38 +0800 [error]: dry run failed: Unknown output plugin 'kafka_buffered'. Run 'gem search -rd fluent-plugin' to find plugins

2015-08-12 17:58:00 +0800 [error]: dry run failed: Unknown output plugin 'kafka_buffered'. Run 'gem search -rd fluent-plugin' to find plugins

My Config:

<source>

    type        tail

    path        /dianyi/log/*/*/*/*.log 

    pos_file    /tmp/facebook.log.pos

    tag         docker.* 

    format none

#    format multiline

#    format_firstline /^2015/

#    format1 /2015(?<method1>[^\n]+)\n/

#    format2 /com\.(?<method2>[^\n]+)\n/

#    format3 /\tat(?<method3>[^\n]+)\n/

    read_from_head      true

</source>

<match docker.dianyi.log.*.*.*.*.log >

  type null

</match>

<filter docker.dianyi.log.*.*.*.*.*.*.*.log>

  type record_transformer

  <record>

    hostname "#{Socket.gethostname}" 

  </record>

</filter>

<match docker.dianyi.log.*.*.*.*.*.*.*.log>

  type                kafka_buffered                        #   ================> I tried and it works well if I change type from kafka_buffered to stdout or elasticsearch.

  zookeeper           172.30.80.90:2181

  default_topic       clp 

#  default_partition_key (string)   :default => nil

  flush_interval      1 

  buffer_type         memory

  output_data_type    json

  output_include_tag  true

  output_include_time true

#  max_send_retries    (integer)    :default => 3

#  required_acks       (integer)    :default => 0

#  ack_timeout_ms      (integer)    :default => 1500

#  compression_codec   (none|gzip|snappy) :default => none

</match>

在 2015年4月14日星期二 UTC+8下午1:51:45，Yogesh Sharma写道：