elasticsearch and tomcat log files

dshe...@teamcmi.com

unread,

Jun 20, 2016, 2:35:20 PM6/20/16

to Fluentd Google Group

Hello All,

I have been having troubles with fluentd trying to send catalina.out entries to elasticsearch. Any help would be appreciated - I don't have much hair left to pull out.

The config file is as follows:

@type tail

format none

path /var/log/tomcat*/localhost_access_log.%Y-%m-%d.txt

pos_file /var/log/td-agent/google-fluentd/pos/tomcat.pos

read_from_head true

tag tomcat-localhost_access_log

</source>

@type tail

format multiline

# Match the date at the beginning of each entry, which can be in one of two

# different formats.

format_firstline /^(\w+\s\d+,\s\d+)|(\d+-\d+-\d+\s)/

format1 /(?<message>.*)/

path /var/log/tomcat*/catalina.out,/var/log/tomcat*/localhost.*.log

pos_file /var/log/td-agent/google-fluentd/pos/tomcat-multiline.pos

read_from_head true

tag tomcat

</source>

@type copy

<store>

type elasticsearch

@log_level trace

host 10.200.10.50

port 9200

index_name fluentd

type_name fluentd

logstash_format true

logstash_prefix fluentd

# flush_interval 10

</store>

<store>

@type stdout

</store>

</match>

I am getting the following in the td-agent.log file when I restart tomcat (Tomcat8, Java8, Debian 8 - Jessie):

2016-06-20 13:26:51 -0500 [info]: reading config file path="/etc/td-agent/td-agent.conf"

2016-06-20 13:26:51 -0500 [info]: starting fluentd-0.12.20

2016-06-20 13:26:51 -0500 [info]: gem 'fluent-mixin-config-placeholders' version '0.3.1'

2016-06-20 13:26:51 -0500 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6'

2016-06-20 13:26:51 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '1.5.0'

2016-06-20 13:26:51 -0500 [info]: gem 'fluent-plugin-mongo' version '0.7.12'

2016-06-20 13:26:51 -0500 [info]: gem 'fluent-plugin-record-modifier' version '0.4.1'

2016-06-20 13:26:51 -0500 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.5.4'

2016-06-20 13:26:51 -0500 [info]: gem 'fluent-plugin-s3' version '0.6.5'

2016-06-20 13:26:51 -0500 [info]: gem 'fluent-plugin-scribe' version '0.10.14'

2016-06-20 13:26:51 -0500 [info]: gem 'fluent-plugin-td' version '0.10.28'

2016-06-20 13:26:51 -0500 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.2'

2016-06-20 13:26:51 -0500 [info]: gem 'fluent-plugin-webhdfs' version '0.4.1'

2016-06-20 13:26:51 -0500 [info]: gem 'fluentd' version '0.12.20'

2016-06-20 13:26:51 -0500 [info]: adding match pattern="tomcat" type="copy"

2016-06-20 13:26:51 -0500 [info]: adding source type="tail"

2016-06-20 13:26:51 -0500 [info]: using configuration file: <ROOT>

...

2016-06-20 13:27:02 -0500 tomcat: {"message":"Jun 20, 2016 1:27:00 PM org.apache.catalina.startup.HostConfig deployDirectory\nINFO: Deploying web application directory /var/lib/tomcat8/webapps/ROOT"}

2016-06-20 13:27:02 -0500 tomcat: {"message":"Jun 20, 2016 1:27:02 PM org.apache.jasper.servlet.TldScanner scanJars\nINFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time."}

2016-06-20 13:27:02 -0500 tomcat: {"message":"Jun 20, 2016 1:27:02 PM org.apache.catalina.startup.HostConfig deployDirectory\nINFO: Deployment of web application directory /var/lib/tomcat8/webapps/ROOT has finished in 1,668 ms"}

2016-06-20 13:27:02 -0500 tomcat: {"message":"Jun 20, 2016 1:27:02 PM org.apache.coyote.AbstractProtocol start\nINFO: Starting ProtocolHandler [\"http-nio-8080\"]"}

2016-06-20 13:27:52 -0500 [warn]: temporarily failed to flush the buffer. next_retry=2016-06-20 13:27:53 -0500 error_class="MultiJson::ParseError" error="Oj::ParseError" plugin_id="object:3fbf26cbb270"

2016-06-20 13:27:52 -0500 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/multi_json-1.12.1/lib/multi_json/adapter.rb:19:in `load'

2016-06-20 13:27:52 -0500 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/multi_json-1.12.1/lib/multi_json.rb:122:in `load'

2016-06-20 13:27:52 -0500 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.17/lib/elasticsearch/transport/transport/serializer/multi_json.rb:24:in `load'

2016-06-20 13:27:52 -0500 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.17/lib/elasticsearch/transport/transport/base.rb:315:in `perform_request'

2016-06-20 13:27:52 -0500 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.17/lib/elasticsearch/transport/transport/http/faraday.rb:20:in `perform_request'

2016-06-20 13:27:52 -0500 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.17/lib/elasticsearch/transport/client.rb:128:in `perform_request'

2016-06-20 13:27:52 -0500 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-api-1.0.17/lib/elasticsearch/api/actions/ping.rb:20:in `block in ping'

2016-06-20 13:27:52 -0500 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-api-1.0.17/lib/elasticsearch/api/utils.rb:197:in `__rescue_from_not_found'

2016-06-20 13:27:52 -0500 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-api-1.0.17/lib/elasticsearch/api/actions/ping.rb:19:in `ping'

2016-06-20 13:27:52 -0500 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.5.0/lib/fluent/plugin/out_elasticsearch.rb:122:in `client'

2016-06-20 13:27:52 -0500 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.5.0/lib/fluent/plugin/out_elasticsearch.rb:279:in `rescue in send'

2016-06-20 13:27:52 -0500 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.5.0/lib/fluent/plugin/out_elasticsearch.rb:277:in `send'

2016-06-20 13:27:52 -0500 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.5.0/lib/fluent/plugin/out_elasticsearch.rb:271:in `write'

2016-06-20 13:27:52 -0500 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/buffer.rb:345:in `write_chunk'

2016-06-20 13:27:52 -0500 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/buffer.rb:324:in `pop'

2016-06-20 13:27:52 -0500 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/output.rb:329:in `try_flush'

2016-06-20 13:27:52 -0500 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/output.rb:140:in `run'

2016-06-20 13:27:53 -0500 [warn]: temporarily failed to flush the buffer. next_retry=2016-06-20 13:27:56 -0500 error_class="MultiJson::ParseError" error="Oj::ParseError" plugin_id="object:3fbf26cbb270"

2016-06-20 13:27:53 -0500 [warn]: suppressed same stacktrace

...

I am assuming that there is some issue with the json that is being sent to the elasticsearch but I have no clue - nor have been able to find anything via Loogle - what could be the problem.

Thanks and have a good one,

Dan

Mr. Fiber

unread,

Jun 20, 2016, 2:42:30 PM6/20/16

to Fluentd Google Group

> I am assuming that there is some issue with the json that is being sent to the elasticsearch

No. Error is ParseError.

https://github.com/elastic/elasticsearch-ruby/blob/master/elasticsearch-transport/lib/elasticsearch/transport/transport/base.rb#L315

From the code, the problem is Elasticsearch retruns a JSON response which cannot be parsed by Oj json parser.

So the problem is Elasticsearch or Elasticsearch ruby client issue, not fluentd.

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

dshe...@teamcmi.com

unread,

Jun 20, 2016, 4:37:59 PM6/20/16

to Fluentd Google Group

Thanks for the quick reply.

The elasticsearch on the server is 5.0.0-alpha3. I will try it tomorrow using 2.3.3 and provide the results.

dshe...@teamcmi.com

unread,

Jun 21, 2016, 7:36:56 AM6/21/16

to Fluentd Google Group

I just tried it on elasticsearch 2.3.3 with no problems. Thanks for pointing me in the right direction.

Reply all

Reply to author

Forward