Receiving warning messages about character encoding.
148 views
Skip to first unread message
Lance N.
Feb 23, 2015, 5:33:34 PM2/23/15
to flu...@googlegroups.com
I'm running the latest td-agent release. This is the full log file. I don't know how this "encoding error" could happen. We only send USASCII 7 text.
Lance
2015-02-12 21:11:35 +0000 [info]: reading config file path="/etc/td-agent/mux.conf"
2015-02-12 21:11:35 +0000 [info]: starting fluentd-0.10.58
2015-02-12 21:11:35 +0000 [info]: gem 'fluent-mixin-config-placeholders' version '0.3.0'
2015-02-12 21:11:35 +0000 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2015-02-12 21:11:35 +0000 [info]: gem 'fluent-mixin-rewrite-tag-name' version '0.1.0'
2015-02-12 21:11:35 +0000 [info]: gem 'fluent-plugin-cadvisor' version '0.2.5'
2015-02-12 21:11:35 +0000 [info]: gem 'fluent-plugin-forest' version '0.3.0'
2015-02-12 21:11:35 +0000 [info]: gem 'fluent-plugin-graphite' version '0.0.5'
2015-02-12 21:11:35 +0000 [info]: gem 'fluent-plugin-mongo' version '0.7.4'
2015-02-12 21:11:35 +0000 [info]: gem 'fluent-plugin-multiprocess' version '0.1.1'
2015-02-12 21:11:35 +0000 [info]: gem 'fluent-plugin-numeric-monitor' version '0.1.7'
2015-02-12 21:11:35 +0000 [info]: gem 'fluent-plugin-record-reformer' version '0.4.0'
2015-02-12 21:11:35 +0000 [info]: gem 'fluent-plugin-rewrite' version '0.0.12'
2015-02-12 21:11:35 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.4.1'
2015-02-12 21:11:35 +0000 [info]: gem 'fluent-plugin-s3' version '0.5.1'
2015-02-12 21:11:35 +0000 [info]: gem 'fluent-plugin-scribe' version '0.10.14'
2015-02-12 21:11:35 +0000 [info]: gem 'fluent-plugin-td' version '0.10.23'
2015-02-12 21:11:35 +0000 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.0'
2015-02-12 21:11:35 +0000 [info]: gem 'fluent-plugin-webhdfs' version '0.4.1'
2015-02-12 21:11:35 +0000 [info]: gem 'fluentd' version '0.10.58'
2015-02-12 21:11:35 +0000 [info]: using configuration file: <ROOT>
<source>
type http
port 8999
</source>
<source>
type forward
port 24224
bind 0.0.0.0
</source>
<source>
type monitor_agent
bind 0.0.0.0
port 24219
</source>
<match post.*.*.*>
type record_reformer
renew_record false
enable_ruby true
service ${tag_parts[1]}.${tag_parts[2]}.${tag_parts[3]}
service_hostname #{ENV['IP']}
tag cluster.${tag_parts[1]}.${tag_parts[2]}.${tag_parts[3]}
datestamp ${time.strftime('%Y-%m-%dT%H:%M:%S.%LZ')}
</match>
<match cluster.*.*.*>
type record_reformer
renew_record false
enable_ruby true
tag cluster_vars.${tag_parts[1]}.${tag_parts[2]}.${tag_parts[3]}
cluster_ip #{ENV['IP']}
</match>
<match cluster_vars.*.*.*>
type rewrite
remove_prefix cluster_vars
add_prefix cluster_stats
</match>
<match cluster_stats.*.*.*>
type copy
<store>
type record_reformer
enable_ruby true
tag upload.${service}
</store>
<store>
type record_reformer
tag monitor_tag.${service}
count 1
</store>
</match>
<match monitor_tag.*.*.*>
type record_reformer
renew_record true
enable_ruby true
tag monitor_second.${tag_parts[1]}.${tag_parts[2]}.${tag_parts[3]}
count 1
</match>
<match monitor_second.*.*.*>
type numeric_monitor
monitor_key count
count_interval 1
aggregate tag
input_tag_remove_prefix monitor_second
output_per_tag yes
tag_prefix monitor_second_filter
</match>
<match monitor_second_filter.*.*.*>
type record_reformer
renew_record true
enable_ruby true
count ${num}
tag monitor_second_filter2.${tag_parts[1]}.${tag_parts[2]}.${tag_parts[3]}
</match>
<match monitor_second_filter2.*.*.*>
type rewrite
remove_prefix monitor_second_filter2
add_prefix monitor_minute
<rule>
key count
pattern ^0$
ignore
</rule>
</match>
<match monitor_minute.*.*.*>
type numeric_monitor
monitor_key count
unit minute
aggregate tag
input_tag_remove_prefix monitor_minute
percentiles 95,99
output_per_tag yes
tag_prefix monitor
</match>
<match monitor.*.*.*>
type record_reformer
renew_record false
enable_ruby true
datestamp ${time.strftime('%Y-%m-%dT%H:%M:%S.%LZ')}
timestamp ${time.strftime('%Y-%m-%d %H:%M:%S')}
monitor_service ${tag_parts[1]}.${tag_parts[2]}.${tag_parts[3]}
tag monitor_stats.tiger.tracks.monitorstats
monitor_hostname #{ENV['IP']}
monitor_num ${num}
service tiger.tracks.monitorstats
</match>
<match monitor_stats.*.*.*>
type rewrite
remove_prefix monitor_stats
add_prefix monitor_out
<rule>
key monitor_service
pattern ^((?!stats).)*$
ignore
</rule>
<rule>
key monitor_hostname
pattern ([^.]+)\..*
replace \\1
</rule>
<rule>
key monitor_num
pattern ^0$
ignore
</rule>
</match>
<match monitor_out.*.*.*>
type copy
<store>
type rewrite
remove_prefix monitor_out
add_prefix upload
</store>
<store>
type record_reformer
enable_ruby true
tag tracks.${monitor_service}
</store>
</match>
<match tracks.*.*.*>
type graphite
host 10.0.0.0
port 2003
tag_for prefix
name_keys num,min,max,avg,sum,p90,p95
</match>
<match upload.{PROD,Prod,ip}.*.*>
type record_reformer
tag upload.prod.${tag_parts[2]}.${tag_parts[3]}
</match>
<match upload.QA.*.*>
type record_reformer
tag upload.qa.${tag_parts[2]}.${tag_parts[3]}
</match>
<match upload.*.*.*>
type rewrite
remove_prefix upload
add_prefix hdfs
</match>
<match hdfs.prod.apps.actions>
type forward
<server>
name hdfs_actions
host 127.0.0.1
port 24225
</server>
</match>
<match hdfs.prod.apps.zendmodo_http>
type forward
<server>
name hdfs_zendmodo_http
host 127.0.0.1
port 24226
</server>
</match>
<match hdfs.prod.apps.zendmodo_other>
type forward
<server>
name hdfs_zendmodo_other
host 127.0.0.1
port 24227
</server>
</match>
<match hdfs.*.*.*>
type forward
<server>
name hdfs_forest
host 127.0.0.1
port 24228
</server>
</match>
<match cluster.**>
type numeric_monitor
monitor_key count
unit minute
aggregate tag
input_tag_remove_prefix cluster_stats
percentiles 90,95
output_per_tag yes
tag_prefix debug
</match>
<match debug.**>
type stdout
</match>
<match **>
type stdout
</match>
</ROOT>
2015-02-12 21:11:35 +0000 [info]: adding source type="http"
2015-02-12 21:11:35 +0000 [info]: adding source type="forward"
2015-02-12 21:11:35 +0000 [info]: adding source type="monitor_agent"
2015-02-12 21:11:35 +0000 [info]: adding match pattern="post.*.*.*" type="record_reformer"
2015-02-12 21:11:35 +0000 [info]: adding match pattern="cluster.*.*.*" type="record_reformer"
2015-02-12 21:11:35 +0000 [info]: adding match pattern="cluster_vars.*.*.*" type="rewrite"
2015-02-12 21:11:35 +0000 [info]: adding match pattern="cluster_stats.*.*.*" type="copy"
2015-02-12 21:11:35 +0000 [info]: adding match pattern="monitor_tag.*.*.*" type="record_reformer"
2015-02-12 21:11:35 +0000 [info]: adding match pattern="monitor_second.*.*.*" type="numeric_monitor"
2015-02-12 21:11:35 +0000 [info]: adding match pattern="monitor_second_filter.*.*.*" type="record_reformer"
2015-02-12 21:11:35 +0000 [info]: adding match pattern="monitor_second_filter2.*.*.*" type="rewrite"
2015-02-12 21:11:35 +0000 [info]: adding match pattern="monitor_minute.*.*.*" type="numeric_monitor"
2015-02-12 21:11:35 +0000 [info]: adding match pattern="monitor.*.*.*" type="record_reformer"
2015-02-12 21:11:35 +0000 [info]: adding match pattern="monitor_stats.*.*.*" type="rewrite"
2015-02-12 21:11:35 +0000 [info]: adding match pattern="monitor_out.*.*.*" type="copy"
2015-02-12 21:11:35 +0000 [info]: adding match pattern="tracks.*.*.*" type="graphite"
2015-02-12 21:11:35 +0000 [info]: adding match pattern="upload.{PROD,Prod,ip}.*.*" type="record_reformer"
2015-02-12 21:11:35 +0000 [info]: adding match pattern="upload.QA.*.*" type="record_reformer"
2015-02-12 21:11:35 +0000 [info]: adding match pattern="upload.*.*.*" type="rewrite"
2015-02-12 21:11:35 +0000 [info]: adding match pattern="hdfs.prod.apps.actions" type="forward"
2015-02-12 21:11:35 +0000 [info]: adding forwarding server 'hdfs_actions' host="127.0.0.1" port=24225 weight=60 plugin_id="object:3f9c40f55318"
2015-02-12 21:11:35 +0000 [info]: adding match pattern="hdfs.prod.apps.zendmodo_http" type="forward"
2015-02-12 21:11:35 +0000 [info]: adding forwarding server 'hdfs_zendmodo_http' host="127.0.0.1" port=24226 weight=60 plugin_id="object:3f9c40f4a3c8"
2015-02-12 21:11:35 +0000 [info]: adding match pattern="hdfs.prod.apps.zendmodo_other" type="forward"
2015-02-12 21:11:35 +0000 [info]: adding forwarding server 'hdfs_zendmodo_other' host="127.0.0.1" port=24227 weight=60 plugin_id="object:3f9c40f469d0"
2015-02-12 21:11:35 +0000 [info]: adding match pattern="hdfs.*.*.*" type="forward"
2015-02-12 21:11:35 +0000 [info]: adding forwarding server 'hdfs_forest' host="127.0.0.1" port=24228 weight=60 plugin_id="object:3f9c40f40f94"
2015-02-12 21:11:35 +0000 [info]: adding match pattern="cluster.**" type="numeric_monitor"
2015-02-12 21:11:35 +0000 [info]: adding match pattern="debug.**" type="stdout"
2015-02-12 21:11:35 +0000 [info]: adding match pattern="**" type="stdout"
2015-02-12 21:11:35 +0000 [info]: listening fluent socket on 0.0.0.0:24224
2015-02-19 15:29:42 +0000 [warn]: emit transaction failed error_class=Encoding::CompatibilityError error=#<Encoding::CompatibilityError: incompatible character encodings: ASCII-8BIT and UTF-8>
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/event.rb:33:in `<<'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/event.rb:33:in `to_msgpack'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/event.rb:33:in `block in to_msgpack_stream'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/event.rb:55:in `call'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/event.rb:55:in `each'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/event.rb:32:in `to_msgpack_stream'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/output.rb:429:in `emit'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/match.rb:36:in `emit'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/engine.rb:160:in `emit_stream'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/engine.rb:140:in `emit'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-rewrite-0.0.12/lib/fluent/plugin/out_rewrite.rb:56:in `block in emit'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/event.rb:55:in `call'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/event.rb:55:in `each'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-rewrite-0.0.12/lib/fluent/plugin/out_rewrite.rb:53:in `emit'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/match.rb:36:in `emit'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/engine.rb:160:in `emit_stream'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/engine.rb:140:in `emit'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-record-reformer-0.4.0/lib/fluent/plugin/out_record_reformer.rb:90:in `block in emit'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/event.rb:55:in `call'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/event.rb:55:in `each'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-record-reformer-0.4.0/lib/fluent/plugin/out_record_reformer.rb:87:in `emit'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/output.rb:33:in `next'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/plugin/out_copy.rb:73:in `emit'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/match.rb:36:in `emit'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/engine.rb:160:in `emit_stream'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/engine.rb:140:in `emit'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-rewrite-0.0.12/lib/fluent/plugin/out_rewrite.rb:56:in `block in emit'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/event.rb:55:in `call'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/event.rb:55:in `each'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-rewrite-0.0.12/lib/fluent/plugin/out_rewrite.rb:53:in `emit'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/match.rb:36:in `emit'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/engine.rb:160:in `emit_stream'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/engine.rb:140:in `emit'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-record-reformer-0.4.0/lib/fluent/plugin/out_record_reformer.rb:90:in `block in emit'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/event.rb:149:in `feed_each'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/event.rb:149:in `each'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-record-reformer-0.4.0/lib/fluent/plugin/out_record_reformer.rb:87:in `emit'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/match.rb:36:in `emit'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/engine.rb:160:in `emit_stream'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/plugin/in_forward.rb:148:in `on_message'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/plugin/in_forward.rb:235:in `call'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/plugin/in_forward.rb:235:in `block in on_read_msgpack'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/plugin/in_forward.rb:234:in `feed_each'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/plugin/in_forward.rb:234:in `on_read_msgpack'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.2.4/lib/cool.io/io.rb:128:in `on_readable'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.2.4/lib/cool.io/io.rb:191:in `on_readable'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.2.4/lib/cool.io/loop.rb:88:in `run_once'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.2.4/lib/cool.io/loop.rb:88:in `run'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/plugin/in_forward.rb:93:in `run'
2015-02-19 15:29:42 +0000 [warn]: emit transaction failed error_class=Encoding::CompatibilityError error=#<Encoding::CompatibilityError: incompatible character encodings: ASCII-8BIT and UTF-8>
2015-02-19 15:29:42 +0000 [warn]: suppressed same stacktrace
2015-02-19 15:29:42 +0000 [warn]: record_reformer: Encoding::CompatibilityError incompatible character encodings: ASCII-8BIT and UTF-8 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/event.rb:33:in `<<'
2015-02-19 15:29:42 +0000 fluent.warn: {"error_class":"Encoding::CompatibilityError","error":"#<Encoding::CompatibilityError: incompatible character encodings: ASCII-8BIT and UTF-8>","message":"emit transaction failed error_class=Encoding::CompatibilityError error=#<Encoding::CompatibilityError: incompatible character encodings: ASCII-8BIT and UTF-8>"}
2015-02-19 15:29:42 +0000 fluent.warn: {"error_class":"Encoding::CompatibilityError","error":"#<Encoding::CompatibilityError: incompatible character encodings: ASCII-8BIT and UTF-8>","message":"emit transaction failed error_class=Encoding::CompatibilityError error=#<Encoding::CompatibilityError: incompatible character encodings: ASCII-8BIT and UTF-8>"}
2015-02-19 15:29:42 +0000 fluent.warn: {"message":"record_reformer: Encoding::CompatibilityError incompatible character encodings: ASCII-8BIT and UTF-8 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/event.rb:33:in `<<'"}
2015-02-12 21:11:35 +0000 [info]: reading config file path="/etc/td-agent/mux.conf"
2015-02-12 21:11:35 +0000 [info]: starting fluentd-0.10.58
2015-02-12 21:11:35 +0000 [info]: gem 'fluent-mixin-config-placeholders' version '0.3.0'
2015-02-12 21:11:35 +0000 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2015-02-12 21:11:35 +0000 [info]: gem 'fluent-mixin-rewrite-tag-name' version '0.1.0'
2015-02-12 21:11:35 +0000 [info]: gem 'fluent-plugin-cadvisor' version '0.2.5'
2015-02-12 21:11:35 +0000 [info]: gem 'fluent-plugin-forest' version '0.3.0'
2015-02-12 21:11:35 +0000 [info]: gem 'fluent-plugin-graphite' version '0.0.5'
2015-02-12 21:11:35 +0000 [info]: gem 'fluent-plugin-mongo' version '0.7.4'
2015-02-12 21:11:35 +0000 [info]: gem 'fluent-plugin-multiprocess' version '0.1.1'
2015-02-12 21:11:35 +0000 [info]: gem 'fluent-plugin-numeric-monitor' version '0.1.7'
2015-02-12 21:11:35 +0000 [info]: gem 'fluent-plugin-record-reformer' version '0.4.0'
2015-02-12 21:11:35 +0000 [info]: gem 'fluent-plugin-rewrite' version '0.0.12'
2015-02-12 21:11:35 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.4.1'
2015-02-12 21:11:35 +0000 [info]: gem 'fluent-plugin-s3' version '0.5.1'
2015-02-12 21:11:35 +0000 [info]: gem 'fluent-plugin-scribe' version '0.10.14'
2015-02-12 21:11:35 +0000 [info]: gem 'fluent-plugin-td' version '0.10.23'
2015-02-12 21:11:35 +0000 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.0'
2015-02-12 21:11:35 +0000 [info]: gem 'fluent-plugin-webhdfs' version '0.4.1'
2015-02-12 21:11:35 +0000 [info]: gem 'fluentd' version '0.10.58'
2015-02-12 21:11:35 +0000 [info]: using configuration file: <ROOT>
<source>
type http
port 8999
</source>
<source>
type forward
port 24224
bind 0.0.0.0
</source>
<source>
type monitor_agent
bind 0.0.0.0
port 24219
</source>
<match post.*.*.*>
type record_reformer
renew_record false
enable_ruby true
service ${tag_parts[1]}.${tag_parts[2]}.${tag_parts[3]}
service_hostname #{ENV['IP']}
tag cluster.${tag_parts[1]}.${tag_parts[2]}.${tag_parts[3]}
datestamp ${time.strftime('%Y-%m-%dT%H:%M:%S.%LZ')}
</match>
<match cluster.*.*.*>
type record_reformer
renew_record false
enable_ruby true
tag cluster_vars.${tag_parts[1]}.${tag_parts[2]}.${tag_parts[3]}
cluster_ip #{ENV['IP']}
</match>
<match cluster_vars.*.*.*>
type rewrite
remove_prefix cluster_vars
add_prefix cluster_stats
</match>
<match cluster_stats.*.*.*>
type copy
<store>
type record_reformer
enable_ruby true
tag upload.${service}
</store>
<store>
type record_reformer
tag monitor_tag.${service}
count 1
</store>
</match>
<match monitor_tag.*.*.*>
type record_reformer
renew_record true
enable_ruby true
tag monitor_second.${tag_parts[1]}.${tag_parts[2]}.${tag_parts[3]}
count 1
</match>
<match monitor_second.*.*.*>
type numeric_monitor
monitor_key count
count_interval 1
aggregate tag
input_tag_remove_prefix monitor_second
output_per_tag yes
tag_prefix monitor_second_filter
</match>
<match monitor_second_filter.*.*.*>
type record_reformer
renew_record true
enable_ruby true
count ${num}
tag monitor_second_filter2.${tag_parts[1]}.${tag_parts[2]}.${tag_parts[3]}
</match>
<match monitor_second_filter2.*.*.*>
type rewrite
remove_prefix monitor_second_filter2
add_prefix monitor_minute
<rule>
key count
pattern ^0$
ignore
</rule>
</match>
<match monitor_minute.*.*.*>
type numeric_monitor
monitor_key count
unit minute
aggregate tag
input_tag_remove_prefix monitor_minute
percentiles 95,99
output_per_tag yes
tag_prefix monitor
</match>
<match monitor.*.*.*>
type record_reformer
renew_record false
enable_ruby true
datestamp ${time.strftime('%Y-%m-%dT%H:%M:%S.%LZ')}
timestamp ${time.strftime('%Y-%m-%d %H:%M:%S')}
monitor_service ${tag_parts[1]}.${tag_parts[2]}.${tag_parts[3]}
tag monitor_stats.tiger.tracks.monitorstats
monitor_hostname #{ENV['IP']}
monitor_num ${num}
service tiger.tracks.monitorstats
</match>
<match monitor_stats.*.*.*>
type rewrite
remove_prefix monitor_stats
add_prefix monitor_out
<rule>
key monitor_service
pattern ^((?!stats).)*$
ignore
</rule>
<rule>
key monitor_hostname
pattern ([^.]+)\..*
replace \\1
</rule>
<rule>
key monitor_num
pattern ^0$
ignore
</rule>
</match>
<match monitor_out.*.*.*>
type copy
<store>
type rewrite
remove_prefix monitor_out
add_prefix upload
</store>
<store>
type record_reformer
enable_ruby true
tag tracks.${monitor_service}
</store>
</match>
<match tracks.*.*.*>
type graphite
host 10.0.0.0
port 2003
tag_for prefix
name_keys num,min,max,avg,sum,p90,p95
</match>
<match upload.{PROD,Prod,ip}.*.*>
type record_reformer
tag upload.prod.${tag_parts[2]}.${tag_parts[3]}
</match>
<match upload.QA.*.*>
type record_reformer
tag upload.qa.${tag_parts[2]}.${tag_parts[3]}
</match>
<match upload.*.*.*>
type rewrite
remove_prefix upload
add_prefix hdfs
</match>
<match hdfs.prod.apps.actions>
type forward
<server>
name hdfs_actions
host 127.0.0.1
port 24225
</server>
</match>
<match hdfs.prod.apps.zendmodo_http>
type forward
<server>
name hdfs_zendmodo_http
host 127.0.0.1
port 24226
</server>
</match>
<match hdfs.prod.apps.zendmodo_other>
type forward
<server>
name hdfs_zendmodo_other
host 127.0.0.1
port 24227
</server>
</match>
<match hdfs.*.*.*>
type forward
<server>
name hdfs_forest
host 127.0.0.1
port 24228
</server>
</match>
<match cluster.**>
type numeric_monitor
monitor_key count
unit minute
aggregate tag
input_tag_remove_prefix cluster_stats
percentiles 90,95
output_per_tag yes
tag_prefix debug
</match>
<match debug.**>
type stdout
</match>
<match **>
type stdout
</match>
</ROOT>
2015-02-12 21:11:35 +0000 [info]: adding source type="http"
2015-02-12 21:11:35 +0000 [info]: adding source type="forward"
2015-02-12 21:11:35 +0000 [info]: adding source type="monitor_agent"
2015-02-12 21:11:35 +0000 [info]: adding match pattern="post.*.*.*" type="record_reformer"
2015-02-12 21:11:35 +0000 [info]: adding match pattern="cluster.*.*.*" type="record_reformer"
2015-02-12 21:11:35 +0000 [info]: adding match pattern="cluster_vars.*.*.*" type="rewrite"
2015-02-12 21:11:35 +0000 [info]: adding match pattern="cluster_stats.*.*.*" type="copy"
2015-02-12 21:11:35 +0000 [info]: adding match pattern="monitor_tag.*.*.*" type="record_reformer"
2015-02-12 21:11:35 +0000 [info]: adding match pattern="monitor_second.*.*.*" type="numeric_monitor"
2015-02-12 21:11:35 +0000 [info]: adding match pattern="monitor_second_filter.*.*.*" type="record_reformer"
2015-02-12 21:11:35 +0000 [info]: adding match pattern="monitor_second_filter2.*.*.*" type="rewrite"
2015-02-12 21:11:35 +0000 [info]: adding match pattern="monitor_minute.*.*.*" type="numeric_monitor"
2015-02-12 21:11:35 +0000 [info]: adding match pattern="monitor.*.*.*" type="record_reformer"
2015-02-12 21:11:35 +0000 [info]: adding match pattern="monitor_stats.*.*.*" type="rewrite"
2015-02-12 21:11:35 +0000 [info]: adding match pattern="monitor_out.*.*.*" type="copy"
2015-02-12 21:11:35 +0000 [info]: adding match pattern="tracks.*.*.*" type="graphite"
2015-02-12 21:11:35 +0000 [info]: adding match pattern="upload.{PROD,Prod,ip}.*.*" type="record_reformer"
2015-02-12 21:11:35 +0000 [info]: adding match pattern="upload.QA.*.*" type="record_reformer"
2015-02-12 21:11:35 +0000 [info]: adding match pattern="upload.*.*.*" type="rewrite"
2015-02-12 21:11:35 +0000 [info]: adding match pattern="hdfs.prod.apps.actions" type="forward"
2015-02-12 21:11:35 +0000 [info]: adding forwarding server 'hdfs_actions' host="127.0.0.1" port=24225 weight=60 plugin_id="object:3f9c40f55318"
2015-02-12 21:11:35 +0000 [info]: adding match pattern="hdfs.prod.apps.zendmodo_http" type="forward"
2015-02-12 21:11:35 +0000 [info]: adding forwarding server 'hdfs_zendmodo_http' host="127.0.0.1" port=24226 weight=60 plugin_id="object:3f9c40f4a3c8"
2015-02-12 21:11:35 +0000 [info]: adding match pattern="hdfs.prod.apps.zendmodo_other" type="forward"
2015-02-12 21:11:35 +0000 [info]: adding forwarding server 'hdfs_zendmodo_other' host="127.0.0.1" port=24227 weight=60 plugin_id="object:3f9c40f469d0"
2015-02-12 21:11:35 +0000 [info]: adding match pattern="hdfs.*.*.*" type="forward"
2015-02-12 21:11:35 +0000 [info]: adding forwarding server 'hdfs_forest' host="127.0.0.1" port=24228 weight=60 plugin_id="object:3f9c40f40f94"
2015-02-12 21:11:35 +0000 [info]: adding match pattern="cluster.**" type="numeric_monitor"
2015-02-12 21:11:35 +0000 [info]: adding match pattern="debug.**" type="stdout"
2015-02-12 21:11:35 +0000 [info]: adding match pattern="**" type="stdout"
2015-02-12 21:11:35 +0000 [info]: listening fluent socket on 0.0.0.0:24224
2015-02-19 15:29:42 +0000 [warn]: emit transaction failed error_class=Encoding::CompatibilityError error=#<Encoding::CompatibilityError: incompatible character encodings: ASCII-8BIT and UTF-8>
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/event.rb:33:in `<<'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/event.rb:33:in `to_msgpack'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/event.rb:33:in `block in to_msgpack_stream'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/event.rb:55:in `call'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/event.rb:55:in `each'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/event.rb:32:in `to_msgpack_stream'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/output.rb:429:in `emit'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/match.rb:36:in `emit'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/engine.rb:160:in `emit_stream'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/engine.rb:140:in `emit'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-rewrite-0.0.12/lib/fluent/plugin/out_rewrite.rb:56:in `block in emit'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/event.rb:55:in `call'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/event.rb:55:in `each'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-rewrite-0.0.12/lib/fluent/plugin/out_rewrite.rb:53:in `emit'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/match.rb:36:in `emit'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/engine.rb:160:in `emit_stream'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/engine.rb:140:in `emit'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-record-reformer-0.4.0/lib/fluent/plugin/out_record_reformer.rb:90:in `block in emit'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/event.rb:55:in `call'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/event.rb:55:in `each'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-record-reformer-0.4.0/lib/fluent/plugin/out_record_reformer.rb:87:in `emit'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/output.rb:33:in `next'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/plugin/out_copy.rb:73:in `emit'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/match.rb:36:in `emit'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/engine.rb:160:in `emit_stream'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/engine.rb:140:in `emit'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-rewrite-0.0.12/lib/fluent/plugin/out_rewrite.rb:56:in `block in emit'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/event.rb:55:in `call'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/event.rb:55:in `each'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-rewrite-0.0.12/lib/fluent/plugin/out_rewrite.rb:53:in `emit'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/match.rb:36:in `emit'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/engine.rb:160:in `emit_stream'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/engine.rb:140:in `emit'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-record-reformer-0.4.0/lib/fluent/plugin/out_record_reformer.rb:90:in `block in emit'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/event.rb:149:in `feed_each'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/event.rb:149:in `each'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-record-reformer-0.4.0/lib/fluent/plugin/out_record_reformer.rb:87:in `emit'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/match.rb:36:in `emit'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/engine.rb:160:in `emit_stream'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/plugin/in_forward.rb:148:in `on_message'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/plugin/in_forward.rb:235:in `call'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/plugin/in_forward.rb:235:in `block in on_read_msgpack'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/plugin/in_forward.rb:234:in `feed_each'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/plugin/in_forward.rb:234:in `on_read_msgpack'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.2.4/lib/cool.io/io.rb:128:in `on_readable'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.2.4/lib/cool.io/io.rb:191:in `on_readable'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.2.4/lib/cool.io/loop.rb:88:in `run_once'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.2.4/lib/cool.io/loop.rb:88:in `run'
2015-02-19 15:29:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/plugin/in_forward.rb:93:in `run'
2015-02-19 15:29:42 +0000 [warn]: emit transaction failed error_class=Encoding::CompatibilityError error=#<Encoding::CompatibilityError: incompatible character encodings: ASCII-8BIT and UTF-8>
2015-02-19 15:29:42 +0000 [warn]: suppressed same stacktrace
2015-02-19 15:29:42 +0000 [warn]: record_reformer: Encoding::CompatibilityError incompatible character encodings: ASCII-8BIT and UTF-8 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/event.rb:33:in `<<'
2015-02-19 15:29:42 +0000 fluent.warn: {"error_class":"Encoding::CompatibilityError","error":"#<Encoding::CompatibilityError: incompatible character encodings: ASCII-8BIT and UTF-8>","message":"emit transaction failed error_class=Encoding::CompatibilityError error=#<Encoding::CompatibilityError: incompatible character encodings: ASCII-8BIT and UTF-8>"}
2015-02-19 15:29:42 +0000 fluent.warn: {"error_class":"Encoding::CompatibilityError","error":"#<Encoding::CompatibilityError: incompatible character encodings: ASCII-8BIT and UTF-8>","message":"emit transaction failed error_class=Encoding::CompatibilityError error=#<Encoding::CompatibilityError: incompatible character encodings: ASCII-8BIT and UTF-8>"}
2015-02-19 15:29:42 +0000 fluent.warn: {"message":"record_reformer: Encoding::CompatibilityError incompatible character encodings: ASCII-8BIT and UTF-8 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/event.rb:33:in `<<'"}
Mr. Fiber
Feb 25, 2015, 3:06:43 AM2/25/15
to flu...@googlegroups.com
Your td-agent 2 seems 2.1.3 but latest td-agent 2 is 2.1.4.
Could you upgrade td-agent2?
td-agent 2.1.4 uses msgpack 0.5.11 and it may resolve this problem.
Masahiro
--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Lance N.
Feb 25, 2015, 9:24:28 PM2/25/15
to flu...@googlegroups.com
Thanks! I'll try that.
Reply all
Reply to author
Forward
0 new messages