fluent 0.14 gelf plugin UndefinedConversionError error="\"\\xC3\" from ASCII-8BIT to UTF-8"

[Guido Lova]

Hi,

Updating fluent 0.12 to 0.14 (0.14.14) i receive this error using gelf plugin (graylog) and i lose many log lines consequently.


2017-04-11 12:22:29 +0200 [warn]: failed to flush the buffer. retry_time=0 next_retry_seconds=2017-04-11 12:22:30 +0200 chunk="54ce1775e1842fe85f88a3285cf45b2c" error_class=Encoding::UndefinedConversionError error="\"\\xC3\" from ASCII-8BIT to UTF-8"

2017-04-11 12:22:29 +0200 [warn]: /opt/fluentd/ruby/lib/ruby/gems/2.4.0/gems/json-2.0.3/lib/json/pure/generator.rb:398:in `encode'
  2017-04-11 12:22:29 +0200 [warn]: /opt/fluentd/ruby/lib/ruby/gems/2.4.0/gems/json-2.0.3/lib/json/pure/generator.rb:398:in `to_json'
  2017-04-11 12:22:29 +0200 [warn]: /opt/fluentd/ruby/lib/ruby/gems/2.4.0/gems/json-2.0.3/lib/json/pure/generator.rb:309:in `block in json_transform'
  2017-04-11 12:22:29 +0200 [warn]: /opt/fluentd/ruby/lib/ruby/gems/2.4.0/gems/json-2.0.3/lib/json/pure/generator.rb:301:in `each'
  2017-04-11 12:22:29 +0200 [warn]: /opt/fluentd/ruby/lib/ruby/gems/2.4.0/gems/json-2.0.3/lib/json/pure/generator.rb:301:in `json_transform'
  2017-04-11 12:22:29 +0200 [warn]: /opt/fluentd/ruby/lib/ruby/gems/2.4.0/gems/json-2.0.3/lib/json/pure/generator.rb:283:in `to_json'
  2017-04-11 12:22:29 +0200 [warn]: /opt/fluentd/ruby/lib/ruby/gems/2.4.0/gems/gelf-3.0.0/lib/gelf/notifier.rb:160:in `notify_with_level!'
  2017-04-11 12:22:29 +0200 [warn]: /opt/fluentd/ruby/lib/ruby/gems/2.4.0/gems/gelf-3.0.0/lib/gelf/notifier.rb:119:in `notify!'
  2017-04-11 12:22:29 +0200 [warn]: /etc/fluent/plugin/out_gelf.rb:120:in `block in write'
  2017-04-11 12:22:29 +0200 [warn]: /opt/fluentd/ruby/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/event.rb:323:in `each'
  2017-04-11 12:22:29 +0200 [warn]: /opt/fluentd/ruby/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/event.rb:323:in `block in each'
  2017-04-11 12:22:29 +0200 [warn]: /opt/fluentd/ruby/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/buffer/memory_chunk.rb:80:in `open'
  2017-04-11 12:22:29 +0200 [warn]: /opt/fluentd/ruby/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/buffer/memory_chunk.rb:80:in `open'
  2017-04-11 12:22:29 +0200 [warn]: /opt/fluentd/ruby/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/event.rb:322:in `each'
  2017-04-11 12:22:29 +0200 [warn]: /etc/fluent/plugin/out_gelf.rb:119:in `write'
  2017-04-11 12:22:29 +0200 [warn]: /opt/fluentd/ruby/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/compat/output.rb:129:in `write'
  2017-04-11 12:22:29 +0200 [warn]: /opt/fluentd/ruby/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/output.rb:1009:in `try_flush'
  2017-04-11 12:22:29 +0200 [warn]: /opt/fluentd/ruby/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/output.rb:1234:in `flush_thread_run'
  2017-04-11 12:22:29 +0200 [warn]: /opt/fluentd/ruby/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/output.rb:398:in `block (2 levels) in start'
  2017-04-11 12:22:29 +0200 [warn]: /opt/fluentd/ruby/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin_helper/thread.rb:66:in `block in thread_create'



Anyone reported this issue ?

thank you

[Mr. Fiber]

That's weird.

Could you show me the actual event to cause this error?

I want to reproduce same problem on my environment.

Masahiro

Listed on SIX Swiss Exchange

Our websites: lastminute.com | bravofly.com | volagratis.com | rumbo.es | jetcost.com | group.lastminute.com

This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden and could be a crime.

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Guido Lova]

I've debugged a bit more reducing the perimeter to the line kind of the particular log that causes this error so i post a reduced config and the log line:

fluent.conf:

<system>
  # equal to -qq option
  log_level debug
  emit_error_log_interval 30
</system>

<source>
  @type tail
  path_key filepath
  path /etc/fluent/appfw/logs/application-*.log
  pos_file /tmp/files_application.pos
  read_from_head true
  tag infile3.app.*
  format multiline
  format_firstline /\d{4}-\d{2}-\d{2}T\d{2}/
  format1 /^(?<message>(?<time>[^ ]*) \| (?<loglevel>[^ ]*) \|.+)/
  time_format %FT%T.%N%z
</source>

<filter infile3.**>
  @type record_transformer
  enable_ruby
  <record>
    appname ${hostname.gsub(/-?[0-9]+$/,'')}
    filetype ${tag_parts[1]}
    myhostname zumpappa2
  </record>
</filter>

<filter infile3.**>
  @type record_transformer
  <record>
    builttag file.${record["appname"]}.${hostname}.${record["filetype"]}.UNUSED
  </record>
</filter>

<match infile3.**>
  @type copy
  <store>
    @type rewrite_tag_filter
    rewriterule1 builttag ^(.+)$ $1
    log_level debug
  </store>
  <store>
    @type rewrite_tag_filter
    rewriterule1 metricstag ^(.+)$ $1
    log_level debug
  </store>
  <store>
    @type rewrite_tag_filter
    rewriterule1 metricstagdetail ^(.+)$ $1
    log_level debug
  </store>
</match>

@include /etc/fluent/forward.conf

forward.conf:

<match file.**>
   @type copy
  #@type forward
  <store>
    type gelf
    host xxxxxxxxxxxxxxxxx
    protocol tcp
    port xxxxx
    flush_interval 12s
    buffer_queue_full_action drop_oldest_chunk
    retry_wait 2
    max_retry_wait 120
    retry_limit 4
    disable_retry_limit false
    flush_at_shutdown false
    buffer_queue_limit 40
    buffer_chunk_limit 64m
  </store>
  <store>
    @type stdout
  </store>
</match>
#

logline:
017-04-12T12:15:22.959+0200 | DEBUG | 9a866acd-7e61-4ff9-b23e-f6c4508a794a | 0n1qg4k | - | - | - | com.xxxxxxx.xxxxxxx.xxxxxxx.xxxxxx.xxxxxxxx.CatalogPaymentMethodRepository.getAllBy | 846BA3F4165C6E571028BB942F01831155C0DC10F2187E44E86009F38CF9044D [description=xxxx, code=01, fees=0, isPreselected=false, description=Acard, code=02, fees=0, isPreselected=false, description=sdkjf Ajkjlk, code=03, fees=0, isPreselected=false, description=AKLJlkj Alkjlkjlkj, code=05, fees=0, isPreselected=false, description=Adkldlkd Aldfkljfdkj, code=06, fees=0, isPreselected=false, description=Gkjljkl Hjlksjdlkj, code=09, fees=-30, isPreselected=true, description=Madskljd, code=10, fees=0, isPreselected=false, description=Mast Jlkjljk, code=11, fees=0, isPreselected=false, description=Llldllldl, code=53, fees=0, isPreselected=false, description=Okjhdskjhsd, code=95, fees=0, isPreselected=false]

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.

[Guido Lova]

Hi Masahiro,

have you tried to reproduce the error ?