Temporarily failed to flush the buffer

[no jihun]

Hi all,

I'm facing an problem with elasticsearch plugin.

buffer files.

-rw-r--r-- 1 irteam irteam 33554123 2015-05-31 20:24 offerwall..q5175e07d725f0a86.buffer

-rw-r--r-- 1 irteam irteam 33554128 2015-05-31 21:29 offerwall..q5175ef49e2970a7f.buffer

-rw-r--r-- 1 irteam irteam 33554141 2015-05-31 22:31 offerwall..q5175fdb307dbb699.buffer

-rw-r--r-- 1 irteam irteam 33554305 2015-05-31 23:34 offerwall..q51760bbbe59715f7.buffer

-rw-r--r-- 1 irteam irteam 33554269 2015-06-01 00:37 offerwall..q517619ae160a6046.buffer

-rw-r--r-- 1 irteam irteam 33554402 2015-06-01 01:44 offerwall..q517627dba39f80aa.buffer

-rw-r--r-- 1 irteam irteam 33554219 2015-06-01 02:54 offerwall..q517636b38c397efc.buffer

-rw-r--r-- 1 irteam irteam 33554326 2015-06-01 04:07 offerwall..q5176465708eb88a9.buffer

-rw-r--r-- 1 irteam irteam 33554374 2015-06-01 05:22 offerwall..q517656a615a1bf2d.buffer

-rw-r--r-- 1 irteam irteam 33554324 2015-06-01 06:37 offerwall..q51766772580109b9.buffer

-rw-r--r-- 1 irteam irteam 33554124 2015-06-01 07:50 offerwall..q5176785db5ed7faa.buffer

-rw-r--r-- 1 irteam irteam 33554329 2015-06-01 09:06 offerwall..q51768897a733ef12.buffer

-rw-r--r-- 1 irteam irteam 33554316 2015-06-01 10:25 offerwall..q51769989731b88bb.buffer

-rw-r--r-- 1 irteam irteam 33554424 2015-06-01 11:42 offerwall..q5176ab304ac6ae4a.buffer

fluent consol log.. It says 

2015-06-01 11:34:45 +0900 [warn]: fluent/output.rb:354:rescue in try_flush: temporarily failed to flush the buffer. next_retry=2015-06-01 11:36:15 +0900 error_class="TypeError" error="no implicit conversion of Array into String" plugin_id="object:3fc32c8d8b94"

  2015-06-01 11:34:45 +0900 [warn]: suppressed same stacktrace

2015-06-01 11:36:15 +0900 [warn]: fluent/output.rb:354:rescue in try_flush: temporarily failed to flush the buffer. next_retry=2015-06-01 11:38:46 +0900 error_class="TypeError" error="no implicit conversion of Array into String" plugin_id="object:3fc32c8d8b94"

  2015-06-01 11:36:15 +0900 [warn]: suppressed same stacktrace

2015-06-01 11:38:46 +0900 [warn]: fluent/output.rb:354:rescue in try_flush: temporarily failed to flush the buffer. next_retry=2015-06-01 11:44:34 +0900 error_class="TypeError" error="no implicit conversion of Array into String" plugin_id="object:3fc32c8d8b94"

  2015-06-01 11:38:46 +0900 [warn]: suppressed same stacktrace

2015-06-01 11:44:34 +0900 [warn]: fluent/output.rb:354:rescue in try_flush: temporarily failed to flush the buffer. next_retry=2015-06-01 11:56:30 +0900 error_class="TypeError" error="no implicit conversion of Array into String" plugin_id="object:3fc32c8d8b94"

  2015-06-01 11:44:34 +0900 [warn]: suppressed same stacktrace

2015-06-01 11:56:30 +0900 [warn]: fluent/output.rb:354:rescue in try_flush: temporarily failed to flush the buffer. next_retry=2015-06-01 12:17:41 +0900 error_class="TypeError" error="no implicit conversion of Array into String" plugin_id="object:3fc32c8d8b94"

and buffer files getting larger. not flushed.

At this situation I have some questions.

1. which side problem this could be. elasticsearch? or fluent?

2. buffer file chunck formatted.. any way to debugg buffer files?

[Mr. Fiber]

> 1. which side problem this could be. elasticsearch? or fluent?

Elasticsearch. Maybe your record has array type in one key but

elasticsearch plugin requires string type.

> 2. buffer file chunck formatted.. any way to debugg buffer files?

What does 'debug' mean? Do you want to touch buffer files directly?

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[no jihun]

thanks Masahiro.

i meant can I figure out which log in buffer makes problemon.

when some buffered log failed to flush,
following logs within the same buffer file handled saparately?

I have tried append new well structured log json to fluent tail target file.
but it didnt flushed to elasticsearch too.

i am sorry to my bad english.

[no jihun]

More information.

I don't know this related with the problem. but,

as you can see about 05/26 05:38 machine's disk fulled.

After noticed the disk full, we removed some files to secure disk space.

   this is the exact time when the buffer stopped flushing.

-rw-r--r-- 1 irteam irteam      12224 2015-05-26 05:38 offerwall..q516edfe2dc46d911.buffer

-rw-r--r-- 1 irteam irteam 33554533 2015-05-26 22:37 offerwall..q516edfe5b6f84bee.buffer

-rw-r--r-- 1 irteam irteam 33554368 2015-05-27 00:26 offerwall..q516fc3bb48a424f6.buffer

-rw-r--r-- 1 irteam irteam 33554377 2015-05-27 02:19 offerwall..q516fdbfce39ea8cd.buffer

-rw-r--r-- 1 irteam irteam 33554414 2015-05-27 04:17 offerwall..q516ff54247649cbb.buffer

-rw-r--r-- 1 irteam irteam 33554430 2015-05-27 06:18 offerwall..q51700fa79c4ba1f7.buffer

2015-05-20 16:08:41 +0900 [info]: fluent/agent.rb:123:add_match: adding match pattern="offerwall.log" type="elasticsearch"

2015-05-20 16:08:41 +0900 [trace]: fluent/plugin.rb:98:register_impl: registered output plugin 'elasticsearch'

2015-05-20 16:08:41 +0900 [info]: fluent/root_agent.rb:142:add_source: adding source type="monitor_agent"

2015-05-20 16:08:41 +0900 [info]: fluent/root_agent.rb:142:add_source: adding source type="tail"

2015-05-20 16:08:41 +0900 [debug]: plugin/in_monitor_agent.rb:203:start: listening monitoring http server on http://0.0.0.0:24220/api/plugins

2015-05-20 16:08:45 +0900 [info]: plugin/out_elasticsearch.rb:67:client: Connection opened to Elasticsearch cluster => {:host=>"1.2.1.7", :port=>9200, :scheme=>"http"}, {:host=>"1.9.8.9", :port=>9200, :scheme=>"http"}, {:host=>"1.9.1.10", :port=>9200, :scheme=>"http"}, {:host=>"1.9.8.11", :port=>9200, :scheme=>"http"}

2015-05-26 05:38:30 +0900 [warn]: fluent/root_agent.rb:189:handle_emits_error: emit transaction failed: error_class=Errno::ENOSPC error="No space left on device @ io_write - /home1/user/.fluentd/buffer/offerwall..b516edfdffd6f6038.buffer" tag="offerwall.log"

  2015-05-26 05:38:31 +0900 [warn]: plugin/in_tail.rb:229:receive_lines: /home1/user/apps/ruby/gems/gems/fluentd-0.12.8/lib/fluent/plugin/buf_file.rb:32:in `write'

  2015-05-26 05:38:31 +0900 [warn]: plugin/in_tail.rb:229:receive_lines: /home1/user/apps/ruby/gems/gems/fluentd-0.12.8/lib/fluent/plugin/buf_file.rb:32:in `<<'

  2015-05-26 05:38:31 +0900 [warn]: plugin/in_tail.rb:229:receive_lines: /home1/user/apps/ruby/gems/gems/fluentd-0.12.8/lib/fluent/buffer.rb:185:in `block in emit'

  2015-05-26 05:38:31 +0900 [warn]: plugin/in_tail.rb:229:receive_lines: /home1/user/apps/ruby-2.2.2/lib/ruby/2.2.0/monitor.rb:211:in `mon_synchronize'

  2015-05-26 05:38:31 +0900 [warn]: plugin/in_tail.rb:229:receive_lines: /home1/user/apps/ruby/gems/gems/fluentd-0.12.8/lib/fluent/buffer.rb:179:in `emit'

  2015-05-26 05:38:31 +0900 [warn]: plugin/in_tail.rb:229:receive_lines: /home1/user/apps/ruby/gems/gems/fluentd-0.12.8/lib/fluent/output.rb:251:in `emit'

  2015-05-26 05:38:31 +0900 [warn]: plugin/in_tail.rb:229:receive_lines: /home1/user/apps/ruby/gems/gems/fluentd-0.12.8/lib/fluent/event_router.rb:88:in `emit_stream'

  2015-05-26 05:38:31 +0900 [warn]: plugin/in_tail.rb:229:receive_lines: /home1/user/apps/ruby/gems/gems/fluentd-0.12.8/lib/fluent/plugin/in_tail.rb:229:in `receive_lines'

  2015-05-26 05:38:31 +0900 [warn]: plugin/in_tail.rb:229:receive_lines: /home1/user/apps/ruby/gems/gems/fluentd-0.12.8/lib/fluent/plugin/in_tail.rb:320:in `call'

  2015-05-26 05:38:31 +0900 [warn]: plugin/in_tail.rb:229:receive_lines: /home1/user/apps/ruby/gems/gems/fluentd-0.12.8/lib/fluent/plugin/in_tail.rb:320:in `wrap_receive_lines'

  2015-05-26 05:38:31 +0900 [warn]: plugin/in_tail.rb:229:receive_lines: /home1/user/apps/ruby/gems/gems/fluentd-0.12.8/lib/fluent/plugin/in_tail.rb:513:in `call'

  2015-05-26 05:38:31 +0900 [warn]: plugin/in_tail.rb:229:receive_lines: /home1/user/apps/ruby/gems/gems/fluentd-0.12.8/lib/fluent/plugin/in_tail.rb:513:in `on_notify'

  2015-05-26 05:38:31 +0900 [warn]: plugin/in_tail.rb:229:receive_lines: /home1/user/apps/ruby/gems/gems/fluentd-0.12.8/lib/fluent/plugin/in_tail.rb:345:in `on_notify'

  2015-05-26 05:38:31 +0900 [warn]: plugin/in_tail.rb:229:receive_lines: /home1/user/apps/ruby/gems/gems/fluentd-0.12.8/lib/fluent/plugin/in_tail.rb:446:in `call'

  2015-05-26 05:38:31 +0900 [warn]: plugin/in_tail.rb:229:receive_lines: /home1/user/apps/ruby/gems/gems/fluentd-0.12.8/lib/fluent/plugin/in_tail.rb:446:in `on_change'

  2015-05-26 05:38:31 +0900 [warn]: plugin/in_tail.rb:229:receive_lines: /home1/user/apps/ruby/gems/gems/cool.io-1.3.0/lib/cool.io/loop.rb:88:in `run_once'

  2015-05-26 05:38:31 +0900 [warn]: plugin/in_tail.rb:229:receive_lines: /home1/user/apps/ruby/gems/gems/cool.io-1.3.0/lib/cool.io/loop.rb:88:in `run'

  2015-05-26 05:38:31 +0900 [warn]: plugin/in_tail.rb:229:receive_lines: /home1/user/apps/ruby/gems/gems/fluentd-0.12.8/lib/fluent/plugin/in_tail.rb:214:in `run'

2015-05-26 05:38:31 +0900 [warn]: fluent/root_agent.rb:189:handle_emits_error: emit transaction failed: error_class=Errno::ENOSPC error="No space left on device @ io_write - /home1/user/.fluentd/buffer/offerwall..b516edfe2dc46d911.buffer" tag="offerwall.log"

  2015-05-26 05:38:31 +0900 [warn]: plugin/in_tail.rb:229:receive_lines: suppressed same stacktrace

2015-05-26 05:38:31 +0900 [warn]: fluent/root_agent.rb:189:handle_emits_error: emit transaction failed: error_class=Errno::ENOSPC error="No space left on device @ io_write - /home1/user/.fluentd/buffer/offerwall..b516edfe2dc46d911.buffer" tag="offerwall.log"

  2015-05-26 05:38:31 +0900 [warn]: plugin/in_tail.rb:229:receive_lines: suppressed same stacktrace

2015-05-26 05:38:31 +0900 [warn]: fluent/root_agent.rb:189:handle_emits_error: emit transaction failed: error_class=Errno::ENOSPC error="No space left on device @ io_write - /home1/user/.fluentd/buffer/offerwall..b516edfe2dc46d911.buffer" tag="offerwall.log"

  2015-05-26 05:38:31 +0900 [warn]: plugin/in_tail.rb:229:receive_lines: suppressed same stacktrace

2015-05-26 05:38:31 +0900 [warn]: fluent/root_agent.rb:189:handle_emits_error: emit transaction failed: error_class=Errno::ENOSPC error="No space left on device @ io_write - /home1/user/.fluentd/buffer/offerwall..b516edfe2dc46d911.buffer" tag="offerwall.log"

2015년 6월 1일 월요일 오후 1시 3분 2초 UTC+9, no jihun 님의 말:

[no jihun]

even though fluentd process restared buffers flushed not at all but still getting larger.

I think there is buffer flush failing scenario when after storage fulled.

because there are 6 fluentd machines but this buffer not flushing problem accurred only on storage fulled 2 machines.
and there is no array types in log.

is there any solution for this situation?

i can provide more information if you need.

thankyou.

[Mr. Fiber]

I'm now joining the event.
I will check your reply deeply.
Please wait...

[Mr. Fiber]

> is there any solution for this situation?

In full storage situation, fluentd with file buffer doesn't work correctly.

In this case, fluentd users use --without-source option to stop incoming events.

After flushed all buffers, restart td-agent with normal setting again.

If you hit disk-full problem by buffer files,

your output destination lacks power.

Masahiro