Re: Failed to flash buffer, exceed maximum insert size

[Masahiro Nakagawa]

Hi Sylvain,

The cause of this issue is buffer_chunk_limit mismatch(*1).

Maybe, out_forward plugin sends large chunk to destination fluentd node.

If you want to use out_mongo in fluentd stream,

you should set "buffer_chunk_limit 10m" in forward plugins.

In japan, Hatena uses this approach to avoid this issue(*2).

*1: https://github.com/fluent/fluentd/blob/master/lib/fluent/buffer.rb#L183

*2: http://blog.stanaka.org/entry/2013/02/22/171053 (in japanese)

P.S.

I think MongoDB's document size limitation is useless feature... 

Masahiro

On Fri, May 17, 2013 at 7:02 AM, Sylvain <syl...@kalache.fr> wrote:

Hello folks!

I'm facing this issue:

2013-05-16 21:55:02 +0000: failed to flush the buffer, retrying. error="Exceded maximum insert size of 16,000,000 bytes" instance=75425090 time=1368741302

  2013-05-16 21:55:02 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/mongo-1.6.4/lib/mongo/collection.rb:975:in `insert_documents'

  2013-05-16 21:55:02 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/mongo-1.6.4/lib/mongo/collection.rb:353:in `insert'

  2013-05-16 21:55:02 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-mongo-0.6.7/lib/fluent/plugin/out_mongo.rb:173:in `operate'

  2013-05-16 21:55:02 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-mongo-0.6.7/lib/fluent/plugin/out_mongo.rb:162:in `write'

  2013-05-16 21:55:02 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.25/lib/fluent/buffer.rb:279:in `write_chunk'

  2013-05-16 21:55:02 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.25/lib/fluent/buffer.rb:263:in `pop'

  2013-05-16 21:55:02 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.25/lib/fluent/output.rb:303:in `try_flush'

  2013-05-16 21:55:02 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.25/lib/fluent/output.rb:120:in `run'

And I'm unsure about the best way to handle that.

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Sylvain]

Hello Masahiro :)

I did not know about that hard MongoDB limit of 16Mo. Thank you!

However something weird is happening

My out_foward is configured to send to : buffer_chunk_limit 1m

While my out_mongo is set to: buffer_chunk_limit 256m

However I added some more information to the debug message and found out that the chunk the out_mongo was trying to process was 320844457bytes (305Megabyte).

One of my "collector"/out_forward must be doing something wrong, is there a way I can debug that?

Found out what is that chunk, where it's coming from etc...?

 

Cheers!

Sylvain

[Masahiro Nakagawa]

Sylvain,

Just weird...

305Megabyte is only once? or sometimes occurred?

In most cases, TimeSlicedOutput plugin stores over 100 MB,

but out_forward is not TimeSlicedOutput...

Do you implement own plugin?