buffer_chunk_limit estimation/configuration

[Karri Niemelä]

Hi. 

What are some general guidelines how to decide on good value for buffer_chunk_limit?

In my case I'm doing inputs using in_tail, read_lines_limit is at default 1000. 

Typical emitted event stream is around 500kB. 

[Satoshi Tagomori]

Hi Karri,

buffer_chunk_limit is the maximum size when Fluentd writes data at once.

The best configuration depends on the type of output plugins.

If you're using Kafka output plugin or other queueing systems, 500KB - 2MB are good in many cases.

For file output plugin, 32MB or more size make better performance.

Moris.

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

田籠 聡 (TAGOMORI Satoshi)
Mail: tago...@gmail.com

[Trevor G]

Related to that, does num_threads impact the memory used in any way?

I've been trying to calculate how much memory will be consumed on a full queue, which I assume is buffer_chunk_limit x buffer_queue_limit.  So for instance, a buffer_chunk_limit of 2MB and a buffer_queue_limit of 128 would consume 256MB on a full queue.

When num_threads (default of 1) is introduced, do they pull from a common queue, or do they have their own queues?  If I have 8 threads in the above example, would I max at 256MB still or would it end up being 2048MB?

I'm finding it hard to find documentation around this math when trying to figure out why my instances are consuming 2GB+ of RAM as opposed to just 256MB.

Thanks,
  Trevor

On Thursday, September 15, 2016 at 12:41:25 AM UTC-7, Satoshi Tagomori wrote:

Hi Karri,

buffer_chunk_limit is the maximum size when Fluentd writes data at once.

The best configuration depends on the type of output plugins.

If you're using Kafka output plugin or other queueing systems, 500KB - 2MB are good in many cases.

For file output plugin, 32MB or more size make better performance.

Moris.

2016-09-15 16:17 GMT+09:00 Karri Niemelä <kak...@gmail.com>:

Hi. 

What are some general guidelines how to decide on good value for buffer_chunk_limit?

In my case I'm doing inputs using in_tail, read_lines_limit is at default 1000. 

Typical emitted event stream is around 500kB. 

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Satoshi Tagomori]

Hi Trevor,

Flush threads pull from a common queue. One buffer chunk queue is created for a buffer, and be shared by one or more flush threads.

Of course, your Fluentd configuration has 2 or more buffered output plugins and its queues, the total consumption of memory/disk spaces

are summation of these buffers.

Moris.

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Trevor G]

Excellent.  Thanks for the explanation of those details.

One final follow up question about the threading--Does the thread remove the data from the queue before or after the data has been delivered?

If the thread removes the data from the queue first and then attempts to transmit the data to the destination, that would allow another chunk to be added to the queue to replace the one that was just pulled.  If the thread cannot post the data to the destination and has a retry policy that has it try a few times, it would need to hold that chunk in memory until the next try.  That would mean that buffer_queue_limit chunks would be in memory, plus an additional buffer_chunk_limit sized chunk for every thread that has a chunk in flight.

If the thread removes the chunk only after it's able to post the data successfully, that would mean that there truly is a max of buffer_queue_limit chunks in memory at any given time for that output plugin.
 
Thanks,
  Trevor

[Mr. Fiber]

> One final follow up question about the threading--Does the thread remove the data from the queue before or after the data has been delivered?

Latter. Fluentd holds flushing chunks until flush succeeded.

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.

[Trevor G]

Great.  Thanks!