Elasticsearch plugin exceeding AWS Elasticsearch 10MB request payload limit

[Aaron Bernardino]

Hi,

I'm using td-agent w/ fluent-plugin-elasticsearch to send  to an AWS Elasticsearch.  I am using a t2.micro.elasticsearch instance type and it has a 10MB limit on the maximum HTTP request payload size.

I've tried setting the buffer chunk limit to smaller values thinking that it might control the size of the HTTP data payload being sent to Elasticsearch when it is flushed.  However, I still get the following error.

2016-07-29 10:52:56 -0400 [warn]: temporarily failed to flush the buffer. next_retry=2016-07-29 10:53:28 -0400 error_class="Elasticsearch::Transport::Transport::Errors::RequestEntityTooLarge" error="[413] {\"Message\":\"Request size exceeded 10485760 bytes\"}" plugin_id="object:3f9ba418f608"

I think I am hitting the 10 MB payload limit of the AWS instance.

Here's a snippet of my elasticsearch plugin configuration:

  <store>

    @type elasticsearch

    host "search-****************************.es.amazonaws.com"

    port 80

    logstash_format true

    include_tag_key true

    tag_key _key

    log_level debug

    # Buffering

    buffer_type file

    buffer_path "/opt/td-agent/buffer/"

    flush_interval 60s

    reload_on_failure true

    max_retry_wait 300s

    retry_wait 15s

    disable_retry_limit

    buffer_chunk_limit 300k

    buffer_queue_limit 4096

  </store>

Any help would be appreciated. Thanks

[Mr. Fiber]

Are there no other warnings in the log?

if output plugin receives 300k+ event chunks from input plugin in one emit,

output plugin tries to flush its chunks directly.

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Aaron Bernardino]

I see. So buffer chunk limit is the number of events, not the size of the chunk. I will try to configure even smaller values and try.
Also I did not see other warning logs. I also did try just setting up an AWS instance with a higher 100MB limit. I did not encounter the error.

Thanks!

[Eduardo Silva]

per documentation buffer_chunk_limit is the size, not the number of events.

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Eduardo Silva

Open Source, Treasure Data

http://www.treasuredata.com/opensource

 

[Mr. Fiber]

> So buffer chunk limit is the number of events, not the size of the chunk.

No. If output plugin receives event stream larger than `buffer_chunk_limit`,

output plugin flushes its chunk immediately to avoid data lost in v0.12.

In master branch, fluentd splits larger chunk into small chunks to avoid this problem.

[Aaron Bernardino]

Ok, so it is the size. Thank you!

[Yasin Amadmia]

Aaron,

If you are using in_tail plugin to source the events, it would be best to use 'read_lines_limit'. I understand that the default 'read_lines_limit' is 1000. So if 1000 lines are available, in_tail would grab them all at once and then if you are on v0.12 or lower (latest td-agent 2.3.2 is actually fluentd v0.12) it will be flushed ignoring the chunk_limit as @repeatedly explained above.