Fluent::BufferQueueLimitError error="queue size exceeds limit"

[Ori Popowski]

Hi,

We're using Fluentd inside a Kubernetes cluster (version 1.3.4).

We have an AWS ElasticSearch Service which is exposed internally to
the cluster via Nginx proxy (Nginx just translates all requests/responses
from search-logging-32yw…gzjk3.eu-west-1.es.amazonaws.com
address to elasticsearch-logging address)

When Fluentd agents start, everything works fine, but one hour later, they
stop sending logs to ElasticSearch and their own logs look like this:

2016-08-11 14:25:27 +0000 [warn]: emit transaction failed: error_class=Fluent::BufferQueueLimitError error="queue size exceeds limit" tag="kubernetes.var.log.containers.user-service-284350070-ydtj0_default_user-service-ebfc038855f5da766c8b64598a17d002103f515290cb0c225793061ff73cb14a.log"
  2016-08-11 14:25:27 +0000 [warn]: suppressed same stacktrace
2016-08-11 14:25:28 +0000 [warn]: emit transaction failed: error_class=Fluent::BufferQueueLimitError error="queue size exceeds limit" tag="kubernetes.var.log.containers.fluentd-elasticsearch-7ekfg_kube-system_fluentd-elasticsearch-9c6927da57653d2efd2bdfbfcca1d11bf0f82589f70c1502e5bd2db108547c51.log"
  2016-08-11 14:25:28 +0000 [warn]: suppressed same stacktrace
2016-08-11 14:25:28 +0000 [warn]: emit transaction failed: error_class=Fluent::BufferQueueLimitError error="queue size exceeds limit" tag="kubernetes.var.log.containers.user-service-284350070-ydtj0_default_user-service-ebfc038855f5da766c8b64598a17d002103f515290cb0c225793061ff73cb14a.log"
  2016-08-11 14:25:28 +0000 [warn]: suppressed same stacktrace
2016-08-11 14:25:28 +0000 [warn]: emit transaction failed: error_class=Fluent::BufferQueueLimitError error="queue size exceeds limit" tag="kubernetes.var.log.containers.image-service-20uuw_default_image-service-d7cbd39346c4075acb24256bfdd77c3cddd02ae82c32bb12f5bfb53f2f42c26c.log"
  2016-08-11 14:25:28 +0000 [warn]: suppressed same stacktrace
2016-08-11 14:25:28 +0000 [warn]: emit transaction failed: error_class=Fluent::BufferQueueLimitError error="queue size exceeds limit" tag="kubernetes.var.log.containers.user-service-284350070-4to1u_default_user-service-afe9225c2fd79f2e96f161ebe5b22815857ca0f8901f17229f29713df23e0f75.log"
  2016-08-11 14:25:28 +0000 [warn]: suppressed same stacktrace
2016-08-11 14:25:28 +0000 [warn]: emit transaction failed: error_class=Fluent::BufferQueueLimitError error="queue size exceeds limit" tag="kubernetes.var.log.containers.billing-service-d7003_default_billing-service-e5441fd1aeea4b647026d1beed9c472b56a0a7746e44fe264aeab85c6724cc21.log"
  2016-08-11 14:25:28 +0000 [warn]: suppressed same stacktrace

I've read in some places that it might be due to buffer_queue_limit being
too low, but I suspect that it's not the problem. If it was, we would miss some
log lines, or maybe most of them, but in our case there's not a single log line
emitted to ElasticSearch.

Our Fluentd version is 0.12.19 and the config file can be found here:
https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/fluentd-elasticsearch/fluentd-es-image/td-agent.conf

I'd appreciate any help, thanks.

[Ori Popowski]

Problem solved.

We haven't seen the start of the log this whole time. Turns out the warnings start with a single 413 Request Entity Too Large message from Nginx which is never repeated the entire log.
It's Nginx way of saying that its client_max_body_size value is too low. Indeed, it turns out the default is 1M while Fluentd's buffer_chunk_limit was set to 2M