file keep on filling up in buffer path and failed to flush the buffer.

[databee]

Hi, I am facing few problem with td agent. All the files from last few months are piling up in buffer path. and now it is failed to flush the buffer. I try to increase the buffer limit and queue limit but not sure why file are still stored in buffer path. 

- Chef td agent version 2.2.0.

- ubuntu 14.04.

- My current match config for collector is 

<match component.*>

  @type copy

  <store>

    @type                 s3

    aws_key_id            xxx

    aws_sec_key           xxx

    s3_bucket             xxx

    s3_region             xxx

    s3_object_key_format  %{path}/${tag}_%{time_slice}_%{index}.%{file_extension}

    path                  logs/component/

    compress              gzip

    buffer_path           /var/log/td-agent/buffer/s3_component

    time_slice_format     %Y-%m-%d/%H/%M

    time_slice_wait       1m

    flush_interval        15m

    buffer_chunk_limit    32m

    buffer_queue_limit    4098

    retry_wait            1s

    max_retry_wait        5s

    disable_retry_limit   true

    storage_class         REDUCED_REDUNDANCY

  </store>

</match>

- I am getting following error 

`2016-11-10 19:59:47 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2016-11-10 19:59:41 +0000 error_class="Errno::EMFILE" error="Too many open files @ rb_sysopen - /tmp/s3-20161110-24004-k538mh" plugin_id="object:3ff2cc3db8"

  2016-11-10 19:59:47 +0000 [warn]: suppressed same stacktrace

2016-11-10 19:59:47 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2016-11-10 19:59:42 +0000 error_class="Errno::EMFILE" error="Too many open files @ rb_sysopen - /tmp/s3-2016-11-10-24004-w5sf44" plugin_id="object:3ff2c8e6d4"

  2016-11-10 19:59:47 +0000 [warn]: suppressed same stacktrace

2016-11-10 19:59:47 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2016-11-10 19:59:52 +0000 error_class="Errno::EMFILE" error="Too many open files @ rb_sysopen - /tmp/s3-20161110-24004-1q9pt5r" plugin_id="object:3ff2c8d6a8"

  2016-11-10 19:59:47 +0000 [warn]: suppressed same stacktrace

2016-11-10 19:59:47 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2016-11-10 19:59:52 +0000 error_class="Errno::EMFILE" error="Too many open files @ rb_sysopen - /tmp/s3-20161110-24004-1x1u77o" plugin_id="object:3ff2c9d1d6"`

[Mr. Fiber]

Thanks for moving to mailing list.

 

>  All the files from last few months are piling up in buffer path.

It means, there is no upload files in S3 directory, right?

> Too many open files

buffer_queue_limit error can't resolve this problem because

you need more resource limit.

This means fluentd can't handle more files.

You need to check why s3 plugin doesn't uplaod buffers.

No other error logs?

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[databee]

There is no other error, No I have file from last 3 month from 20 server for 5 type of logs into buffer path. I tried to change to buffer size limit but still not working. 

compress              gzip
buffer_path           /var/log/td-agent/buffer/s3_component
time_slice_format     %Y-%m-%d/%H/%M
time_slice_wait       1m
flush_interval        15m

buffer_chunk_limit    252m
buffer_queue_limit    2m
retry_wait            300s
max_retry_wait        900s
disable_retry_limit   true
storage_class         REDUCED_REDUNDANCY

No sure, what is going on. 

Error message still the same. 

I am trying to remove all limit and deleting the buffer, then restart the td agent. will see if it works.

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.

[Mr. Fiber]

> time_slice_format %Y-%m-%d/%H/%M

This means you create files at each minutes.

If your flush takes 1+ minute, buffer is never decreased.

Maybe, checking current buffer setting is better.

For reducing IO latency, increasing num_threads is one way.

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.

[databee]

Thank you, I am no longer getting the failed to flush error after removing all config for flush interval and time slice wait. but i am getting the following error now. 

2016-11-11 23:31:32 +0000 [warn]: Size of the emitted data exceeds buffer_chunk_limit.

2016-11-11 23:31:32 +0000 [warn]: This may occur problems in the output plugins ``at this server.``

2016-11-11 23:31:32 +0000 [warn]: To avoid problems, set a smaller number to the buffer_chunk_limit

2016-11-11 23:31:32 +0000 [warn]: in the forward output ``at the log forwarding server.``

2016-11-11 23:31:35 +0000 [warn]: no patterns matched tag="fluent.info"

2016-11-11 23:32:02 +0000 [warn]: Size of the emitted data exceeds buffer_chunk_limit.

2016-11-11 23:32:02 +0000 [warn]: This may occur problems in the output plugins ``at this server.``

2016-11-11 23:32:02 +0000 [warn]: To avoid problems, set a smaller number to the buffer_chunk_limit

2016-11-11 23:32:02 +0000 [warn]: in the forward output ``at the log forwarding server.``

So, is the default buffer_chunk_limit is higher and I need to minimize that or I need to increase the limit.