Fluentd not flushing in memory buffer before shutting down

448 views
Skip to first unread message

Pranav Buch

unread,
Feb 16, 2020, 10:50:44 PM2/16/20
to Fluentd Google Group
Hi Guys, 

I have a fluentd running with HTTP input and http output and STDOUT output. My pipeline looks like below : 



    <source>
      @type http
      port 8888
      bind 127.0.0.1
      body_size_limit 2m
      keepalive_timeout 10s
    </source>
    <match some-logs>
      @type copy
      <store>
        @type stdout
        <format>
        output_type json
        </format>
      </store>
      <store ignore_error>
        @type http
        <buffer>
          flush_interval 1s
          overflow_action drop_oldest_chunk
          chunk_limit_size 2mb
          flush_thread_count 10
        </buffer>
        endpoint https://output:6065/
        <format>
        @type json
        </format>
      </store>
    </match>


Usecase - If I shutdown fluentd when its retrying to post one of the messages to "endpoint", it simply shutsdown without any error or anything else reported. Please find below logs indicating the same : 


================================

2020-02-17 03:36:09 +0000 [warn]: #0 failed to flush the buffer. retry_time=6 next_retry_seconds=2020-02-17 03:36:41 +0000 chunk="59ebd3dae6f84709455240c95ac13d5b" error_class=SocketError error="Failed to open TCP connection to output:6065 (getaddrinfo: Temporary failure in name resolution)"
  2020-02-17 03:36:09 +0000 [warn]: #0 /usr/local/lib/ruby/2.6.0/net/http.rb:949:in `rescue in block in connect'
  2020-02-17 03:36:09 +0000 [warn]: #0 /usr/local/lib/ruby/2.6.0/net/http.rb:946:in `block in connect'
  2020-02-17 03:36:09 +0000 [warn]: #0 /usr/local/lib/ruby/2.6.0/timeout.rb:76:in `timeout'
  2020-02-17 03:36:09 +0000 [warn]: #0 /usr/local/lib/ruby/2.6.0/net/http.rb:945:in `connect'
  2020-02-17 03:36:09 +0000 [warn]: #0 /usr/local/lib/ruby/2.6.0/net/http.rb:930:in `do_start'
  2020-02-17 03:36:09 +0000 [warn]: #0 /usr/local/lib/ruby/2.6.0/net/http.rb:919:in `start'
  2020-02-17 03:36:09 +0000 [warn]: #0 /usr/local/lib/ruby/2.6.0/net/http.rb:605:in `start'
  2020-02-17 03:36:09 +0000 [warn]: #0 /fluentd/plugins/out_http.rb:209:in `send_request'
  2020-02-17 03:36:09 +0000 [warn]: #0 /fluentd/plugins/out_http.rb:117:in `write'
  2020-02-17 03:36:09 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.4.2/lib/fluent/plugin/output.rb:1125:in `try_flush'
  2020-02-17 03:36:09 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.4.2/lib/fluent/plugin/output.rb:1425:in `flush_thread_run'
  2020-02-17 03:36:09 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.4.2/lib/fluent/plugin/output.rb:454:in `block (2 levels) in start'
  2020-02-17 03:36:09 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.4.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2020-02-17 03:36:47 +0000 [warn]: #0 failed to flush the buffer. retry_time=7 next_retry_seconds=2020-02-17 03:37:43 +0000 chunk="59ebd3dae6f84709455240c95ac13d5b" error_class=SocketError error="Failed to open TCP connection to output:6065 (getaddrinfo: Temporary failure in name resolution)"
  2020-02-17 03:36:47 +0000 [warn]: #0 suppressed same stacktrace
2020-02-17 03:37:40 +0000 [info]: Received graceful stop
2020-02-17 03:37:41 +0000 [info]: #0 fluentd worker is now stopping worker=0
2020-02-17 03:37:41 +0000 [info]: #0 shutting down fluentd worker worker=0
2020-02-17 03:37:41 +0000 [info]: #0 shutting down input plugin type=:http plugin_id="object:3ff50485274c"
2020-02-17 03:37:41 +0000 [info]: #0 shutting down output plugin type=:copy plugin_id="object:3ff50520b360"
2020-02-17 03:37:41 +0000 [info]: #0 shutting down output plugin type=:stdout plugin_id="object:3ff50487de4c"
2020-02-17 03:37:41 +0000 [info]: #0 shutting down output plugin type=:http plugin_id="object:3ff503efd9f8"
2020-02-17 03:37:41 +0000 [info]: Worker 0 finished with status 0


====================================

At the time of shutdown, the endpoint was available, my expectation is before shutting down, it should have tried to flush once again instead it drops everything that was buffered and shuts down.

Can anyone see any issues with my configuration or anything else?


Balaji K

unread,
Feb 17, 2020, 4:49:06 AM2/17/20
to flu...@googlegroups.com
Hi,

Not sure if you have given this a try "flush_at_shutdown true", we use this for the same purpose. Hope this helps..

rgds
Balaji Kamal Kannadassan

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fluentd/ea2127ba-cf44-474d-893f-1ded2f138e93%40googlegroups.com.


--
Balaji Kamal Kannadassan
plot #70, Door #28,
Majestic Colony Main Road,
Vlasaravakkam,
Chennai-87

Pranav Buch

unread,
Feb 17, 2020, 5:24:44 PM2/17/20
to flu...@googlegroups.com
Hi,

 I believe thats a default for memory buffer, isn’t that?

Regards,
Pranav Buch

On 17 Feb 2020, at 8:49 pm, Balaji K <kbala...@gmail.com> wrote:



Pranav Buch

unread,
Feb 26, 2020, 12:46:42 AM2/26/20
to Fluentd Google Group
Bump!

I have tried setting that manually as well, no luck !
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.


--
Balaji Kamal Kannadassan
plot #70, Door #28,
Majestic Colony Main Road,
Vlasaravakkam,
Chennai-87

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.

Pranav Buch

unread,
Feb 26, 2020, 12:53:02 AM2/26/20
to Fluentd Google Group
Adding debug logs if they are of any help : 

============================================================


2020-02-26 05:50:33 +0000 [debug]: #0 Post data to https://output:6065/ with chunk(59f742d29f5b8a5b173a64a6c99853e1)
2020-02-26 05:50:39 +0000 [debug]: #0 taking back chunk for errors. chunk="59f742d29f5b8a5b173a64a6c99853e1"
2020-02-26 05:50:39 +0000 [warn]: #0 failed to flush the buffer. retry_time=4 next_retry_seconds=2020-02-26 05:50:47 +0000 chunk="59f742d29f5b8a5b173a64a6c99853e1" error_class=SocketError error="Failed to open TCP connection to logstash:6065 (getaddrinfo: Temporary failure in name resolution)"
  2020-02-26 05:50:39 +0000 [warn]: #0 /usr/local/lib/ruby/2.6.0/net/http.rb:949:in `rescue in block in connect'
  2020-02-26 05:50:39 +0000 [warn]: #0 /usr/local/lib/ruby/2.6.0/net/http.rb:946:in `block in connect'
  2020-02-26 05:50:39 +0000 [warn]: #0 /usr/local/lib/ruby/2.6.0/timeout.rb:76:in `timeout'
  2020-02-26 05:50:39 +0000 [warn]: #0 /usr/local/lib/ruby/2.6.0/net/http.rb:945:in `connect'
  2020-02-26 05:50:39 +0000 [warn]: #0 /usr/local/lib/ruby/2.6.0/net/http.rb:930:in `do_start'
  2020-02-26 05:50:39 +0000 [warn]: #0 /usr/local/lib/ruby/2.6.0/net/http.rb:919:in `start'
  2020-02-26 05:50:39 +0000 [warn]: #0 /usr/local/lib/ruby/2.6.0/net/http.rb:605:in `start'
  2020-02-26 05:50:39 +0000 [warn]: #0 /fluentd/plugins/out_http.rb:209:in `send_request'
  2020-02-26 05:50:39 +0000 [warn]: #0 /fluentd/plugins/out_http.rb:117:in `write'
  2020-02-26 05:50:39 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.4.2/lib/fluent/plugin/output.rb:1125:in `try_flush'
  2020-02-26 05:50:39 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.4.2/lib/fluent/plugin/output.rb:1425:in `flush_thread_run'
  2020-02-26 05:50:39 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.4.2/lib/fluent/plugin/output.rb:454:in `block (2 levels) in start'
  2020-02-26 05:50:39 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.4.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2020-02-26 05:50:41 +0000 [info]: fluent/log.rb:322:info: Received graceful stop
2020-02-26 05:50:41 +0000 [debug]: #0 fluentd main process get SIGTERM
2020-02-26 05:50:41 +0000 [debug]: #0 getting start to shutdown main process
2020-02-26 05:50:42 +0000 [info]: #0 fluentd worker is now stopping worker=0
2020-02-26 05:50:42 +0000 [info]: #0 shutting down fluentd worker worker=0
2020-02-26 05:50:42 +0000 [debug]: #0 calling stop on input plugin type=:http plugin_id="object:3feef6550d14"
2020-02-26 05:50:42 +0000 [debug]: #0 calling stop on output plugin type=:copy plugin_id="object:3feef71f775c"
2020-02-26 05:50:42 +0000 [debug]: #0 calling stop on output plugin type=:stdout plugin_id="object:3feef657a948"
2020-02-26 05:50:42 +0000 [debug]: #0 calling stop on output plugin type=:http plugin_id="object:3feef51095b8"
2020-02-26 05:50:42 +0000 [debug]: #0 preparing shutdown input plugin type=:http plugin_id="object:3feef6550d14"
2020-02-26 05:50:42 +0000 [info]: #0 shutting down input plugin type=:http plugin_id="object:3feef6550d14"
2020-02-26 05:50:42 +0000 [debug]: #0 preparing shutdown output plugin type=:copy plugin_id="object:3feef71f775c"
2020-02-26 05:50:42 +0000 [info]: #0 shutting down output plugin type=:copy plugin_id="object:3feef71f775c"
2020-02-26 05:50:42 +0000 [debug]: #0 preparing shutdown output plugin type=:http plugin_id="object:3feef51095b8"
2020-02-26 05:50:42 +0000 [debug]: #0 preparing shutdown output plugin type=:stdout plugin_id="object:3feef657a948"
2020-02-26 05:50:42 +0000 [info]: #0 shutting down output plugin type=:stdout plugin_id="object:3feef657a948"
2020-02-26 05:50:42 +0000 [info]: #0 shutting down output plugin type=:http plugin_id="object:3feef51095b8"
2020-02-26 05:50:42 +0000 [debug]: #0 calling after_shutdown on input plugin type=:http plugin_id="object:3feef6550d14"
2020-02-26 05:50:42 +0000 [debug]: #0 calling after_shutdown on output plugin type=:copy plugin_id="object:3feef71f775c"
2020-02-26 05:50:42 +0000 [debug]: #0 calling after_shutdown on output plugin type=:stdout plugin_id="object:3feef657a948"
2020-02-26 05:50:42 +0000 [debug]: #0 calling after_shutdown on output plugin type=:http plugin_id="object:3feef51095b8"
2020-02-26 05:50:42 +0000 [debug]: #0 closing input plugin type=:http plugin_id="object:3feef6550d14"
2020-02-26 05:50:42 +0000 [debug]: #0 closing output plugin type=:copy plugin_id="object:3feef71f775c"
2020-02-26 05:50:42 +0000 [debug]: #0 closing output plugin type=:stdout plugin_id="object:3feef657a948"
2020-02-26 05:50:42 +0000 [debug]: #0 closing output plugin type=:http plugin_id="object:3feef51095b8"
2020-02-26 05:50:42 +0000 [debug]: #0 closing buffer instance=70295546411220
2020-02-26 05:50:42 +0000 [debug]: #0 calling terminate on input plugin type=:http plugin_id="object:3feef6550d14"
2020-02-26 05:50:42 +0000 [debug]: #0 calling terminate on output plugin type=:copy plugin_id="object:3feef71f775c"
2020-02-26 05:50:42 +0000 [debug]: #0 calling terminate on output plugin type=:stdout plugin_id="object:3feef657a948"
2020-02-26 05:50:42 +0000 [debug]: #0 calling terminate on output plugin type=:http plugin_id="object:3feef51095b8"
2020-02-26 05:50:42 +0000 [info]: fluent/log.rb:322:info: Worker 0 finished with status 0

============================================================

Balaji K

unread,
Feb 28, 2020, 5:26:33 AM2/28/20
to flu...@googlegroups.com
Sorry Pranav, I don't have further input on the same. We added above said flag for us and didn't face any problem. Did you check the reason for below log..

2020-02-26 05:50:39 +0000 [warn]: #0 failed to flush the buffer. retry_time=4 next_retry_seconds=2020-02-26 05:50:47 +0000 chunk="59f742d29f5b8a5b173a64a6c99853e1" error_class=SocketError error="Failed to open TCP connection to logstash:6065 (getaddrinfo: Temporary failure in name resolution)"

rgds
Balaji Kamal Kannadassan

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.


--
Balaji Kamal Kannadassan
plot #70, Door #28,
Majestic Colony Main Road,
Vlasaravakkam,
Chennai-87

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fluentd/896bdbcf-4fdc-4cfc-9892-691360ef5d85%40googlegroups.com.

Pranav Buch

unread,
Feb 28, 2020, 6:59:54 PM2/28/20
to flu...@googlegroups.com
Reply all
Reply to author
Forward
0 new messages