Buffered Logs

[Yasin Amadmia]

Dear All,

Wanted to check what will be the expected behaviour of fluentd in case if say an output plugin is erroring out (i.e. not sending any events to final destination, eg. mongo store) and then to fix that one has to restart fluentd agent. Will all the events that were not sent be resent after the restart (presuming they were buffered and the buffer is preserved across restarting agent)? 

Thanks,

Yasin.

[Mr. Fiber]

Hi Yasin,

You can check buffer behaviour in this article: http://docs.fluentd.org/articles/buffer-plugin-overview

> presuming they were buffered and the buffer is preserved across restarting agent

It depends on buffer type.

If you want to keep events during server restart,

you should use file buffer.

Masahiro

This e-mail and any attachments are confidential and intended solely for the use of the addressee only. If you have received this message in error, you must not copy, distribute or disclose the contents; please notify the sender immediately and delete the message.

This message is attributed to the sender and may not necessarily reflect the view of Travis Perkins Trading Company Limited, 733503, Lodge Way House, Lodge Way, Harlestone Road, Northampton, NN5 7UG or its parent company Travis Perkins plc (Registered in England No. 824821; Lodge Way House, Lodge Way, Harlestone Road, Northampton, NN5 7UG; VAT number 408556737) and any of its subsidiaries. Agreements binding Travis Perkins Trading Company Limited may not be concluded by means of e-mail communication.

E-mail transmissions are not secure and Travis Perkins Trading accepts no responsibility for changes made to this message after it was sent. Whilst steps have been taken to ensure that this message is virus free, Travis Perkins Trading accepts no liability for infection and recommends that you scan this e-mail and any attachments.

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Yasin Amadmia]

Thanks.

So does it mean I have to (i can) add buffer related options to any plugin that supports buffered output ?

ex: fluent-plugin-aws-elasticsearch-service

<match es.**>

  type "aws-elasticsearch-service"

  type_name "access_log"

  logstash_format true

  include_tag_key true

  tag_key "@log_name"

  flush_interval 1s

  #buffer specific configs

  buffer_type file

  buffer_path /path

  buffer_chunk_limit 256m

  buffer_queue_limit 128

  flush_interval 60s

  ....

  <endpoint>

    url https://CLUSTER_ENDPOINT_URL

    region eu-west-1

    # access_key_id "secret"

    # secret_access_key "seekret"

  </endpoint>

</match>

[Mr. Fiber]

> So does it mean I have to (i can) add buffer related options to any plugin that supports buffered output ?

Yes.