Fluentd buffer type

[MING LIU]

Hi，I’m using fluentd to collect production environment logs. Due to the large amount of data,   I use buffer plugin , but there is a backlog on the three aggregations, almost 80G. I guess in buffer section, using @type file to save data in the disk cost too long. So I wonder can i use @type memory . Will it OOM？Or it can turn to save data in disk when memory utilization is high?

[cosmo09...@gmail.com]

Hi,

From Fluentd document, using file type buffer should be buffering up to 64GB and using memory type buffer should be using up to 512MB memory:
https://docs.fluentd.org/configuration/buffer-section#buffering-parameters

If your using disk has more than 64GB free space when using file buffer, you can send 80GB backlog.
Fluentd does not copy entire log files. Instead, step by step tailing and buffering on sending data.

Best,

Hiroshi

2021年3月10日水曜日 10:36:37 UTC+9 liuming...@gmail.com: