Fluentd as a log aggregator and forwarder

[Sahil Sareen]

Hey everyone

I want to use fluentd as a log aggregator from ~30 sources, ~350 events per minute.
Can someone suggest the configuration to be used for buffering and high availability, like no events should be missed and all forwarded to destination, need for a standby fluentd agent and how to configure?

Thanks

Sahil

[Mr. Fiber]

Hi Sahil,

At first, check this article: http://docs.fluentd.org/articles/high-availability

Fluentd's out_forward plugin supports load balancing,, at-most-once/at-least-once,

active-standby/active-active model and more.

Please check plugin artcile http://docs.fluentd.org/articles/out_forward

We recommend to use multiple aggregators for load-balancing or active-standby.

And for avoiding log lost, `require_ack_response true` is also useful and

using file buffer for persistent disk buffering.

Buffering parameter depends on your traffic: http://docs.fluentd.org/articles/buffer-plugin-overview

For throughput, longer flush_interval, 60s - 300s, and larger buffer chunk, 8+mb ,is good.

For low latency, use smaller values instead.

Setting <secondary> is also important when destination outage.

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sahil Sareen]

Thanks for your response, I've done this setup and have a few more questions.

Regarding high availability configuration and active/standby model:
I have just single source of each log type, Have two types of (different) logs each with a SINGLE source.(Attached sketch to explain the same)

I think the Forwarder - Aggregator model doesn't help me due to this, as that seems to be useful if there exist multiple sources of logically same category of data as on the documentation.

So I've just used a forwarder(with SNS plugin) to forward to SNS.

Here is my configuration for HTTP source(same for the Syslog source but using syslog plugin for source): https://paste.fedoraproject.org/363285/62520571/

From my understanding if the td-agent goes down, events would be buffered on file_buffer(upto 8m<default>). If the forward to SNS fails events will be written to secondary(standby) file which is log tailed by fluentd for retry and serves as a backup to avoid data loss.

The data scale is 250K events/day ~ 3 events/second ~ 500MB/day(for both streams together).

Can you please suggest if I can improve my setup for best availability/performance?

Thanks

Sahil

[Mr. Fiber]

> From my understanding if the td-agent goes down, events would be buffered on file_buffer(upto 8m<default>)

8m x buffer_queue_limit(default is 256).

> If the forward to SNS fails events will be written to secondary(standby)

Yes, logs are sent to secondary after reached retry_limit.

> The data scale is 250K events/day ~ 3 events/second ~ 500MB/day(for both streams together).

> Can you please suggest if I can improve my setup for best availability/performance?

fluentd's default setting can handle 500MB/day.

For better setup, change buffer parameters for your disk capacity and

flush / retry releated parameters for your traffic is better.