How to keep messages order when using aggregator

[Shafreeck Sea]

Hi:

I am suffering the performance problem when writing logs directly from fluentd to kafka, so I am given a suggestion that introducing aggregators to offload the agent. However, I am wondering how to keep message orders when load balancing to multiple aggregators. It looks like we do not have a client affinity algorithm to do load balance. 

[Mr. Fiber]

> However, I am wondering how to keep message orders when load balancing to multiple aggregators.

Currently, fluentd itself doesn't provide it.

In addition, out_kafka_buffered also doesn't gurantee the message order when num_threads is larger than 1.

If your kafka cluster support kafka connect, kafka-connect-fluentd is one way to improve the performance.

https://github.com/fluent/kafka-connect-fluentd

Masahiro

 

On Wed, Nov 15, 2017 at 2:44 PM, Shafreeck Sea <shaf...@gmail.com> wrote:

Hi:

I am suffering the performance problem when writing logs directly from fluentd to kafka, so I am given a suggestion that introducing aggregators to offload the agent. However, I am wondering how to keep message orders when load balancing to multiple aggregators. It looks like we do not have a client affinity algorithm to do load balance. 

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Shafreeck Sea]

Got it, thanks !

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.