Multi-Datacenter Replication

[Xin Jin]

Hi all,

I have a few questions about multi-datacenter solution using Kafka:

One solution for a data pipeline over a few datacents (DCs) consists of a series of kafkas with mirrormaker over DCs with replications from one DC to another DC and combine them together. Confluent platform doesn’t provide this whole solution, though the enterprise version does provide some sort of solution.

1.) How difficulty or complex if we develop this solution based on the current kafka?

2.) Are there any open sources that can provide further help? I have done some research on this topic and not find any of them, except the uReplicator that make some improvement over mirrormaker. Is this because it is too complicated or the current tools are enough?

I really appreciate your comments/experiences.

Thanks,

Xin

[Ewen Cheslack-Postava]

On Fri, Jul 28, 2017 at 2:12 PM, Xin Jin <xjin...@gmail.com> wrote:

Hi all,

I have a few questions about multi-datacenter solution using Kafka:

One solution for a data pipeline over a few datacents (DCs) consists of a series of kafkas with mirrormaker over DCs with replications from one DC to another DC and combine them together. Confluent platform doesn’t provide this whole solution, though the enterprise version does provide some sort of solution.

Confluent Platform does ship with MirrorMaker as it is part of Kafka, but we recommend Replicator for the reasons described below.

 

1.) How difficulty or complex if we develop this solution based on the current kafka?

Replicator provides a bunch of benefits over MM, which are outlined here: http://docs.confluent.io/current/connect/connect-replicator/docs/connect_replicator.html#features In particular, topics are created with the correct number of partitions, the number of partitions are tracked and updated if they are changed over time, and those topics have the same topic configurations applied (e.g. whether using 'delete' or 'compact' policy). You would need to modify/enhance MM to get the same functionality.

 

2.) Are there any open sources that can provide further help? I have done some research on this topic and not find any of them, except the uReplicator that make some improvement over mirrormaker. Is this because it is too complicated or the current tools are enough?

uReplicator is the only other open source tool I'm aware of.

-Ewen

 

I really appreciate your comments/experiences.

Thanks,

Xin

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/0cb0e974-d76c-449a-8074-a0260a795dfe%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Xin Jin]

Hi Ewen,

Thank you very much.

Xin

On Monday, July 31, 2017 at 8:48:11 AM UTC-7, Ewen Cheslack-Postava wrote:

On Fri, Jul 28, 2017 at 2:12 PM, Xin Jin <xjin...@gmail.com> wrote:

Hi all,

I have a few questions about multi-datacenter solution using Kafka:

One solution for a data pipeline over a few datacents (DCs) consists of a series of kafkas with mirrormaker over DCs with replications from one DC to another DC and combine them together. Confluent platform doesn’t provide this whole solution, though the enterprise version does provide some sort of solution.

Confluent Platform does ship with MirrorMaker as it is part of Kafka, but we recommend Replicator for the reasons described below.

 

1.) How difficulty or complex if we develop this solution based on the current kafka?

Replicator provides a bunch of benefits over MM, which are outlined here: http://docs.confluent.io/current/connect/connect-replicator/docs/connect_replicator.html#features In particular, topics are created with the correct number of partitions, the number of partitions are tracked and updated if they are changed over time, and those topics have the same topic configurations applied (e.g. whether using 'delete' or 'compact' policy). You would need to modify/enhance MM to get the same functionality.

 

2.) Are there any open sources that can provide further help? I have done some research on this topic and not find any of them, except the uReplicator that make some improvement over mirrormaker. Is this because it is too complicated or the current tools are enough?

uReplicator is the only other open source tool I'm aware of.

-Ewen

 

I really appreciate your comments/experiences.

Thanks,

Xin

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.