Issues While Using Kafka Mirror Maker

[Rachit]

Hi,

We have been trying to move from our old kafka cluster to new 3 broker cluster and were looking for ways to copy all old partitions onto the new cluster. Kafka mirrormaker seems the best option for doing this, but we faced a few problems while implementing:

On destination (new cluster), We ran following command

bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config config/consumer.properties --producer.config config/producer.properties --whitelist T.abc

This triggers kafka consumer, which the consumes data. We can see that on Kafka-manager also, but the producer doesn't startup, i.e no new topic is created on the destination broker.

(auto.create.topics.enable=true) has been already set.

Only when some new data is posted in the topic T.abc on source, the topic gets created, but the old data is still not present, only the new data gets mirrored.

If someone has used mirroring, kindly suggest how to make it work? 

[konst...@confluent.io]

Hi Rachit, 

from what you describe it sounds like your issue is related to MirrorMaker's (MM) consumer settings instead. 

Specifically the property auto.offset.reset.

This property by default instructs the consumer to fetch records from the latest offset and on. That's why you see only newly created data (w.r.t to when you start MM). 

You should set 

auto.offset.reset=earliest 

in order to replicate your existing records. If you tell MM to use the old consumer (which I don't suggest unless you have a specific reason to do that) the equivalent value for this property is smallest.

Regards,

Konstantine

[Rachit]

Hi Constantine,

Thanks for replying,

As suggested by you, we set the auto.offset.reset=smallest, and on starting mirror, Although the topic does show up on destination, but the console shows up following warnings continuously

[2016-11-11 09:41:20,401] WARN Error while fetching metadata with correlation id 0 : {T.HelloWorld=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

[2016-11-11 09:41:20,508] WARN Error while fetching metadata with correlation id 1 : {T.HelloWorld=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

[2016-11-11 09:41:20,614] WARN Error while fetching metadata with correlation id 2 : {T.HelloWorld=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

[2016-11-11 09:41:20,719] WARN Error while fetching metadata with correlation id 3 : {T.HelloWorld=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

[2016-11-11 09:41:20,826] WARN Error while fetching metadata with correlation id 4 : {T.HelloWorld=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

[2016-11-11 09:41:20,932] WARN Error while fetching metadata with correlation id 5 : {T.HelloWorld=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

[2016-11-11 09:41:21,037] WARN Error while fetching metadata with correlation id 6 : {T.HelloWorld=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

[2016-11-11 09:41:21,143] WARN Error while fetching metadata with correlation id 7 : {T.HelloWorld=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient) 

Following is the screenshot of consumer at source cluster:

[Konstantine Karantasis]

Sounds like a connectivity issue. 

Check the FAQ here

Depending on which version you use, you might want to check the *advertised* properties for your brokers too: https://kafka.apache.org/documentation#brokerconfigs

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/de993f28-10b2-44ca-b680-d512f48f61a1%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.