Mirror maker thread failure

[haim]

Hello

Currently we have 2 remote Kafka Clusters (0.9) which works ok, they are both located in US.

Kafka Mirror is running on both cluster, and topics are mirrored between clusters as configured.

I tries to add a 3rd cluster which is in India.

This cluster seems to works ok, but mirror from this cluster to any of the US is failing with errors:

when tested with Kafka 0.9 got errors of "batch expired" .

I tried to adjust the options in consumer and producer (batch.size, request.timeout.ms)

This didn't work, and the mirror proc would crash

I then upgraded only the new remote cluster to Kafka_2.11-1.0.0

Now i'm getting:

[2017-12-12 15:37:12,964] FATAL [mirrormaker-thread-2] Mirror maker thread failure due to  (kafka.tools.MirrorMaker$MirrorMakerThread)

kafka.common.ConsumerRebalanceFailedException: group1_in-kafka2-1513092912419-a3ad0cbf can't rebalance after 4 retries

        at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:670)

        at kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:968)

        at kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.<init>(ZookeeperConsumerConnector.scala:1002)

        at kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:176)

        at kafka.tools.MirrorMaker$MirrorMakerOldConsumer.init(MirrorMaker.scala:520)

        at kafka.tools.MirrorMaker$MirrorMakerThread.run(MirrorMaker.scala:421)

[2017-12-12 15:37:12,997] FATAL [mirrormaker-thread-7] Mirror maker thread exited abnormally, stopping the whole mirror maker. (kafka.tools.MirrorMaker$MirrorMakerThread)

[2017-12-12 15:37:12,997] FATAL [mirrormaker-thread-4] Mirror maker thread exited abnormally, stopping the whole mirror maker. (kafka.tools.MirrorMaker$MirrorMakerThread)

[2017-12-12 15:37:12,997] FATAL [mirrormaker-thread-1] Mirror maker thread exited abnormally, stopping the whole mirror maker. (kafka.tools.MirrorMaker$MirrorMakerThread)

[2017-12-12 15:37:12,997] FATAL [mirrormaker-thread-2] Mirror maker thread exited abnormally, stopping the whole mirror maker. (kafka.tools.MirrorMaker$MirrorMakerThread)

[2017-12-12 15:37:12,997] FATAL [mirrormaker-thread-5] Mirror maker thread exited abnormally, stopping the whole mirror maker. (kafka.tools.MirrorMaker$MirrorMakerThread)

[2017-12-12 15:37:12,997] FATAL [mirrormaker-thread-3] Mirror maker thread exited abnormally, stopping the whole mirror maker. (kafka.tools.MirrorMaker$MirrorMakerThread)

[2017-12-12 15:37:12,997] FATAL [mirrormaker-thread-0] Mirror maker thread exited abnormally, stopping the whole mirror maker. (kafka.tools.MirrorMaker$MirrorMakerThread)

[2017-12-12 15:37:12,997] FATAL [mirrormaker-thread-6] Mirror maker thread exited abnormally, stopping the whole mirror maker. (kafka.tools.MirrorMaker$MirrorMakerThread)

The only thing i noticed is that the latency from India to US is about 180ms, while between the US clusters it's about 10ms

Can you advise ?

Thank you