mirroring with kafka-mirror-maker usage with Confluent

[Nicolas Phung]

Hello,

My use case is the following: I'm trying to move data from one kafka cluster (integration) to another kafka cluster (here's my dev for testing purpose). I have read the documentation here http://confluent.io/docs/current/kafka/deployment.html#mirroring.

I got the following configuration for consumer.properties :

zookeeper.connect=integration.local:2181

zookeeper.connection.timeout.ms=6000

group.id=test-consumer-group

auto.offset.reset=smallest

And configuration for producer.properties:

metadata.broker.list=localhost:9092

producer.type=sync

compression.codec=none

serializer.class=kafka.serializer.DefaultEncoder

When I'm running the command :

kafka-mirror-maker --consumer.config consumer.properties --producer.config producer.properties --whitelist 'analytics-v2' 

And it generates the following :

Picked up JAVA_TOOL_OPTIONS: -javaagent:/usr/share/java/jayatanaag.jar 

[2015-06-04 14:19:15,496] WARN Error while fetching metadata [{TopicMetadata for topic analytics-v2 -> 

No partition metadata for topic analytics-v2 due to kafka.common.LeaderNotAvailableException}] for topic [analytics-v2]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)

[2015-06-04 14:19:15,516] WARN Error while fetching metadata [{TopicMetadata for topic analytics-v2 -> 

No partition metadata for topic analytics-v2 due to kafka.common.LeaderNotAvailableException}] for topic [analytics-v2]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)

[2015-06-04 14:19:15,517] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: analytics-v2 (kafka.producer.async.DefaultEventHandler)

[2015-06-04 14:19:15,628] WARN Error while fetching metadata [{TopicMetadata for topic analytics-v2 -> 

No partition metadata for topic analytics-v2 due to kafka.common.LeaderNotAvailableException}] for topic [analytics-v2]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)

[2015-06-04 14:19:15,631] WARN Error while fetching metadata [{TopicMetadata for topic analytics-v2 -> 

No partition metadata for topic analytics-v2 due to kafka.common.LeaderNotAvailableException}] for topic [analytics-v2]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)

[2015-06-04 14:19:15,635] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: analytics-v2 (kafka.producer.async.DefaultEventHandler)

[2015-06-04 14:19:15,807] WARN Error while fetching metadata [{TopicMetadata for topic analytics-v2 -> 

No partition metadata for topic analytics-v2 due to kafka.common.LeaderNotAvailableException}] for topic [analytics-v2]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)

[2015-06-04 14:19:15,813] WARN Error while fetching metadata [{TopicMetadata for topic analytics-v2 -> 

No partition metadata for topic analytics-v2 due to kafka.common.LeaderNotAvailableException}] for topic [analytics-v2]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)

[2015-06-04 14:19:15,815] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: analytics-v2 (kafka.producer.async.DefaultEventHandler)

[2015-06-04 14:19:15,919] WARN Error while fetching metadata [{TopicMetadata for topic analytics-v2 -> 

No partition metadata for topic analytics-v2 due to kafka.common.LeaderNotAvailableException}] for topic [analytics-v2]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)

[2015-06-04 14:19:15,929] WARN Error while fetching metadata [{TopicMetadata for topic analytics-v2 -> 

No partition metadata for topic analytics-v2 due to kafka.common.LeaderNotAvailableException}] for topic [analytics-v2]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)

[2015-06-04 14:19:15,930] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: analytics-v2 (kafka.producer.async.DefaultEventHandler)

[2015-06-04 14:19:16,035] WARN Error while fetching metadata [{TopicMetadata for topic analytics-v2 -> 

No partition metadata for topic analytics-v2 due to kafka.common.LeaderNotAvailableException}] for topic [analytics-v2]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)

[2015-06-04 14:19:16,036] ERROR Failed to send requests for topics analytics-v2 with correlation ids in [0,8] (kafka.producer.async.DefaultEventHandler)

[2015-06-04 14:19:16,038] FATAL [mirrormaker-producer-0] Producer thread failure due to  (kafka.tools.MirrorMaker$ProducerThread)

kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries.

at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)

at kafka.producer.Producer.send(Producer.scala:77)

at kafka.producer.OldProducer.send(BaseProducer.scala:62)

at kafka.tools.MirrorMaker$ProducerThread.run(MirrorMaker.scala:309)

[2015-06-04 14:19:16,055] FATAL [mirrormaker-producer-0] Producer thread exited abnormally, stopping the whole mirror maker. (kafka.tools.MirrorMaker$ProducerThread)

Is there any specifics I'm missing for Kafka Confluent ? This topic is build with the schema-registry and message in this topic are in Avro. Do you have some sample I can use to do this properly ?

I look around a bit for the stack and mostly it says you have issues with zookeeper. But when I'm using consumer/producer from my apps or kafka-avro-console-consumer/producer, everything works as expected with no zookeeper error.

Regards,

Nicolas PHUNG

[Ewen Cheslack-Postava]

LeaderNotAvailableException indicates that the the partition analytics-v2 doesn't currently have a leader, either because election is currently in progress or all the replicas are down. It looks like something is wrong with the destination cluster.

Can you try running kafka-topics.sh for that topic to see what state that partition is in?

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/72c81991-6d71-4119-8821-3ebbaa6039dc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Thanks,
Ewen

[Thomas R]

Hi Ewen,

Le jeudi 4 juin 2015 20:39:57 UTC+2, Ewen Cheslack-Postava a écrit :

LeaderNotAvailableException indicates that the the partition analytics-v2 doesn't currently have a leader, either because election is currently in progress or all the replicas are down. It looks like something is wrong with the destination cluster.

Can you try running kafka-topics.sh for that topic to see what state that partition is in?

Here is the output of the topic and its partitions:

kafka-topics  --zookeeper localhost:2181 --describe --topic analytics-v2
Topic:analytics-v2    PartitionCount:400    ReplicationFactor:2    Configs:
    Topic: analytics-v2    Partition: 0    Leader: 2    Replicas: 2,1    Isr: 2,1
    Topic: analytics-v2    Partition: 1    Leader: 1    Replicas: 1,2    Isr: 1,2
    Topic: analytics-v2    Partition: 2    Leader: 2    Replicas: 2,1    Isr: 2,1
    Topic: analytics-v2    Partition: 3    Leader: 1    Replicas: 1,2    Isr: 1,2
    Topic: analytics-v2    Partition: 4    Leader: 2    Replicas: 2,1    Isr: 2,1
    Topic: analytics-v2    Partition: 5    Leader: 1    Replicas: 1,2    Isr: 1,2
    Topic: analytics-v2    Partition: 6    Leader: 2    Replicas: 2,1    Isr: 2,1
    [...]
    Topic: analytics-v2    Partition: 390    Leader: 2    Replicas: 2,1    Isr: 2,1
    Topic: analytics-v2    Partition: 391    Leader: 1    Replicas: 1,2    Isr: 1,2
    Topic: analytics-v2    Partition: 392    Leader: 2    Replicas: 2,1    Isr: 2,1
    Topic: analytics-v2    Partition: 393    Leader: 1    Replicas: 1,2    Isr: 1,2
    Topic: analytics-v2    Partition: 394    Leader: 2    Replicas: 2,1    Isr: 2,1
    Topic: analytics-v2    Partition: 395    Leader: 1    Replicas: 1,2    Isr: 1,2
    Topic: analytics-v2    Partition: 396    Leader: 2    Replicas: 2,1    Isr: 2,1
    Topic: analytics-v2    Partition: 397    Leader: 1    Replicas: 1,2    Isr: 1,2
    Topic: analytics-v2    Partition: 398    Leader: 2    Replicas: 2,1    Isr: 2,1
    Topic: analytics-v2    Partition: 399    Leader: 1    Replicas: 1,2    Isr: 1,2

seems that partitions are ok and scaled between our two servers so we don't understand what we are missing.

Thomas

[Nicolas Phung]

Hi Ewen,

Have you got success with using the kafka-mirror-maker with Kafka confluent ? We don't need additionnal configuration to be compatible with the schema registry ?

Regards,
Nicolas PHUNG