Getting "Commit cannot be completed due to group rebalance" from MirrorMaker

2,311 views

Skip to first unread message

Stuart Wong

unread,

Mar 22, 2016, 6:28:50 PM3/22/16

to Confluent Platform

Hello again.

I'm using MirrorMaker from the Kafka 0.9.0 stack with the `--new-consumer` flag and getting the following:

Mar 22 22:08:39 myserver01.example.com sh[26005]: [2016-03-22 22:08:39,060] WARN block.on.buffer.full config is deprecated and will be removed soon. Please use max.block.ms (org.apache.kafka.clients.producer.KafkaProducer)

Mar 22 22:10:26 myserver01.example.com sh[26005]: [2016-03-22 22:10:26,075] ERROR Error ILLEGAL_GENERATION occurred while committing offsets for group mirrormaker-cor-1 (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)

Mar 22 22:10:26 myserver01.example.com sh[26005]: [2016-03-22 22:10:26,089] FATAL [mirrormaker-thread-0] Mirror maker thread failure due to (kafka.tools.MirrorMaker$MirrorMakerThread)

Mar 22 22:10:26 myserver01.example.com sh[26005]: org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed due to group rebalance

Mar 22 22:10:26 myserver01.example.com sh[26005]: at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle(ConsumerCoordinator.java:552)

Mar 22 22:10:26 myserver01.example.com sh[26005]: at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle(ConsumerCoordinator.java:493)

Mar 22 22:10:26 myserver01.example.com sh[26005]: at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:665)

Mar 22 22:10:26 myserver01.example.com sh[26005]: at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:644)

Mar 22 22:10:26 myserver01.example.com sh[26005]: at org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:167)

Mar 22 22:10:26 myserver01.example.com sh[26005]: at org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:133)

Mar 22 22:10:26 myserver01.example.com sh[26005]: at org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107)

Mar 22 22:10:26 myserver01.example.com sh[26005]: at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.onComplete(ConsumerNetworkClient.java:380)

Mar 22 22:10:26 myserver01.example.com sh[26005]: at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:288)

Mar 22 22:10:26 myserver01.example.com sh[26005]: at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:320)

Mar 22 22:10:26 myserver01.example.com sh[26005]: at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:213)

Mar 22 22:10:26 myserver01.example.com sh[26005]: at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:193)

Mar 22 22:10:26 myserver01.example.com sh[26005]: at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:163)

Mar 22 22:10:26 myserver01.example.com sh[26005]: at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.commitOffsetsSync(ConsumerCoordinator.java:358)

Mar 22 22:10:26 myserver01.example.com sh[26005]: at org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:942)

Mar 22 22:10:26 myserver01.example.com sh[26005]: at kafka.tools.MirrorMaker$MirrorMakerNewConsumer.commit(MirrorMaker.scala:546)

Mar 22 22:10:26 myserver01.example.com sh[26005]: at kafka.tools.MirrorMaker$.commitOffsets(MirrorMaker.scala:340)

Mar 22 22:10:26 myserver01.example.com sh[26005]: at kafka.tools.MirrorMaker$MirrorMakerThread.maybeFlushAndCommitOffsets(MirrorMaker.scala:438)

Mar 22 22:10:26 myserver01.example.com sh[26005]: at kafka.tools.MirrorMaker$MirrorMakerThread.run(MirrorMaker.scala:399)

Mar 22 22:10:26 myserver01.example.com sh[26005]: [2016-03-22 22:10:26,101] ERROR Error ILLEGAL_GENERATION occurred while committing offsets for group mirrormaker-cor-1 (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)

Mar 22 22:10:26 myserver01.example.com sh[26005]: Exception in thread "mirrormaker-thread-0" org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed due to group rebalance