Kafka consumer stuck in balancing

[Zhuo Chen]

Hi Guys,

We are running some consumers for topic "QUATTRO.TM.DISCOUNTING.SYNC". It seems like they are always stuck with balancing and it doesn't show any offset info either. Then I ran a local consumer to the same topic and wanted to invoke the balance again to make it work but failed. The log showed it wan't assign to any partition. A few days ago I had the same problem but the describe group command at least showed the offset and lag, but for this time, nothing. 

Could anyone help with it? Thanks in advance.

Describe group:

PS C:\Users\ZC90764\kafka_2.11-1.1.0\bin\windows> .\kafka-consumer-groups.bat --bootstrap-server orsxrsnaappr01:9092,ors

xrsnaappr02:9092,orsxrsnaappr03:9092 --describe --group QUATTRO.TM.DISCOUNTING.SYNC

Note: This will not show information about old Zookeeper-based consumers.

Warning: Consumer group 'QUATTRO.TM.DISCOUNTING.SYNC' is rebalancing.

Local consumer log:

54844 [Thread-20-DiscountingSyncKafkaSpout-executor[2 2]] INFO  o.a.k.c.c.i.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=QUATTRO.TM.DISCOUNTING.SYNC] Discovered group coordinator lnyratxrsap1p.nam.nsroot.net:9092 (id: 2147483646 rack: null)

54861 [Thread-20-DiscountingSyncKafkaSpout-executor[2 2]] INFO  o.a.k.c.c.i.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=QUATTRO.TM.DISCOUNTING.SYNC] Revoking previously assigned partitions []

54861 [Thread-20-DiscountingSyncKafkaSpout-executor[2 2]] INFO  o.a.s.k.s.KafkaSpout - Partitions revoked. [consumer-group=QUATTRO.TM.DISCOUNTING.SYNC, consumer=org.apache.kafka.clients.consumer.KafkaConsumer@77fa9ee, topic-partitions=[]]

54861 [Thread-20-DiscountingSyncKafkaSpout-executor[2 2]] INFO  o.a.k.c.c.i.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=QUATTRO.TM.DISCOUNTING.SYNC] (Re-)joining group

Describe topic:

Topic:170484.PROD.TM.SNAPSHOT.TRIGGER   PartitionCount:6        ReplicationFactor:6     Configs:

        Topic: 170484.PROD.TM.SNAPSHOT.TRIGGER  Partition: 0    Leader: 4       Replicas: 4,5,6,1,2,3   Isr: 5,1,6,2,3,4

        Topic: 170484.PROD.TM.SNAPSHOT.TRIGGER  Partition: 1    Leader: 5       Replicas: 5,6,1,2,3,4   Isr: 5,1,6,2,3,4

        Topic: 170484.PROD.TM.SNAPSHOT.TRIGGER  Partition: 2    Leader: 6       Replicas: 6,1,2,3,4,5   Isr: 5,1,6,2,3,4

        Topic: 170484.PROD.TM.SNAPSHOT.TRIGGER  Partition: 3    Leader: 1       Replicas: 1,2,3,4,5,6   Isr: 5,1,6,2,3,4

        Topic: 170484.PROD.TM.SNAPSHOT.TRIGGER  Partition: 4    Leader: 2       Replicas: 2,3,4,5,6,1   Isr: 5,1,6,2,3,4

        Topic: 170484.PROD.TM.SNAPSHOT.TRIGGER  Partition: 5    Leader: 3       Replicas: 3,4,5,6,1,2   Isr: 5,1,6,2,3,4