Kafka consumer error after node failure

333 views
Skip to first unread message

ar...@alcide.io

unread,
Jul 30, 2017, 5:03:35 AM7/30/17
to Confluent Platform
Hi,

We're testing a node failure scenario, on a 3 instance kafka cluster, with replication factor of 2.

After deleting an instance, the consumer is constantly failing.

The consumer is using kafka streams to read messages

Thanks

This is the consumer failure log:

08:09:19.667 [ComponentsActivityEventsStream-608b9f05-0911-4b14-a1b1-37247747686a-StreamThread-2] INFO  o.a.k.s.p.i.StreamThread - stream-thread [ComponentsActivityEventsStream-608b9f05-0911-4b14-a1b1-37247747686a-StreamThread-2] Created active task 0_0 with assigned partitions [components-activity-0]

08:09:19.667 [ComponentsActivityEventsStream-608b9f05-0911-4b14-a1b1-37247747686a-StreamThread-2] INFO  o.a.k.s.p.i.StreamThread - stream-thread [ComponentsActivityEventsStream-608b9f05-0911-4b14-a1b1-37247747686a-StreamThread-2] partition assignment took 40075 ms.

        current active tasks: []

        current standby tasks: []

08:09:19.667 [ComponentsActivityEventsStream-608b9f05-0911-4b14-a1b1-37247747686a-StreamThread-2] ERROR o.a.k.c.c.i.ConsumerCoordinator - User provided listener org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener for group ComponentsActivityEventsStream failed on partition assignment

org.apache.kafka.streams.errors.StreamsException: Store ComponentsActivityStore's change log (ComponentsActivityEventsStream-ComponentsActivityStore-changelog) does not contain partition 0

        at org.apache.kafka.streams.processor.internals.StoreChangelogReader.validatePartitionExists(StoreChangelogReader.java:87)

        at org.apache.kafka.streams.processor.internals.ProcessorStateManager.register(ProcessorStateManager.java:165)

        at org.apache.kafka.streams.processor.internals.AbstractProcessorContext.register(AbstractProcessorContext.java:100)

        at org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore.init(RocksDBSegmentedBytesStore.java:110)

        at org.apache.kafka.streams.state.internals.ChangeLoggingSegmentedBytesStore.init(ChangeLoggingSegmentedBytesStore.java:72)

        at org.apache.kafka.streams.state.internals.MeteredSegmentedBytesStore.init(MeteredSegmentedBytesStore.java:65)

        at org.apache.kafka.streams.state.internals.RocksDBWindowStore.init(RocksDBWindowStore.java:95)

        at org.apache.kafka.streams.state.internals.CachingWindowStore.init(CachingWindowStore.java:69)

        at org.apache.kafka.streams.processor.internals.AbstractTask.initializeStateStores(AbstractTask.java:201)

        at org.apache.kafka.streams.processor.internals.StreamTask.<init>(StreamTask.java:140)

        at org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:1234)

        at org.apache.kafka.streams.processor.internals.StreamThread$TaskCreator.createTask(StreamThread.java:294)

        at org.apache.kafka.streams.processor.internals.StreamThread$AbstractTaskCreator.retryWithBackoff(StreamThread.java:254)

        at org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:1313)

        at org.apache.kafka.streams.processor.internals.StreamThread.access$1100(StreamThread.java:73)

        at org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener.onPartitionsAssigned(StreamThread.java:183)

        at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:265)

        at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:363)

        at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:310)

        at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:297)

        at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078)

        at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043)

        at org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:582)

        at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:553)

        at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:527)

08:09:19.673 [ComponentsActivityEventsStream-608b9f05-0911-4b14-a1b1-37247747686a-StreamThread-2] INFO  o.a.k.s.p.i.StreamThread - stream-thread [ComponentsActivityEventsStream-608b9f05-0911-4b14-a1b1-37247747686a-StreamThread-2] Shutting down

08:09:19.674 [ComponentsActivityEventsStream-608b9f05-0911-4b14-a1b1-37247747686a-StreamThread-2] INFO  o.a.k.c.p.KafkaProducer - Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.

08:09:19.681 [ComponentsActivityEventsStream-608b9f05-0911-4b14-a1b1-37247747686a-StreamThread-2] INFO  o.a.k.s.p.i.StreamThread - stream-thread [ComponentsActivityEventsStream-608b9f05-0911-4b14-a1b1-37247747686a-StreamThread-2] Removing all active tasks []

08:09:19.681 [ComponentsActivityEventsStream-608b9f05-0911-4b14-a1b1-37247747686a-StreamThread-2] INFO  o.a.k.s.p.i.StreamThread - stream-thread [ComponentsActivityEventsStream-608b9f05-0911-4b14-a1b1-37247747686a-StreamThread-2] Removing all standby tasks []

08:09:19.681 [ComponentsActivityEventsStream-608b9f05-0911-4b14-a1b1-37247747686a-StreamThread-2] INFO  o.a.k.s.p.i.StreamThread - stream-thread [ComponentsActivityEventsStream-608b9f05-0911-4b14-a1b1-37247747686a-StreamThread-2] Removing all standby tasks []

08:09:19.681 [ComponentsActivityEventsStream-608b9f05-0911-4b14-a1b1-37247747686a-StreamThread-2] INFO  o.a.k.s.p.i.StreamThread - stream-thread [ComponentsActivityEventsStream-608b9f05-0911-4b14-a1b1-37247747686a-StreamThread-2] Stream thread shutdown complete

08:09:19.681 [ComponentsActivityEventsStream-608b9f05-0911-4b14-a1b1-37247747686a-StreamThread-2] WARN  o.a.k.s.p.i.StreamThread - stream-thread [ComponentsActivityEventsStream-608b9f05-0911-4b14-a1b1-37247747686a-StreamThread-2] Unexpected state transition from ASSIGNING_PARTITIONS to DEAD.

08:09:19.681 [ComponentsActivityEventsStream-608b9f05-0911-4b14-a1b1-37247747686a-StreamThread-2] WARN  i.a.k.BaseEventsStream - uncaught exception in stream thread ComponentsActivityEventsStream

org.apache.kafka.streams.errors.StreamsException: stream-thread [ComponentsActivityEventsStream-608b9f05-0911-4b14-a1b1-37247747686a-StreamThread-2] Failed to rebalance.

        at org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:589)

        at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:553)

        at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:527)

Caused by: org.apache.kafka.streams.errors.StreamsException: Store ComponentsActivityStore's change log (ComponentsActivityEventsStream-ComponentsActivityStore-changelog) does not contain partition 0

        at org.apache.kafka.streams.processor.internals.StoreChangelogReader.validatePartitionExists(StoreChangelogReader.java:87)

        at org.apache.kafka.streams.processor.internals.ProcessorStateManager.register(ProcessorStateManager.java:165)

        at org.apache.kafka.streams.processor.internals.AbstractProcessorContext.register(AbstractProcessorContext.java:100)

        at org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore.init(RocksDBSegmentedBytesStore.java:110)

        at org.apache.kafka.streams.state.internals.ChangeLoggingSegmentedBytesStore.init(ChangeLoggingSegmentedBytesStore.java:72)

        at org.apache.kafka.streams.state.internals.MeteredSegmentedBytesStore.init(MeteredSegmentedBytesStore.java:65)

        at org.apache.kafka.streams.state.internals.RocksDBWindowStore.init(RocksDBWindowStore.java:95)

        at org.apache.kafka.streams.state.internals.CachingWindowStore.init(CachingWindowStore.java:69)

        at org.apache.kafka.streams.processor.internals.AbstractTask.initializeStateStores(AbstractTask.java:201)

        at org.apache.kafka.streams.processor.internals.StreamTask.<init>(StreamTask.java:140)

        at org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:1234)

        at org.apache.kafka.streams.processor.internals.StreamThread$TaskCreator.createTask(StreamThread.java:294)

        at org.apache.kafka.streams.processor.internals.StreamThread$AbstractTaskCreator.retryWithBackoff(StreamThread.java:254)

        at org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:1313)

        at org.apache.kafka.streams.processor.internals.StreamThread.access$1100(StreamThread.java:73)

        at org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener.onPartitionsAssigned(StreamThread.java:183)

        at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:265)

        at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:363)

        at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:310)

        at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:297)

        at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078)

        at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043)

        at org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:582)

        ... 2 common frames omitted

08:09:19.682 [pool-2-thread-1] WARN  i.a.k.BaseEventsStream - ComponentsActivityEventsStream - Kafka Error. Exiting application

08:09:19.683 [Thread-5] INFO  o.a.k.s.KafkaStreams - stream-client [ComponentsActivityEventsStream-608b9f05-0911-4b14-a1b1-37247747686a] State transition from REBALANCING to PENDING_SHUTDOWN.

08:09:19.684 [Thread-3] INFO  o.a.i.i.u.t.G - Invoking shutdown hook...

08:09:19.685 [Thread-4] INFO  o.a.k.s.KafkaStreams - stream-client [activity-processor-09d9172b-e87f-4396-80c1-c0a74d46f428] State transition from RUNNING to PENDING_SHUTDOWN.

08:09:19.685 [kafka-streams-close-thread] INFO  o.a.k.s.p.i.StreamThread - stream-thread [activity-processor-09d9172b-e87f-4396-80c1-c0a74d46f428-StreamThread-1] Informed thread to shut down

08:09:19.685 [kafka-streams-close-thread] INFO  o.a.k.s.p.i.StreamThread - stream-thread [activity-processor-09d9172b-e87f-4396-80c1-c0a74d46f428-StreamThread-1] State transition from RUNNING to PENDING_SHUTDOWN.

08:09:19.685 [kafka-streams-close-thread] INFO  o.a.k.s.p.i.StreamThread - stream-thread [ComponentsActivityEventsStream-608b9f05-0911-4b14-a1b1-37247747686a-StreamThread-2] Informed thread to shut down

08:09:19.686 [kafka-streams-close-thread] WARN  o.a.k.s.p.i.StreamThread - stream-thread [ComponentsActivityEventsStream-608b9f05-0911-4b14-a1b1-37247747686a-StreamThread-2] Unexpected state transition from DEAD to PENDING_SHUTDOWN.

08:09:19.686 [kafka-streams-close-thread] INFO  o.a.k.s.KafkaStreams - stream-client [ComponentsActivityEventsStream-608b9f05-0911-4b14-a1b1-37247747686a] Stopped Kafka Streams process.

08:09:19.686 [Thread-5] INFO  o.a.k.s.KafkaStreams - stream-client [ComponentsActivityEventsStream-608b9f05-0911-4b14-a1b1-37247747686a] State transition from PENDING_SHUTDOWN to NOT_RUNNING.



Reply all
Reply to author
Forward
0 new messages