Kafka Stream KTable -> Initializing State Stores help.

[Adam K]

Hello,

New to Kafka Streams. We've created two apps that create streams, one that uses a StateStore explicitiy -- not within a KTable. 

The KTable stream app seems to have no issues with restarts, The one with the StateStore seems to have a terrible time getting Initialized. It often seems to timeout and retry again. I'd appreciate any pointers as to what is going on. How can I stop the process from taking so long to initialize???

Thanks!

[StreamThread-2] INFO org.apache.kafka.streams.processor.internals.assignment.TaskAssignor - stream-thread [StreamThread-2] Assigned with: {ee3a32ad-6885-4d76-9793-73c0874b766a=[activeTasks: ([0_0, 0_1, 0_2, 0_3]) assignedTasks: ([0_0, 0_1, 0_2, 0_3]) prevActiveTasks: ([]) prevAssignedTasks: ([]) capacity: 2.0 cost: 2.0]}

[StreamThread-2] INFO org.apache.kafka.streams.processor.internals.StreamPartitionAssignor - stream-thread [StreamThread-2] Completed validating internal topics in partition assignor

[StreamThread-2] INFO org.apache.kafka.streams.processor.internals.StreamPartitionAssignor - stream-thread [StreamThread-2] Completed validating internal topics in partition assignor

[StreamThread-2] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Successfully joined group redsky-test with generation 17

[StreamThread-1] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Successfully joined group redsky-test with generation 17

[StreamThread-2] INFO org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Setting newly assigned partitions [redsky-new-pdp-part-deux3-2, redsky-new-pdp-part-deux3-0] for group redsky-test

[StreamThread-2] INFO org.apache.kafka.streams.processor.internals.StreamThread - stream-thread [StreamThread-2] New partitions [[redsky-new-pdp-part-deux3-2, redsky-new-pdp-part-deux3-0]] assigned at the end of consumer rebalance.

[StreamThread-1] INFO org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Setting newly assigned partitions [redsky-new-pdp-part-deux3-3, redsky-new-pdp-part-deux3-1] for group redsky-test

[StreamThread-1] INFO org.apache.kafka.streams.processor.internals.StreamThread - stream-thread [StreamThread-1] New partitions [[redsky-new-pdp-part-deux3-3, redsky-new-pdp-part-deux3-1]] assigned at the end of consumer rebalance.

[StreamThread-1] INFO org.apache.kafka.streams.processor.internals.StreamThread - stream-thread [StreamThread-1] Creating active task 0_1 with assigned partitions [[redsky-new-pdp-part-deux3-1]]

[StreamThread-2] INFO org.apache.kafka.streams.processor.internals.StreamThread - stream-thread [StreamThread-2] Creating active task 0_0 with assigned partitions [[redsky-new-pdp-part-deux3-0]]

[StreamThread-2] INFO org.apache.kafka.streams.processor.internals.StreamTask - task [0_0] Initializing state stores

[StreamThread-1] INFO org.apache.kafka.streams.processor.internals.StreamTask - task [0_1] Initializing state stores

[Matthias J. Sax]

Do you know about standby tasks? Those should help

http://docs.confluent.io/current/streams/architecture.html#fault-tolerance


-Matthias

> --
> You received this message because you are subscribed to the Google
> Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to confluent-platf...@googlegroups.com
> <mailto:confluent-platf...@googlegroups.com>.
> To post to this group, send email to confluent...@googlegroups.com
> <mailto:confluent...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/confluent-platform/8ba0ebbc-fbc8-4fe3-8771-b59d3485dde4%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/8ba0ebbc-fbc8-4fe3-8771-b59d3485dde4%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.