Re: Streams not starting at all

[Matthias J. Sax]

Not sure. Maybe some timeout issue?

> [2017-11-25 00:39:37,911] INFO [GroupCoordinator 1001]: Member dev-gbohub-daily-d3c35ef2-003d-44d3-9cd0-53955e7c8ad5-StreamThread-1-consumer-d1b3ddd2-42d0-406b-94d2-077272f7c44f in group dev-gbohub-daily has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)

It seems that the thread does not report back to the broker.

> 16:42:59.357 [dev-gbohub-daily-e93fb046-9ca4-40cb-8a93-947e524b3005-StreamThread-1] DEBUG o.a.k.s.p.i.StreamPartitionAssignor - stream-thread [dev-gbohub-daily-e93fb046-9ca4-40cb-8a93-947e524b3005-StreamThread-1-consumer] Constructed client metadata {e93fb046-9ca4-40cb-8a93-947e524b3005=ClientMetadata{hostInfo=null, consumers=[dev-gbohub-daily-e93fb046-9ca4-40cb-8a93-947e524b3005-StreamThread-1-consumer-e12100bb-5a61-42d5-bd69-87bb96e8a316], state=[activeTasks: ([]) standbyTasks: ([]) assignedTasks: ([]) prevActiveTasks: ([]) prevAssignedTasks: ([]) capacity: 1]}} from the member subscriptions.

One thread does not get anything assigned. Maybe missing metadata? Does
the input topic exist?


You are starting a single Kafka Stream instance?


-Matthias



On 11/24/17 4:52 PM, Hyun Joon Seol wrote:
> Recently my kafka streams applications does not do anything when started.
>
> Whenever I start it, it just stops at 
> 16:42:55.908
> [dev-gbohub-daily-e93fb046-9ca4-40cb-8a93-947e524b3005-StreamThread-1]
> INFO  o.a.k.c.c.i.AbstractCoordinator - [Consumer
> clientId=dev-gbohub-daily-e93fb046-9ca4-40cb-8a93-947e524b3005-StreamThread-1-consumer,
> groupId=dev-gbohub-daily] (Re-)joining group
>
> The consumer group command first returns
> CALA-AM02046714:confluent-3.3.0 seolh001$ bin/kafka-consumer-groups
> --bootstrap-server $BROKERS --describe --group dev-gbohub-daily
> Note: This will only show information about consumers that use the Java
> consumer API (non-ZooKeeper-based consumers).
>
> Consumer group 'dev-gbohub-daily' has no active members.
>
> TOPIC                          PARTITION  CURRENT-OFFSET 
> LOG-END-OFFSET  LAG        CONSUMER-ID                                 
>      HOST                           CLIENT-ID
>
> and then a few moments later, it returns
> CALA-AM02046714:confluent-3.3.0 seolh001$ bin/kafka-consumer-groups
> --bootstrap-server $BROKERS --describe --group dev-gbohub-daily
> Note: This will only show information about consumers that use the Java
> consumer API (non-ZooKeeper-based consumers).
>
> Error: Consumer group 'dev-gbohub-daily' does not exist.
>
> The broker logs keep returning this:
> [2017-11-25 00:39:24,911] INFO [GroupCoordinator 1001]: Preparing to
> rebalance group dev-gbohub-daily with old generation 2
> (__consumer_offsets-7) (kafka.coordinator.group.GroupCoordinator)
> [2017-11-25 00:39:27,911] INFO [GroupCoordinator 1001]: Stabilized group
> dev-gbohub-daily generation 3 (__consumer_offsets-7)
> (kafka.coordinator.group.GroupCoordinator)
> Nov 25 00:39:27 ip-10-180-171-105 kafka: [2017-11-25 00:39:27,911] INFO
> [GroupCoordinator 1001]: Stabilized group dev-gbohub-daily generation 3
> (__consumer_offsets-7) (kafka.coordinator.group.GroupCoordinator)
> Nov 25 00:39:24 ip-10-180-171-105 kafka: [2017-11-25 00:39:24,911] INFO
> [GroupCoordinator 1001]: Preparing to rebalance group dev-gbohub-daily
> with old generation 2 (__consumer_offsets-7)
> (kafka.coordinator.group.GroupCoordinator)
> Nov 25 00:39:37 ip-10-180-171-105 kafka: [2017-11-25 00:39:37,911] INFO
> [GroupCoordinator 1001]: Preparing to rebalance group dev-gbohub-daily
> with old generation 3 (__consumer_offsets-7)
> (kafka.coordinator.group.GroupCoordinator)
> Nov 25 00:39:37 ip-10-180-171-105 kafka: [2017-11-25 00:39:37,911] INFO
> [GroupCoordinator 1001]: Group dev-gbohub-daily with generation 4 is now
> empty (__consumer_offsets-7) (kafka.coordinator.group.GroupCoordinator)
> Nov 25 00:39:37 ip-10-180-171-105 kafka: [2017-11-25 00:39:37,911] INFO
> [GroupCoordinator 1001]: Member
> dev-gbohub-daily-d3c35ef2-003d-44d3-9cd0-53955e7c8ad5-StreamThread-1-consumer-d1b3ddd2-42d0-406b-94d2-077272f7c44f
> in group dev-gbohub-daily has failed, removing it from the group
> (kafka.coordinator.group.GroupCoordinator)
> [2017-11-25 00:39:37,911] INFO [GroupCoordinator 1001]: Preparing to
> rebalance group dev-gbohub-daily with old generation 3
> (__consumer_offsets-7) (kafka.coordinator.group.GroupCoordinator)
> [2017-11-25 00:39:37,911] INFO [GroupCoordinator 1001]: Member
> dev-gbohub-daily-d3c35ef2-003d-44d3-9cd0-53955e7c8ad5-StreamThread-1-consumer-d1b3ddd2-42d0-406b-94d2-077272f7c44f
> in group dev-gbohub-daily has failed, removing it from the group
> (kafka.coordinator.group.GroupCoordinator)
>
> I'm attaching full debug logs for the Kafka Streams app. Can anyone
> guide me to the right path? I've spent a lot of time trying to figure
> out why this happens but I'm at a dead end it seems.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to confluent-platf...@googlegroups.com
> <mailto:confluent-platf...@googlegroups.com>.
> To post to this group, send email to confluent...@googlegroups.com
> <mailto:confluent...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/confluent-platform/3e363908-32d3-40ce-a05d-a40941de742a%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/3e363908-32d3-40ce-a05d-a40941de742a%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

[Hyun Joon Seol]

This was due to an input topic missing, if anybody landed here with a search! 

Thanks Matthias.

> an email to confluent-platform+unsub...@googlegroups.com
> <mailto:confluent-platform+unsub...@googlegroups.com>.