What triggers creation of __consumer_offsets topic?

[Alfred Landrum]

While I'm currently working on using librdkafka, I believe this question is relevant for any third party client due to reading this exchange:

https://github.com/Shopify/sarama/issues/227

I'm testing with a single 0.8.2.1 broker. When I start the broker with no prior state (so totally fresh zookeeper & no topics), and I use librdkafka to send an OffsetFetchRequest for a test topic & consumer group, it receives the 'not coordinator for consumer' error.

If I then use the kafka-console-producer & kafka-console-consumer to push & pull data using a different topic and consumer group (specifying "offsets.storage=kafka"), I see that the __consumer_offsets topic has been created.  I can then issue a OffsetFetchRequest with the original topic & group, I see the expected reply of success with an offset of -1.

So how is the consumer api triggering __consumer_offsets creation? 

[Willem van Bergen]

Based on my experiments today (see https://github.com/Shopify/sarama/pull/411) the very first consumer management API request you do, will trigger the topic to be created so it can store offsets. While this is in progress, it returns `ConsumerCoordinatorNotAvailable`. On a Travis VM this takes about 4 seconds. Once the topic is cerated, all subsequent requests will be successful.

I deal with this by just waiting a bit after receiving ConsumerCoordinatorNotAvailable, and trying again.

Willem

--
You received this message because you are subscribed to the Google Groups "kafka-clients" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kafka-client...@googlegroups.com.
To post to this group, send email to kafka-...@googlegroups.com.
Visit this group at http://groups.google.com/group/kafka-clients.
To view this discussion on the web visit https://groups.google.com/d/msgid/kafka-clients/0643fdd2-641e-49eb-bd7e-47f9811b5492%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alfred Landrum]

Thanks for the tip. So I let my test program run for a bit, sending nothing but offset fetch requests, and the broker never created __consumer_topics.  I then tried sending a consumer metadata request, and that did lead to the __consumer_topics topic showing.

[Dana Powers]

also note that this is for kafka-backed offsets tracking, which is the new OffsetFetch/Commit api v1 (and above). api v0 will continue to use zookeeper-backed offsets [until deprecated!] and does not use a ConsumerCoordinator.

-Dana

To view this discussion on the web visit https://groups.google.com/d/msgid/kafka-clients/22b0ef35-5242-4231-9f4b-b51205a5bdf9%40googlegroups.com.