Kafka Current offset is not getting updated for the consumer group.

[Gopa R]

We use KIS for druid ingestion. I'm noticing the kafka current offset is not getting updated for the consumer group but in the overload log I could see the kafka offset is getting reset and data is getting loaded as expected. No error in any of the logs.

Not sure what could be the problem.

Thanks in advance.
 

[Gopa R]

I saw comments "druid kafka indexing task keeps topic partition offset by itself,  don't commit it to kafka, Therefore, you can't get information from kafka." Is this true?

[Peter Marshall]

Hi!  I'm not sure I understand the issue .... but maybe it will help to know that Druid is just a sink - it is a normal consumer, subscribing to the stream.  It records where it is in the stream in its own data - this offset is updated safely to guarantee exactly once ingestion.  You don't have to do anything in particular...

[Abraham Sultan]

Hi Peter, do you know where does Druid record where it is in the stream?

[Peter Marshall]

Hey Abraham - sorry for the late reply.

I've been reliably informed that Druid stores the committed offsets (read and published segments) in the Metadata Database (druid_dataSources?).

On each iteration of the supervisor's run loop, the supervisor fetches the list of partitions from Kafka and determines the starting offset for each partition (either based on the last processed offset if continuing, or starting from the beginning or ending of the stream if this is a new topic).