kafka managed offset question

[kirkblo...@gmail.com]

Hopefully this is a quick question due to a PEBCAK :). I'd like to track offsets via the kafka offset manager, not zookeeper, since we are monitoring our kafka consumers using Burrow (plus I'd like to just get away from offsets in ZK anyway). Whenever I restart secor, it always starts reading from offset 0, and Burrow never shows a secor consumer in it's list.

Hunting through code, it looks like offsets are committed after a successful upload in Uploader.uploadFiles(), but I can only locate code in secor for committing offsets via zookeeper and can't find any reference in the KafkaClient code to actually commit offsets to the kafka offset managed topics. Is kafka topic offset tracking functionality working at the moment?

In secor.common.properties I've got the following set:

kafka.dual.commit.enabled=false
kafka.offsets.storage=kafka

Thanks in advance!

[kirkblo...@gmail.com]

FWIW I'm working off of 0.16-SNAPSHOT....

[Paula Ta-Shma]

I think this is a good question and am also interested in using the kafka offset manager instead of zookeeper. As far as I can tell Secor relies on zookeeper for offset management. How easy would it be to use Kafka's newer interface to update committed offsets at the granularity of individual partitions instead of writing to zookeeper directly ? In that case they would be stored wherever the Kafka offsets are configured to be stored. 

[kis...@wreally.com]

Did anybody find an answer to this question? I have the exact same doubt - I also went through the code and it seems like Secor does not use Kafka's offset management at all. Offset autocommit is disabled, and there are no manual commits as well. There might very well be a good reason behind this, but this was very surprising to me.

It would be great if one of the Secor devs can explain a little about this design decision.

Thanks,

KN.

[vijayanbala...@gmail.com]

Hi,
Even i am interested in managing offset in kafka than zookeeper . I am not able to find any reference on how to make this change. Please update here if anyone manage to implement this.

Thanks,
Vijayan B

[hc...@pinterest.com]

Secor does not use kafka offset mgmt out of the box, everything is committed to ZK.  Several reasons:

1. Secor was developed prior to kafka offset management feature was in place;

2. The primary reason for kafka to use its own internal topic for offset management is for large topic (e.g. a topic with a thousand partitions) management, in reality most people's topic does not have that many partitions;

3. We had used kafka's internal topic for offset mgmt in other projects, it was not easy to use.  You lose the visibility into what's stored there.  For ZK storage, there are so many ZK tools out there for you to browse the stored offsets (and make quick edit if you need to manually move the offset).  If you are using kafka's internal topic, every update to that topic is rolled up and compacted together to generate the final result, it's very hard to reason what's the current value and how to modify the value without writing some code;