How to consume messages from a specific offset using Rest Consumer

[Nguyen Cao]

Hi all,

I am looking into the documentation of Kafka REST Proxy (http://docs.confluent.io/2.0.0/kafka-rest/docs/intro.html). There is one place saying use low-level read operations to retrieve messages at specific offsets. However, in the API documentation I could not find instruction on how to include offset parameters in the request. Am I missing something? 

Another related use case is that if somehow our consumers hang out/die for a few hours, instead of starting processing from the beginning, can I specify a specific timeframe (for example within 24hours, etc.) in the REST consumes to get these unprocessed messages?

Many thanks,

Nguyen

 

[Alex Loddengaard]

Hi Nguyen,

You can specify the offset to start consuming at. See here:

http://docs.confluent.io/2.0.0/kafka-rest/docs/api.html#get--topics-(string-topic_name)-partitions-(int-partition_id)-messages?offset=(int)[&count=(int)]

Note that the above API call is per-partition. If you have more than one partition, you'll need to make this call for each partition (likely in parallel, from a different consumer -- partitions exist for parallelism).

Also, you may find this API valuable:

http://docs.confluent.io/2.0.0/kafka-rest/docs/api.html#post--consumers-(string-group_name)

In particular, the auto.offset.reset configuration. "largest" means the most recent message will be consumed first. "smallest" means the earliest message will be consumed first. You can read more about this here:

http://mail-archives.apache.org/mod_mbox/kafka-users/201504.mbox/%3CCAE1jLMM0EiTTaeP5ruRMQcQ...@mail.gmail.com%3E

I hope this answers your question.

Alex

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/d55fa551-e2dd-4ba1-9e01-c5eb1529976c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nguyen Cao]

Many thanks, Alex. It's just what I am looking for. Still surprised I have not found it before.

Nguyen