How to consume Kafka topic messages periodically. Avoid reading records that are added after you s

[anand reddy akidi]

Hi,

How to consume Kafka topic messages periodically. I would like start cron job every 15 mins to consume messages but It should consume only the messages which are added to the topic before starting the job ( avoid reading records that are added after you start consuming ).

Regards

Anand.

[Ewen Cheslack-Postava]

You can write a consumer application that uses the consumer's endOffsets API http://kafka.apache.org/0110/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#endOffsets(java.util.Collection) to determine the set of messages that are in Kafka at the start of the job and then stop processing once you've consumed up to that offset.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/0380fad8-daa5-4e6b-9b01-dfc0712a74bf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[anand reddy akidi]

Hi,

Could you please help me with some sample code or example?

Regards

Anand.

On Friday, July 28, 2017 at 2:08:51 PM UTC-6, Ewen Cheslack-Postava wrote:

You can write a consumer application that uses the consumer's endOffsets API http://kafka.apache.org/0110/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#endOffsets(java.util.Collection) to determine the set of messages that are in Kafka at the start of the job and then stop processing once you've consumed up to that offset.

On Fri, Jul 28, 2017 at 11:01 AM, anand reddy akidi <anandred...@gmail.com> wrote:

Hi,

How to consume Kafka topic messages periodically. I would like start cron job every 15 mins to consume messages but It should consume only the messages which are added to the topic before starting the job ( avoid reading records that are added after you start consuming ).

Regards

Anand.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.