Difference between connect-offsets and __consumer_offsets?

[Renukaradhya H D]

Hi,

I am running Kafka-Connect in distributed mode.

I need to know where my consumer group(Sink Connector) offsets are stored?

I also want to know why we have two topics connect-offsets and __consumer_offsets for storing offset?

Please let me know the differences.

Regards,

Aradhya

[Gwen Shapira]

If you have a sink connector and it commits offsets normally to Kafka
(not all of them do - HDFS connector writes offsets to HDFS), it will
use __consumer_offsets topic.

connect-offsets topic is used for source connectors to keep track of
"logical offsets" in the source system - for example IDs in database
tables, location in files, etc.

Hope this helps.

> --
> You received this message because you are subscribed to the Google Groups
> "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to confluent-platf...@googlegroups.com.
> To post to this group, send email to confluent...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/confluent-platform/037a59be-33eb-4c72-be29-18e469265b47%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

[Renukaradhya H D]

 It makes sense.

Thanks Gwen.

On Tuesday, February 28, 2017 at 3:35:16 AM UTC+5:30, Gwen Shapira wrote:

If you have a sink connector and it commits offsets normally to Kafka
(not all of them do - HDFS connector writes offsets to HDFS), it will
use __consumer_offsets topic.

connect-offsets topic is used for source connectors to keep track of
"logical offsets" in the source system - for example IDs in database
tables, location in files, etc.

Hope this helps.

On Mon, Feb 27, 2017 at 4:08 AM, Renukaradhya H D
<aradhya...@gmail.com> wrote:
> Hi,
> I am running Kafka-Connect in distributed mode.
>
> I need to know where my consumer group(Sink Connector) offsets are stored?
>
> I also want to know why we have two topics connect-offsets and
> __consumer_offsets for storing offset?
>
> Please let me know the differences.
>
> Regards,
> Aradhya
>
> --
> You received this message because you are subscribed to the Google Groups
> "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to confluent-platform+unsub...@googlegroups.com.

[Yeshwanth Kumar]

Hello, 

Can you point me where exactly the offset logic is applied in any sink connectors? Little curious on how offset is applied and then it continues from that specific point.

Thanks,
Yesh

[Ewen Cheslack-Postava]

Sink connectors use the normal __consumer_offsets topic. SinkTasks automatically have their offsets reloaded using the normal consumer mechanisms (so there's not really code to point to beyond https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java#L378-L401 which is where the consumer is constructed and defaults to reset offsets to "earliest"). If an implementation wants to override that behavior, they can invoke one of the methods to reset offsets when they are being initialized, either http://kafka.apache.org/0102/javadoc/org/apache/kafka/connect/sink/SinkTaskContext.html#offset(java.util.Map) or http://kafka.apache.org/0102/javadoc/org/apache/kafka/connect/sink/SinkTaskContext.html#offset(org.apache.kafka.common.TopicPartition,%20long).

-Ewen

 It makes sense.

> email to confluent-platform+unsubscribe@googlegroups.com.

> To post to this group, send email to confluent...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/confluent-platform/037a59be-33eb-4c72-be29-18e469265b47%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent-platform@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/ac723d00-12cc-4f00-8c2a-f093a847ef61%40googlegroups.com.