What's the maximum size of each offset in a Kafka topic partition?

[Chaitra]

Hello,

1) I'm confused as to what the maximum size of each offset is? 

2) Suppose I have a producer that's reading continuously from a 10 Gb file, so will all the data from this file go to one offset or it'll be broken down into many offsets? If so, how are they broken down?

Thank you!

[Ewen Cheslack-Postava]

The offset is incremented for each record produced to Kafka. So the answer to your question depends on how your producer splits up that file. If it takes 1 byte at a time and converts it into a record, your offsets will increase by 10 billion for the 10Gb file. If you put all 10Gb into a single record you'll only increase the offset in Kafka by 1.

Fundamentally, the only maximum offset imposed by Kafka is that it has to be a 64-bit value. So it could be as large as LONG_MAX.

-Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/95a37628-d293-464a-b8a1-6f259a813c67%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.