Log compaction for a specific topic
94 views
Skip to first unread message
Marek Svitok
Oct 7, 2016, 9:48:41 AM10/7/16
to Confluent Platform
Hi all.
I've read about log compaction here: https://cwiki.apache.org/confluence/display/KAFKA/Log+Compaction and here http://kafka.apache.org/documentation.html#compaction
and it says that it is possible to enabled log compaction for a specific topic. How can I enable log compaction while defining my KStream?
Thanks,
Marek.
I've read about log compaction here: https://cwiki.apache.org/confluence/display/KAFKA/Log+Compaction and here http://kafka.apache.org/documentation.html#compaction
and it says that it is possible to enabled log compaction for a specific topic. How can I enable log compaction while defining my KStream?
Thanks,
Marek.
Marek Svitok
Oct 7, 2016, 9:51:34 AM10/7/16
to Confluent Platform
Or maybe more precise question would be: are the internal topics - made by KTable compacted by default?
Thanks.
Dňa piatok, 7. októbra 2016 15:48:41 UTC+2 Marek Svitok napísal(-a):
Thanks.
Dňa piatok, 7. októbra 2016 15:48:41 UTC+2 Marek Svitok napísal(-a):
Marek Svitok
Oct 7, 2016, 10:16:36 AM10/7/16
to Confluent Platform
Ok I have an answer found here : http://www.confluent.io/blog/distributed-real-time-joins-and-aggregations-on-user-activity-events-using-kafka-streams/
KTables are backed by state stores in Kafka Streams. These state stores are local to your application (more precisely: they are local to the instances of your application, of which there can be one or many), which means that interacting with these state stores does not require talking over the network, so read and write operations are very fast. For the sake of brevity we will not go into the technical details here how state stores are partitioned across the instances of your application to achieve scalability, or how Kafka Streams ensures that these local state stores can be fully recovered in the face of machine failures to prevent data loss (hint: state stores are but tables, and any changes to them can be captured in a changelog stream, which can be persisted to Kafka in log-compacted topics, from which they can be restored).
Dňa piatok, 7. októbra 2016 15:51:34 UTC+2 Marek Svitok napísal(-a):
KTables are backed by state stores in Kafka Streams. These state stores are local to your application (more precisely: they are local to the instances of your application, of which there can be one or many), which means that interacting with these state stores does not require talking over the network, so read and write operations are very fast. For the sake of brevity we will not go into the technical details here how state stores are partitioned across the instances of your application to achieve scalability, or how Kafka Streams ensures that these local state stores can be fully recovered in the face of machine failures to prevent data loss (hint: state stores are but tables, and any changes to them can be captured in a changelog stream, which can be persisted to Kafka in log-compacted topics, from which they can be restored).
Dňa piatok, 7. októbra 2016 15:51:34 UTC+2 Marek Svitok napísal(-a):
Michael Noll
Oct 10, 2016, 6:35:40 AM10/10/16
to confluent...@googlegroups.com
Yes, they are compacted by default.
--
--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/3dd93112-1f2d-46e8-bcf1-9483b6c3f1a1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
Michael G. Noll
Product Manager | Confluent
Reply all
Reply to author
Forward
0 new messages