Kafka Streams automatically created topics

[Andy Chambers]

Hey Folks,

It seems like a kafka streams app will automatically create "helper" topics (e.g. those used by reducers, and repartition nodes), with the correct partitions. Have I got that right? Anyone mind pointing me to the apache jira that resolved this so I can learn more about how it works? We had a work-around in place and I'd like to make sure I can safely strip out the work-around and just rely on kafka streams itself to create these topics.

Cheers,

Andy

[Matthias J. Sax]

Hi,

Yes, your understanding is correct. Some topics get created by Streams
automatically since the first version. Thus, no JIRA for this part (ie,
internal changelog topics to back up state stores).

Some more auto-topic creation was introduced here (for data
re-partitioning): https://issues.apache.org/jira/browse/KAFKA-3561

However, some part of this code got updated in the mean time -- so the
PR is not 100% up to date any more.

Be also aware of this issue:
https://issues.apache.org/jira/browse/KAFKA-4601

> We had a work-around in place and I'd like to make sure I can
> safely strip out the work-around and just rely on kafka streams itself
> to create these topics.

A work-around for what?


-Matthias

> --
> You received this message because you are subscribed to the Google
> Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to confluent-platf...@googlegroups.com
> <mailto:confluent-platf...@googlegroups.com>.
> To post to this group, send email to confluent...@googlegroups.com
> <mailto:confluent...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/confluent-platform/71e99fef-5d1c-447e-b093-712025d2f3ca%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/71e99fef-5d1c-447e-b093-712025d2f3ca%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.