Cleaning up KeyValueStore data

[Tianxiang Xiong]

After creating a persistent KeyValueStore, what's the best way to clean it up, i.e. remove its data? 

The documentation says:

Keep all key-value entries off-heap in a local database, although for durability all entries are recorded in a Kafka topic that can be read to restore the entries if they are lost.

But it doesn't say what topic the entries are stored in. Presumably cleaning up involves deleting the topic?

[Matthias J. Sax]

You can use bin/kafka-streams-application-reset to clean up application.

See
https://www.confluent.io/blog/data-reprocessing-with-kafka-streams-resetting-a-streams-application/

and

http://docs.confluent.io/current/streams/developer-guide.html#application-reset-tool


Topic are named like <applicationID>-<operatorName>-changelog


-Matthias


On 1/5/17 3:08 PM, 'Tianxiang Xiong' via Confluent Platform wrote:
> After creating a persistent KeyValueStore
> <https://kafka.apache.org/0100/javadoc/index.html?org/apache/kafka/streams/state/KeyValueStore.html>,

> what's the best way to clean it up, i.e. remove its data?
>
> The documentation says:
>
> Keep all key-value entries off-heap in a local database, although
> for durability all entries are recorded in a Kafka topic that can be
> read to restore the entries if they are lost.
>
>

> But it doesn't say /what/ topic the entries are stored in. Presumably

> cleaning up involves deleting the topic?
>
>

> --
> You received this message because you are subscribed to the Google
> Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to confluent-platf...@googlegroups.com
> <mailto:confluent-platf...@googlegroups.com>.
> To post to this group, send email to confluent...@googlegroups.com
> <mailto:confluent...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/confluent-platform/94f8b8ca-2fdf-4eb2-af7a-0d38e5516dd3%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/94f8b8ca-2fdf-4eb2-af7a-0d38e5516dd3%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

[Tianxiang Xiong]

I'll try the tool, thanks.

Does deleting the topic backing a KeyValueStore serve to effective clean it up? Or is there data in other places that needs to be gotten rid off?

On Thursday, 5 January 2017 15:59:27 UTC-8, Matthias J. Sax wrote:

You can use bin/kafka-streams-application-reset to clean up application.

See
https://www.confluent.io/blog/data-reprocessing-with-kafka-streams-resetting-a-streams-application/

and

http://docs.confluent.io/current/streams/developer-guide.html#application-reset-tool


Topic are named like <applicationID>-<operatorName>-changelog


-Matthias


On 1/5/17 3:08 PM, 'Tianxiang Xiong' via Confluent Platform wrote:
> After creating a persistent KeyValueStore
> <https://kafka.apache.org/0100/javadoc/index.html?org/apache/kafka/streams/state/KeyValueStore.html>,
> what's the best way to clean it up, i.e. remove its data?
>
> The documentation says:
>
>     Keep all key-value entries off-heap in a local database, although
>     for durability all entries are recorded in a Kafka topic that can be
>     read to restore the entries if they are lost.
>
>
> But it doesn't say /what/ topic the entries are stored in. Presumably
> cleaning up involves deleting the topic?
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to confluent-platform+unsub...@googlegroups.com
> <mailto:confluent-platform+unsub...@googlegroups.com>.

[Tianxiang Xiong]

It seems that I can get rid of the state store's data by deleting the application's local state in /tmp/kafka-streams/<application.id>/. Should this be the case? Is KeyValueStore data global data or local data?

[Matthias J. Sax]

Just deleting the local state directory is not sufficient, because it
will be re-created on restart using the backup changelog topic (that is
the ground of truth -- the state is actually just a cache for the
changelog).

Thus, you also need to delete the corresponding changelog topic.

bin/kafka-streams-application-reset does all this for you.


-Matthias

> > an email to confluent-platf...@googlegroups.com
> > <mailto:confluent-platf...@googlegroups.com>.

> > To post to this group, send email to
> confluent...@googlegroups.com
> > <mailto:confluent...@googlegroups.com>.
> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/confluent-platform/94f8b8ca-2fdf-4eb2-af7a-0d38e5516dd3%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/94f8b8ca-2fdf-4eb2-af7a-0d38e5516dd3%40googlegroups.com>
>
> >

> <https://groups.google.com/d/msgid/confluent-platform/94f8b8ca-2fdf-4eb2-af7a-0d38e5516dd3%40googlegroups.com?utm_medium=email&utm_source=footer

> <https://groups.google.com/d/msgid/confluent-platform/94f8b8ca-2fdf-4eb2-af7a-0d38e5516dd3%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
> > For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to confluent-platf...@googlegroups.com
> <mailto:confluent-platf...@googlegroups.com>.

> To post to this group, send email to confluent...@googlegroups.com
> <mailto:confluent...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/confluent-platform/942fa3c9-3353-43c7-ac17-4726f0d8d615%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/942fa3c9-3353-43c7-ac17-4726f0d8d615%40googlegroups.com?utm_medium=email&utm_source=footer>.