How to access state store

Shannon Ma

unread,

Nov 22, 2016, 1:32:25 PM11/22/16

to Confluent Platform

Hi,

I have been trying to have state store working, in my stream1 i save to the store which is working. Within stream1 i cannot access the store. So I put stream1 to a topic, and start another stream, stream2 to read from the topic, in the logic i want to access the store, i am getting

Exception in thread "StreamThread-2" org.apache.kafka.streams.errors.StreamsException: Exception caught in process. taskId=0_0, processor=KSTREAM-SOURCE-0000000000, topic=TXN_DATA_P_INT, partition=0, offset=0

at org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:200)

at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:436)

at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:242)

Caused by: org.apache.kafka.streams.errors.InvalidStateStoreException: the state store, txnwindowstore, may have migrated to another instance.

at org.apache.kafka.streams.state.internals.StreamThreadStateStoreProvider.stores(StreamThreadStateStoreProvider.java:42)

at org.apache.kafka.streams.state.internals.QueryableStoreProvider.getStore(QueryableStoreProvider.java:46)

at org.apache.kafka.streams.KafkaStreams.store(KafkaStreams.java:378)

at com.etcc.demo.RtTxnDuplicateIndentifier$4.apply(RtTxnDuplicateIndentifier.java:348)

at com.etcc.demo.RtTxnDuplicateIndentifier$4.apply(RtTxnDuplicateIndentifier.java:339)

at org.apache.kafka.streams.kstream.internals.KStreamFlatMapValues$KStreamFlatMapValuesProcessor.process(KStreamFlatMapValues.java:41)

at org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:82)

at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:204)

at org.apache.kafka.streams.kstream.internals.KStreamMap$KStreamMapProcessor.process(KStreamMap.java:43)

at org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:82)

at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:204)

at org.apache.kafka.streams.processor.internals.SourceNode.process(SourceNode.java:66)

at org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:181)

Thanks

Shannon

Matthias J. Sax

unread,

Nov 22, 2016, 2:36:51 PM11/22/16

to confluent...@googlegroups.com

If you access stores, you have to keep in mind the the overall state is
shared over all running Streams instances and you local state only
contains data for some keys (ie, partitions assigned to this instance).
Furthermore, if you start new instances or stop running instanced (or
failure) a rebalance can happen migrating store to different instances.

> Caused by: org.apache.kafka.streams.errors.InvalidStateStoreException:
> the state store, txnwindowstore, may have migrated to another instance.

Thus, you meta data about which keys are located at what instance can
change. You need to refresh your meta data to recover from this error
and try to access the store afterwards.

Check out
http://docs.confluent.io/current/streams/developer-guide.html#interactive-queries
and
https://www.confluent.io/blog/unifying-stream-processing-and-interactive-queries-in-apache-kafka/
and
https://github.com/confluentinc/examples/blob/3.1.x/kafka-streams/src/main/java/io/confluent/examples/streams/interactivequeries/kafkamusic/KafkaMusicExample.java

-Matthias

> --
> You received this message because you are subscribed to the Google
> Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to confluent-platf...@googlegroups.com
> <mailto:confluent-platf...@googlegroups.com>.
> To post to this group, send email to confluent...@googlegroups.com
> <mailto:confluent...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/confluent-platform/8f5b51f5-ec51-4fc4-92b1-8763c6c4ed99%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/8f5b51f5-ec51-4fc4-92b1-8763c6c4ed99%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

signature.asc

Shannon Ma

unread,

Nov 22, 2016, 2:52:13 PM11/22/16

to Confluent Platform

Right now I only have one instance.

--->

You need to refresh your meta data to recover from this error
and try to access the store afterwards.

--->

How do i refresh metadata?

Matthias J. Sax

unread,

Nov 22, 2016, 3:03:04 PM11/22/16

to confluent...@googlegroups.com

If you have only one running instance, I assume you are not dealing with
store meta data so there is nothing to refresh at this point.

However, the error might occur during bootstrapping, too. This means,
that you try to query the store, before it is available for querying.
Currently, you can only catch the exception and retry until the store
gets online. This is not a nice solution, but right now there is no API
to check if a store is ready or not.

-Matthias

> --
> You received this message because you are subscribed to the Google
> Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to confluent-platf...@googlegroups.com
> <mailto:confluent-platf...@googlegroups.com>.
> To post to this group, send email to confluent...@googlegroups.com
> <mailto:confluent...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/confluent-platform/cf7d423f-9227-40a1-abad-26058aac08fe%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/cf7d423f-9227-40a1-abad-26058aac08fe%40googlegroups.com?utm_medium=email&utm_source=footer>.

signature.asc

Shannon Ma

unread,

Nov 23, 2016, 10:49:12 AM11/23/16

to Confluent Platform

Thanks Matthias, it is working now.

I have couple questions

1, store.fetch(), i dont need to give exact time range (window start/end boundary), all the windows fall between start/end will return, right?

2, windowed store will be gone after retention, right? is it deleted from db?

3, where is the db file? can i delete if i need to restart fresh?

Thanks

Shannon

Damian Guy

unread,

Nov 23, 2016, 12:05:37 PM11/23/16

to Confluent Platform

Hi Shannon,

1. Yes that is correct

2. The window store is actually multiple RocksDB (Segments) stores on disk. Old windows are automatically removed when a Segment expires, i.e., the stores are closed and the data is removed from disk. Segment duration is calculated as Math.max(retentionPeriod, (numSegments - 1), MIN_SEGMENT_INTERVAL); Where MIN_SEGMENT_INTERVAL is 60 seconds.

3. The DB file is stored under the state directory as specified by StreamsConfig.STATE_DIR_CONFIG - the default is /tmp/kafka-streams. You can delete the data under here if you wish to restart from the beginning. You want to make sure you have shutdown your application first, though. You can also use the reset-tool, described here: https://www.confluent.io/blog/data-reprocessing-with-kafka-streams-resetting-a-streams-application/

Thanks,

Damian

--

You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/44695125-18d8-4834-aac7-ff303fcd37d3%40googlegroups.com.

Shannon Ma

unread,

Nov 23, 2016, 12:33:47 PM11/23/16

to Confluent Platform

Thanks, one more question on the window, if i have a 5 hour window with retention of 10 Hours, so i have two 5 H windows.Is it based on current time (the first time)?

So if current time is 11 am, then we have two windows 6-11 and 1-6, is that right?

Michael Noll

unread,

Nov 23, 2016, 2:04:18 PM11/23/16

to confluent...@googlegroups.com

FYI: If you are using Confluent Platform, then the state directory defaults to `/var/lib/kafka-streams` -- cf. http://docs.confluent.io/current/streams/developer-guide.html#optional-configuration-parameters.

On Wed, Nov 23, 2016 at 6:33 PM, Shannon Ma <shan...@gmail.com> wrote:

Thanks, one more question on the window, if i have a 5 hour window with retention of 10 Hours, so i have two 5 H windows.Is it based on current time (the first time)?

So if current time is 11 am, then we have two windows 6-11 and 1-6, is that right?

--

You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/535365a3-9467-4061-9ac8-8e2cbafaa4ee%40googlegroups.com.

Matthias J. Sax

unread,

Nov 23, 2016, 4:43:55 PM11/23/16

to confluent...@googlegroups.com

One thing to add about fetch()

They query you issue returns all windows with:
fromTime <= WindowStartTime <= toTime

WindowEndTime is not considered in fetch()

-Matthias

> <mailto:confluent-platf...@googlegroups.com>.

> To post to this group, send email to
> confluent...@googlegroups.com

> <mailto:confluent...@googlegroups.com>.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/confluent-platform/44695125-18d8-4834-aac7-ff303fcd37d3%40googlegroups.com

> <https://groups.google.com/d/msgid/confluent-platform/44695125-18d8-4834-aac7-ff303fcd37d3%40googlegroups.com?utm_medium=email&utm_source=footer>.

> For more options, visit https://groups.google.com/d/optout.
>

> --
> You received this message because you are subscribed to the Google
> Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to confluent-platf...@googlegroups.com

> <mailto:confluent-platf...@googlegroups.com>.

> To post to this group, send email to confluent...@googlegroups.com

> <mailto:confluent...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/confluent-platform/CAJikTEUU2Gadn-6Xd_GKjCU1_RBnXP%3D-cxPmMYOWYAisP2N4Vg%40mail.gmail.com
> <https://groups.google.com/d/msgid/confluent-platform/CAJikTEUU2Gadn-6Xd_GKjCU1_RBnXP%3D-cxPmMYOWYAisP2N4Vg%40mail.gmail.com?utm_medium=email&utm_source=footer>.

signature.asc

Shannon Ma

unread,

Nov 30, 2016, 11:37:59 AM11/30/16

to Confluent Platform

Thanks, can you answer my question on the window time/range, also I like to know if i stop the stream for sometime (say couple hours), how does it affect the window store.

> send an email to confluent-platform+unsub...@googlegroups.com
> <mailto:confluent-platform+unsub...@googlegroups.com>.

> To post to this group, send email to
> confluent...@googlegroups.com
> <mailto:confluent...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/confluent-platform/44695125-18d8-4834-aac7-ff303fcd37d3%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/44695125-18d8-4834-aac7-ff303fcd37d3%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to confluent-platform+unsub...@googlegroups.com
> <mailto:confluent-platform+unsub...@googlegroups.com>.

Matthias J. Sax

unread,

Nov 30, 2016, 3:48:35 PM11/30/16

to confluent...@googlegroups.com

What do you mean by "stop the stream". Do you mean that ther is no new
input data (ie, you stop an external Producer) or do you mean that you
stop your Kafka Streams application?

-Matthias

> <javascript:>

> > <mailto:shan...@gmail.com <javascript:>>> wrote:
> >
> > Thanks Matthias, it is working now.
> >
> > I have couple questions
> >
> > 1, store.fetch(), i dont need to give exact time range (window
> > start/end boundary), all the windows fall between start/end will
> > return, right?
> >
> > 2, windowed store will be gone after retention, right? is it
> deleted
> > from db?
> >
> > 3, where is the db file? can i delete if i need to restart fresh?
> >
> >
> > Thanks
> > Shannon
> >
> > --
> > You received this message because you are subscribed to the
> Google
> > Groups "Confluent Platform" group.
> > To unsubscribe from this group and stop receiving emails from it,
> > send an email to

> confluent-platf...@googlegroups.com <javascript:>
> > <mailto:confluent-platf...@googlegroups.com
> <javascript:>>.

> > To post to this group, send email to

> > confluent...@googlegroups.com <javascript:>
> > <mailto:confluent...@googlegroups.com <javascript:>>.

> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/confluent-platform/44695125-18d8-4834-aac7-ff303fcd37d3%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/44695125-18d8-4834-aac7-ff303fcd37d3%40googlegroups.com>
>
> >

> <https://groups.google.com/d/msgid/confluent-platform/44695125-18d8-4834-aac7-ff303fcd37d3%40googlegroups.com?utm_medium=email&utm_source=footer

> <https://groups.google.com/d/msgid/confluent-platform/44695125-18d8-4834-aac7-ff303fcd37d3%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
> > For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "Confluent Platform" group.
> > To unsubscribe from this group and stop receiving emails from it,
> send

> > an email to confluent-platf...@googlegroups.com
> <javascript:>
> > <mailto:confluent-platf...@googlegroups.com
> <javascript:>>.

> > To post to this group, send email to confluent...@googlegroups.com

> <javascript:>
> > <mailto:confluent...@googlegroups.com <javascript:>>.

> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/confluent-platform/CAJikTEUU2Gadn-6Xd_GKjCU1_RBnXP%3D-cxPmMYOWYAisP2N4Vg%40mail.gmail.com
> <https://groups.google.com/d/msgid/confluent-platform/CAJikTEUU2Gadn-6Xd_GKjCU1_RBnXP%3D-cxPmMYOWYAisP2N4Vg%40mail.gmail.com>
>
> >

> <https://groups.google.com/d/msgid/confluent-platform/CAJikTEUU2Gadn-6Xd_GKjCU1_RBnXP%3D-cxPmMYOWYAisP2N4Vg%40mail.gmail.com?utm_medium=email&utm_source=footer

> <https://groups.google.com/d/msgid/confluent-platform/CAJikTEUU2Gadn-6Xd_GKjCU1_RBnXP%3D-cxPmMYOWYAisP2N4Vg%40mail.gmail.com?utm_medium=email&utm_source=footer>>.
>
> > For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to confluent-platf...@googlegroups.com
> <mailto:confluent-platf...@googlegroups.com>.

> To post to this group, send email to confluent...@googlegroups.com
> <mailto:confluent...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/confluent-platform/3e5edf99-e303-43fd-a5d5-58d0f3eee1f0%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/3e5edf99-e303-43fd-a5d5-58d0f3eee1f0%40googlegroups.com?utm_medium=email&utm_source=footer>.

signature.asc

Shannon Ma

unread,

Nov 30, 2016, 6:11:03 PM11/30/16

to Confluent Platform

Stop the Kafka Streams application.

> confluent-platform+unsub...@googlegroups.com <javascript:>
> > <mailto:confluent-platform+unsub...@googlegroups.com

> <javascript:>>.
> > To post to this group, send email to
> > confluent...@googlegroups.com <javascript:>
> > <mailto:confluent...@googlegroups.com <javascript:>>.
> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/confluent-platform/44695125-18d8-4834-aac7-ff303fcd37d3%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/44695125-18d8-4834-aac7-ff303fcd37d3%40googlegroups.com>
>
> >
> <https://groups.google.com/d/msgid/confluent-platform/44695125-18d8-4834-aac7-ff303fcd37d3%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/confluent-platform/44695125-18d8-4834-aac7-ff303fcd37d3%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
> > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "Confluent Platform" group.
> > To unsubscribe from this group and stop receiving emails from it,
> send

> > an email to confluent-platform+unsub...@googlegroups.com
> <javascript:>
> > <mailto:confluent-platform+unsub...@googlegroups.com

> <javascript:>>.
> > To post to this group, send email to confluent...@googlegroups.com
> <javascript:>
> > <mailto:confluent...@googlegroups.com <javascript:>>.
> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/confluent-platform/CAJikTEUU2Gadn-6Xd_GKjCU1_RBnXP%3D-cxPmMYOWYAisP2N4Vg%40mail.gmail.com
> <https://groups.google.com/d/msgid/confluent-platform/CAJikTEUU2Gadn-6Xd_GKjCU1_RBnXP%3D-cxPmMYOWYAisP2N4Vg%40mail.gmail.com>
>
> >
> <https://groups.google.com/d/msgid/confluent-platform/CAJikTEUU2Gadn-6Xd_GKjCU1_RBnXP%3D-cxPmMYOWYAisP2N4Vg%40mail.gmail.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/confluent-platform/CAJikTEUU2Gadn-6Xd_GKjCU1_RBnXP%3D-cxPmMYOWYAisP2N4Vg%40mail.gmail.com?utm_medium=email&utm_source=footer>>.
>
> > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to confluent-platform+unsub...@googlegroups.com
> <mailto:confluent-platform+unsub...@googlegroups.com>.

Matthias J. Sax

unread,

Nov 30, 2016, 7:08:46 PM11/30/16

to confluent...@googlegroups.com

If you stop your application, it should not have any impact and on
restart, if should just continue to work.

-Matthias

> > confluent-platf...@googlegroups.com <javascript:>
> <javascript:>
> > > <mailto:confluent-platf...@googlegroups.com
> <javascript:>

> > > an email to confluent-platf...@googlegroups.com
> <javascript:>
> > <javascript:>
> > > <mailto:confluent-platf...@googlegroups.com
> <javascript:>

> > <https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>>.
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "Confluent Platform" group.
> > To unsubscribe from this group and stop receiving emails from it,
> send

> > an email to confluent-platf...@googlegroups.com
> <javascript:>
> > <mailto:confluent-platf...@googlegroups.com

> <javascript:>>.
> > To post to this group, send email to confluent...@googlegroups.com
> <javascript:>
> > <mailto:confluent...@googlegroups.com <javascript:>>.
> > To view this discussion on the web visit
> >

> https://groups.google.com/d/msgid/confluent-platform/3e5edf99-e303-43fd-a5d5-58d0f3eee1f0%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/3e5edf99-e303-43fd-a5d5-58d0f3eee1f0%40googlegroups.com>
>
> >
> <https://groups.google.com/d/msgid/confluent-platform/3e5edf99-e303-43fd-a5d5-58d0f3eee1f0%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/confluent-platform/3e5edf99-e303-43fd-a5d5-58d0f3eee1f0%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>

> > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to confluent-platf...@googlegroups.com
> <mailto:confluent-platf...@googlegroups.com>.

> To post to this group, send email to confluent...@googlegroups.com
> <mailto:confluent...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/confluent-platform/f91264a0-8503-47b9-bbbb-afe610a115de%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/f91264a0-8503-47b9-bbbb-afe610a115de%40googlegroups.com?utm_medium=email&utm_source=footer>.

signature.asc

Shannon Ma

unread,

Nov 30, 2016, 11:59:59 PM11/30/16

to Confluent Platform

Right, i know it will work, I just like to understand a little bit more how the windows advance/rebalance/etc.

> > confluent-platform+unsub...@googlegroups.com <javascript:>
> <javascript:>
> > > <mailto:confluent-platform+unsub...@googlegroups.com

> > > an email to confluent-platform+unsub...@googlegroups.com
> <javascript:>
> > <javascript:>
> > > <mailto:confluent-platform+unsub...@googlegroups.com

> <javascript:>
> > <javascript:>>.
> > > To post to this group, send email to
> confluent...@googlegroups.com
> > <javascript:>
> > > <mailto:confluent...@googlegroups.com <javascript:>>.
> > > To view this discussion on the web visit
> > >
> >
> https://groups.google.com/d/msgid/confluent-platform/CAJikTEUU2Gadn-6Xd_GKjCU1_RBnXP%3D-cxPmMYOWYAisP2N4Vg%40mail.gmail.com
> <https://groups.google.com/d/msgid/confluent-platform/CAJikTEUU2Gadn-6Xd_GKjCU1_RBnXP%3D-cxPmMYOWYAisP2N4Vg%40mail.gmail.com>
>
> >
> <https://groups.google.com/d/msgid/confluent-platform/CAJikTEUU2Gadn-6Xd_GKjCU1_RBnXP%3D-cxPmMYOWYAisP2N4Vg%40mail.gmail.com
> <https://groups.google.com/d/msgid/confluent-platform/CAJikTEUU2Gadn-6Xd_GKjCU1_RBnXP%3D-cxPmMYOWYAisP2N4Vg%40mail.gmail.com>>
>
> >
> > >
> >
> <https://groups.google.com/d/msgid/confluent-platform/CAJikTEUU2Gadn-6Xd_GKjCU1_RBnXP%3D-cxPmMYOWYAisP2N4Vg%40mail.gmail.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/confluent-platform/CAJikTEUU2Gadn-6Xd_GKjCU1_RBnXP%3D-cxPmMYOWYAisP2N4Vg%40mail.gmail.com?utm_medium=email&utm_source=footer>
>
> >
> <https://groups.google.com/d/msgid/confluent-platform/CAJikTEUU2Gadn-6Xd_GKjCU1_RBnXP%3D-cxPmMYOWYAisP2N4Vg%40mail.gmail.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/confluent-platform/CAJikTEUU2Gadn-6Xd_GKjCU1_RBnXP%3D-cxPmMYOWYAisP2N4Vg%40mail.gmail.com?utm_medium=email&utm_source=footer>>>.
>
> >
> > > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>
> > <https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>>.
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "Confluent Platform" group.
> > To unsubscribe from this group and stop receiving emails from it,
> send

> > an email to confluent-platform+unsub...@googlegroups.com
> <javascript:>
> > <mailto:confluent-platform+unsub...@googlegroups.com

> <javascript:>>.
> > To post to this group, send email to confluent...@googlegroups.com
> <javascript:>
> > <mailto:confluent...@googlegroups.com <javascript:>>.
> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/confluent-platform/3e5edf99-e303-43fd-a5d5-58d0f3eee1f0%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/3e5edf99-e303-43fd-a5d5-58d0f3eee1f0%40googlegroups.com>
>
> >
> <https://groups.google.com/d/msgid/confluent-platform/3e5edf99-e303-43fd-a5d5-58d0f3eee1f0%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/confluent-platform/3e5edf99-e303-43fd-a5d5-58d0f3eee1f0%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
> > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to confluent-platform+unsub...@googlegroups.com
> <mailto:confluent-platform+unsub...@googlegroups.com>.

Matthias J. Sax

unread,

Dec 1, 2016, 12:38:08 AM12/1/16

to confluent...@googlegroups.com

Not sure what information you are looking for.

If you use a windowed operation, windows are kept store in a changelog
topic with a combined key of (record-key,window-id) and the value the
current aggregate result of the window. Each time, you update the
aggregate a new record is appended to the changelog topic.

The changelog topic is configured to be log compacted, thus, after
compaction triggers old aggregation result are deleted and only the
latest aggregation result for each window is kept. Additionally, a
retention time is configured on the changelog topic, too, to delete old
window after window retention time passed.

Retention time is based on record metadata timestamps. Thus, as long as
no new data is written to a topic, and thus potentially advancing the
max-timestamp within a topic, no data will be deleted. Only if newer
data (i.e., higher metadata timestamp) gets written, log rolling might
trigger and old data is deleted if retention time passed.

Furthermore, for aggregations KTable also use an internal state to cache
windows for faster updates. This internal state uses RocksDB that allows
to have a state larger than available main memory, because RocksDB can
use the disk, too.

Does this help? If not, please ask a more specific question.

-Matthias

> > > confluent-platf...@googlegroups.com
> <javascript:> <javascript:>
> > <javascript:>
> > > >
> <mailto:confluent-platf...@googlegroups.com <javascript:>

> confluent-platf...@googlegroups.com <javascript:>

> > <javascript:>
> > > <javascript:>
> > > >
> <mailto:confluent-platf...@googlegroups.com <javascript:>

> > > <https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>
> > <https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>>>.
> > >
> > > --
> > > You received this message because you are subscribed to the
> Google
> > > Groups "Confluent Platform" group.
> > > To unsubscribe from this group and stop receiving emails
> from it,
> > send

> > > an email to confluent-platf...@googlegroups.com
> <javascript:>
> > <javascript:>
> > > <mailto:confluent-platf...@googlegroups.com

> <javascript:>
> > <javascript:>>.
> > > To post to this group, send email to
> confluent...@googlegroups.com
> > <javascript:>
> > > <mailto:confluent...@googlegroups.com <javascript:>>.
> > > To view this discussion on the web visit
> > >
> >

> https://groups.google.com/d/msgid/confluent-platform/3e5edf99-e303-43fd-a5d5-58d0f3eee1f0%40googlegroups.com

> <https://groups.google.com/d/msgid/confluent-platform/3e5edf99-e303-43fd-a5d5-58d0f3eee1f0%40googlegroups.com>
>
> >
> <https://groups.google.com/d/msgid/confluent-platform/3e5edf99-e303-43fd-a5d5-58d0f3eee1f0%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/3e5edf99-e303-43fd-a5d5-58d0f3eee1f0%40googlegroups.com>>
>
> >
> > >
> >
> <https://groups.google.com/d/msgid/confluent-platform/3e5edf99-e303-43fd-a5d5-58d0f3eee1f0%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/confluent-platform/3e5edf99-e303-43fd-a5d5-58d0f3eee1f0%40googlegroups.com?utm_medium=email&utm_source=footer>
>
> >
> <https://groups.google.com/d/msgid/confluent-platform/3e5edf99-e303-43fd-a5d5-58d0f3eee1f0%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/confluent-platform/3e5edf99-e303-43fd-a5d5-58d0f3eee1f0%40googlegroups.com?utm_medium=email&utm_source=footer>>>.
>
> >
> > > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>

> > <https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>>.
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "Confluent Platform" group.
> > To unsubscribe from this group and stop receiving emails from it,
> send

> > an email to confluent-platf...@googlegroups.com
> <javascript:>
> > <mailto:confluent-platf...@googlegroups.com

> <javascript:>>.
> > To post to this group, send email to confluent...@googlegroups.com
> <javascript:>
> > <mailto:confluent...@googlegroups.com <javascript:>>.
> > To view this discussion on the web visit
> >

> https://groups.google.com/d/msgid/confluent-platform/f91264a0-8503-47b9-bbbb-afe610a115de%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/f91264a0-8503-47b9-bbbb-afe610a115de%40googlegroups.com>
>
> >
> <https://groups.google.com/d/msgid/confluent-platform/f91264a0-8503-47b9-bbbb-afe610a115de%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/confluent-platform/f91264a0-8503-47b9-bbbb-afe610a115de%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
> > For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to confluent-platf...@googlegroups.com
> <mailto:confluent-platf...@googlegroups.com>.

> To post to this group, send email to confluent...@googlegroups.com
> <mailto:confluent...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/confluent-platform/0c8f1b9f-57b0-444e-b72a-3a84ec5bb186%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/0c8f1b9f-57b0-444e-b72a-3a84ec5bb186%40googlegroups.com?utm_medium=email&utm_source=footer>.

signature.asc

Shannon Ma

unread,

Dec 1, 2016, 12:45:08 PM12/1/16

to Confluent Platform

Thanks for the detailed explanation, very helpful, i will ask later if i have more specific question.

Shannon

> > > confluent-platform+unsub...@googlegroups.com
> <javascript:> <javascript:>
> > <javascript:>
> > > >
> <mailto:confluent-platform+unsub...@googlegroups.com <javascript:>

> confluent-platform+unsub...@googlegroups.com <javascript:>
> > <javascript:>
> > > <javascript:>
> > > >
> <mailto:confluent-platform+unsub...@googlegroups.com <javascript:>

> > > an email to confluent-platform+unsub...@googlegroups.com
> <javascript:>
> > <javascript:>
> > > <mailto:confluent-platform+unsub...@googlegroups.com

> <javascript:>
> > <javascript:>>.
> > > To post to this group, send email to
> confluent...@googlegroups.com
> > <javascript:>
> > > <mailto:confluent...@googlegroups.com <javascript:>>.
> > > To view this discussion on the web visit
> > >
> >
> https://groups.google.com/d/msgid/confluent-platform/3e5edf99-e303-43fd-a5d5-58d0f3eee1f0%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/3e5edf99-e303-43fd-a5d5-58d0f3eee1f0%40googlegroups.com>
>
> >
> <https://groups.google.com/d/msgid/confluent-platform/3e5edf99-e303-43fd-a5d5-58d0f3eee1f0%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/3e5edf99-e303-43fd-a5d5-58d0f3eee1f0%40googlegroups.com>>
>
> >
> > >
> >
> <https://groups.google.com/d/msgid/confluent-platform/3e5edf99-e303-43fd-a5d5-58d0f3eee1f0%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/confluent-platform/3e5edf99-e303-43fd-a5d5-58d0f3eee1f0%40googlegroups.com?utm_medium=email&utm_source=footer>
>
> >
> <https://groups.google.com/d/msgid/confluent-platform/3e5edf99-e303-43fd-a5d5-58d0f3eee1f0%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/confluent-platform/3e5edf99-e303-43fd-a5d5-58d0f3eee1f0%40googlegroups.com?utm_medium=email&utm_source=footer>>>.
>
> >
> > > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>
> > <https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>>.
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "Confluent Platform" group.
> > To unsubscribe from this group and stop receiving emails from it,
> send

> > an email to confluent-platform+unsub...@googlegroups.com
> <javascript:>
> > <mailto:confluent-platform+unsub...@googlegroups.com

> <javascript:>>.
> > To post to this group, send email to confluent...@googlegroups.com
> <javascript:>
> > <mailto:confluent...@googlegroups.com <javascript:>>.
> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/confluent-platform/f91264a0-8503-47b9-bbbb-afe610a115de%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/f91264a0-8503-47b9-bbbb-afe610a115de%40googlegroups.com>
>
> >
> <https://groups.google.com/d/msgid/confluent-platform/f91264a0-8503-47b9-bbbb-afe610a115de%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/confluent-platform/f91264a0-8503-47b9-bbbb-afe610a115de%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
> > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to confluent-platform+unsub...@googlegroups.com
> <mailto:confluent-platform+unsub...@googlegroups.com>.

jpava...@gmail.com

unread,

Apr 30, 2018, 3:52:13 PM4/30/18

to Confluent Platform

Hello Matthias,

I have the same issue. I am updating state store in one class and trying to read in a different java class. How to refresh meta manually?

> an email to confluent-platform+unsub...@googlegroups.com
> <mailto:confluent-platform+unsub...@googlegroups.com>.

Reply all

Reply to author

Forward