retention period for windows in Kafka Streams

[Clemens Valiente]

Quoting the documentation:

 http://docs.confluent.io/3.0.0/streams/concepts.html#windowing

Windowing operations are available in the Kafka Streams DSL, where users can specify a retention period for the window. This allows Kafka Streams to retain old window buckets for a period of time in order to wait for the late arrival of records whose timestamps fall within the window interval. If a record arrives after the retention period has passed, the record cannot be processed and is dropped.

 http://docs.confluent.io/3.0.0/streams/developer-guide.html#windowing-a-stream

Windowing is a common prerequisite for stateful transformations which group records in a stream, for example, by their timestamps. A local state store is usually needed for a windowing operation to store recently received records based on the window interval, while old records in the store are purged after the specified window retention period.

I tried to find out how to specify the mentioned retention period. Unfortunately the two quoted paragraphs only link to each other and don't reveal anything on how to actually specify the retention period. I went through the source code of Kafka streams and still couldn't quite figure it out.

So my question is:

- how do I specify the interval after which to drop old windows from the state store?

- Can I specify the behaviour of what happens if a message arrives for a window after that window was dropped from the state? (e.g. similar to withLateFirings triggers in cloud pub/sub)

Thanks a lot!

Clemens

[Damian Guy]

Hi Clemens,

You can specify the retention interval by calling Windows.until(..)

For example:

TimeWindows.of("name",ONE_DAY).until(size * 7)

Records arriving after the window has expired are dropped. You currently can't specify any behaviour for this.

Thanks,

Damian 

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/c4eb8fc1-9c7f-4776-84ce-350b15b30884%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.