Process historical time series data with Kafka Streams

[Mindaugas Zickus]

Hi All,

I wonder if Kafka streams is suitable for processing historical time series data.

Our use case: we have clickstream histories(time series) of multiple users and would like to calculate user specific sliding window aggregates over past periods to create features to train machine learning models. 

Is it possible to bulk load historical time series data to Kafka Streams directly from some NoSql database or it is necessary to load data to Kafka topic first?

thanks, Mindis

[Matthias J. Sax]

Hi,

Streams is tightly coupled with Kafka and using the DSL you can only
read data from a Kafka topic. To load data into Kafka, you could use
Kafka connect.

As an alternative, you can use the low-lever Processor API in which you
can do basically anything you like -- of course, you need to write
custom code and lose some benefits the DSL offers.

About historical data and windows: this works with no problems (using
DSL again). Kafka Streams supports event time windows. See Confluent
docs for more details:
http://docs.confluent.io/3.0.0/streams/concepts.html#time


-Matthias

> --
> You received this message because you are subscribed to the Google
> Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to confluent-platf...@googlegroups.com
> <mailto:confluent-platf...@googlegroups.com>.
> To post to this group, send email to confluent...@googlegroups.com
> <mailto:confluent...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/confluent-platform/bf8b8bd3-f87a-48f8-888a-206906bbd90c%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/bf8b8bd3-f87a-48f8-888a-206906bbd90c%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.