Debezium CDC streams and joins

[David Kalosi]

Hi everybody,

I have the following use case we would like to implement and since we have just started exploring streaming apps I need some help :)

We have 3 CDC streams on the input from debezium representing 3 MySQL tables from a source database. 

Those 3 tables are one logical loan entity where one stream is the actual loan account table, the second stream represents the installments (one to many, one loan having multiple installments) and the third stream is just some additional data enriching the actual loanaccount (one on one relationship). 

We need to take those 3 captured streams and create a denormalized model that we can store in elasticsearch - so effectively merging those 3 streams into one model. 

The idea is was 

debezium (kafka connect source) -> kafka streams (transformation / join) -> elastic search (kafka connect sink).

What is the best approach (if any) to model those in streams ? After reading through the docs the CDC events are basically a KTable since each event represents the complete change set for the given record. 

Question is how to window / join them using streams.

thanks for any suggestions / help

[Chen Yuequan]

hi, David

any suggestion for this question?

thanks

best regards

Yuequan Chen

在 2017年3月10日星期五 UTC+8上午5:22:30，David Kalosi写道：

[Robin Moffatt]

Did you look at KSQL? 

https://www.confluent.io/blog/ksql-open-source-streaming-sql-for-apache-kafka/

https://speakerdeck.com/rmoff/look-ma-no-code-building-streaming-data-pipelines-with-apache-kafka

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/df0ace30-86c5-4bee-917b-86d45d5e1381%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.