Delete records from a ksqldb table/state store

[Anup Tiwari]

Hi Team,

Do we have any way where we can define TTL for record expiry.

Actually Use Case is something on the lines that whenever a user perform an event=X first time in his life on an App, we will start counting event=Y for that user and we will keep on counting event=Y till next N days since first click and after that we don't need data of that user.
So can we delete/expire that record from the state store ?

Regards,
Anup Tiwari

[Ben Ellis]

From what I've read:

1) you could use a windowed store to back your table, and specify retention policy, after which records expire

2) or you could manually push tombstones to the topic used as primary input for your topic, that should cause related entries in the state store to be deleted

3) or you could fork/extend RocksDB config so it includes a TTL (but you'd still have to solve how to prune the changelog topics)

4) or via Kafka Streams you could schedule a periodic job that emits tombstones for entries in your state store older than some TTL

Option 1 is easiest, but requires you to use some form of windowing.

Your use case doesn't quite fit option 1. You could use a session window so that orphaned clicks would be automatically pruned, but you'd need something like Kafka Streams app to trigger issuing a tombstone when, while a session window is open, a click arrives that is over X amount of time since the beginning of that window. I don't see how to do that with KSQL alone. If someone knows a way to do that I'd be interested to learn about it.

--
You received this message because you are subscribed to the Google Groups "ksqldb-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ksql-users+...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/ksql-users/CAH8KkAq9f%3DR7N91NMZjA64V3-YMZU43mP5C48iO%3DC3uFNSCk_w%40mail.gmail.com.