KSQL and Kubernetes

[Josh W]

Hello Everyone,

I have a question about how to manage ksqlDB Server instances in Kubernetes. 

Lets say we have a ksqlDB Server instance running a query that consumes from 2 topics, joins the data together, and outputs that data to a new topic.

When Kubernetes decides to take that instance of ksqlDB Server down and start it elsewhere. How do we make sure that we do not re-process old messages from the input topics?

We do not have complete control over the GroupID, and can only specify 'Earliest' and 'Latest' for new consumers.

Thanks,

- Josh W

[Almog Gavra]

Hello Josh - ksqlDB creates a groupId for each query. If you issue a query with "earliest" or "latest", that setting only applies if ksql is creating a consumer group for the very first time. If you're interested, you can see the consumer group information for a query by issuing a DESCRIBE EXTENDED on the source.

- Almog

--
You received this message because you are subscribed to the Google Groups "ksqldb-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ksql-users+...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/ksql-users/68b1612c-5339-4286-85b8-baee0fb3c4c6n%40googlegroups.com.

[Almog Gavra]

Yup! (Assuming you are running in "interactive" mode: https://docs.ksqldb.io/en/latest/concepts/ksqldb-architecture/#ksqldb-deployment-modes). In general, our architecture docs here have a pretty good overview of how this would work.

On Thu, Jan 14, 2021 at 12:01 PM Josh W <wieb...@gmail.com> wrote:

Hi Almog,

Thank you for the response.

So if a ksqlDB Server instance was to be restarted we would not have to re-issue the queries?

Thanks,

-Josh

To view this discussion on the web, visit https://groups.google.com/d/msgid/ksql-users/b67fc0c8-47e9-452a-9ac3-735a593ebf45n%40googlegroups.com.

[Josh W]

Hi Almog, 

Thank you for clearing that up for me!

This applies to interactive mode because headless mode will always require a command to be issued to the server via SQL file?

-Josh

[Almog Gavra]

No problem - the specifics I had in mind about queryID generation is specific to interactive mode (though I suppose everything I wrote applies to both). Headless will also not re-run queries on restarts, but the mechanism is slightly different (namely, if the sql file is re-ordered the queries will be restarted - it generates a queryID based on the position in the file).

To view this discussion on the web, visit https://groups.google.com/d/msgid/ksql-users/de9a7f4c-4021-437d-93ed-96d58c347395n%40googlegroups.com.