Information needed for KsqlDB Upgrade

[Anup Tiwari]

Hi Team,

We are using new to ksqldb and using ksqldb of confluent 5.5.0(community version) and planning to upgrade it to confluent 6.0.

I found below 2 links for same :-

1. upgrading-to-ksqldb-6-0-from-ksqldb-5-5  :- Here I can't find steps, the only thing mentioned is Breaking changes.

2. ksqlDB upgrade :- Here i can see steps but it seems we have to set this up in all together a new cluster(kafka + ksqldb) and post changes, terminate old one. So may i know if it is necessary or i am missing something? Because I thought it should be like any other software upgrade in a current environment.

Let me know if i am missing something and for 2nd link, how can we see ksqldb version inside confluent package.

Note :- We have an open source setup for kafka_2.13-2.6.0(using tarball) and confluent 5.5.0(community version) ksqldb(using tarball).

Regards,
Anup Tiwari

[Victoria Xia]

Hi Anup,

I've opened a PR to clear up confusion between the two types of upgrades: in-place upgrades, and those that involve rebuilding state from scratch. I'm hoping to have this merged to the live docs soon: https://github.com/confluentinc/ksql/pull/6866.

The other source of confusion is the two sets of versions we have for ksqlDB. The second link you provided is for community ksqlDB versions, whereas the first is for versions of ksqlDB bundled with Confluent Platform. You can see the mapping between the two versions here: https://docs.confluent.io/platform/current/installation/versions-interoperability.html#ksqldb It's on us to reconcile the two sets of documentation to reduce confusion here. Thanks for raising this.

As for your specific question, the upgrade from ksqlDB 5.5 to ksqlDB 6.0 can be performed in-place. Your existing queries will continue to run from where they left off, though you'll have to update query statements for new queries to adjust for syntax changes between the new versions.

Hope this helps,

Victoria

[Anup Tiwari]

Thank you for an update.

As you have mentioned in the previous email that " the upgrade from ksqlDB 5.5 to ksqlDB 6.0 can be performed in-place." so can you please share steps / link for same ? Because in my 1st link, I can't find steps, the only thing mentioned is Breaking changes.

Regards,
Anup Tiwari

--
You received this message because you are subscribed to the Google Groups "ksqldb-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ksql-users+...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/ksql-users/9bf6b835-115d-4aca-b509-11aa28e5092an%40googlegroups.com.

[Victoria Xia]

An in-place upgrade simply refers to stopping your ksqlDB servers and restarting them with the new version. In other words, a regular upgrade. (This information is added in the PR I linked, and should be live on the docs site shortly.)

Best,

Victoria

[Anup Tiwari]

Hey Victoria,

When you say "in-place upgrade simply refers to stopping your ksqlDB servers and restarting them with the new version" then does this mean that i have to stop current ksqlDB 5.5 and download confluent 6 tar ball and copy configs of ksqlDB 5.5 to ksqlDB 6 and start newer version ?

If this interpretation is correct then will ksqlDB 6 build all tables / streams from scratch using the command and changelog topic of kafka ?

I went through the PR link but I didn't understand the answer to the above question and hence asked. If i have missed something then apologies for that  .

Regards,
Anup Tiwari

To view this discussion on the web, visit https://groups.google.com/d/msgid/ksql-users/ce404f92-308b-44bd-a337-840183aa3442n%40googlegroups.com.

[Anup Tiwari]

Hi Victoria / Team, 

Can you please revert on trail mail doubts? 

[Almog Gavra]

Hello Anup - your understanding is correct. ksqlDB will rebuild its state based on the command topic so long as you configure it with the same service id. You can think of an "in place upgrade" identical to restarting a 5.5 server, except with newer code.

To view this discussion on the web, visit https://groups.google.com/d/msgid/ksql-users/CAH8KkApVoWN_BDLTvKQFN%2BOApGDGSGJ%3D533MvTbK5kCHzFa88w%40mail.gmail.com.

[Anup Tiwari]

Thanks Almog / Victoria for help.

Hi Team,

Just wanted to tell you that our upgrade activity  was successful. We came across 2 problems(which were actually missed from our side only :) ) and solved it so just wanted to share those along with upgrade steps with the community.

Steps of "In-Place" upgrade :-

1. Download the latest confluent version.

2. Set it up with all required configs. In our case we mostly copied all configs from the current version(which is running) to the new version.

3. Take Backup and then copy all UDF JAR to new version .(OPTIONAL STEP In case you don't have any UDF).

4. Stop ksqlDB process of current version(5.5 in our case) and start newer version(6 in our case) with latest KsqlDB config file which you created in step2.

5. KsqlDB will start executing queries and rebuild all states using command topic because the value of "ksql.service.id" is the same.

6. Now KsqlDB with the latest version is up and running.

Issues which we faced :-

1. We have made some UDFs like ARRAY_EXCEPT , ARRAY_LENGTH , ARRAY_REMOVE etc since they were not available with Confluent KsqlDB 5.5 so when we started newer version, it was giving an error saying "KsqlException: UdfFactory not compatible with existing factory. function: ARRAY_EXCEPT existing". This error was because Confluent KsqlDB 6 has these functions as in built so it was creating conflict with our UDF functions. So to solve this we created a new udf jar by eliminating those inbuilt functions and so KsqlDB was up and running.

2. We have made few environment variables and alias in "~/.bashrc" file which were still pointing to older KsqlDB version so we logged in to ksql console from command line, it wasn't able to execute "describe" command since ksql cli console was pointing to older version(5.5) and ksqlDB server process was of newer version (confluent 6) so this was creating conflict. So to solve this we made changes in environment variables and aliases in "~/.bashrc" file so now ksql cli was also of a newer version.

Regards,
Anup Tiwari

[Victoria Xia]

Hi Anup,

Thanks for the update, and for sharing these great details! I will open a PR to get your details into the public docs, unless you'd like to do the honors yourself? :)

Best,

Victoria

[Anup Tiwari]

I would love to do that. Can you guide me how to do that?

Regards,
Anup Tiwari

[Victoria Xia]

This is the file containing the docs that need to be updated: https://github.com/confluentinc/ksql/blob/master/docs/operate-and-deploy/installation/upgrading.md You can open a PR against the repository with your improvements, and someone from the team can help review and merge them. 

Here's a link to our contributing guide, though many of the steps about IDE setup are not necessary for PRs that only contain docs updates rather than code changes: https://github.com/confluentinc/ksql/blob/master/CONTRIBUTING.md

Looking forward to your contribution!

Thanks,

Victoria

[Anup Tiwari]

Hi Victoria,

I have raised a PR. Let me know if I have missed something.

Regards,
Anup Tiwari

[Victoria Xia]

Thanks for the contribution, Anup! I just left a review. Let's follow up on the PR itself.

Best,

Victoria