Moving to Java 11 as a baseline for Debezium 1.6

[Gunnar Morling]

Hi all,

$subject already states pretty much what I would like to discuss: for Debezium 1.6 (due by the end of Q2), we're considering to move to Java 11 or newer as a required runtime.

The reason for this is that first dependencies are moving to Java 11 as a baseline, so we cannot really stay on 8, unless we stop updating these dependencies, which isn't desirable in terms of bugfixes etc. At this point this concerns a few test only dependencies for our downstream testing on K8s/OpenShift, the MySQL binlog reader client (which already had upgraded to 11, but we could make the case for going back to 8 at least for a while), and the Debezium Quarkus extension for the outbox pattern. But we expect that more and more dependencies will move to require 11 eventually; It has been nearly three years since Java 11 came out, and it's less than six months until the next LTS version (Java 17), so we feel the time for upgrading has come eventually.

The impact of this change on users is expected to be fairly low, as Kafka Connect runs with Java 11 without any issues. E.g. the upstream Debezium container images for Kafka Connect etc. have been based on Java 11 already for quite some time. The biggest impact I could see is on users of the Debezium embedded mode, which will have to move their application to Java 11, too.

If you have any questions or concerns about this change, please let me know. Right now, we're planning to do this change for Debezium 1.6 Alpha1, which is to be released by the end of this month.

Thanks a lot for any feedback,

--Gunnar

[Gunnar Morling]

Hi all,

Since there were no concerns raised about moving the baseline to JDK 11, we're planning to move forward with this change for Debezium 1.6 Alpha1. There was one interesting question off-list, and this is whether Debezium would work with later versions of Java, too, such as the currently stable version 16.

We haven't formalized any strategy here yet, but I think the following is a reasonable goal:

* Always make sure Debezium can be used with the latest LTS version of Java (i.e. right now Java 11)

* Best effort to make sure later stable versions work too (i.e. right now 16, where in fact there seems to be an issue with the binlog client dependency which is under investigation)

* Only minimal efforts to support early access builds (i.e. right now 17-ea; we try to have CI jobs, but so far there always were some issues, e.g. due to the increased class file version number not supported by some older ASM version pulled in by a dependency, etc.)

I believe this provides a reasonable combination between trying to support latest Java versions while keeping efforts and time spent under control. If there's concerns/feedback/questions on this proposal, please reply here. I think it'd make sense to materialize that info in the docs eventually.

--Gunnar

[Gunnar Morling]

Hey all,

After some more consideration, we decided to slightly adjust course in this matter: Debezium 1.6 will require JDK 11 or newer for *building* Debezium (as there are some testing infrastructure components which require 11), but it can be *run* on Java 1.8 or newer.

The reason being that the Debezium Cassandra connector currently uses version 3.x of the Cassandra driver, which itself cannot be used with 11 yet. While we theoretically could move the other connectors to 11 already, that'd make testing and general handling much more complex. Unfortunately, upgrading to Cassandra driver 4.x isn't a simple drop-in replacement either, but requires some work (what exactly, still needs to be clarified), and we probably should not squeeze this into 1.6. @Bingqin et al, if you have any insights here -- or even plans by yourself for upgrading to 4.x -- please chime in.

Once the Cassandra connector is ready to be used with JDK 11, we'll revisit the issue and re-evaluate the decision about upgrading the required runtime version to Java 11.

Best,

--Gunnar

[chr...@wepay.com]

Hey Gunnar,

Glad to hear about the delay; we were going through some back bending to accommodate it.

Re C* 4.x, I don't even think it's released yet. I see only beta here:

https://cassandra.apache.org/download/

As you said, migrating to 4.x is a substantial undertaking (though it does give us some useful features like node-local reads for snapshots and ability to read active write-ahead log file). I suspect it's going to take a while for 4.x to be released and matured. Not sure how that'll affect DBZ overall.

Cheers,

Chris

[Bingqin Zhou]

Hi Gunnar,

Thanks for bringing this up. As Chris said, we're planning to upgrade to Cassandra 4.0 but are waiting for Cassandra 4.x to be released as well. Currently the latest stable version of Cassandra Driver is 3.11.0 (https://mvnrepository.com/artifact/com.datastax.cassandra/cassandra-driver-core).

Bingqin Zhou

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/2d73d38f-2b12-45aa-80e0-3a2eea6b1078n%40googlegroups.com.

[Gunnar Morling]

Hey Chris,

chr...@wepay.com schrieb am Freitag, 23. April 2021 um 20:04:58 UTC+2:

Hey Gunnar,

Glad to hear about the delay; we were going through some back bending to accommodate it.

I see. Sorry for the forth and back. 

Re C* 4.x, I don't even think it's released yet. I see only beta here:

https://cassandra.apache.org/download/

Doh, completely had missed that. Seems we're bound to stay on 1.8 for a while then, unless they release a 3.x version which is compatible with 11+ (unfortunately, this affects the driver *and* the DB itself, as we start it in-process for testing AFAIK. We may mitigate the latter using containers, though). 

The reason for pushing for the move to 11 is that first dependencies are about to start that move. E.g. the binlog client already had done the jump, but after talking to the maintainer, this got reverted for now. But I think it's going to happen eventually. At least, with the requirement and ability to build on 11, we're ready to make the jump ourselves eventually. We already have a test dependency requiring 11+ (hence the requirement for 11 at build time).

As you said, migrating to 4.x is a substantial undertaking (though it does give us some useful features like node-local reads for snapshots and ability to read active write-ahead log file). I suspect it's going to take a while for 4.x to be released and matured. Not sure how that'll affect DBZ overall.

Good question. I could see user interest in the connector to support 4.x once it's out, which we should address. The more troublesome situation though would be if one of the dependencies forces us to move to 11, while we still cannot move yet for the Cassandra connector, as it's still on 3.x. In that case, we'd have to build different connectors with different compatibility levels, which complicates the set-up a bit. But we can cross that bridge when we get there :) 

Cheers,

Chris

Best,

--Gunnar