[VOTE] 3.6.0 RC1

[Satish Duggana]

Hello Kafka users, developers and client-developers,

This is the second candidate for the release of Apache Kafka 3.6.0. Some of the major features include:

* KIP-405 : Kafka Tiered Storage
* KIP-868 : KRaft Metadata Transactions
* KIP-875: First-class offsets support in Kafka Connect
* KIP-898: Modernize Connect plugin discovery
* KIP-938: Add more metrics for measuring KRaft performance
* KIP-902: Upgrade Zookeeper to 3.8.1
* KIP-917: Additional custom metadata for remote log segment

Release notes for the 3.6.0 release:
https://home.apache.org/~satishd/kafka-3.6.0-rc1/RELEASE_NOTES.html

*** Please download, test and vote by Saturday, September 23, 8am PT

Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~satishd/kafka-3.6.0-rc1/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~satishd/kafka-3.6.0-rc1/javadoc/

* Tag to be voted upon (off 3.6 branch) is the 3.6.0 tag:
https://github.com/apache/kafka/releases/tag/3.6.0-rc1

* Documentation:
https://kafka.apache.org/36/documentation.html

* Protocol:
https://kafka.apache.org/36/protocol.html

* Successful Jenkins builds for the 3.6 branch:
There are a few runs of unit/integration tests. You can see the latest at https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/. We will continue running a few more iterations.
System tests:
We will send an update once we have the results.

Thanks,
Satish.

[Divij Vaidya]

Hey Satish

My comments about documentation misses from RC0 vote thread [1] are
still not addressed (such as missing metric documentation, formatting
problems etc). Could you please mention why we shouldn't consider them
as blockers to make RC1 as the final release?

[1] https://lists.apache.org/thread/cokoxzd0jtgjtrlxoq7kkzmvpm75381t

> --
> You received this message because you are subscribed to the Google Groups "kafka-clients" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kafka-client...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/kafka-clients/CAM-aUZ%3DuJ-SKeVFtBZwBjhLHKw4CbxF_ws%2BvQqaymGHFsC%2Bmdg%40mail.gmail.com.

[Satish Duggana]

Hi Divij,

Thanks for calling out documentation issues. Some of them are being addressed in the existing documentation PRs. We will keep addressing the remaining documentation issues by making respective changes in kafka-site repo for 3.6.0 release.

I think documentation is important but I do not consider documentation issues as release blockers. I want others to review and test the release candidate sooner to finalize the RC for 3.6.0 release. 

~Satish.

[Satish Duggana]

Thanks Greg for verifying the release including the earlier blocker(KAFKA-15473) verification.

~Satish.

On Wed, 20 Sept 2023 at 22:30, Greg Harris <greg....@aiven.io.invalid> wrote:

Hi all,

I verified the functionality of KIP-898 and the recent fix for
KAFKA-15473 with the following steps:

1. I started a 3.5.1 broker, and a 3.5.1 worker with most (>400)
publicly available plugins installed
2. I captured the output of /connector-plugins
3. I upgraded the worker to 3.6.0-rc1
4. I captured the output of /connector-plugins with various settings
of plugin.discovery
5. I ran the migration script to add manifests to my plugins
6. I captured the output of /connector-plugins with various settings
of plugin.discovery
7. I downgraded the worker to 3.5.1
8. I diffed the output of /connector-plugins across the different
cases and observed the expected changes.
    a. When plugins are migrated for 3.6.0, all modes produce identical results.
    b. When plugins are not migrated for 3.6.0, only_scan and
hybrid_warn produce identical results, hybrid_fail crashes, and
service_load is missing plugins
    c. When upgrading from 3.5.1 I see that plugins with invalid
constructors are hidden, AK plugins now have versions, multi-interface
plugins now show each interface type, and plugins using AppInfoParser
change versions.
    d. The startup logs now include descriptive errors for invalid
plugins that otherwise would have been thrown at runtime
    d. The fix for KAFKA-15473 prevents duplicates
    e. The output for 3.5.1 after downgrading is identical to before.

+1 (non-binding)

Thanks Satish for running the release!

[Josep Prat]

Hi Satish,

I ran the following validation steps:

- Built from source with Java 11 and Scala 2.13

- Verified Signatures and hashes of the artifacts generated

- Navigated through Javadoc including links to JDK classes

- Run the unit tests

- Run integration tests

- Run the quickstart in KRaft and Zookeeper mode

I +1 this release (non-binding)

Thanks for your efforts!

--

Josep Prat Open Source Engineering Director, Aiven josep...@aiven.io   |   +491715557497 aiven.io   |

Aiven Deutschland GmbH Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen Amtsgericht Charlottenburg, HRB 209739 B

[Satish Duggana]

Thanks Luke for reporting KRaft issue[1].

I am not sure whether it is a release blocker for 3.6.0. Need input
from other KRaft experts also to finalize the decision. Even if we
adopt a fix, do not we need to bake it for some time before it is
pushed to production to avoid any regressions as this change is in the
critical paths?

1. https://issues.apache.org/jira/browse/KAFKA-15489

Thanks,
Satish.

On Sat, 23 Sept 2023 at 03:08, Luke Chen <sho...@gmail.com> wrote:
>
> Hi Satish,
>
> I found the current KRaft implementation will have "split brain" issue when
> network partition happens, which will cause inconsistent metadata returned
> from the controller.
> Filed KAFKA-15489 <https://issues.apache.org/jira/browse/KAFKA-15489> for
> this issue, and PR <https://github.com/apache/kafka/pull/14428> is ready
> for review.
>
> Even though this is not a regression issue (this has already existed since
> the 1st release of KRaft feature), I think this is an important issue since
> KRaft is announced production ready.
> Not sure what other people's thoughts are.
>
> Thank you.
> Luke

> > [image: Aiven] <https://www.aiven.io>
> >
> > *Josep Prat*
> > Open Source Engineering Director, *Aiven*
> > josep...@aiven.io | +491715557497
> > aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud
> > >
> > <https://www.linkedin.com/company/aiven/> <
> > https://twitter.com/aiven_io>
> > *Aiven Deutschland GmbH*

[Ismael Juma]

Given that this is not a regression and there have been no reports for over a year, I think it's ok for this to land in 3.6.1.

Ismael