[VOTE] 0.11.0.0 RC2

Ismael Juma

unread,

Jun 22, 2017, 9:16:52 PM6/22/17

to d...@kafka.apache.org, Kafka Users, kafka-clients

Hello Kafka users, developers and client-developers,

This is the third candidate for release of Apache Kafka 0.11.0.0.

This is a major version release of Apache Kafka. It includes 32 new KIPs. See the release notes and release plan (https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+0.11.0.0) for more details. A few feature highlights:

* Exactly-once delivery and transactional messaging

* Streams exactly-once semantics

* Admin client with support for topic, ACLs and config management

* Record headers

* Request rate quotas

* Improved resiliency: replication protocol improvement and single-threaded controller

* Richer and more efficient message format

Release notes for the 0.11.0.0 release:

http://home.apache.org/~ijuma/kafka-0.11.0.0-rc2/RELEASE_NOTES.html

*** Please download, test and vote by Tuesday, June 27, 6pm PT

Kafka's KEYS file containing PGP keys we use to sign the release:

http://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):

http://home.apache.org/~ijuma/kafka-0.11.0.0-rc2/

* Maven artifacts to be voted upon:

https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:

http://home.apache.org/~ijuma/kafka-0.11.0.0-rc2/javadoc/

* Tag to be voted upon (off 0.11.0 branch) is the 0.11.0.0 tag:

https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=8698fa1f41102f1664b05baa4d6953fc9564d91e

* Documentation:

http://kafka.apache.org/0110/documentation.html

* Protocol:

http://kafka.apache.org/0110/protocol.html

* Successful Jenkins builds for the 0.11.0 branch:

Unit/integration tests: https://builds.apache.org/job/kafka-0.11.0-jdk7/187/

System tests: pending (will send an update tomorrow)

/**************************************

Thanks,

Ismael

Ismael Juma

unread,

Jun 22, 2017, 9:37:11 PM6/22/17

to d...@kafka.apache.org, Kafka Users, kafka-clients

A quick note on notable changes since rc1:

1. A significant performance improvement if transactions are enabled: https://github.com/apache/kafka/commit/f239f1f839f8bcbd80cce2a4a8643e15d340be8e

2. Fixed a controller regression if many brokers are started simultaneously: https://github.com/apache/kafka/commit/c0033b0e0b9e56242752c82f15c6388d041914a1

3. Fixed a couple of Connect regressions: https://github.com/apache/kafka/commit/c029960bf4ae2cd79b22886f4ee519c4af0bcc8b and https://github.com/apache/kafka/commit/1d65f15f2b656b7817eeaf6ee1d36eb3e2cf063f

4. Fixed an import log cleaner issue: https://github.com/apache/kafka/commit/186e3d5efc79ed803f0915d472ace77cbec88694

Full diff: https://github.com/apache/kafka/compare/5b351216621f52a471c21826d0dec3ce3187e697...0.11.0.0-rc2

Ismael

Ismael Juma

unread,

Jun 26, 2017, 6:53:07 AM6/26/17

to Vahid S Hashemian, d...@kafka.apache.org, kafka-clients, Kafka Users

Hi Vahid,

Sorry for not replying to the previous email, I had missed it. A couple of questions:

1. Is this also happening in trunk? Seems like it should be the case for months and seemingly no-one reported it until the RC stage.

2. Is it correct that this only happens when compiling on Windows without Cygwin?

I believe the following PR should fix it, please verify:

https://github.com/apache/kafka/pull/3431

Ismael

On Fri, Jun 23, 2017 at 8:25 PM, Vahid S Hashemian <vahidha...@us.ibm.com> wrote:

Hi Ismael,

Not sure if my response on RC1 was lost or this issue is not a show-stopper:

I checked again and with RC2, tests still fail in my Windown 64 bit environment.

:clients:checkstyleMain
[ant:checkstyle] [ERROR] C:\Users\User\Downloads\kafka-0.11.0.0-src\clients\src\main\java\org\apache\kafka\common\protocol\Errors.java:89:1: Class Data Abstraction Coupling is 57 (max allowed is 20) classes [ApiExceptionBuilder, BrokerNotAvailableException, ClusterAuthorizationException, ConcurrentTransactionsException, ControllerMovedException, CoordinatorLoadInProgressException, CoordinatorNotAvailableException, CorruptRecordException, DuplicateSequenceNumberException, GroupAuthorizationException, IllegalGenerationException, IllegalSaslStateException, InconsistentGroupProtocolException, InvalidCommitOffsetSizeException, InvalidConfigurationException, InvalidFetchSizeException, InvalidGroupIdException, InvalidPartitionsException, InvalidPidMappingException, InvalidReplicaAssignmentException, InvalidReplicationFactorException, InvalidRequestException, InvalidRequiredAcksException, InvalidSessionTimeoutException, InvalidTimestampException, InvalidTopicException, InvalidTxnStateException, InvalidTxnTimeoutException, LeaderNotAvailableException, NetworkException, NotControllerException, NotCoordinatorException, NotEnoughReplicasAfterAppendException, NotEnoughReplicasException, NotLeaderForPartitionException, OffsetMetadataTooLarge, OffsetOutOfRangeException, OperationNotAttemptedException, OutOfOrderSequenceException, PolicyViolationException, ProducerFencedException, RebalanceInProgressException, RecordBatchTooLargeException, RecordTooLargeException, ReplicaNotAvailableException, SecurityDisabledException, TimeoutException, TopicAuthorizationException, TopicExistsException, TransactionCoordinatorFencedException, TransactionalIdAuthorizationException, UnknownMemberIdException, UnknownServerException, UnknownTopicOrPartitionException, UnsupportedForMessageFormatException, UnsupportedSaslMechanismException, UnsupportedVersionException]. [ClassDataAbstractionCoupling]
[ant:checkstyle] [ERROR] C:\Users\User\Downloads\kafka-0.11.0.0-src\clients\src\main\java\org\apache\kafka\common\protocol\Errors.java:89:1: Class Fan-Out Complexity is 60 (max allowed is 40). [ClassFanOutComplexity]
[ant:checkstyle] [ERROR] C:\Users\User\Downloads\kafka-0.11.0.0-src\clients\src\main\java\org\apache\kafka\common\requests\AbstractRequest.java:26:1: Class Fan-Out Complexity is 43 (max allowed is 40). [ClassFanOutComplexity]
[ant:checkstyle] [ERROR] C:\Users\User\Downloads\kafka-0.11.0.0-src\clients\src\main\java\org\apache\kafka\common\requests\AbstractResponse.java:26:1: Class Fan-Out Complexity is 42 (max allowed is 40). [ClassFanOutComplexity]
:clients:checkstyleMain FAILED

FAILURE: Build failed with an exception.

Thanks.
--Vahid

Ismael Juma

unread,

Jun 26, 2017, 4:54:18 PM6/26/17

to d...@kafka.apache.org, kafka-clients, Kafka Users

Hi Vahid,

Can you please check which test fails first? The errors you mentioned can happen if a test fails and doesn't clean-up properly.

Ismael

On Mon, Jun 26, 2017 at 8:41 PM, Vahid S Hashemian <vahidha...@us.ibm.com> wrote:

Hi Ismael,

To answer your questions:

1. Yes, the issues exists in trunk too.

2. I haven't checked with Cygwin, but I can give it a try.

And thanks for addressing this issue. I can confirm with your PR I no
longer see it.
But now that the tests progress I see quite a few errors like this in
core:

kafka.server.ReplicaFetchTest > classMethod FAILED
java.lang.AssertionError: Found unexpected threads,
allThreads=Set(ZkClient-EventThread-268-127.0.0.1:56565,
ProcessThread(sid:0 cport:56565):, metrics-mete
r-tick-thread-2, SessionTracker, Signal Dispatcher, main, Reference
Handler, ForkJoinPool-1-worker-1, Attach Listener, ProcessThread(sid:0
cport:59720):, ZkClie
nt-EventThread-1347-127.0.0.1:59720, kafka-producer-network-thread |
producer-1, Test worker-SendThread(127.0.0.1:56565), /127.0.0.1:54942 to
/127.0.0.1:54926 w
orkers Thread 2, Test worker, SyncThread:0,
NIOServerCxn.Factory:/127.0.0.1:0, Test worker-EventThread, Test
worker-SendThread(127.0.0.1:59720), /127.0.0.1:5494
2 to /127.0.0.1:54926 workers Thread 3,
ZkClient-EventThread-22-127.0.0.1:54976, ProcessThread(sid:0
cport:54976):, Test worker-SendThread(127.0.0.1:54976), Fin
alizer, metrics-meter-tick-thread-1)

I tested on a VM and a physical machine, and both give me a lot of errors
like this.

> ------------------------------

Jason Gustafson

unread,

Jun 26, 2017, 5:25:02 PM6/26/17

to d...@kafka.apache.org, kafka-clients, Kafka Users

+1 from me. I verified the artifacts and tested basic producing and consuming. I also verified the procedure for upgrading to the new message format.

-Jason

Ismael Juma

unread,

Jun 26, 2017, 6:54:30 PM6/26/17

to Vahid S Hashemian, Kafka Users, d...@kafka.apache.org, kafka-clients

Hi Vahid,

There are a few known issues when running Kafka on Windows. A PR with some fixes is: https://github.com/apache/kafka/pull/3283. The fact that the index cannot be accessed indicates that it may be a similar issue. I suggest we move this discussion to the relevant JIRAs instead of the release thread.

Ismael

On Mon, Jun 26, 2017 at 11:25 PM, Vahid S Hashemian <vahidha...@us.ibm.com> wrote:

Hi Ismael,

This is the output of core tests from the start until the first failed test.

kafka.admin.AdminRackAwareTest > testAssignmentWithRackAwareWithUnevenRacks PASSED

kafka.admin.AdminRackAwareTest > testAssignmentWith2ReplicasRackAware PASSED

kafka.admin.AdminRackAwareTest > testAssignmentWithRackAwareWithUnevenReplicas PASSED

kafka.admin.AdminRackAwareTest > testSkipBrokerWithReplicaAlreadyAssigned PASSED

kafka.admin.AdminRackAwareTest > testAssignmentWithRackAware PASSED

kafka.admin.AdminRackAwareTest > testRackAwareExpansion PASSED

kafka.admin.AdminRackAwareTest > testAssignmentWith2ReplicasRackAwareWith6Partitions PASSED

kafka.admin.AdminRackAwareTest > testAssignmentWith2ReplicasRackAwareWith6PartitionsAnd3Brokers PASSED

kafka.admin.AdminRackAwareTest > testGetRackAlternatedBrokerListAndAssignReplicasToBrokers PASSED

kafka.admin.AdminRackAwareTest > testMoreReplicasThanRacks PASSED

kafka.admin.AdminRackAwareTest > testSingleRack PASSED

kafka.admin.AdminRackAwareTest > testAssignmentWithRackAwareWithRandomStartIndex PASSED

kafka.admin.AdminRackAwareTest > testLargeNumberPartitionsAssignment PASSED

kafka.admin.AdminRackAwareTest > testLessReplicasThanRacks PASSED

kafka.admin.AclCommandTest > testInvalidAuthorizerProperty PASSED

kafka.admin.ConfigCommandTest > testScramCredentials PASSED

kafka.admin.ConfigCommandTest > shouldParseArgumentsForTopicsEntityType PASSED

kafka.admin.ConfigCommandTest > testUserClientQuotaOpts PASSED

kafka.admin.ConfigCommandTest > shouldAddTopicConfig PASSED

kafka.admin.ConfigCommandTest > shouldAddClientConfig PASSED

kafka.admin.ConfigCommandTest > shouldDeleteBrokerConfig PASSED

kafka.admin.DeleteConsumerGroupTest > testGroupWideDeleteInZKDoesNothingForActiveConsumerGroup PASSED

kafka.admin.ConfigCommandTest > testQuotaConfigEntity PASSED

kafka.admin.ConfigCommandTest > shouldNotUpdateBrokerConfigIfMalformedBracketConfig PASSED

kafka.admin.ConfigCommandTest > shouldFailIfUnrecognisedEntityType PASSED

kafka.admin.AdminTest > testBasicPreferredReplicaElection PASSED

kafka.admin.ConfigCommandTest > shouldNotUpdateBrokerConfigIfNonExistingConfigIsDeleted PASSED

kafka.admin.AdminTest > testPreferredReplicaJsonData PASSED

kafka.admin.BrokerApiVersionsCommandTest > checkBrokerApiVersionCommandOutput PASSED

kafka.admin.ReassignPartitionsCommandTest > shouldRemoveThrottleReplicaListBasedOnProposedAssignment PASSED

kafka.admin.ReassignPartitionsCommandTest > shouldFindMovingReplicasMultipleTopics PASSED

kafka.admin.ReassignPartitionsCommandTest > shouldNotOverwriteExistingPropertiesWhenLimitIsAdded PASSED

kafka.admin.ReassignPartitionsCommandTest > shouldFindMovingReplicasMultipleTopicsAndPartitions PASSED

kafka.admin.ReassignPartitionsCommandTest > shouldRemoveThrottleLimitFromAllBrokers PASSED

kafka.admin.ReassignPartitionsCommandTest > shouldFindMovingReplicas PASSED

kafka.admin.ReassignPartitionsCommandTest > shouldFindMovingReplicasMultiplePartitions PASSED

kafka.admin.ReassignPartitionsCommandTest > shouldSetQuotaLimit PASSED

kafka.admin.ReassignPartitionsCommandTest > shouldFindMovingReplicasWhenProposedIsSubsetOfExisting PASSED

kafka.admin.ReassignPartitionsCommandTest > shouldUpdateQuotaLimit PASSED

kafka.admin.ReassignPartitionsCommandTest > shouldFindTwoMovingReplicasInSamePartition PASSED

kafka.admin.ReassignPartitionsCommandTest > shouldNotOverwriteEntityConfigsWhenUpdatingThrottledReplicas PASSED

kafka.admin.ConfigCommandTest > shouldNotUpdateBrokerConfigIfMalformedEntityName PASSED

kafka.admin.ConfigCommandTest > shouldSupportCommaSeparatedValues PASSED

kafka.admin.ConfigCommandTest > shouldNotUpdateBrokerConfigIfMalformedConfig PASSED

kafka.admin.DeleteConsumerGroupTest > testGroupTopicWideDeleteInZKDoesNothingForActiveGroupConsumingMultipleTopics PASSED

kafka.admin.AddPartitionsTest > testReplicaPlacementAllServers PASSED

kafka.admin.ConfigCommandTest > shouldParseArgumentsForBrokersEntityType PASSED

kafka.admin.ConfigCommandTest > shouldAddBrokerConfig PASSED

kafka.admin.AdminTest > testReassigningNonExistingPartition PASSED

kafka.admin.ConfigCommandTest > testQuotaDescribeEntities PASSED

kafka.admin.AdminTest > testGetBrokerMetadatas PASSED

kafka.admin.ConfigCommandTest > shouldParseArgumentsForClientsEntityType PASSED

kafka.admin.AclCommandTest > testAclCli PASSED

kafka.admin.ReassignPartitionsIntegrationTest > testRackAwareReassign PASSED

kafka.admin.AdminTest > testBootstrapClientIdConfig PASSED

kafka.admin.ReassignPartitionsClusterTest > shouldExecuteThrottledReassignment FAILED
java.nio.file.FileSystemException: C:\Users\IBM_AD~1\AppData\Local\Temp\kafka-719085320148197500\my-topic-0\00000000000000000000.index: The process cannot access the file because it is being used by another process.

From the error message, it sounds like one of the prior tests does not do a proper clean-up?!

Jun Rao

unread,

Jun 27, 2017, 1:26:45 AM6/27/17

to Ismael Juma, Vahid S Hashemian, Kafka Users, d...@kafka.apache.org, kafka-clients

Hi, Ismael,

Thanks for running the release. +1. Verified quickstart on the 2.11 binary.

Jun

--
You received this message because you are subscribed to the Google Groups "kafka-clients" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kafka-clients+unsubscribe@googlegroups.com.
To post to this group, send email to kafka-...@googlegroups.com.
Visit this group at https://groups.google.com/group/kafka-clients.
To view this discussion on the web visit https://groups.google.com/d/msgid/kafka-clients/CAD5tkZYey%3DLSri2NpGnh1RNGfCTtZRJyCNMOU7cF0nZW5Ec38g%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

Ismael Juma

unread,

Jun 27, 2017, 8:53:19 AM6/27/17

to Kafka Users, d...@kafka.apache.org, kafka-clients

Thanks Jeff, that's helpful. To be clear, this should not affect the Java client (max.request.size = 1 MB) or librdkafka-based clients (message.max.bytes = 1MB) with the default settings. I'm a bit surprised that Sarama doesn't have a similar mechanism. Seems like we'll have to live with that.

Ismael

On Mon, Jun 26, 2017 at 9:51 PM, Jeff Chao <jc...@heroku.com> wrote:

Hi,

Heroku has been doing additional performance testing on (1) log compaction
and, separately (2) Go clients with older message format against 0.11-rc2
brokers with new message format.

For log compaction, we've tested with messages using a single key, messages
using unique keys, and messages with a bounded key range. There were no
notable negative performance impacts.

For client testing with old format vs new format, we had Sarama Go async
producer clients speaking their older client protocol versions and had
messages producing in a tight loop. This resulted in a high percentage of
errors, though some messages went through:

Failed to produce message kafka: Failed to produce message to topic
rc2-topic: kafka server: Message was too large, server rejected it to avoid
allocation error.

Although this is to be expected as mentioned in the docs (
http://kafka.apache.org/0110/documentation.html#upgrade_11_message_format)
where in aggregate messages may become larger than max.message.bytes from
the broker, we'd like to point out that this might be confusing for users
running older clients against 0.11. That said, users can however work
around this issue by tuning their request size to be less than
max.message.bytes.

This, along with the testing previously mentioned by Tom wraps up our
performance testing. Overall, we're a +1 (non-binding) for this release,
but wanted to point out the client issue above.

Thanks,
Jeff

On Mon, Jun 26, 2017 at 12:41 PM, Vahid S Hashemian <

vahidha...@us.ibm.com> wrote:

> Hi Ismael,
>

> Thanks.
> --Vahid

>
>
>
>
> From: Ismael Juma <ism...@gmail.com>
> To: Vahid S Hashemian <vahidha...@us.ibm.com>
> Cc: d...@kafka.apache.org, kafka-clients
> <kafka-...@googlegroups.com>, Kafka Users <us...@kafka.apache.org>
> Date: 06/26/2017 03:53 AM
> Subject: Re: [VOTE] 0.11.0.0 RC2
>
>
>

> > ------------------------------

Guozhang Wang

unread,

Jun 27, 2017, 3:53:59 PM6/27/17

to Kafka Users, d...@kafka.apache.org, kafka-clients

My vote seems to be eaten by dev@kafka mailing list, have tried a couple of times but they never show up in mailing list archive.

So retrying again to send to all three mailing list.

Guozhang

On Mon, Jun 26, 2017 at 9:36 PM, Guozhang Wang <wang...@gmail.com> wrote:

+1

Verified 0110 web docs and java docs; verified quick start with 2.11 / 2.12 scala versions.

One minor observation: on the web docs we show the cmd for 2.11 scala version; we'd better make it templated with the version number.

1
2
> tar -xzf kafka_2.11-0.10.2.0.tgz
> cd kafka_2.11-0.10.2.0

On Mon, Jun 26, 2017 at 3:53 PM, Ismael Juma <ism...@gmail.com> wrote:

> ------------------------------

--
-- Guozhang

--

-- Guozhang

Guozhang Wang

unread,

Jun 27, 2017, 5:43:26 PM6/27/17

to us...@kafka.apache.org, kafka-clients

+users and clients as well.

---------- Forwarded message ----------
From: Guozhang Wang <wang...@gmail.com>
Date: Tue, Jun 27, 2017 at 2:42 PM
Subject: Re: [VOTE] 0.11.0.0 RC2

To: "d...@kafka.apache.org" <d...@kafka.apache.org>

+1

Verified 0110 web docs and java docs; verified quick start with 2.11 / 2.12 scala versions.

One minor observation: on the web docs we show the cmd for 2.11 scala version; we'd better make it templated with the version number, or just mentioning the following code snippet assumes downloaded 2.11 version.

Guozhang

On Mon, Jun 26, 2017 at 3:53 PM, Ismael Juma <ism...@gmail.com> wrote:

> ------------------------------

--

-- Guozhang

--

-- Guozhang

Ismael Juma

unread,

Jun 27, 2017, 7:40:59 PM6/27/17

to d...@kafka.apache.org, Kafka Users, kafka-clients

This vote passes with 7 +1 votes (3 binding) and no 0 or -1 votes.

+1 votes

PMC Members: Jun, Gwen, Guozhang

Committers: Jason, Ismael
Community: Tom Crayford, Jeff Chao

0 votes: none

-1 votes: none

Thanks to everyone who tested, voted and/or contributed to the release.

I'll continue with the release process and the release announcement will follow in the next few days.

Ismael

P.S. Guozhang's vote was in a separate thread due to some delivery issues when replying to this thread.

On Fri, Jun 23, 2017 at 2:16 AM, Ismael Juma <ism...@juma.me.uk> wrote:

Reply all

Reply to author

Forward