Welcome!

This is a mailing list for kafka client developers to share tips, tricks,

problems, and so on, related to writing client libraries to interact with

the apache kafka server.

Below are some ideas that were floated by Jay Kreps on the kafka

users mailing list regarding how to improve the kafka ecosystem:

One of the short term goals will be to improve the documentation on how

to implement a client.

The Kafka Protocol is described here:

https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol

The kafka server team has tried to make the protocol spec not just a dry

document but also have it share best practices, rationale, and intentions. It
could potentially be even better as there is really a range of options
from a very simple quick implementation to a more complex highly
optimized version. It would be good to really document some of the
options and tradeoffs.

Another goal should be to come up with a standard way of documenting

the features of kafka clients. In an ideal world it would be possible to

get the same information (author, language, feature set, download link,

source code, etc) for all clients. It would be great to standardize the

documentation for the client as well. For example having one or two

basic examples that are repeated for every client in a standardized way.

This would let someone come to the Kafka site who is not a java developer,

and click on the link for their language and view examples of interacting with
Kafka in the language they know using the client they would eventually
use.

Another Idea: Build a Kafka Client Compatibility Kit (KCCK) :-) The idea is this:
anyone who wants to implement a client would implement a simple
command line program with a set of standardized options. The
compatibility kit would be a standard set of scripts that ran their
client using this command line driver and validate its behavior. E.g.
for a producer it would test that it correctly can send messages, that
the ordering is retained, that the client correctly handles
reconnection and metadata refresh, and compression. The output would
be a list of features that passed are certified, and perhaps basic
performance information. This would be an easy way to help client
developers write correct clients, as well as having a standardized
comparison for the clients that says that they work correctly.

Explain me on how Group Consumer Rebalance Works

Hasantha