New plugin proposal / repository request: Kafka Events

[lucamilanesio]

Hi all,

I've started a new plugin to publish Gerrit events to Apache Kafka, inspired by the RabbitMQ one.

Why using Kafka instead of RabbitMQ?

- performance

- scalability

- distribution

- fault tolerance

- stream processing

Can anyone create a 'kafka-events' repository on gerrit-review?

Thank you in advance.

Luca.

[David Pursehouse]

Done:  https://gerrit-review.googlesource.com/#/admin/projects/plugins/kafka-events

The group 'plugins-kafka-events' has the Owner permission on refs/*.  Members of that group are you and me; let me know if anyone else should be added.

[lucamilanesio]

Thanks David, first initial version is now pushed for review:

https://gerrit-review.googlesource.com/#/c/88772/

Apache Kafka is much faster than RabbitMQ and automatically manages a lot of stuff such as:

- reconnection in case of failure

- reallocation of partitions across the survivors

- replication and quorum

- coordination with Zookeeper

... and has an out-of-the-box events backup solution on S3.

Luca.

[Magnus Bäck]

On Friday, October 14, 2016 at 00:44 CEST,

lucamilanesio <luca.mi...@gmail.com> wrote:

> Thanks David, first initial version is now pushed for review:
> https://gerrit-review.googlesource.com/#/c/88772/
> Apache Kafka is much faster than RabbitMQ and automatically manages a
> lot of stuff such as:
> - reconnection in case of failure

How is this a feature of the broker? If the RabbitMQ event plugin
doesn't reconnect automatically that's definitely something that
needs to be fixed.

> - reallocation of partitions across the survivors
> - replication and quorum

Why would you say that RabbitMQ doesn't deal with replication?

--
Magnus Bäck | Software Engineer, Firmware Platform Tools
magnu...@axis.com | Axis Communications

[Luca Milanesio]

> On 21 Oct 2016, at 08:53, Magnus Bäck <magnu...@axis.com> wrote:
>
> On Friday, October 14, 2016 at 00:44 CEST,
> lucamilanesio <luca.mi...@gmail.com> wrote:
>
>> Thanks David, first initial version is now pushed for review:
>> https://gerrit-review.googlesource.com/#/c/88772/
>> Apache Kafka is much faster than RabbitMQ and automatically manages a
>> lot of stuff such as:
>> - reconnection in case of failure
>
> How is this a feature of the broker? If the RabbitMQ event plugin
> doesn't reconnect automatically that's definitely something that
> needs to be fixed.

The RabbitMQ plugin does it "by hand" whilst with Kafka the reconnection and reallocation of the partitions to the consumers is automatic :-)
They both have it, with different effort in the client code for managing it.

>
>> - reallocation of partitions across the survivors
>> - replication and quorum
>
> Why would you say that RabbitMQ doesn't deal with replication?

RabbitMQ is a master-slave replication whilst Kafka is master-master (or I would say peer-to-peer).
I am aiming of providing a "events backbone" for a forthcoming Gerrit multi-master setup ... and thus the backbone has to be multi-master by design.