FYI
I'm currently scanning the Internet for best practices with CQRS/ES to put in use.
And I came across this part of Greg Young's talk on ES (from 38:38 until around 41:54 ~ only 3:20 in total)
I will not try to beat explanation in the video, but I'll summarize for those who what to read the point...
He distinguishes two types of subscription models:
- Producer-driven (e.g. RabbitMQ)
- Consumer-driven (e.g. Kafka)
The problem with Producer-driven subscription system is that it requires additional "control channel" for a consumer to the producer.
Basically, every time a consumer needs data from an event store, it has to use "control channel" to instruct producer to prepare a queue and events in it for this consumer.
The opposite approach is Consumer-driven subscription system where the position in the stream is remembered by the consumer itself.
In this case, the consumer simply positions its own "iterator" anywhere in the stream (the "iterator" is called "offset" in Kafka) and read events sequentially.
Obviously, the absence of "control channel" in the second case is not true - for the consumer-driven subscription the "control channel" is simply elegant built-in "offset"/"iterator" thing.
There are few reasons I'm "enriching" this thread by my post:
- (Correct me if I'm wrong) It makes a perfect additional clarification for "Event Bus versus Event Store" words by Allard. Indeed, if Kafka is an Event Store with its elegant Consumer-driven subscription approach Event Bus becomes redundant as the customer will use private "offset" to consume events.
- And I also remember mailing lists topics which went something like this "How do I replay both historical events from t(0) to t(then) and then all events happened since t(then) until t(now) seamlessly?" - this is what also discussed in the video piece above. Kafka is the natural solution here.
- It also seems like Kafka is the perfect out-of-the-box "Record and Replay" server I've been thinking about (see "Record and Replay – realtime playback of system state" thread - https://groups.google.com/d/msg/axonframework/KDbdy4MN8bA/Gx7n4n7gBgAJ). Obviously, I'm after support for Kafka.