Kafka and Axon Framework thoughts

[Yordis Prieto]

I know Greg probably will dislike this thread from just the title, and I kind of feel

ashamed as a student.

But, as much as I understand the concepts at the architecture level, I am

struggling to move forward.

Not because I don't know how to implement f(state, event) -> state  but because

I don't have anyone that could double check my thoughts and it is quite hard to

know if I am even going for the right path and when you get into complicated

topics your simple f(state, event) -> state doesn't do the trick anymore.

I am trying to find some frameworks that at least I can double check my thoughts

again other people and see where my implementation overlaps.

What are your thoughts about https://axoniq.io/ ?

What you don't like from Axon?

Right now, I am trying to put GraphQL API layer that sends all the mutations (commands)

to a Kafka command stream and then bifurcate into multiple streams based on the domain

model or aggregate root I guess.

I am following this setup https://github.com/capitalone/cqrs-manager-for-distributed-reactive-services

I would love to gather some feedbacks on Kafka and ES and how you are using it in production

right now, especially how you are managing the topics, what I keep struggling is in the aggregates

stream management

[Tomas Ptacnik]

Hi,

in our company we have a component that we call "event splitter" which publishes events into a specific topic based on event metadata.

Each event contains this metadata

- event type (it's basically an aggregate name)

- distribution key

- group topics (optional)

All events are published into one Kafka topic called "events" and then the event splitter republishes them into separate topic(s) based on the "event type" and optionally into other group topics if given.

Distribution key is used for partitioning.

The event splitter is not only republishing the events, but most importantly also validating events against our central events schema repository. Only events with valid schema are published by event splitter.

Best,

Tomas

--
You received this message because you are subscribed to the Google Groups "DDD/CQRS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dddcqrs+u...@googlegroups.com.
Visit this group at https://groups.google.com/group/dddcqrs.
To view this discussion on the web visit https://groups.google.com/d/msgid/dddcqrs/77b800e1-5e4f-42c3-9b6f-8911cb9a2f19%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ben Kloosterman]

First question I have is what's wrong with simpleCqrs ? 

When it doubt its better to start simple,  frame works tend to introduce a lot of poorly understood behavior and make it easy to introduce complexities you don't even need or are ready for ( Sagas/ Snapshotting etc) . Most CQRS / DDD frameworks i have seen are very complex. If your not doing a  big system have  a look at Postgres Marten as its still pretty light but cant do too much and they avoid  the read/ write model by using  1db..  

One issue with the above model Kafka model is its hard (very) to validate..  You want to validate before you create the event and the read model is eventually consistent. Especially parents and children. 

eg 2 very common sequential events
Create Parent
Create Child 

 A UI you can sometimes trust, much less so for an API . So how do you validate if the parent exists , when the create parent message is in the kafka stream waiting to be processed ? 
Do you fail if its not in the query store and force the consumer to retry until its inserted.. ( bad api ) 
Do you read all events and try to find a create  message ?  ( performance with a lot of events and gets harder with more complex validation) 
Do you block till its in the query side ( very poor performance / scalability)
There are more but most are unattractive. 

With millions of streams  you can simply go  does stream exist or build up the state from reading a *few* events to validate and prevent creation of bad events. There is much discussion on set based validation.

We use 
Event Store -> Build memory state models at startup for small ones ( < 1M events) and CosmosDB for larger ones . 
We only have a small part of our domain use events. 
We fire events with state to Eventhub with Kafka for other parties/ services to consume ( we used to use a 2nd EventStore for this but are moving to a cloud managed solution)

Future we are arguing / thinking about
- the idea of having events as the primary storage and playing them into elastic search for querying replacing the read/ query model.
- Do you archive events and lose the ability to create the whole read model ? Or do you / can you do accounting / Bank time based segregation eg 2018 events build 2018 query model.  

Ben

[Yordis Prieto]

> First question I have is what's wrong with simpleCqrs ? 

I am dealing with 56K RPM (and a lot of database transactions), the good thing is that most of those aggregates have a short lifespan but still more than 100 aggregates (big system)

So I can't simply roll with SimpleCQRS if I want to make it to production, right now I am just focused on enabling the infrastructure so others can take advantages of it, which

is why I don't want to use any framework as well (but again I need to double check what I am doing)

I have to think about snapshots, how to manage a lot of streams, and even how to deal with multi-zones

> One issue with the above model Kafka model is its hard (very) to validate..  You want to validate before you create the event and the read model is eventually consistent. Especially parents and children. 

This is why I want to put GraphQL to have some validation layer in front of the message bus.

> . A UI you can sometimes trust, much less so for an API . So how do you validate if the parent exists , when the create parent message is in the kafka stream waiting to be processed ? 

I got lost here.

> Do you read all events and try to find a create  message ?  ( performance with a lot of events and gets harder with more complex validation) 

What do you mean?

I would probably send to a centralize Command Event stream that then bifurcate to a better stream more focus in particular aggregate, still thinking what should I do.

> With millions of streams  you can simply go  does stream exist or build up the state from reading a *few* events to validate and prevent creation of bad events. There is much discussion on set based validation.

Yeah this topic about valdiation, hopefully GraphQL will help me a lot with it

--

You received this message because you are subscribed to the Google Groups "DDD/CQRS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dddcqrs+u...@googlegroups.com.
Visit this group at https://groups.google.com/group/dddcqrs.

To view this discussion on the web visit https://groups.google.com/d/msgid/dddcqrs/CAJT18iaPkGOqS_c%3DvSDVhadaxD1x0%3DMEU3-Spph5RDVDAPstuA%40mail.gmail.com.

[Tomas Ptacnik]

Probably important to mention that we don't use Kafka as an event store, but as a message bus.

We have only a few services from many on our platform that are event sourced and those services have each its own database where one table represents an event store.

Remark there: We already experienced cases where one central event store would save us some troubles.

On the other hand I'm not sure if the single point of failure is worth the risk. It probably depends on how reliable the central event store would be.

T.

[Yordis Prieto]

> those services have each its own database where one table represents an event store.

Good to know this

To view this discussion on the web visit https://groups.google.com/d/msgid/dddcqrs/CAJ7md6WgZcGZX7nTvzkHqEZF9AG%2BRjkFhSJ-ZNHjHP71m4ULng%40mail.gmail.com.

[Ch Pe]

>   we don't use Kafka as an event store, but as a message bus

Do you have a subscription model where bounded contexts subscribe to specific types of messages or does each bounded context evaluate every message in a kafka stream and decide what to do?

To view this discussion on the web visit https://groups.google.com/d/msgid/dddcqrs/CAJ7md6WgZcGZX7nTvzkHqEZF9AG%2BRjkFhSJ-ZNHjHP71m4ULng%40mail.gmail.com.

[Tomas Ptacnik]

Every service subscribe only to messages/topics it's interested in.

To view this discussion on the web visit https://groups.google.com/d/msgid/dddcqrs/CACL874OKp_y6zEmxaUw1-6Pv6oeH2c8uwkeStszgZjuOtLZHfQ%40mail.gmail.com.

[Yordis Prieto]

My take away from this article https://www.confluent.io/blog/put-several-event-types-kafka-topic/

at least in Kafka land

- Topic per Aggregate type.

- Partition key based on the aggregate ID.

Just sharing back the progress

[Tom Widmer]

A few notes about using Kafka as an event store

* Use one stream per aggregate type, with aggregate id as the shard key
* Use permanent retention, or stream the events somewhere more permanent (S3 for example)
* Use another stream for commands, with aggregate id as the shard key again. This allows ensures you don’t process in parallel commands for the same aggregate. You can send responses to a reply stream so the api can respond synchronously to clients
* command processors keep snapshots for each aggregate up to date, possibly caching them in RAM. You need up to date snapshots since there’s no way to read events for just one aggregate instance. You validate the command against the snapshot, write events, send the response then save the snapshot (or batch saves), then process the next command.

Kind regards,
Tom

[Yordis Prieto]

Tom amazing response,

I am quite happy that I was going for the right path thou, this type of feedbacks are the one I am looking for.

Thaks a lot

--
You received this message because you are subscribed to the Google Groups "DDD/CQRS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dddcqrs+u...@googlegroups.com.
Visit this group at https://groups.google.com/group/dddcqrs.

To view this discussion on the web visit https://groups.google.com/d/msgid/dddcqrs/7e0d8034-7e00-4e33-a932-3d26d7cdb5e0%40googlegroups.com.

[Greg Young]

Good luck ;-)

BTW remember to do power pull testing ...

Virus-free. www.avast.com

To view this discussion on the web visit https://groups.google.com/d/msgid/dddcqrs/CAAKqwiEwcGeY48k0rfn6S2F4hdqNOmXrP-Nt2xmbLfYP5ts5-Q%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Studying for the Turing test

[Sandor Agafonoff]

This is going to be an opinionated response, because I'm slightly over the conversation about using Kafka as an event store.

TL;DR
It's not practical, given the alternatives.

Here is how simple it is to see.

Storing events in Kafka by aggregate type, I.e events for multiple aggregates in a single stream, is not scalable as far as the command side goes.

Storing a single aggregate per topic, which would be required to solve the above, is not scalable based on Kafka's core design.

End of discussion.

The frustrating part is, you have the makers of Kafka running around saying event sourcing us a use case. Let's investigate this.

If you look at the material from Kafka, they have a slightly different version of event sourcing. They have confined their definition to be, that you can derive new applications "downstream" of the publisher, I.e the read side of the equation. Now, this is true, but they have no mention of how to use Kafka as an event store for the command, or write side. That's not a mistake. Kafka itself does not have the features baked in to achieve the guarantees required by an event sourced application. Optimistic concurrency, is but one.

While it is true, you can do a bunch of stuff, to get it "working" such as using a single writer, you gotta ask, is it worth it?

There are better features baked into your existing rdbms for this than Kafka.

On the other side though. I love Kafka for distributing events downstream. It's absolutely perfect for this, it's what it was designed for. Combine this with Kafka connect and CDC, you can quite easily have the write side operating with the required level of consistency, and keeping your write and publish action atomic.

So stop hacking the tool, Kafka, to meet your needs. There are much more practical alternatives, but definitely consider Kafka for your event distribution, it wins at that.

[Tom Widmer]

How about if you put commands onto a command topic for the aggregate type, that is also sharded by aggregate id? Then you don’t have a single command handler any more. I.e. with Kinesis streams you can have 500 shards per aggregate type. If you add fan out, you could have 500*500 shards. With Kafka, you can have as many shards as you need I think.

> --
> You received this message because you are subscribed to a topic in the Google Groups "DDD/CQRS" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/dddcqrs/P1JCAX344G0/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to dddcqrs+u...@googlegroups.com.

> Visit this group at https://groups.google.com/group/dddcqrs.

> To view this discussion on the web visit https://groups.google.com/d/msgid/dddcqrs/ee4e8bf6-6f41-4b09-8cbe-5149d6afa445%40googlegroups.com.

[A Holbreich]

Hi Sandor, i liked your answer very much.
could you recomend some event stores implementations. Maybe it doesn't matter but i'm in a the world of JVM based services.

[Ben Kloosterman]

On Thu, Jun 13, 2019 at 12:39 AM Yordis Prieto <yordis...@gmail.com> wrote:

> First question I have is what's wrong with simpleCqrs ? 

I am dealing with 56K RPM (and a lot of database transactions), the good thing is that most of those aggregates have a short lifespan but still more than 100 aggregates (big system)

So I can't simply roll with SimpleCQRS if I want to make it to production, right now I am just focused on enabling the infrastructure so others can take advantages of it, which

is why I don't want to use any framework as well (but again I need to double check what I am doing)

Simple CQRS does all that as its just a light weight wrapper.

I have to think about snapshots, how to manage a lot of streams, and even how to deal with multi-zones

> One issue with the above model Kafka model is its hard (very) to validate..  You want to validate before you create the event and the read model is eventually consistent. Especially parents and children. 

This is why I want to put GraphQL to have some validation layer in front of the message bus.

So you need to build message bus to graphQL without eventual consistency .. thats hard.  

> . A UI you can sometimes trust, much less so for an API . So how do you validate if the parent exists , when the create parent message is in the kafka stream waiting to be processed ? 

I got lost here.

> Do you read all events and try to find a create  message ?  ( performance with a lot of events and gets harder with more complex validation) 

What do you mean?

I would probably send to a centralize Command Event stream that then bifurcate to a better stream more focus in particular aggregate, still thinking what should I do.

All this is complex .The architectural complexity will not result in achieving production readiness fast. 

> With millions of streams  you can simply go  does stream exist or build up the state from reading a *few* events to validate and prevent creation of bad events. There is much discussion on set based validation.

Yeah this topic about valdiation, hopefully GraphQL will help me a lot with it

It wont help , nothing out of the box will,  its a conceptual issue.  

No one i know of uses Kafka to store events due to difficulty in querying and building up state esp for validation and logic BEFORE the event is created which requires previous events,  

Ben

[Ben Kloosterman]

Like Sandor im talking about Kafka for event storage, its fine for commands, but you can also just have a simpler API without a command bus calling the BL. In most cases all the command bus is an API -> command ->  Bus -> Command Handler ->   Domain .. Often API  -> Domain will do  and be a lot more familiar to the team ( since the API calls message on the wire is a command and the controller the handler) . There is no issue here its just a question do you want a lot of plumbing or a command bus for other services to write vs api calls.
 
The key to me is when you validate the API call or Command Message  because calling the read side will have issues for some very common cases due to the fact  the data is not immediately there. Now for sets there are quite a few articles here http://danielwhittaker.me/2017/10/09/handle-set-based-consistency-validation-cqrs/  with the best suggestion being  get your domain to deal with it. You definitely don't want these issues for an aggregate for validation and decisions.  Everyone is attracted to the idea of a managed event store but all the implementations I have seen struggle with either
- Guaranteed order
- Querying small amount of events fast not reading a whole stream it may work in some domains but when it changes  you may have to change your whole system
- Eventual consistency
- Longer term storage

Which leaves a lot of complex code which needs time to mature to deal with the mismatch. CQRS aggregates and read models are simple but its very easy to created complex and ill fitting architectures in fact most frameworks do this by being feature rich eg do you really need a command bus ? 

I do think its possible to build some event store say on PostgreSQL like Marten for some better acceptance in Enterprise architecture  ( and no one wants to manage data anymore) and plug in to components like Kafka/Blobs/S3 but these are not there yet and building a  high perf wrapper on an existing db is not trivial. ( See https://abdullin.com/lokad-cqrs-retrospective/ for this and a good story on richer  frameworks) 

Ben

[Yordis Prieto]

Let me guess something, I think I am getting the heck around it

Kinesis will not work either?

On Tuesday, June 11, 2019 at 10:50:04 PM UTC-7, Yordis Prieto wrote:

[Sandor Agafonoff]

Kinesis has a maximum retention policy of 168hrs. So you're right it won't work. Great for distributing events downstream though.

[Yordis Prieto]

At this point, I will assume that you assume that I will be using Kafka or Kinesis as my Event Store.

I care to have the message bus and the stream architecture out of those tools rather than caring about Event Store.

In the case of Kafka, the retention could be forever and I can dump everything into S3

In the case of Kinesis, the retention could be 168hrs and I can dump everything into S3

So it is hard for me to understand where you are coming from when you say Kafka will no work when for the most part

big companies are leveraging the technology just fine, the same with Kinesis, where it makes sense.

I wouldn't use Kafka as my Event Store (I would probably use S3 or any other database), but I would definitely try

to use Kinesis or Kafka for my message bus and all these messes of streams.

I started doing everything with just Erlang and GenServer, and Actor model and those things,

until I realized that I am adding complexity to my plate for no reason dealing with things that those tools

already figure it out, or at least they are one step ahead of me.

So again, when you say Kafka will no scale, Kafka will no work and so on, I am having such a hard time

understanding you.

Maybe I am missing something

On Mon, Jun 17, 2019 at 2:33 PM Sandor Agafonoff <3lev...@gmail.com> wrote:

Kinesis has a maximum retention policy of 168hrs. So you're right it won't work. Great for distributing events downstream though.

--

You received this message because you are subscribed to the Google Groups "DDD/CQRS" group.

To unsubscribe from this group and stop receiving emails from it, send an email to dddcqrs+u...@googlegroups.com.

Visit this group at https://groups.google.com/group/dddcqrs.

To view this discussion on the web visit https://groups.google.com/d/msgid/dddcqrs/8bf5c40c-e724-4241-b177-88d30147542d%40googlegroups.com.

[Sandor Agafonoff]

The question was for feedback around ES and Kafka, no?

The feedback is what you have. Kafka and kinesis are no good for the event store, but great for your event distribution. To be honest, you can hook up Kafka connect with CDC and have your average rdbms pump messages into Kafka. It's fine.

So I was not assuming that you were using it, rather clarifying the use case of Kafka with ES. So when I say Kafka won't scale, you are right, I am talking about Kafka as an event store.

Now when you say you can dump the events to any simple storage, you are right. I'm just not sure where your aggregates or command handlers are building/reading their current state from.

When you build an aggregate or command handler state, where do you read it from?

To view this discussion on the web visit https://groups.google.com/d/msgid/dddcqrs/CAAKqwiGEyr02Bc1h3Tt0ZQcrtZTM67TMpS%3Dpjf1%2BrmLpEG5BdQ%40mail.gmail.com.

[Yordis Prieto]

Perfect!

Just want to make sure that I didn't misunderstand because I wouldn't use Kafka for

my Event Store for sure 

On Tuesday, June 11, 2019 at 10:50:04 PM UTC-7, Yordis Prieto wrote:

[Greg Young]

What are your requirements that you would need it at all if you have an event store already? Pretty much all support subscriptions of varying types ... Sounds like just an additional moving piece.

--

You received this message because you are subscribed to the Google Groups "DDD/CQRS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dddcqrs+u...@googlegroups.com.
Visit this group at https://groups.google.com/group/dddcqrs.

To view this discussion on the web visit https://groups.google.com/d/msgid/dddcqrs/de0619e9-d6e1-469f-866c-c417ea4e0d67%40googlegroups.com.

[Ben Kloosterman]

So it is hard for me to understand where you are coming from when you say Kafka will no work when for the most part

big companies are leveraging the technology just fine, the same with Kinesis, where it makes sense.

Big companies dont use Kafka to store events there is a reason it has a 7 day limit by default - that's how most people use it.  It is an inter-system / inter service  bus ..  With Event Sourcing the Events are your source of truth / data store for a app / micro service this is a fundamental difference.   Your readdb / view model is not that critical - you delete and rebuild from events when needed. When Kafka is your data store you need far more extensive querying then just  what is the next bunch of messages.

No one has any issues about scaling Kafka / Kinesis as a bus  but querying events for an aggregate thats a different matter. 

Re scaling .. 

The usage of Kafka is  consumers querying for next set of events  this is what it is optimized for. 

With CQRS you may have 100M aggregates and you get the producer querying the store with a query like GetEvents for aggregate id 1234 , type = "My aggregate".  If you have 100M streams each query will just return the events for those aggregates  eg 2 events . If you have a type stream you will return all events of the type instead of 2 you may be returning 1G  events every time you  go  GetEvents for type which is normally when an event arrives .  Kafka does not support having 100M streams its not how its designed to be used.   

To get around this is not easy you may rely on complex caching  ( > 10GB)  or Tomas event slitter by key and then cache. Either way you are building a non out of the box complex system to deal with this mismatch and your solving a problem that DBs not Kafka was designed to handle. 
   

I wouldn't use Kafka as my Event Store (I would probably use S3 or any other database), but I would definitely try

to use Kinesis or Kafka for my message bus and all these messes of streams.

If your using S3 to store and retrieve the events than you have no issue with Kafka scaling as this is your event store. 

My comments above  indicate this is not trivial .  Normally every time an event arrives you need to fetch all events for an aggregate to build up state or validate. Doing a hundred or a thousand blob fetches when a message comes in can be challenging as most systems dont process the next event till the previous one is completed..   
This goes down several paths 
1) You store all events for an aggregate in a mutable S3 object / blob . Obviously this requires locking .  see  Raynad   /lokad-cqrs-retrospective/    append blobs.  Note you need to write these your self in your domain you cant get Kafka to do this else you introduce eventual consistency which is really bad for validation. 
2) Each event is a object/ blob this is what the Kafka / out of the box tools do . So you now need to build an index / way of fetching all the events for an aggregate for the write/ command side. 
3) Use DynamoDB its probably the cheapest dev cost way to store events in a hosted service and supports Type/Id lookups which return multiple records .  You have the basics. For more complex cases you have to write the plumbing .    You may have consistency  issues i havent used DynamoDB .
4) PostgreSQL with Marten can also be used but its design is not mature and has some perf issues better for small projects.

I started doing everything with just Erlang and GenServer, and Actor model and those things,

until I realized that I am adding complexity to my plate for no reason dealing with things that those tools

already figure it out, or at least they are one step ahead of me.

Yes Erlang and Actor are special things for special needs.  Using a tool like Kafka for something  its not designed for is the same thing . 

So again, when you say Kafka will no scale, Kafka will no work and so on, I am having such a hard time

understanding you.

Maybe I am missing something

Its not about the tools its about the concepts.  Have you built a small prototype form simple CQRS  and looked at validation to ensure bad domain events are not created  or increasing performance  ?  

Generally when people are first exposed to CQRS    many focus on the read model and treat it like an existing DB so they quickly move to having shorter term / transient events and adding logic on the read side  ( rather than simple projected views) and then run into eventual consistency issues / validation / complex read model issues.  This is not CQRS  but a strange hybrid i have serious doubts about and probably quite a few of the CQRS failures ..  In those cases i would not have a write model at all.  Why do you want event souring ? That should help me more as i have prob missed some earlier conversations. 

Most problems i see are not found till significantly into an implementation
- Scaling , the sequential processing nature introduce issues. 
- Validation - we trusted the consumer - then moved to read side lookup validation both had serious issues.  
- Many devs don't understand the pattern and why the write side is needed.

Note its perfectly acceptable to run event sourcing like the banks / accounting software and have events in period blocks but this is very different form keeping events for say a rolling  180 days where you loose the ability to recreate the view/ read state  and instead rely on a backed up read db. In those systems i can go get events for customer , for a month 6 years  ago and rebuild the state ( which means i can  fix projection bugs in the past) .


Ben

[João Bragança]

> Just want to make sure that I didn't misunderstand because I wouldn't use Kafka for my Event Store for sure 

The sad reality is that certain vendors are pushing their products that are clearly not event store databases as databases.

--

You received this message because you are subscribed to the Google Groups "DDD/CQRS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dddcqrs+u...@googlegroups.com.
Visit this group at https://groups.google.com/group/dddcqrs.
To view this discussion on the web visit https://groups.google.com/d/msgid/dddcqrs/de0619e9-d6e1-469f-866c-c417ea4e0d67%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Sent from my regular computer

http://twitter.com/thefringeninja

http://www.thefringeninja.com/

[Ben Kloosterman]

On Tue, Jun 18, 2019 at 7:56 PM João Bragança <joao...@braganca.name> wrote:

> Just want to make sure that I didn't misunderstand because I wouldn't use Kafka for my Event Store for sure 

The sad reality is that certain vendors are pushing their products that are clearly not event store databases as databases.

Yeah I have hit that as well. and people with very little experience delivering these solutions in the area think it will magically solve their problems - I try to keep an open mind to new things but we have all been there before.

Ben 

[Yordis Prieto]

What about people from the NYT? I think 

They put all the information into Kafka topics and call it a day.

Do they all have little experience and don’t know what they are doing?

I will no defend Kafka as my Event Store since it will no work for me, but it is mind boggling how you speak with certainly about this topic.

Just depends, it is totally fine to use Kafka retentions as my database, just depends.

For me has more helpful to read about the organization of Kafka streams and how I could architect them, since this was the struggle.

Btw, would you prefer to have multiple event store or a centralized one?

--

You received this message because you are subscribed to the Google Groups "DDD/CQRS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dddcqrs+u...@googlegroups.com.
Visit this group at https://groups.google.com/group/dddcqrs.

To view this discussion on the web visit https://groups.google.com/d/msgid/dddcqrs/CAJT18iarCWtusbTJ%2BSTaX%2BKGXiAkUxWAbhGkn6DJ_ewL0f7YWA%40mail.gmail.com.

[Ben Kloosterman]

On Wed, Jun 19, 2019 at 2:04 PM Yordis Prieto <yordis...@gmail.com> wrote:

What about people from the NYT? I think 

They put all the information into Kafka topics and call it a day.

Do they all have little experience and don’t know what they are doing?

I am a bit annoyed .. and I did not mean or say all ( people is not all people :-) ) . it depends on your event query pattern and how atomic your data is if all your events are atomic without past future dependencies a lot of things become easier  but this is rare in my experience and you may not hit the issues trill well into production which is nasty..   There is a common pattern to send changes and then have historical data but this is not CQRS , it normally involves crud data  changes , another place is  EDA / event processors  .   

For me has more helpful to read about the organization of Kafka streams and how I could architect them, since this was the struggle.

Btw, would you prefer to have multiple event store or a centralized one?

We use 1 per service and 1 acting us our router / event distributor which is supposed to have a 90 day limit but doesn't . In our case the fact a service owns its data is important for replays to other systems after bugs etc. Especially as all that is between teams in different countries,.   It is not necessarily the best way .  

Ben


 

[Greg Young]

Sorry for the long response... Hopefully this clarifies a bit...

Kafka again does *not* have a durability assurance. To be fair with running N instances and ensuring N-? have the same data you will probably be fine! possibly needing to dedupe etc (depending on setup but not horrific), its unlikely to have an issue (and it can be worked around!). To be fair anything that has a durability assurance can still end up in problems with possible data loss! Drives/drivers .. lie about durability and you really need to test the setup doing *actual power pulls* before assuming it to be true! :) I have also heard at varying points they have supported an assurance and have not checked recently (months) so it may exist today, however it has not in the past.

Now as for articles @ NYT use case you mention, it is pushing +- 300 articles (messages)/day from their description, this is a quite different use case to most systems...

"Let's start with The New York Times. “NYTimes.com publishes roughly 150 articles aday (Monday-Saturday), 250 articles on Sunday and 65 blog posts per day,” said Danielle Rhoades Ha, a spokeswoman for the Times, in an email."  https://www.confluent.io/blog/publishing-apache-kafka-new-york-times/

This is from their site. Now let's read through what they are mainly using it for ... *DISTRIBUTING/REBUILDING*. (note I am off an old post here it would not surprise me if these #s are higher due to further information etc being included just using the #s included in the post).

They are putting 300 articles/day then using it for queuing/distributing/rebuilding. What happens if you got really unlucky! and it went down in the few ms after one was published. Oh you just republish (the articles are held in another system! its being used as a long queue for publishing/rebuilding not as the actual book of record). This is a quite reasonable use case and even a republish is not the end of the world in the highly unlikely event (5-6 9s or more depending on number of nodes etc) of a failure. I bet they can even republish *all articles* if they want to (or articles from a given day etc), nice safety net there *just in case* :)

How many events/day are you pushing? Is it more than 300? Can they be republished or is *THIS* the book of record? Many systems I have dealt with are dealing with 5000-10000+/second and *losing them* is a major issue as the event storage *IS* the book of record (think credit card transactions as a bank for an example). In the NYT use case (and many others!) losing them is not a big deal, its not the book of record (they can be recreated by repushing from the originating system!). The key here is *what is the book of record*?

"Do they all have little experience and don’t know what they are doing?"

Not at all their usage here is totally reasonable. That their problem space and requirements are quite likely entirely different than your desired usage is where the issues brought up are ...

To view this discussion on the web visit https://groups.google.com/d/msgid/dddcqrs/B82D62EC-DEF9-433D-A951-357CD45C0A6A%40gmail.com.

For more options, visit https://groups.google.com/d/optout.