CQRS / ES framework

[Sriram Ethiraj]

Im C# Dev, new to DDD/CQRS/ES, Do you people recommend any CQRS/ES framework like EventFlow ? Wonder every one has their own ?

Thanks

Sri

[Ryan Erskine]

I'm also interested in this. We just started trying to use EventFlow. One of the big drawbacks that we are going to have to fix is that there is no guarantee that messages will get published. See: https://github.com/eventflow/EventFlow/issues/609

[Tomas Ptacnik]

Hi,

we have a few Golang services with CQRS,ES and we have no framework in place. All custom built. 

But we had a guy in the team that already had experience with such development from his previous job.

Best, T.

On Tue, 2 Apr 2019, 12:59 Ryan Erskine, <ryan.j....@gmail.com> wrote:

I'm also interested in this. We just started trying to use EventFlow. One of the big drawbacks that we are going to have to fix is that there is no guarantee that messages will get published. See: https://github.com/eventflow/EventFlow/issues/609

--
You received this message because you are subscribed to the Google Groups "DDD/CQRS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dddcqrs+u...@googlegroups.com.
Visit this group at https://groups.google.com/group/dddcqrs.
For more options, visit https://groups.google.com/d/optout.

On Tue, 2 Apr 2019, 12:59 Ryan Erskine, <ryan.j....@gmail.com> wrote:

I'm also interested in this. We just started trying to use EventFlow. One of the big drawbacks that we are going to have to fix is that there is no guarantee that messages will get published. See: https://github.com/eventflow/EventFlow/issues/609

--
You received this message because you are subscribed to the Google Groups "DDD/CQRS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dddcqrs+u...@googlegroups.com.
Visit this group at https://groups.google.com/group/dddcqrs.
For more options, visit https://groups.google.com/d/optout.

[Greg Young]

What precisely do you want the framework to *do*? 

For the domain model side I could write you something in a few hours tops. Projections are a bit more difficult but it depends on your scope and what is needed (eg many read models or one? Physically located together or distributed? Selective replays or entire model replays? Parallel implementation or serial? What kind of monitoring etc? What kind of notifications?). Getting both up and running should be on the order of hours->days though even with 100% custom code (depends how far you want to go on projections). Can you provide more details of what you *need* overall?

Cheers,

Greg

--

You received this message because you are subscribed to the Google Groups "DDD/CQRS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dddcqrs+u...@googlegroups.com.
Visit this group at https://groups.google.com/group/dddcqrs.
For more options, visit https://groups.google.com/d/optout.

--

Studying for the Turing test

[Sriram Ethiraj]

Thank you all for your inputs, I don't have any specific thing in mind, just wanted to see how people are solving their problems,

my 2 cents, if we have a defined framework for starters like me,it helps more where i can see the overall picture via help documents (like eventflow)

looks like pro's move towards their custom ones.

Thanks and Regards
Sriram

[Greg Young]

Its not so much that I want a custom one its that I don't want the complexities of one at all there is a material difference. 

I don't want to be able to write a custom extension to determine how to select my event handling methods in my domain object (I know they are called Apply, they are always called Apply). I don't want to use some convoluted auto-wrapping of my handlers with some bizarre way of describing their ordering via attributes with their dependencies all hooked through a container. I would rather have my +- 20-30 lines of code that does this. I don't want when there is some issue in the wrapping and wire ups to lose 2-3 days debugging through the 5000-10000 lines of code responsible for it :) I like it in code, its quick to write, never breaks in weird non-obvious ways (or emits wrong IL etc ... well ok I have caught the C# compiler doing that once or twice), and a Jr right out of school can understand it.

Make sense?

[Sriram Ethiraj]

i agree your every single statement. i went through that pain too.

[Phil Sandler]

Up front disclaimer: I agree with you in general.

But the lack of a lowest-common-denominator-but-battle-tested framework for .Net impedes adoption for those who don't have a lot of background in ES .  Being able to quickly spin something up using a reliable set of tools that are well documented and have decent community support would help ES get a foot in the door on a lot of projects/teams.

The best analogy I can think of is Prism in the MVVM space, from back in the days when I did WPF and Silverlight development.  After a few projects using Prism and other MVVM frameworks, I understood what I needed (and what I didn't need) well enough to get the benefits out of rolling my own solution.

A simple (or even highly opinionated) framework for the .Net masses could make big difference for adoption.

On Wednesday, April 3, 2019 at 10:11:42 AM UTC-5, Greg Young wrote:

Its not so much that I want a custom one its that I don't want the complexities of one at all there is a material difference. 

I don't want to be able to write a custom extension to determine how to select my event handling methods in my domain object (I know they are called Apply, they are always called Apply). I don't want to use some convoluted auto-wrapping of my handlers with some bizarre way of describing their ordering via attributes with their dependencies all hooked through a container. I would rather have my +- 20-30 lines of code that does this. I don't want when there is some issue in the wrapping and wire ups to lose 2-3 days debugging through the 5000-10000 lines of code responsible for it :) I like it in code, its quick to write, never breaks in weird non-obvious ways (or emits wrong IL etc ... well ok I have caught the C# compiler doing that once or twice), and a Jr right out of school can understand it.

Make sense?

On Wed, Apr 3, 2019 at 10:51 AM Sriram Ethiraj <sri...@gmail.com> wrote:

Thank you all for your inputs, I don't have any specific thing in mind, just wanted to see how people are solving their problems,

my 2 cents, if we have a defined framework for starters like me,it helps more where i can see the overall picture via help documents (like eventflow)

looks like pro's move towards their custom ones.

On Tue, Apr 2, 2019 at 4:24 PM Greg Young <gregor...@gmail.com> wrote:

What precisely do you want the framework to *do*? 

For the domain model side I could write you something in a few hours tops. Projections are a bit more difficult but it depends on your scope and what is needed (eg many read models or one? Physically located together or distributed? Selective replays or entire model replays? Parallel implementation or serial? What kind of monitoring etc? What kind of notifications?). Getting both up and running should be on the order of hours->days though even with 100% custom code (depends how far you want to go on projections). Can you provide more details of what you *need* overall?

Cheers,

Greg

On Mon, Mar 18, 2019 at 1:49 AM Sriram Ethiraj <sri...@gmail.com> wrote:

Im C# Dev, new to DDD/CQRS/ES, Do you people recommend any CQRS/ES framework like EventFlow ? Wonder every one has their own ?

Thanks

Sri

--
You received this message because you are subscribed to the Google Groups "DDD/CQRS" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ddd...@googlegroups.com.

Visit this group at https://groups.google.com/group/dddcqrs.
For more options, visit https://groups.google.com/d/optout.

--

Studying for the Turing test

--
You received this message because you are subscribed to the Google Groups "DDD/CQRS" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ddd...@googlegroups.com.

Visit this group at https://groups.google.com/group/dddcqrs.
For more options, visit https://groups.google.com/d/optout.

--

Thanks and Regards
Sriram

--
You received this message because you are subscribed to the Google Groups "DDD/CQRS" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ddd...@googlegroups.com.

Visit this group at https://groups.google.com/group/dddcqrs.
For more options, visit https://groups.google.com/d/optout.

[Greg Young]

What *precisely* would be in this "highly opinionated framework"? I think you will quickly find this is where the issues are ;-)

To unsubscribe from this group and stop receiving emails from it, send an email to dddcqrs+u...@googlegroups.com.

Visit this group at https://groups.google.com/group/dddcqrs.
For more options, visit https://groups.google.com/d/optout.

[Phil Sandler]

As someone who is (much) more a consumer of frameworks than a designer of frameworks, I'm not sure I'm the best person to answer that question.  I understand where you're coming from--I can think of at least one .Net ES framework that I was excited to try but was too opinionated for my taste; or, to be more precise, I didn't agree with the opinions/decisions the framework developers made.

 

But in my own experience, using NEventStore for my first ES implementation made a huge difference in getting the ball rolling.  It allowed me to focus more on modelling the domain and less on writing plumbing code.  At the time, I would not have had the confidence to write it myself, and a project that has benefitted greatly from ES would have had to have been architected using a more traditional approach.

 

To be fair to your point--I had to write all the projection code from scratch, and the lessons learned during that process were immeasurable.  

[Ben Kloosterman]

IMHO this is too heavy just use something like SimpleCQRS and an in memory read model to start with grow from there and make sure you don't have a CRUD domain.  Especially if your new,  you can get right into the domain ASAP , it does not take too long to use something like Event Store. 

The biggest issue I see if people get into Sagas , Replays , SnapShots and complex eventually consistent readmodels / projections when its best to avoid these things until you need them. A library with so  many features is going to be a burden for devs not familiar with CQRS/ES when some more adventurous types people these feature. 

The things they implement maybe a few lines for the consumer but they have a heavy technical burden, people make mistakes and when it goes wrong you don't want to be in a position where you need to know how a large complex lib works . 
KISS.

Also consider the pattern where you fetch data built from events but have public properties on an aggregate for queries this works well in document style query by id systems and means you dont have an eventually consistent read model ( or you just have some small ones) .  When you have lots of query requirements then build your eventually consistent read model though im leaning more to things like Elastic Search.. 

Ben

[Greg Young]

It's a huge architectural mistake to have "consistent read models" you would literally be negligent to do this under almost all circumstances. I can struggle to even come up with a valid one but it's a struggle. There are ways of assuring receiving information for an operation but there is a drastic difference between that and the entire model!

--

You received this message because you are subscribed to the Google Groups "DDD/CQRS" group.

To unsubscribe from this group and stop receiving emails from it, send an email to dddcqrs+u...@googlegroups.com.

[Raoul Duke]

what are the right boundaries for read consistency? “it depends,” i know, i know...

[Ben Kloosterman]

It's a huge architectural mistake to have "consistent read models" .

I could have been clearer .. The emphasis was more on the complex  in "complex eventually consistent readmodels".   Agree a separate DB will pretty much be eventually consistent  or limited to 100tps - often the DB becomes a traditional DB with lots of logic ..  rather than simple views.  Have had lots of  issues with event play requiring a consistent state. 

Note:   I cant see anything wrong with doing consistent reads from events ( transactions) banking and accounting systems have been doing it for decades and its consistent.  Its just you cant build GP  queries on it but sometimes you don't have a requirement for that ( and this is where I think Elastic search is better just feed the events in - those systems are designed to take streams  ) . 

Ben

[boaby...@gmail.com]

Just my 2 cents that deviates a bit from the original question...

The problem with looking for a "CQRS framework" is that CQRS is really just another name for quite a common way we've all been building many applications since the traditional years. In fact I can hardly think of any production systems I've worked on that can't be described as CQRS/ES.

It could well be a synonym for SOA. I mean, if you look at most SOA systems, you’ll see close parallels with all the CQRS/ES/DDD vocabulary. And because SOA is a pretty established discipline, you can easily find many mature tools/frameworks for your specific needs (e.g. SOA, SIEM, BI), but you hardly see the label “CQRS”, because to say "CQRS project" is rather meaningless in itself. Your application may end up looking CQRS in the end, but it’s a rather inconsequential outcome, not a goal on its own. It's sort of like saying "a framework to build a yellow car".

Let's look at typical SOA patterns and see how they relate to CQRS/ES/DDD concepts (in bold), and there are useful tools/frameworks that helps specific needs.

Let's pick a commerce side of an organisation, you'll typically have some services e.g. CRM, e-commerce, billing-systems, CPQ (configure, pricing, quoting), product-master, provisioning, dunning, accounting, identity, export-control, taxes, etc. Some of them are custom-built, some off-the-shelf (salesforce, netsuite, Xero, Avatax, etc). These things are essentially Bounded Contexts. Each of these services perform a very specific set of business functions (i.e. Commands, e.g. "place an order", "allocate revenue" etc). And they have their own specialised views of common business concepts (e.g. the noun "customers" and "orders" are modelled differently between CRM, billing-system, and accounting systems).

Integrating all these services with each other can become a mess really quickly - to perform a simplest task you'll need to issue commands to dozens of services. A common SOA pattern is to put an ESB in the middle (e.g. mulesoft, serviceMix) that acts as the orchestrator between services. ESBs gives you all the adapters you can ask for to connect with all your various services. These adapters are your Anti Corruption Layers, and you define transformers (Context Mapping) to transform these services into a universal business-domain-model (BDM), i.e. Ubiquitous Language. These BDM activities are pushed into a data stream (e.g. kinesis, kafka, scoop, flume). So now every service owner only needs to worry about pushing/listening to BDM, rather than how to talk to hundreds of different services.

All these events are eventually ingested into an MDM (Master-Data-Management, e.g. tibco, talend, informatica) that creates unified master-records by correlating data from all those services within the entire organisation (i.e. Projections) and reconcile any inconsistencies. The clean master-record output is stored in a data-warehouse in various shapes for different querying needs (Read Model), e.g. redshift, spark, elastic-search, graphql.

Now to complete the loop, all the front-end applications will make Queries to these data-warehouse, so in a single graphql query you can get a 360-degree view of a customer across everything e.g. billing-systems, CRM, provisioning, e-commerce, entitlements, etc. From these front-end applications you can then initiate business operations via the ESB that will issue Commands to individual services, which completes the circle.

Now there are other things too e.g. customer insights, ML, BI, SIEM that basically work on event-sourcing, i.e. it lets you construct information on-demand by processing historical raw events (structured/unstructured), and there are special purpose tools out there e.g. GA/GTM, snowplow, optimizely, splunk, mahout, spark.

PS: there are CQRS frameworks like Axon. My point is that there are also a lot of other products out there that help you in SOA and micro-services environments - they just don't typically wear any CQRS badge.

On Thursday, 4 April 2019 01:51:23 UTC+11, Sriram Ethiraj wrote:

Thank you all for your inputs, I don't have any specific thing in mind, just wanted to see how people are solving their problems,

my 2 cents, if we have a defined framework for starters like me,it helps more where i can see the overall picture via help documents (like eventflow)

looks like pro's move towards their custom ones.

On Tue, Apr 2, 2019 at 4:24 PM Greg Young <gregor...@gmail.com> wrote:

What precisely do you want the framework to *do*? 

For the domain model side I could write you something in a few hours tops. Projections are a bit more difficult but it depends on your scope and what is needed (eg many read models or one? Physically located together or distributed? Selective replays or entire model replays? Parallel implementation or serial? What kind of monitoring etc? What kind of notifications?). Getting both up and running should be on the order of hours->days though even with 100% custom code (depends how far you want to go on projections). Can you provide more details of what you *need* overall?

Cheers,

Greg

On Mon, Mar 18, 2019 at 1:49 AM Sriram Ethiraj <sri...@gmail.com> wrote:

Im C# Dev, new to DDD/CQRS/ES, Do you people recommend any CQRS/ES framework like EventFlow ? Wonder every one has their own ?

Thanks

Sri

--
You received this message because you are subscribed to the Google Groups "DDD/CQRS" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ddd...@googlegroups.com.

Visit this group at https://groups.google.com/group/dddcqrs.
For more options, visit https://groups.google.com/d/optout.

--

Studying for the Turing test

--
You received this message because you are subscribed to the Google Groups "DDD/CQRS" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ddd...@googlegroups.com.

Visit this group at https://groups.google.com/group/dddcqrs.
For more options, visit https://groups.google.com/d/optout.

[Greg Young]

"Now there are other things too e.g. customer insights, ML, BI, SIEM that basically work on event-sourcing, i.e. it lets you construct information on-demand by processing historical raw events (structured/unstructured), and there are special purpose tools out there e.g. GA/GTM, snowplow, optimizely, splunk, mahout, spark."

I have never heard of *anyone* using something like splunk for anything resembling eventsourcing nor have I see any indication that it can do anything resembling eventsourcing

To unsubscribe from this group and stop receiving emails from it, send an email to dddcqrs+u...@googlegroups.com.

Visit this group at https://groups.google.com/group/dddcqrs.
For more options, visit https://groups.google.com/d/optout.

[Boaby Smith]

Not sure we have the same definition of event-sourcing.

https://martinfowler.com/eaaDev/EventSourcing.html

“Event Sourcing ensures that all changes to application state are stored as a sequence of events. Not just can we query these events, we can also use the event log to reconstruct past states, and as a foundation to automatically adjust the state to cope with retroactive changes.”

Splunk is an excellent tool for that - it allows you to get data such as operational & business metrics, user behavior, fraud and threat detections - those are not application states that are simply retrieved from somewhere, instead it reconstruct them on the fly by processing through dozens of TBs/day of raw events very effortlessly. It allows you to retroactively go back to any particular point in time, or build a time-series of certain derived facts that you’re interested in.

I know that’s a very specific purpose (dev-ops and analytics) - you won’t use it for all-things event-sourcing. But that’s exactly my point. There are mature tools out there that really really excels at their specialised purpose, and they don’t usually carry “CQRS” or “ES” badge.

Another example of specialised ES tool is billing platforms. It collects billable events (e.g. usages), and generates real-time/historical billing data from those events. Most platforms allow you to plug your own code to build your own custom logic, but in the end this is a tool that’s designed for a specific use-case, and it doesn’t carry “ES” mentioned because that’s not what matters.

I think what’s important is to identify what you actually want to achieve. The fact that the end result fits the description of CQRS or event-sourcing is incidental - not a goal.

I’ve been in a couple of “CQRS projects” where everything is overly complicated and full of engineering DIY madness because they were do hung up with CQRS, i.e. they set out to build the project with CQRS in mind. It didn’t end well.

Which is ironic because most projects that never had the word “CQRS” ever mentioned have all usually fit CQRS just fine (i.e. you send commands to A, and query back from B), and they run well too.

[Greg Young]

re Splunk:

By this logic any log is Event Sourcing, even syslog. Its fine if you want to have it but you have delved off into nonsensical definitions. I would not consider the log from my webserver to be an event sourced system.

"Another example of specialised ES tool is billing platforms. It collects billable events (e.g. usages), and generates real-time/historical billing data from those events. Most platforms allow you to plug your own code to build your own custom logic, but in the end this is a tool that’s designed for a specific use-case, and it doesn’t carry “ES” mentioned because that’s not what matters."

These *are* often event sourced and have been said to be Event Sourced since the very beginning no? They often handle how they serialize events in different ways though many are even built up explicitly on event logs (I have worked on quite a few! they are all over 20-30 years old :O built on the *same ideas*)

[Boaby Smith]

“Storing” the logs is not event sourcing. “Processing” the logs to reconstruct a quantitative view about your application at a given point in time - that’s event sourcing.

Besides, we don’t push only logs to splunk. We also push application events and analytics tracking data to it too. We also install connectors to our netsuite and billing system so all billing events and invoices are also fed to splunk.

It basically tracks everything that happens in the whole company, you can derive almost every information by processing the right events.

You don’t start off looking at this problem thinking “what ES tool should I use”? You should think “what SIEM or analytics tools out there”? Because there’s no general-purpose ES tool that can come close to Splunk at performing this particular task.

[Boaby Smith]

Also tools like kafka shows up a lot in ES discussions.

What do you think linkedin created and use kafka for?

Ingesting logs.

[Sandor Agafonoff]

I investigated Kafka heavily to use for event sourcing. It's not a good fit. It's great for logs, and yes, you can later derive many useful analytics etc. But in terms of the application which is producing events, needing to read the same events to create the next state, it's not possible, given the consistency properties of Kafka. In terms of downstream applications using the events produced, it is fine.

So, to event source an application, you need an isolated event store per service, and then a broadcast mechanism to Kafka or splunk, or whatever, for downstream applications.

I also know Kafka will tell you that you can use streams to achieve event sourcing, but it's not really possible.

[Greg Young]

The biggest one to mention is that kafka is not durable which is kind of important for events in an event sourced systems, its meant as a transport not as a store.

--
You received this message because you are subscribed to the Google Groups "DDD/CQRS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dddcqrs+u...@googlegroups.com.
Visit this group at https://groups.google.com/group/dddcqrs.
For more options, visit https://groups.google.com/d/optout.

[Boaby Smith]

Specifically, kafka is primarily useful only for ingesting events.
It’s not very useful at processing or even storing them. That’s the realm of where ever those events are ingested into (hdfs, spark, redshift)

[Boaby Smith]

I.e. it is possible to achieve event-sourcing with kafka. But it’s only a tool to collect events rapidly from your services into your data lakes where the actual processing happens.

[Kalin Maldjanski]

But in terms of the application which is producing events, needing to read the same events to create the next state, it's not possible, given the consistency properties of Kafka.

Why is that exactly? Kafka offers topics and partitions and you have the freedom to manage the partitions as you wish and you have guaranteed consistency within a partition.

 The biggest one to mention is that kafka is not durable

Greg, what do you mean by that exactly?  You can have infinite message retention unlike many other message brokers.

On Sunday, April 7, 2019 at 6:55:07 AM UTC+2, Greg Young wrote:

The biggest one to mention is that kafka is not durable which is kind of important for events in an event sourced systems, its meant as a transport not as a store.

On Sun, Apr 7, 2019 at 12:34 AM Sandor Agafonoff <3lev...@gmail.com> wrote:

I investigated Kafka heavily to use for event sourcing. It's not a good fit. It's great for logs, and yes, you can later derive many useful analytics etc. But in terms of the application which is producing events, needing to read the same events to create the next state, it's not possible, given the consistency properties of Kafka. In terms of downstream applications using the events produced, it is fine.

So, to event source an application, you need an isolated event store per service, and then a broadcast mechanism to Kafka or splunk, or whatever, for downstream applications.

I also know Kafka will tell you that you can use streams to achieve event sourcing, but it's not really possible.

--
You received this message because you are subscribed to the Google Groups "DDD/CQRS" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ddd...@googlegroups.com.

Visit this group at https://groups.google.com/group/dddcqrs.
For more options, visit https://groups.google.com/d/optout.

[Alexander Langer]

Am 07.04.19 um 10:38 schrieb Kalin Maldjanski:

> But in terms of the application which is producing events, needing
> to read the same events to create the next state, it's not possible,
> given the consistency properties of Kafka.
>
>
> Why is that exactly? Kafka offers topics and partitions and you have the
> freedom to manage the partitions as you wish and you have guaranteed
> consistency within a partition.

https://medium.com/serialized-io/apache-kafka-is-not-for-event-sourcing-81735c3cf5c

[Kalin Maldjanski]

Thx for the reply and the article. 

My limited experience with it shows that there are solutions to those and if you set the right design/configuration and understand the tradeoffs you can have a great advantage of Kafka advanced scalability, fault-tolerance and streaming capabilities.

My 2 cents/solutions on mentioned issues in the article.

2) Consistent writes

This really depends how you design your aggregates/writers/producers also optimistic concurrency control is not trivial and not elegant at all. I would rather use a 'single writer'. For example I would never write to the same entity or topic/key pair (key defines the partition) from >1 writer nodes. I would have a node responsible for that particular aggregate (like the actors akka). 

In practice using the right partitioning with this approach should not have any performance impact as mentioned in the article, on the contrary - you will have the right sharding that guarantees strong consistency and ready for scaling out.

1) Loading current state 

This could be indeed a pain point. However, as I mentioned above you can have better design of your partitioning which suits your needs (i.e. aggregate id as the partition key) and thus solve the 'topic discoverability' + the 'read all events' issue to 'read some events' compromise. 

Of course you cannot have too many topics nor partitions as both result in more RAM and CPU and then you will end-up sharing those streams i.e. you still need to go by some events that you don't care but this overhead should be negligible depending of the amount of events you have. In case it isn't then you indeed could use some indexing/database in front to query fast your events or use snapshots. 

Also practically you would not want to load-up you aggregate every time you have a request but keep it in memory for some time (again like an actor model approach)

I would love to hear some feedback from a real life production use case :)

[Andrew Young]

Confluence likes to market Kafka as a possible Event Store but if you ever really tried to use it as such, you will realize the pain in trying to make it work. Kafka is great for streaming messages in-flight but I would look elsewhere for an Event Store.

Case in point: just try getting all the events for an Aggregate ID in a topic/partition and you'll see the pain quickly. 

[Greg Young]

 The biggest one to mention is that kafka is not durable

"Greg, what do you mean by that exactly?  You can have infinite message retention unlike many other message broke"

Sorry for using what might be a formal term. Durability is an assurance often discussed with things such as databases/message queue. If I pull the power plug, can I lose data? With kafka the answer is *yes*. I have not reviewed their source recently but as of the last time I did they were *still not durable* though made varying claims that they were. This is a crucial thing to be considering when looking at messaging queues, especially if comparing performance, let me rephrase it differently, once I have put a message in a queue is it assured to not be *lost*.

[Kalin Maldjanski]

Thought the formal term was obvious as far as I know you have a few configuration options to manage durability guarantees :  flush.ms, min.insync.replicas and acks for producers. 

However, unlike you I haven't looked deeply in the code to double check that behaviour and I now will probably do so.

[Greg Young]

None of those things make something durable. Durability is defined as me making sure information is persistent on disk before I acknowledge your operation. Durability is a simple yet rigid formal definition. You can't be "usually" or "occasionally" durable. 

--

You received this message because you are subscribed to the Google Groups "DDD/CQRS" group.

To unsubscribe from this group and stop receiving emails from it, send an email to dddcqrs+u...@googlegroups.com.

Visit this group at https://groups.google.com/group/dddcqrs.
For more options, visit https://groups.google.com/d/optout.

[Greg Young]

correction: persistent including for a power-cycle ... There are other mechanisms than disk and I should not exclude them.

There are many systems that do similar things ... They are not durable, they focus on the lowering the probability of a failure often to the point of it not mattering. This is a completely legitimate strategy! There are reasons why I may even prefer this! They are not however durable. Remember durability has a formal definition. There is not "usually durable" etc. 

BTW on a side note many people insist on running durable systems then run them on non-durable hardware. OK! These people we can throw rocks at :P

[Kalin Maldjanski]

haha nice yoda quote :)

OK point taken :)

What I meant is for example min.insync.replicas combined with acks=all ensures data is even replicated on another node(s) when successful write is confirmed. 

Or if you set flush.messages=1 kafka will fsync after every message which should bring it to the formal durability definition 

Again, I don't know how is this actually implemented :)

To unsubscribe from this group and stop receiving emails from it, send an email to ddd...@googlegroups.com.

Visit this group at https://groups.google.com/group/dddcqrs.
For more options, visit https://groups.google.com/d/optout.

--

Studying for the Turing test

[Greg Young]

"Or if you set flush.messages=1 kafka will fsync after every message which should bring it to the formal durability definition "

Actually I don't think it will! IIRC they fsync asynchronously! Thus a response could be sent without an fsync. I would need to dig through code on that though, been a while.

To unsubscribe from this group and stop receiving emails from it, send an email to dddcqrs+u...@googlegroups.com.

Visit this group at https://groups.google.com/group/dddcqrs.
For more options, visit https://groups.google.com/d/optout.

[Andrew Young]

https://kafka.apache.org/documentation/#design_filesystem

All data is immediately written to a persistent log on the filesystem without necessarily flushing to disk. In effect this just means that it is transferred into the kernel's pagecache.

Basically, it is written to "disk" but NOT written to disk. :)

You received this message because you are subscribed to a topic in the Google Groups "DDD/CQRS" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/dddcqrs/7ZbPGJ0Re5w/unsubscribe.
To unsubscribe from this group and all its topics, send an email to dddcqrs+u...@googlegroups.com.

[Greg Young]

There are some options around this IIRC... But yes in general its not durable.

[Boaby Smith]

If I pull the power plug, can I lose data? With kafka the answer is *yes*. 

That would never be acceptable. Kafka has always handled replications from the very start. They recommend that for prod you'd never want it any less than 3 (eventually, e.g. DR) and 2 synchronous-replicas (i.e. kafka will only ack your writes once at least 2 replicas are committed).

once I have put a message in a queue is it assured to not be *lost*.

I'd say that's an absolute requirement for any prod-worthy tool. "Durable" or "persistent", I don't think losing messages in the case of failures is ever acceptable. Kafka had been battle-tested in many production systems long before they graduated from apache incubation status, so I think we're fine there.

--

You received this message because you are subscribed to the Google Groups "DDD/CQRS" group.

To unsubscribe from this group and stop receiving emails from it, send an email to dddcqrs+u...@googlegroups.com.

[Kalin Maldjanski]

haha :) yep confusing indeed but what they mean by 'persistent log on the filesystem' is the kernel page caches, but without fsync(moving the data to a non-volatile storage) there could be no guarantee the data could be retrieved after reboot/crash.

in essence what I think they try to point out is simply they skip in-memory cache and use only the kernel buffers.

As a result of these factors using the filesystem and relying on pagecache is superior to maintaining an in-memory cache or other structure

 And also they advise to let the system handles flushes but use replication acks to re-assure your events are persistent elsewhere 

flush.ms - In general we recommend you not set this and use replication for durability and allow the operating system's background flush capabilities as it is more efficient.

but yea ... :)

You received this message because you are subscribed to a topic in the Google Groups "DDD/CQRS" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/dddcqrs/7ZbPGJ0Re5w/unsubscribe.

To unsubscribe from this group and all its topics, send an email to ddd...@googlegroups.com.

[Hendry Luk]

Well if your server explodes the answer is also *yes*: you can lose data. As can any other databases.

The key is as always: redundancy.

And Kafka in particular, coming originally from hadoop root (in fact it’s still running on zookeeper), is built specifically for massively-decentralised, highly fault-tolerant, self-healing environment. After all, hadoop’s core philosophy is to run entirely on cheap disposable commodity hardware that’s pre-assumed to inevitably break all the time. Cattle not pets.

So if you’re worried about “lost” messages, I’d say kafka probably fares better than most.

Having said that, I agree with many comments here: it is designed for streaming, not a permanent data store. Possible, but it has a very restrictive set of functionalities and administrative capabilities to be your primary production database.

--

You received this message because you are subscribed to the Google Groups "DDD/CQRS" group.

To unsubscribe from this group and stop receiving emails from it, send an email to dddcqrs+u...@googlegroups.com.

[dharmj...@gmail.com]

Hi Tomas,

I am very new to DDD/CQRS/ES so please ignore if I ask something fundamental. I have also started working on this subject and thinking of using golang. How do you model various DDD constructs in golang(AggregateRoot,Agrregates,Entities,Value Objects etc.)

Thanks,

Dharmjit Singh

On Tuesday, 2 April 2019 21:46:57 UTC+5:30, Tomas Ptacnik wrote:

Hi,

we have a few Golang services with CQRS,ES and we have no framework in place. All custom built. 

But we had a guy in the team that already had experience with such development from his previous job.

Best, T.

On Tue, 2 Apr 2019, 12:59 Ryan Erskine, <ryan.j...@gmail.com> wrote:

I'm also interested in this. We just started trying to use EventFlow. One of the big drawbacks that we are going to have to fix is that there is no guarantee that messages will get published. See: https://github.com/eventflow/EventFlow/issues/609

--
You received this message because you are subscribed to the Google Groups "DDD/CQRS" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ddd...@googlegroups.com.

Visit this group at https://groups.google.com/group/dddcqrs.
For more options, visit https://groups.google.com/d/optout.

On Tue, 2 Apr 2019, 12:59 Ryan Erskine, <ryan.j...@gmail.com> wrote:

I'm also interested in this. We just started trying to use EventFlow. One of the big drawbacks that we are going to have to fix is that there is no guarantee that messages will get published. See: https://github.com/eventflow/EventFlow/issues/609

--
You received this message because you are subscribed to the Google Groups "DDD/CQRS" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ddd...@googlegroups.com.

[Greg Young]

To be fair O_DIRECT is absolutely worth a discussion ...

<< runs ...

vary far away ...

further than the cache can see ...

or could it ...

shit ...

To unsubscribe from this group and stop receiving emails from it, send an email to dddcqrs+u...@googlegroups.com.

Visit this group at https://groups.google.com/group/dddcqrs.
For more options, visit https://groups.google.com/d/optout.

[Tomas Ptacnik]

Hi Dharmjit,

we model it as structs with methods. 

Similar to classes with methods in object oriented languages like Java.

Be prepared that if you do DDD with go, you will have to either give up idiomatic golang or you will have to do DDD differently than you would do it for example in Java.

Reasons are for example:

- visibilities in golang are a bit different than in other languages

- golang has implicit interfaces vs explicit interfaces in other languages

- there might be more :)

Best,

Tomas

To unsubscribe from this group and stop receiving emails from it, send an email to dddcqrs+u...@googlegroups.com.

Visit this group at https://groups.google.com/group/dddcqrs.

To view this discussion on the web visit https://groups.google.com/d/msgid/dddcqrs/6e793246-f1c3-41fc-95be-0e7262ef1cf5%40googlegroups.com.