Alternatives to CQRS and event sourcing

[Johan 't Hart]

Hi all

We are considering to use CQRS and eventsourcing for our new project, which we want to build in the cloud.
We already decided to go with microservices and docker/Kubernetes.
And although I'm very enthusiastic about event sourcing, my manager is not convinced to go for it because he claims it is not mainstream enough and he thinks some cons are hard to handle. E.g.
- never be able to delete data,
- forever and quickly growing disk space needs
- database maintainer can't fix database errors easily
- no managed event store in Azure

Hence he wants me (as an architect) to consider and compare alternative architectures.
Problem is, when crud is not enough (because e.g. of the reactive nature of the app), and when we do want to use the microservices architecture, I can't find alternatives. I look around a lot, but when I search for an alternative to CRUD in MSA, the result is always CQRS with some form of event sourcing.

So hence my question: is there a serious (mainstream) alternative for CQRS when CRUD is not enough? If it's only for comparing the pros and cons.

I realize this might not be the best place to ask this question. But hey, I can at least try...

Greeting
Johan

[Matt Cooke]

There is a lot of suggestions here:

https://microservices.io/patterns/microservices.html

Including things like https://microservices.io/patterns/data/database-per-service.html

You could choose to output events or some other type of messages to provide inter-service communication (if needed) without using event sourcing within each service.  You can also create one or more eventually consistent read model without event sourcing - in fact it’s quite common to asyncronously load data (ETL) into a unified reporting/OLAP store from multiple primary databases for reporting and analytics.

You can also use CQRS whilst using one of more database(s)/datastore(s) as the authoritative model instead of an event store.

Also some services could be based on event sourcing and work with others that are not.

I should caveat I’m not an expert in Microservices or event sourcing - so all the above is mainly theoretical understanding or just basic logic...

Kind regards,

Matt

Sent from my iPhone

--
You received this message because you are subscribed to the Google Groups "DDD/CQRS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dddcqrs+u...@googlegroups.com.
Visit this group at https://groups.google.com/group/dddcqrs.
For more options, visit https://groups.google.com/d/optout.

[Vikas Pandey]

May be for a new project a microservices architecture is not really best thing. You could start with Monolith first strategy and then once you have a matured system then you could try to break it in microservices.

Take a look at following blog for some pros and cons.

https://martinfowler.com/bliki/MonolithFirst.html

Regards

Vikas

[Matthew Cooke]

I second that about starting with a monolith. I would consider using DDD modelling (avoiding every concept directly referencing every other concept in the model) as this helps make it easier to pull out into microservices later should you wish.

[Chris Richardson]

Hi,

I think there are two issues.

The first issue is, as others have pointed out, what architecture to use: monolithic or microservices. If you are a small team working then most likely the monolithic architecture is the best choice. Especially, if the domain is unfamiliar. You don't have the complex issues that the microservices architecture solves and its drawbacks will slow you down. If, however, you are a large team developing a complex application then the microservice architecture is likely a good choice.

If you have chosen the microservice architecture, then the second issue you need to address is how to solve the distributed data management problems you have created for yourself - see https://microservices.io/patterns/data/database-per-service.html. You will very likely need to use the Saga pattern to maintain data consistency between services and CQRS pattern to implement queries that retrieve data from multiple services.

Both patterns rely on some form of asynchronous messaging between services.  When using orchestration-based sagas, services communicate using request/asynchronous response style messaging. However, when using orchestration-based sagas and CQRS, services publish domain events when data (ie. aggregates) change. 

Event sourcing is one way to publish domain events when data changes. However, it's not the only way. You might, for example, not need the other benefits of event sourcing, such as aggregate history. An application can use 'regular' persistence and  publish events explicitly. You need do need to make sure that this happens as part of a database transaction. See the transactional messaging pattern.

I hope this helps.

Chris

-- 

Learn microservices - http://learn.microservices.io

Microservices application platform http://eventuate.io

Consulting and training http://chrisrichardson.net

[Johan 't Hart]

Thanks all for these quick replies! It is very useful.

As a summary the advice is to stay with a monolith and only go for microservices when you need to.
Sounds like good advice. However we already decided to go for microservices and are building them already for almost a year now. (Which was probably a decision taken too soon in hindsight. But it is going to be a complex application anyway and we identified some domains already.)

I've scanned lots of the very good website https://microservices.io already to look for alternatives to event sourcing and CQRS in an MSA context. But also here it is in my understanding summarized like:

- In MSA you best use database per service instead of shared DB
- To do interservice 'transactions' you need saga pattern
- When using saga pattern you need events for state changes
- events for state changes must be done atomically together with the actual state change
- for that there are a few possibilities (which I might be able to use as a comparison), of which event sourcing is I think the best approach
- because an event store is an inefficient read model, you need CQRS to have (eventual) consistent denormalized efficient read models

So it looks like, MSA always leads to event sourcing with CQRS this way... Although there may be less ideal alternatives for event sourcing like transaction log trailing...

Anyway, thanks for the info! I think I have enough to go on with.

[Chris Richardson]

On Sun, Oct 7, 2018 at 3:45 PM Johan 't Hart <jop...@gmail.com> wrote:

Thanks all for these quick replies! It is very useful.

As a summary the advice is to stay with a monolith and only go for microservices when you need to.
Sounds like good advice. However we already decided to go for microservices and are building them already for almost a year now. (Which was probably a decision taken too soon in hindsight. But it is going to be a complex application anyway and we identified some domains already.)

I've scanned lots of the very good website https://microservices.io already to look for alternatives to event sourcing and CQRS in an MSA context. But also here it is in my understanding summarized like:

- In MSA you best use database per service instead of shared DB
- To do interservice 'transactions' you need saga pattern
- When using saga pattern you need events for state changes

Choreography-based sagas, yes. Orchestration-based sagas, no

 

- events for state changes must be done atomically together with the actual state change
- for that there are a few possibilities (which I might be able to use as a comparison), of which event sourcing is I think the best approach

Not really. Event sourcing is an option. It has benefits but it also has significant downsides.

Often using traditional persistent with explicit event publishing works fine.

For example, you can use regular schema migrations and since events are transient you have far fewer issues with event schema evolution than you do with event sourcing.

- because an event store is an inefficient read model, you need CQRS to have (eventual) consistent denormalized efficient read models

MSA often needs CQRS. Event Sourcing - very likely - if you need non-PK based queries.

 

So it looks like, MSA always leads to event sourcing with CQRS this way...

Not really. For reasons already mentioned.

 

Although there may be less ideal alternatives for event sourcing like transaction log trailing...

Explicit event publishing + transaction log tailing has trade-offs. I wouldn't say it's less ideal. See above.

Chris

[Rickard Öberg]

Hi Johan!

We're going deep into "it depends" territory here. See below.

On Sun, Oct 7, 2018 at 10:27 PM Johan 't Hart <jop...@gmail.com> wrote:
> We are considering to use CQRS and eventsourcing for our new project, which we want to build in the cloud.
> We already decided to go with microservices and docker/Kubernetes.
> And although I'm very enthusiastic about event sourcing, my manager is not convinced to go for it because he claims it is not mainstream enough and he thinks some cons are hard to handle. E.g.
> - never be able to delete data,

I delete data all the time. I do a blue/green deployment where a new
cluster subscribes to the old, and makes transformation of data on the
way. Aggregating data, in order to compress the events, is one thing
we do.

> - forever and quickly growing disk space needs

This depends a bit on your storage format. JSON is the most common,
but you can also use something like protobuf. But sure, this is one of
the main considerations with eventsourcing.

> - database maintainer can't fix database errors easily

For us it's the opposite. If we find out we messed up with how we
projected events into the database we fix the issue, deploy, and then
rebuild the databases from scratch (takes about 1.5h currently). No
patching, and we know the database is consistent with the code
deployed.

/Rickard

[Greg Young]

- never be able to delete data,

You can delete data its just a much larger thought. If you think you can't its likely worth more research before building such a system.

- forever and quickly growing disk space needs

Not really and to be fair which is greater, the number of GB/disk/year or your data retention?

- database maintainer can't fix database errors easily

They can its just different. It can actually be the same but likely causes more issues ...

- no managed event store in Azure

there are event stores built on azure managed services

--
You received this message because you are subscribed to the Google Groups "DDD/CQRS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dddcqrs+u...@googlegroups.com.
Visit this group at https://groups.google.com/group/dddcqrs.
For more options, visit https://groups.google.com/d/optout.

--

Studying for the Turing test

[Gottfried Szing]

Hi

Have you done MS or ES or CQRS already before? Do you know your domain very well? 

Because if you dont say to all of them YES, I would go to monolith first and then step by step move on to more advanced topics. E.g. starting with a monolith to do a deep dive into your domain. Then if you are familiar with the domain, add CQRS for persistence and maybe do ES as well. But remember: doing MS does not mean to do CQRS or ES automatically. It goes well together, but it adds a lot of complexity. 

From my point of view is doing microservice with a lot of experience with CQRS/ES in an unknown domain a bad start. You have to do a lot at once to get started.

Br

[Johan 't Hart]

Thanks again for the answers.

A little bit of background. Actually we are very familiar with the domain at hand. It is a telephony application that we are developing for years now. But currently it is a somewhat monolithic single tenant application running on prem at our customers.
The 'new project' is actually to move this software to the cloud and host it ourselves instead of the customers. So we basically want to do saas.
Here we decided to go for a MSA architecture and make it multitenant. And since the frontend is very reactive in nature and must aggregate different communication channels, I figured event sourcing would be a good fit here.

So it is not unfamiliarity with the domain, and not with the reactiveness of it. MSA, CQRS, event sourcing and multitenancy however are new for us.

However we are doing research for quite a while now but can't get a 'go' from our manager because of his concerns.

But I think this thread already yielded a few considerations and options to be able to compare some architectural approaches, so thanks again!