CQRS wo/ Event Sourcing + DB Views

[Mike Rowley]

The current system I'm working on uses a relational database to store current state and both read and write operations are done on this model.  We are looking at a redo (or partial redo) and want to use the benefits CQRS brings by separating the read (query) and write (commands) models.

Introducing eventual consistency would cause a lot of grief considering how the current UI has been implemented and the level of understanding of event sourcing mainly amongst the supporting staff is limited.

The system I am working with does not require any significant scaling so I'm debating if "cheating" by leveraging DB views to create the read models is an acceptable approach that anyone has used?

The alternative is to have separate read models that are updated transactionally when an event is published (transactionally in order to avoid eventual consistency concerns).  Seems like a lot more work than simply creating DB views....  Obviously there is overhead with any joins and data massaging for the DB views but I feel that initially at least the write and read models will be very similar and therefore the views should be fairly straight forward.

Thoughts?

[João Bragança]

Before you separate things at the data model level, you should separate them at the object / interface level first. Then, just roll another implementation of your public interface MyReadModelRepository/Facade/Service/Whatever when you need to.

Then by all means, start off with a view. When (or IF) that is too slow, move the hand rolled SQL into the application (to avoid the table scan). And when that's too slow, make it eventually consistent. Or use a document db. Or run it in memory.

--
You received this message because you are subscribed to the Google Groups "DDD/CQRS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dddcqrs+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Sent from my regular computer

http://twitter.com/thefringeninja

http://www.thefringeninja.com/

[Mike Rowley]

Interesting point about hand rolled SQL vs DB Views.

The end result is the read model should spit out "denormalized(ish)" DTOs (perhaps specific to each view/context/screen) so I guess we should consider if using DB views is entirely necessary.

However, my initial train of thought was we might have an ORM mapping over the read layer (thinking of .NET LINQ over DB), not the most optimized approach but in our case it would help in readability and reduce the need to create and manage SQL statements and transformation classes [SQL result -> DTO]  (I know one might argue the same about maintaining mapping files).  In this case, it might be beneficial to have the DB views so that mapping can take place over already denormalized views.

[Ben Kloosterman]

I refactored a small amount of code (5K lines) in a green field app still at pre alpha to move to CQRS and the cost was much larger than I thought .. If you have no need for scaling and an existing system what is the driver for the change ??… If its performance you can partition the DB , run on faster disks or optimize some key queries..

 

And for eventual consistency you really need to ask the business users once it has been implemented . It’s much easier for a green field app because the expectations are not there.

 

Ben

 

--

[Mike Rowley]

Thanks for your response Ben and I agree that one must evaluate if the extra effort of maintaining both write and read models truly provides any net benefit.  

However, I have seen benefits from CQRS from past projects and truly think that separating read/write concerns will actually reduce the complexity of this system.  I was just hoping to get input from others who may have also considered using DB Views for the read models and any issues / advantages they might be able to share.

[Ben Kloosterman]

I don’t think there is extra effort in maintaining a read and write model but I do think the conversion cost of the domain will swallow any form of benefit in lower maintenance…

 

Re using views..  you can do things at the DB end eg replicate it and just do read requests from 1 db but this is not CQRS  - CQRS ( and its benefits) refer’s to the domain not the data..  I did try something similar once with views but it fell apart  because  de normalized views didn’t work properly with Foreign Keys / child objects ( eg where it needs to add one  so huge queries with lots of Outer if not null joins  which are not allowed on Indexed views see http://blog.sqlauthority.com/2010/06/29/sql-server-outer-join-not-allowed-in-indexed-views/ ) . 

 

If you have an existing system that is too complex  ,  I would break it down with DDD  or  SOA  before considering CQRS . Or just use DDD with CQRS just for the part with the hard logic.   SOA has the benefit that you can use the whole system as is and don’t need a massive and risky change to the domain logic  , just add some facade’s and cut what is not needed when you have the time and put new requirements into new services if appropriate. Once this is done most of the domains (not all) will probably be too simple for CQRS

 

Ben

[Greg Young]

I would love to hear why it was a "much larger cost than you thought"
maybe you could share what you refactored to? At its core moving from
a regular domain model two CQRS + ES is refactoring two methods to
messages and writing projections to 1nf instead of writing/reading
from 3nf.

--
Le doute n'est pas une condition agréable, mais la certitude est absurde.

[Ben Kloosterman]

A bit hard to describe in 1 email... I will give some more updates later.

None of this code was really Command domain logic and 5% of it will end up
in the domain , it has lots of random generation and is pretty tightly
coupled with the DB via repositories ( the program isn't but this
generation code is ). So instead of it querying something from 1 DB and
writing it in a new DB with a slightly different schema which later parts
of the process access , I now have to create commands and make sure the
information is around longer as I can't really query the output DB (
eventual consistency) . I didn't want 100 trivial commands so created 10
big ones ( which make sense ) which then go to a process manager that
create the aggregates from them. I also bypassed the domain entirely for
some static data and that code needed only minor changes. This process
manager also needs to read events and handle failure though it's likely I
will skip it as for failure its likely we will fix the bug and restart the
process.. The fact that it was difficult to change because this code
pulled data directly from the repository caused me to change it so the
information is pushed/injected that probably added 20%. The code also had
quite a bit of DependencyResolver.Current.GetService code which was a pain..


I had a choice between making the commands include all the information or
just the keys and relevant information and put static data in the domain but
chose to not put the static data in the command domain. Right now probably
80% of the logic is in the creation of the commands for the CQRS domain ( eg
random generation ) , for the final system it will probably be 50/50 with
most of the remaining 'logic ' being lookups , validation , creating
commands and random generation .

I haven't got to the projections yet ( except 15 static data tables which
stayed the same and 10 untested projections) but there will probably be
100 of them...

I found creating the actual logic probably cost me 100% more than RAD style
/ 30% more than OO style but I get most of this back in testability and
maintenance as the logic is quite complex . And the cost won't go grow as
much as the complexity grows ie I feel more in control . So I think CQRS
+ ES is the right choice for this project.

Surprisingly I also moved to event sourcing which I thought was bad early
on.. I think for my app this was actually cheaper. Can't remember the exact
reasons but loading the whole aggregate via serialization worked better
than eager loading the aggregate. ( and you don't need to worry about having
a schema for the write model )

I think a lot of large refactoring's will cause similar costs ( there is
always some painful code like CRUD or integration etc) , is there such a
thing as an easy large refactoring ? It wasn't too bad I thought it would be
a week or two but it looks like 4-6 weeks is closer to the truth. Im not
including the costs of the CQRS domain itself just refactoring this
generation code. I built my own CQRS framework based on your one , but once
you want Sagas etc there is a lot more involved , I looked at the existing
frameworks and didn't like any of them give me something light please with
Sagas but no Bus , Dependency injection or fancy DB stuff.

If you have a full domain of 50 -100 or so classes , breaking it into say
10 Aggregate's needed for consistency will not be trivial. You could make
the whole domain 1 Aggregate but you need to create commands with no help
from those classes , then do the hydration and your domain is the same as
before .. you still need to break it up and that's the hard bit.

Ben

[Mike Rowley]

Thinking about this a little more I was reminded of an issue I had on a previous project.  For reads this project was using the popular "DB -> ORM -> Domain -> Mapper -> DTO" approach for reads.  As Greg has noted in his excellent CQRS Viddler video we often get a lot of "accidental complexity" with this approach specifically for us we ended up having to do a lot of ORM optimization just to get some simple DTOs on the screen.

In order to avoid the ORM complexity we started to create DB views that were representative of our DTOs and retrieved these DTOs using simple ADO.NET.

However, one problem we faced with this approach (and one that would also occur if using DB views for the CQRS read side) is when we need to provide a calculated value.

For example, consider Order which has many OrderItems.  Order has a calculated value OrderTotal = OrderItems.Sum(x => x.Total).  When mapping a DTO from the domain this is simple: OrderDto.OrderTotal is simply mapped to Order.OrderTotal.  

However, creating a DB view of Order would now become more complex because we would now need to include a SUM of the Order Items in SQL.  Simple enough, but calculated values are not always a simple summation.  Perhaps Order status is dependent on the state of its items.  Now we must duplicate more complex logic in SQL in order to produce the correct values in the DTO DB View.  Being SQL there is little chance that this calculation will be verified with unit tests and we now have duplicate logic (domain & SQL) and any change has to be synchronized.

So, I'm starting to conclude that using DB Views for a read model isn't a great approach.  Although it is a simple way to avoid eventual consistency while providing separation for reads / writes it could introduce major headaches for anything but the most simple projections.

If the goal is still to avoid "Eventual Consistency" on the read model then I think the unfortunate strategy is to ensure that Command Execution and subsequent Read Model updates are done within a transaction.

Perhaps introducing Eventual Consistency with UI trickery may be a better approach after all.......

[Bennie Kloosteman]

Depending on your threading model  , one way to avoid eventual consitancy  is to create a transaction scope for each command  ...easy to do... 

Introducing eventual consitancy in the same system where you had ACID before would need a LOT of business buy in   before i would attempt it. 

Two biggest  issues  I had with views are 

1)  indexed views limitatons ( and you want indexed views)  

2)   Foreign keys .. Obviously you have a normalized DB and you want to create a de nornalised table  now with Hydrating logic you can go 

if ( subitem != null) 

     FIllSubitemFields() ..

With a view its a pain .. you can do an if  not  null outerjoin but then you cant used indexed views.  You can also use Exits but the end result is you end up getting more and more logic into the views and quite complex queries. For a small project no problem but for lots of tables it would be risky.  

--