Always event sourcing?

[Nils Kilden-Pedersen]

After having started 2 projects using event sourcing, I decided to do CQRS using a RDB backend for the write model.

But it appears to not offer any simplicity, but rather appears to make things more complex. E.g.:

• Requires 2PC (alternatively, still have an event store, which seems like a hybrid that's neither here nor there)
• Requires an ORM (alternatively, hand-parsing events to construct SQL)

Comments, advice, rants, insults?

[John Thornborrow]

Right tool for right job. There is no single answer/solution to every problem. 

Many variables to this equation:

1. "Size" of application.

2. Frequency of change.

3. Does it require something more than "just get it working"?

4. Do they pay you enough/Can you afford the time to explore all contexts?

5. etc

:)

[Nils Kilden-Pedersen]

On Sat, Feb 11, 2012 at 11:12 AM, John Thornborrow <thornbor...@gmail.com> wrote:

Right tool for right job. There is no single answer/solution to every problem. 

Many variables to this equation:

1. "Size" of application.

2. Frequency of change.

3. Does it require something more than "just get it working"?

4. Do they pay you enough/Can you afford the time to explore all contexts?

5. etc

:)

No offense :-), but there you are merely repeating the "arguments" I've heard before instead of addressing the specific point I was making, which was an attempt to counter those very arguments.

[Philip Jander]

It depends :)
Within one "autonomous business component" (not neccessarily in Udi's sense), meaning one piece of software that shares infrastructure, I try to use a single persistence mechanism only. If there are event sourced parts of the model and others that only require crud operations, I use Created, Updated (somtimes combined as Upsert) and Deactivated events. The disadvantages are a bloated storage since every event carries the complete data, even if a part of it is unchanged, and the additional projection required. So far, I feel that the advantages outweigh the additional complexity.
However, if possibly, I try to separate CRUD parts of the domain into their own (A)BCs so that they can be implemented using a simpler mechanism.

Cheers
Phil

[Paco Wensveen]

1. It does not require 2 phase commit
2. It does not require an ORM

[Philip Jander]

also, you can find part of an answer in Udi's blog post of yesterday:
http://www.udidahan.com/2012/02/10/udi-greg-reach-cqrs-agreement

[Nils Kilden-Pedersen]

n Sat, Feb 11, 2012 at 11:51 AM, Paco Wensveen <pac...@gmail.com> wrote:

1. It does not require 2 phase commit

How do you safely publish events without it?

 

2. It does not require an ORM

Correct, I did give an unappealing alternative. Can you perhaps recommend more appealing options?

[Paco Wensveen]

Explained in some short statements:
- Use a database with only one table containing the events.
- When a command is handled, the result can be 0 or more events for
only one aggregate root.
- The events have to be saved and published.
- Save the events by inserting one table row containing all the events
- Publish the events over the bus
- Update the row with a bool field marking that the events are published
- Use idempotent command handlers and aggregate roots
- The most important way to query the event store is asking for all
events about a single aggregate root. Get all rows with the aggregate
root id, and with the published field true.

Explained in pseudo code
void HandleCommand(command)
{
// get all the previous events
var events = sql("select * from events where aggregateRootId =
{command.AggregateRootId} and IsPublished = 1");

// create the aggregate root and restore the state by applying the
previous events with some (naming) convention
var aggregateRoot = createAndApplyEvents<SomeAggregateRootType>(events);

// this is the only statement in the command handler that cannot be
moved to some general abstraction for all commands
aggregateRoot.DoSomethingWithTheCommandInAnIdempotentWay(command.Parameter);

// The unhandled events are the events that are not published or stored yet
var unHandledEvents = aggregateRoot.GetUnhandledEvents();

var batchId = Guid.NewGuid();

// add the events to the event store
query("insert into Events IsPublished = 1, BatchId = {batchId},
AggregateRootId = {command.aggregaterootid}, Batch = {
mapToBinaryOrJsonOrWhatever(events)}")

// send the unhandled events to the event handlers
bus.publish(unhandledEvents);

// markt the batch as published
query("update Events set IsPublished = 1 where BatchId = {batchId})
}

This means 3 sql statements are needed for the whole event store and
transactions are not used:
- add batch of events (insert)
- mark batch of events as published (update)
- get events by aggregateroot (select)

Of course, you can add more options to the event store, like replaying
events in different ways or snapshots.

[Nils Kilden-Pedersen]

Paco, I suggest you re-read my original post. I think you're entirely missing the point.

[John Thornborrow]

None taken! File it under "comments" or even "insults" if you like :p. I didn't see a specific point :)

Unless when you say "using an RDB for write model" you mean a relational data-model and not just using an RDBMS for the Event Store?

The more times I read it, the more I think that was my misunderstanding.

The project I'm on at the moment (for a logistics system) is in this exact state, and yes, it does have complexity which would be non-extant if we used event sourcing, particularly so when bi-directional relationships are being maintained (i.e. when child/parent is reversible, and the child can also be the parent.) and having the usual "no, I don't need everything related, I just wanted that small slice of data" 

J.

[Paco Wensveen]

I just reread your first message, but I don't get the point. Maybe you can add some examples.

Op 11 feb. 2012 21:56 schreef "Nils Kilden-Pedersen" <nil...@gmail.com> het volgende:

[@yreynhout]

It's unclear whether you are using an RDBMS as an eventstore or as a
structural datastore. In either case, you are wrong wrt 2PC being
required.

On 11 feb, 16:19, Nils Kilden-Pedersen <nil...@gmail.com> wrote:
> After having started 2 projects using event sourcing, I decided to do CQRS
> using a RDB backend for the write model.
> But it appears to not offer any simplicity, but rather appears to make
> things more complex. E.g.:
>

>    - Requires 2PC (alternatively, still have an event store, which seems

>    like a hybrid that's neither here nor there)

>    - Requires an ORM (alternatively, hand-parsing events to construct SQL)
>
> Comments, advice, rants, insults?

[Nils Kilden-Pedersen]

On Sat, Feb 11, 2012 at 3:56 PM, John Thornborrow <thornbor...@gmail.com> wrote:

Unless when you say "using an RDB for write model" you mean a relational data-model and not just using an RDBMS for the Event Store?

Right. I should have been more specific on that point, probably due to the fact that I'm using MongoDB for the event store normally.

 

The project I'm on at the moment (for a logistics system) is in this exact state, and yes, it does have complexity which would be non-extant if we used event sourcing, particularly so when bi-directional relationships are being maintained (i.e. when child/parent is reversible, and the child can also be the parent.) and having the usual "no, I don't need everything related, I just wanted that small slice of data" 

Ok, that's interesting. I assume you're not using event sourcing then due to politics/legacy/whatever?

[Nils Kilden-Pedersen]

On Sat, Feb 11, 2012 at 4:15 PM, Paco Wensveen <pac...@gmail.com> wrote:

I just reread your first message, but I don't get the point. Maybe you can add some examples.

As John pointed out, I was being unclear when I wrote "RDB". What I meant was using an RDB for a traditional data model, not as an event store.

So to rephrase: It appears to me that using a traditional data model (in an RDB environment) for the write side brings about complexity that doesn't appear to exist when using event sourcing, which seems to be the opposite of what is normally argued, namely that event sourcing should only be used if needed, competitive advantage, etc.

Hope that's clearer.

[Nils Kilden-Pedersen]

On Sat, Feb 11, 2012 at 4:15 PM, @yreynhout <yves.r...@gmail.com> wrote:

It's unclear whether you are using an RDBMS as an eventstore or as a
structural datastore. In either case, you are wrong wrt 2PC being
required.

I hope my last posts have clarified my original post.

When you state that 2PC is not required, do you mean strictly speaking, i.e. you don't *have* to use it (if your application is a toy), or do you mean literally that it's not required, even for mission critical applications? If the latter, please elaborate how you deal with failure between RDB commit and event publishing.

[Joshua Ramirez]

I would pay close attention to this distributed podcast.

http://distributedpodcast.com/2011/episode-9-when-to-avoid-cqrsclarified 

Short and ignorant answer: you are probably gold plating.

[Nils Kilden-Pedersen]

This is not about whether to do
CQRS or not. It's about, when using CQRS, whether to do ES or traditional model.

[John Thornborrow]

That's it, legacy. :) 

The main complexity we have had with our model/ORM is the prolific use of natural keys (most of which are composite) which for one reason or another have not worked well with the usual ORM tools. Performance has been a tricky one, too - because of relational-ORMs usual trait of "pull everything that is related to this object, even if I don't need it right now". Lazy-loading was/is possible, but still has its drawbacks such as if we need to serialise an object (our domain is not quite encapsulated just yet.. but we're getting there!) or the plain case that it means a new query for every related object. Was difficult to get a go-between where we can specify at query time what to loazy-load and what not to lazy-load.

As for the 2PC problem, that's something I'd be interested in learning about too. We actually have a 4PC due to needing to update other legacy systems and if any fail, they should all fail/rollback - hence why I'm eager to learn a way around this!

J.

[John Thornborrow]

I've just read my reply back, and I think it's obvious that we've basically suffered from the usual problems presented to us by any complex relational model. CQRS or not.

[Jonathan Matheus]

If you listen the distributed podcast episode Joshua mentioned or Udi's recent post, he mentions that CQRS is not a top level architecture. He mentions splitting your system into logical boundaries of coupling (ie. a business component) and making an informed decision if CQRS and event sourcing is the right architecture for that part of your system.

That said, if you are using CQRS in a part of your system and you are using eventually consistent read models that are populated by events, then I think that event sourcing is NOT gold plating. 

Like you mentioned, one of the benefits of using event sourcing when applying CQRS in a BC is that I can focus on designing my events and commands and I don't have to worry about designing a relational model on top of that that's just going to change when my requirements change. I just don't want to deal with the headache an ORM is going to bring.

I believe there is a tangible benefit to being able to create views without having to create ETL scripts or guess about how my traditional data model ended up in the state that it did. I also think it's pretty scary, when using CQRS, figuring out how I'm going to keep all these view models up to date and correct without being able to recover messages. Udi mentions that you can do the same thing with an audit queue, logging, and other tools & you totally can. However, I don't want to deal with 4 or 5 ways of doing things when I have a solution that works well.

The bad:

One complexity when using event sourcing is the amount of events that are stored over time. You'll have to come up with a strategy to keep backup / storage costs down. One option is archiving events that you don't currently need in your system. One option is to archive events from streams (or AR if you're using DDD with event sourcing) whose lifetime has ended in your BC. The decision will be different from BC to BC, but the solution will be the same. You solve it once and there you go. That's not the case with a traditional data model. You essentially have to design from scratch from BC to BC. It's not a one time problem.

Another complexity is replaying for streams / ARs that have very long lifetimes, meaning they have lots of events. This shouldn't happen that often if you're modeling your BCs carefully. Your consistency boundaries should be VERY small. Therefore their lifetimes are very small. If you do see this problem, see if you can reduce the consistency boundary. If you can't, resort to snapshots. The complexity there is that you now have to deal with when to snapshot and how to change them when your requirements change. These problems are offered out of the box in many frameworks. One way to deal with changes in requirements with snapshots is to delete all of your snapshots & re-snapshot when there's a unrecoverable breaking change in your snapshot (not often). Again, that's a script I write once, not a model I have to design every time I face a new BC.

As far a 2PC, using a SQL store isn't forcing you to use the DTC. There are ways to get around 2PC regardless of technology. SQL Azure, for instance doesn't support DTC.

I understand the opinions that people have that CQRS is gold plating. You definitely want to see if a small CRUD area of your system warrants defining events and commands for, having command handlers, view cache handlers, eventually consistent UI workflows. If you do decide to use CQRS though, I wouldn't do it w/o storing my events.

Hope I didn't just go on a crazy man rant,

Jonathan Matheus

[@yreynhout]

You are assuming "save events" and "publish events" happen in the same
tx. Split it up. 2 txs.

On 11 feb, 22:25, Nils Kilden-Pedersen <nil...@gmail.com> wrote:

[Dennis]

In regards to 2PC, publishing events CAN be handled external to the
transaction to an event store. One of the purposes for a
"sequencer" (a unique auto-incremented number) for each event, is to
be able to "chase" the new events. The sequencer is a different from
the event version. The version is for tracking the order of events for
a given event stream. Given the relaxed consistency for the read side,
a separate service can be used to manage the consistency boundary (in
a relaxed manner) for ensuring events have been successfully ACK'd and
the "last known sequence" has been updated. Also note, the last know
sequence does NOT have to be stored with the write or read side.

ORM... No ORM is needed for the write or read store even w/ an RDBMS
data store. The write is straight forward. 1 table for the events and
likely 1 table for the event stream head. The read side should be
denormalized an nothing more than simple SELECT * FROM...

That said, I would highly recommend at least starting with a NOSQL
data store. Less friction. You can always switch later.

On Feb 11, 10:19 am, Nils Kilden-Pedersen <nil...@gmail.com> wrote:
> After having started 2 projects using event sourcing, I decided to do CQRS
> using a RDB backend for the write model.
> But it appears to not offer any simplicity, but rather appears to make
> things more complex. E.g.:
>

>    - Requires 2PC (alternatively, still have an event store, which seems

>    like a hybrid that's neither here nor there)

>    - Requires an ORM (alternatively, hand-parsing events to construct SQL)
>
> Comments, advice, rants, insults?

[Nils Kilden-Pedersen]

On Sat, Feb 11, 2012 at 11:47 PM, Jonathan Matheus <jmat...@gmail.com> wrote:

That said, if you are using CQRS in a part of your system and you are using eventually consistent read models that are populated by events, then I think that event sourcing is NOT gold plating. 

That's what I've realizing too, or at least I think I am, hence this post to verify or debunk that notion.

 

Like you mentioned, one of the benefits of using event sourcing when applying CQRS in a BC is that I can focus on designing my events and commands and I don't have to worry about designing a relational model on top of that that's just going to change when my requirements change. I just don't want to deal with the headache an ORM is going to bring.

Indeed. I dropped ORMs 7 years ago. Not worth it, IMO.

 

One complexity when using event sourcing is the amount of events that are stored over time.

Right, but it's not necessarily a very complex problem to deal with, particularly when compared to the alternative. Storage is cheap, bla bla bla.

 

Hope I didn't just go on a crazy man rant,

No, very helpful.

[Nils Kilden-Pedersen]

On Sun, Feb 12, 2012 at 6:33 AM, @yreynhout <yves.r...@gmail.com> wrote:

You are assuming "save events" and "publish events" happen in the same
tx. Split it up. 2 txs.

There is no "save events" when using non-ES. This is why I'm claiming that you need 2PC if not using ES.

[Nils Kilden-Pedersen]

Dennis, I think you, like other people, misunderstood my original post. I apologize for being unclear. Please read follow-up for clarification.

[Jonathan Matheus]

Actually, frameworks like NServiceBus support message forwarding, where you can forward received messages to another queue. 

One option outside of event sourcing is to forward to an audit queue which will "save" an event in case you need them. This also comes with its own set of complexities, but it's def an option.

[Nils Kilden-Pedersen]

The case that is crucial to deal with is this:

1. You persist you data model changes into a standard RDB model (non-ES)
2. Your app crashes
3. Never happens: Event publishing (events are now gone forever)

I don't see how you avoid this scenario without 2PC.

I haven't considered any kind of event logging or persistence of events, because if you're doing that you can just as well do ES. Doing both just makes things more complicated.

[Joshua Ramirez]

Please forgive my earlier post's terse nature. I dot consider myself an expert on these matters (yet ;) and I want to avoid pontificating on something I know little about.

However, despite the podcast's misleading title, the entire last half of the podcast is mostly about when to use event sourcing if you've decided to use CQRS.

I still assert that you should listen to it. It's very good material.

[Nils Kilden-Pedersen]

On Sun, Feb 12, 2012 at 1:11 PM, Joshua Ramirez <joshuaramir...@gmail.com> wrote:

However, despite the podcast's misleading title, the entire last half of the podcast is mostly about when to use event sourcing if you've decided to use CQRS.

I still assert that you should listen to it. It's very good material.

Ok, I will try to find time. Thanks for clarifying.

[Nuno Lopes]

Hi,

So to rephrase: It appears to me that using a traditional data model (in an RDB environment) for the write side brings about complexity that doesn't appear to exist when using event sourcing, which seems to be the opposite of what is normally argued, namely that event sourcing should only be used if needed, competitive advantage, etc.

If you find 2PC between a relational database and a messaging framework you can the the following.

1) Make you relational model

2) Instead of publishing the event on save, write the event in some separate table in the same transactions. Such table would of course reside in the same database where the model resides.

3) Provided trigger are supported by you RDBMS, we can use a trigger to publish the event and so on (AFTER). We eliminate some latency that might be created by the communication between your system and some messaging component. That is all. So you don't necessarily need 2PC. 

Cheers,

Nuno

[Greg Young]

You just moved the queue to the database so you can do a local transaction instead of 2PC :)

--
Le doute n'est pas une condition agréable, mais la certitude est absurde.

[Nuno Lopes]

Hi,

You just moved the queue to the database so you can do a local transaction instead of 2PC :)

Exactly, but it answers part of the concerns as described. Looks stupid doesn't it :) 

Haven't tried it but on SQL Server 2008 one has already a CREATE QUEUE facility: http://msdn.microsoft.com/en-us/library/ms190495.aspx

How much of this is necessary, it depends on the person that decides.

[Nils Kilden-Pedersen]

On Mon, Feb 13, 2012 at 5:02 AM, Nuno Lopes <nbpl...@gmail.com> wrote:

If you find 2PC between a relational database and a messaging framework you can the the following.

1) Make you relational model

2) Instead of publishing the event on save, write the event in some separate table in the same transactions. Such table would of course reside in the same database where the model resides.

3) Provided trigger are supported by you RDBMS, we can use a trigger to publish the event and so on (AFTER). We eliminate some latency that might be created by the communication between your system and some messaging component. That is all. So you don't necessarily need 2PC. 

Indeed, but now I'm persisting the events into a data store anyway, which seems to me to be the worst possible solution, i.e. doing both event storage AND relational domain model maintenance. 

[Nils Kilden-Pedersen]

On Mon, Feb 13, 2012 at 5:04 AM, Greg Young <gregor...@gmail.com> wrote:

You just moved the queue to the database so you can do a local transaction instead of 2PC :)

You beat me to it :-)

[Nuno Lopes]

Hi,

Indeed, but now I'm persisting the events into a data store anyway, which seems to me to be the worst possible solution, i.e. doing both event storage AND relational domain model maintenance. 

I took that you wanted to maintain your relational database model and still use CQRS while avoiding 2PC. So I gave you a solution. But you haven't really stated your options so your technical assessment over what was suggested  is somewhat understandable.

Do you usually work in green field projects with the power to choose any technical component? My experience is that green field projects compared with brown field projects are relatively rare. I usually work on brown-field lately. This might be what you just need for those occasions were you need to avoid 2PC and retain some control on your side of the equation. Just a thought.

Cheers,

Nuno

PS: I think a lot of assessments made over CQRS are relatively naive for brown field projects considering todays corporate landscape. But I may be wrong. Sometimes the best solution is one that people can understand in 5 minutes.

[Nils Kilden-Pedersen]

On Mon, Feb 13, 2012 at 11:50 AM, Nuno Lopes <nbpl...@gmail.com> wrote:

Hi,

Indeed, but now I'm persisting the events into a data store anyway, which seems to me to be the worst possible solution, i.e. doing both event storage AND relational domain model maintenance. 

I took that you wanted to maintain your relational database model and still use CQRS while avoiding 2PC. So I gave you a solution. 

Fair enough.

 

Do you usually work in green field projects with the power to choose any technical component?

I do currently, yes. And having only done CQRS with event sourcing, I decided to try a smaller project with old school RDB data model backend, mostly because I kept reading that ES was gold plating. And then I encountered those issues that made it seem like ES is not gold plating, just candy all the way.

 

My experience is that green field projects compared with brown field projects are relatively rare. I usually work on brown-field lately. This might be what you just need for those occasions were you need to avoid 2PC and retain some control on your side of the equation. Just a thought.

Yeah, I can see that it might be an option in a legacy/political environment. Thanks.

PS: I think a lot of assessments made over CQRS are relatively naive for brown field projects considering todays corporate landscape. But I may be wrong. Sometimes the best solution is one that people can understand in 5 minutes.

Yes, DDD + CQRS and particularly ES does require some outside-box thinking, so it very much is environment dependent.

[Nuno Lopes]

Hi,

Yes, DDD + CQRS and particularly ES does require some outside-box thinking, so it very much is environment dependent.

Applying any new technique into a contexts where you need to leverage on the existing technical and human infrastructure requires a great deal of out-of-the-box thinking from the one bringing it too. It's not like you can just use the regular approaches to the technique all the time. So it really goes both ways. It's a learning process.

As for applying new techniques and technologies things get a bit easier when we don't have legacy. Then again, a lot green field projects go bust, so different challenges need to be dealt with. Only very few in comparison actually have the chance to mature.

This is to say, that the idea that legacy is a problem to DDD+ES because of the so called out-of-the-box thinking requirement is wrong IMHO. Legacy is instead an opportunity, because you have a base to compare your solutions with.

Cheers,

Nuno

[Julian Dominguez]

This is a very interesting topic. I do find myself struggling a lot
when trying to use TDD as my design approach, if I try to do CQRS
without ES.
With ES is very easy just to unit test the write model, because you
don't need a DB (event store) or anything like that, you just test
that the events are what you expected (which make my tests very
descriptive to my domain).
On the other hand, if I were to use CQRS alone, and DO NOT publish
intention revealing events (for example, I might have my read model
being read only SQL views put together from my 3NF write model), then
it is very hard to write unit tests. The whole idea of having CQRS is
to completely encapsulate the querying mechanisms from the update
model, so the domain should care less on how how is the data actually
persisted, but that is very hard to accomplish in unit tests, where
you have to assert the outcome of a command (for example, by looking
into the persisted data, or at least inspecting the data that should
be persisted)
I find that CQRS could start you in a path where you can later decide
to distribute your data sources (and have a DB for writes & a
different for reads). Nevertheless, if doing TDD, then unit tests for
the write model using the different distribution mechanisms would be
written in a very different way (assertions would be very different),
as you will have to deal in the tests with the integration mechanisms
themselves (as the persisted data would need to be very much in sync
with the notification mechanisims, if any).
On the other hand, with ES you just test the events, and there is no
need to care in you domain model tests about how those events are
later published and stored, as this is a bit of infrastructure that
will always be the same.

So in summary, IF you want to do TDD, am I the only one that thinks ES
along with CQRS is not goldplating, and in fact it makes the code that
uses CQRS much much easier to implement, and enables behavioral unit
testing?

-Julian