Using command sourcing or event sourcing?

[netfocus]

In LMAX architecture, there are the following main components: input event + business logic processor(BLP) + output event.

As i know, LMAX use the input event to rebuild the whole BLP, not the output event. But in CQRS+Event Sourcing architecture, i found that it use the output event to rebuild aggregate root. And the input event in LMAX is silimar with the command in CQRS, it represents what user want to do, and the output event represents what has happened in domain aggregate, it is produced by aggregate.

So as you can see, LMAX use the input event sourcing (command sourcing), and CQRS use output event sourcing.

In LMAX, i know that we must journal the input event before the BLP processing them, the reason for that is we can use these input events to

rebuild the whole BLP in memory by using the event sourcing pattern.

But in CQRS, we don't persist the commands before the command was handled by the command handler. Instead, we persist the output event which is produced by aggregate. In fact, in my opinion, in CQRS architecture, if we want to ensure the output event must be persisted after the aggregate changed, the dicrectly way is persist the output event in the same context transaction synchronizely, but this will result we cannot implement the in-memory domain model like LMAX BLP. Unless we persist the output event asynchronizely, but if use this async way how can we ensure the event was persisted successfully?

Does anyone can explain what is the advantage and disadvantage between these two different approaches?

To help us to discussion this question, i write the following sample code:

public class ChangeNoteTitleCommand

{

public Guid NoteId { get; set; }

public string NewTitle { get; set; }

}

public class NoteTitleChanged

{

public Guid NoteId { get; private set; }

public string NewTitle { get; private set; }

public NoteTitleChanged(Guid noteId, string newTitle)

{

this.NoteId = noteId;

this.NewTitle = newTitle;

}

}

public class Note : AggregateRootBase

{

public string Title { get; private set; }

public ChangeTitle(string newTitle)

{

ApplyEvent(new NoteTitleChanged(this.Id, newTitle))

}

private void OnNoteTitleChanged(NoteTitleChanged evnt)

{

this.Title = evnt.NewTitle;

}

}

Some explainations:

ChangeNoteTitleCommand handler will call Note.ChangeTitle method to execute the change note title business logic.

ChangeTitle method will produce a NoteTitleChanged domain event, and OnNoteTitleChanged private method will handle this event and update the state of the current note.

[Greg Young]

Are they really two different architectures? 

--
You received this message because you are subscribed to the Google Groups "DDD/CQRS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dddcqrs+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Le doute n'est pas une condition agréable, mais la certitude est absurde.

[netfocus]

At least from the replay point, they are different. LMAX replay input events, but CQRS replay output events. The different is: input event is more like command in CQRS, isn't it?

If we replay output events (produced by aggregate), and if we want to implement in-memory mode like LMAX, how do we ensure all the output will be persisted?

To unsubscribe from this group and stop receiving emails from it, send an email to dddcqrs+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Greg Young]

You are discussing the difference between a command processor and a
downstream event processor. I think in your comparing you need to look
for the forest instead of the trees.

Command is a message.
Event is a message.

An event is something that has happened.
A command is something you want me to do.

I must accept an event.
I can say no to a command.

Sometimes in systems we prefer to model as commands. Sometimes we
prefer to model as events. Neither completely changes your
architecture. Very often when receiving an event I will actually
produce a new event as well (like a stock market)
OrderPlaced->TradeOccured.

As for how to process. They are basically identical. On a single
machine you can pretty easily be processing (consistently and durable)
10-20k commands or events per second. You can get much higher with
work (bottleneck is generally the durability).

Cheers,

Greg

> --
> You received this message because you are subscribed to the Google Groups
> "DDD/CQRS" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to dddcqrs+u...@googlegroups.com.

[netfocus]

Thanks, greg.

As you said, for event we must accept, but for command we can say no. And thanks for the clarification for command and event.

So do you mean we should rebuild the domain model using events not commands? Yes, i think that's right. And i made a design which

can allow us implement event sourcing + in-memory mode. When the case of a user want to change a note's title. He sends a ChangeNoteTitleCommand. The following process i design:

1. Send ChangeNoteTitleCommand to input disruptor;
2. Command handler execute method called by input disruptor;
3. Note.ChangeTitle method called by command handler execute method;
4. NoteTitleChanged domain event is created and raised in note.ChangeTitle method;
5. The infrastructure framework send the above NoteTitleChanged event to input disruptor when the raise method is called;
6. Journal event handler called by input disruptor to persist the event;
7. Another event handler called by input disruptor to really apply all the note state changes according with the event.
8. A third event handler called by input disruptor to send the event to the output disruptor;
9. All the external event handlers are called by the output disruptor; for example, some external event handlers will update the CQRS query side data;

And now, i may understand that in LMAX, its input events maybe means the event in the above step 5. So in fact, maybe the input event and output event is just the same event in LMAX architecture, right?

[Bennie Kloosteman]

LMAX "architecture" is a specific architecture on how to address
thread contention for a specific problem and that is high speed low
latency trading at a massive scale - It says nothing about overall
design , modularity etc etc because most high speed trading systems
are mostly isolated systems . LMAX uses event sourcing but not CQRS ..
but that does really matter , the lesson i got from LMAX and why its
important for WHEN you need high speed transactions

- think about cache usage its very important ( which also goes well
with breaking apps into smaller async message pumps services )
- Often in parallelism , contention for most designs greatly reduce
performance to the point where a single thread not blocking can do
massive amounts of work and exceed the potential of most parallel
systems ( Unless they are very carefully written)
- You minimize IO ( and persistence)
- Which means you minimize the DB LMAX uses files for its event source
since no DB kept up.

I don't think there is anything new here ( Disruptor is basically a
ring buffer as used in many device drivers) , except for pointing out
again the massive contention in most modern shared collections and
the increasing importance of cache usage and TLB flushes on context
switches.

Anyway to do LMAX in CQRS you need to have asynchronous calls to the
repository and any other IO .. and the IO needs to be done on
different threads to your command processing -with minimal or no
locking. This leaves you with 2 choices

1) Send more information with the command
2) Divide commands into 2 sub commands / phases , one command tells
the IO thread what information to get and the call back creates the
second command, the second command does the business logic with all
the information ..

Agree with Greg this is only worth it if you have > 10-20K commands
per machine and you have an IO /DB system that can support such high
command processing rates.. and if that is the case , there is a case
for such an architecture instead of distributed ( eg if you have an
SOA / distributed environment than you should go distributed if not
consider LMAX)

Ben

[netfocus]

Thanks for your reply, ben.

You have mentioned that when to use LMAX and when should not. And furthermore, you mentioned some key features of LMAX.

For your 2 sub commands phase, I don't know very well. But I think I've more clear than before that when we want to implement event sourcing and in-memory architecture. There are at least two key points we should follow:

1. ensure to persist the events before the domain state changes;
   
2. use the events to rebuild the domain state when the system restart;
   

The only bottleneck is the first point. That means how fast we persist the event, then how fast our system is.

Now, let us get back to my original question. What kinds of event we should use to rebuild the domain state? Currently, in my opinion, I guess that only the events which are created and produced by domain can be used to rebuild the domain state. The command (in CQRS architecture) probably should not be used to rebuild the domain state, right?

[Jonathan Curtis]

I don't know much about LMAX, but there must be an assumption that you can only replay commands when there has been no change in business logic. Otherwise you will get different results (unless that is what you want...). Typically in a business domain, you don't want to change the past.

[Kijana Woodard]

Stipulation: I don't know much about LMAX either. 

netfocus said LMAX used "input events".

Given that they deal with algorithmic trading, their "input events" are likely something akin to "market ticks". If so, their input events really are Events in the sense that you can't "reject" them. If the market says GOOG is up 1/16, you can't/won't say "no it isn't". This is quite different from a PlaceOrder command from a user.

My guess is they want to have different BLP's operating against the same tick history and then look at the results to find winning formulas.

--

[Greg Young]

The are a backend. The events are like orderplaced (results in
orderplaced->ordercancelled->tradeoccured->etc).

But its very common that such a system is not really "the book of
record". A warehouse system is the quintessential example of this. The
computer is not the book of record, the warehouse is!

[Bennie Kloosteman]

> Now, let us get back to my original question. What kinds of event we should
> use to rebuild the domain state? Currently, in my opinion, I guess that only
> the events which are created and produced by domain can be used to rebuild
> the domain state.

Correct

>The command (in CQRS architecture) probably should not be
> used to rebuild the domain state, right?
>

Correct and the order of LMAX itself is irrelevant ( and nor really an
important part of LMAX architecture) and i would not use it ( It is
just event processor getting market events and more importantly events
based on market depth scans and some dummy trades) , however the
KEY to LMAX is slow things ( IO !) like most GUIs and node.js are on
another thread , that way the command processing thread (s) stays
running has few/no locks , context switches and hence has a high L1
and L2 cache hit and does huge amounts of work . Messaging between
threads is done in a non locking manner via a ring buffer (
"disruptor").

re 2 sub commands . The real key is in a typical CQRS system you go

1.Command Processor
send command
handle command
bring AR/entity into memory
run logic
fire events

2.Event Processor
apply changes
save changes
handle errors

Which i would retain as it has many advantages . However handle
command normally has IO to bring in the information needed into memory
( eg create the relevant aggregate) and obviously save changes so
these cannot be done on the command processor thread ! The easiest way
is as i described in the last email.

Once you do the above ( pull out IO) then you have to handle errors
and failures in my case an exception is thrown and callers to the
command processor can hook these events ( which are fired on worker
thread so the caller can not interfere with the command processor ) .

Note on a parallel systems the IO is normally done on the same thread
( but the OS often will just block it and use io completion threads
behind the scenes) , it is difficult to say what is better on a high
performance system eg some of the factors

- Performance vs ease of coding
- Do you handcraft your parallel locking
- Do you use a lib to make parrallelism easy eg RX
- Do you have an existing CQRS framework which is mature for many threads
- Do you have a mature SOA environment
- Developer skills
- What DB solution do you use

LMAX on CQRS has a significant cost in terms of non standard
infrastructure ( eg call backs to create new commands ) but once
written its pretty easy to code for , eg probably < 10% overhead to a
standard CQRS system. Parallel optimized systems would require less
infrastructure code but be much more expensive to find and tune the
bottle necks.

I would also add CQRS systems have lower contention than traditional
systems so benefit less from LMAX but i think the highest performing
systems would be mostly LMAX with some hand crafted parallelism
.though such a system would need an incredible and custom event store.

If your still interested in LMAX CQRS send me a message and i will
mail you some code when im back in shanghai.

Ben

[Robert Friberg]

Howdy folks,

Interesting discussion!

Command sourcing (CS) with a single in-memory projection has a name, system prevalence, and a substantial, though perhaps not thriving, culture.

There are many implementations: prevayler for java, bamboo and LiveDb for .NET (by me), madeleine for ruby and perlvayler for perl to name a few.

Fowler calls it MemoryImage, and describes it well here: http://martinfowler.com/bliki/MemoryImage.html

The focus is on domain state persistence (think database), so commands (think stored procedures) must be deterministic with regards to the prevalent system,

which means no external dependencies, not even the system clock! External domain behavior is implemented at a higher level, for example service or controller.

Commands are persisted before execution, allowing for greater query throughput because the write lock can be obtained after the disk write.

In LiveDB commands that fail are discarded or rolled back (rollback marker appended to the log).

So before execution, commands serve as requests which can be rejected and after execution they serve as facts: "This command was executed"

So I would say ES and CS are variations of the same architecture but coming from wildly different origins.

Robert Friberg

http://livedomain.codeplex.com/

http://livedb.devrex.se/

[Henrik Feldt]

My takeaway from LMAX was that they primarily operate on a single aggregate (the exchange) that is highly available and has billions of events while most CQRS system have many ARs and fewer events per AR.

[Nuno Lopes]

I think one could use the Disruptor to publish and distribute messages within the same bounded context instead of queues. Each thread would have only one message handler reponsible to find the aggregate and execute the command or transform the event into one or more commands puting them back in the queue. 

To minimize contention, each thread could be responsible for a range of Aggregates that could be defined based on Aggregate Type or other heuristic. So basically you woul not have two threads "using" the same aggregate instance. You could also have all Aggregates in memory.

And so on ...

Just some points.

[Greg Young]

Is a ring buffer no longer a queue?

[Nuno Lopes]

One can say that it is.

[Bennie Kloosteman]

Thats just a question of scale a sensor system could generate similar
events... and in many cases data can be converted to events.

ie I may be out of touch but AFAIK , the exchange itself often does
not make most of the events you normally do a market depth scan (
since the exchange does not tell you bids ) which return multiple
lists ( as these scans are done all the time and in parralel) which
are then converted to events .

Anyway to me this is not the take away , the main takeaway is that
parallel ( and worse distributed) does not always give the best
performing results...its intestesting that they first build a
parallel actor based system but got better results from a single
thread. I dont really see how you can take stuff from the business
logic of LMAX as the logic is not much different to a number of
similar systems .

Ben

[Bennie Kloosteman]

On Wed, Feb 20, 2013 at 1:07 AM, Nuno Lopes <nbpl...@gmail.com> wrote:
> I think one could use the Disruptor to publish and distribute messages
> within the same bounded context instead of queues. Each thread would have
> only one message handler reponsible to find the aggregate and execute the
> command or transform the event into one or more commands puting them back in
> the queue.

Yes correct but not sure why a thread per handler is needed.. you can
just code the appropriate handler. In most cases all you need is to
increase the data with the command a bit or split the command into 2 .
Command A gets the data and creates a new command with a reference to
an in memory aggragte (via a callback on a different thread once the
data has been retrieved) .

>
> To minimize contention, each thread could be responsible for a range of
> Aggregates that could be defined based on Aggregate Type or other heuristic.
> So basically you woul not have two threads "using" the same aggregate
> instance. You could also have all Aggregates in memory.

If their is 1 thread than this is not needed ( it is an issue in a
parralel or distributed system which is idempotent messages /
optomistic concurency- retry are needed ).. With a single thread it
can not be an issue if the thread only runs 1 command at a time and
completes all work on a command before starting the next . Most
commands will finish when it hits an exception , schedules a
callback , or adds a message to an output queue - there is no IO /
long running tasks so all commands should complete in micro seconds
) . Good old simple single threaded programming.

Ben

[Nuno Lopes]

In meant that one handler per thread yet each handling multiple types messages. Unlike one handler per type of event that we see in CQRS. If things get complex this handler could dispatch the event to subsquent specific handlers per type.

[John Lonergan]

Jonathan's point for me is the key distinguishing factor between command sourcing / event sourcing.

If I do a release that changes the command business logic then when my system restarts I want the state of the system restored to that prior to the release.
If I were to use comand sourcing then this wouldn't work as the new logic would be applied to all the historic commands during replay.

If however I go for rebuilding my state from the stream of historic events these are not impacted by logic changes.

Sure if I introduce a new command then I can support both the old and the new command logic.
But that's inconvenient when only the logic has changed.

Also from an integration point of view its far more useful to me to integrate downstream systems using the event stream rather than the command stream.