CQRS Meta Model

[michael-hamburg]

Hi all,

Everyone talks about CQRS but it's hard to define what it exactly is.

What makes up a command? What is a query? There are many descriptions
in the web but there is no precise definition. That's what a "meta
model" is about - It defines exact the structure of a model like CQRS.

So I tried to describe the "CQRS domain" using -you already guessed
it- DDD to start a discussion about it - Here is a first draft:

http://www.metacqrs.org/metacqrs-ddd.html

(Methods are currently hidden - Only the static structure is visible.)

I hope it's not too difficult to understand - It's sometimes weird to
talk about things in the "meta sphere"... ;-) Don't expect it to be
perfect - It's only a quick first draft!

Any comments welcome.

Cheers,
Michael

[Tom Janssens]

Here is a simple attempt developers should understand:

      var UI = new UserInterfaceApp();

      var Qry = new QueryApp();

      var Cmd = new CommandProcessorApp();

      UI.OnRunCommand(cmd) = Cmd.RunCommand(cmd);

      UI.RequestViewData(requestinfo) = Qry.RequestViewData(requestinfo);

      Cmd.OnChangesEmitted(changes) = Qry.ProcessChanges(changes);

Might be a bit to simple, but I think it might work!

[Angel Java Lopez]

Hi people!

Michael, nice to see a meta model here! 

Hmm... I'm still a bit sleepy, here at Argentina, and a newbie to CQRS ... but I don't understand the last three lines. What is returned by UI.OnRunCommand(cmd) to be a left value?

Angel "Java" Lopez

@ajlopez

[michael-hamburg]

Hi Tom,

My idea is more to define rules like "An aggregate 1-N commands" - You
can find that relationship between the Aggregate and the Command in
the UML diagram.

Cheers,
Michael

[michael-hamburg]

Hi Angel,

> Michael, nice to see a meta model here!

Tnx. Next step is to add the methods for the aggregate roots. I'll
also add the Java source code. Maybe someone can translate it to C#
for the .NET folks!?

Cheers,
Michael

[Kalle Launiala]

Michael,

Thanks a lot for doing this!

Some models define possible entitie-trees belonging to each aggregate
roots, what's common take on those? The ones I saw were like tree-
hierarchy-relation of entities. I don't personally have enough CQRS
design experience to know.

Kalle

On 19 marras, 12:17, michael-hamburg <reg.google....@future-invent.de>
wrote:

[michael-hamburg]

Hi Kalle,

I know what you mean - The current diagrams shows a more "type based"
view of the model. I'll think it should be possible to create another
perspective that focuses more on the aggregates, child entities and
the relations to other aggregate roots. I'll get back after I created
such a view.

Cheers,
Michael

[Kalle Launiala]

Hi,

I want to underline, that I'm not sure if the entities are really
required or even desired in aggregate root.

I made an example model for CQRS where I somewhat forgot the entities;
partially on purpose, because I wanted to understand them properly.
They reflect rich domain model a lot, and hence might be even
undesired "echo" of something that is not necessarily required or
desired in CQRS+ES model.

I'll make an effort to bring C# in, if you don't find anyone else to
take on the C# side. I can combine example for T4 generators with your
model and while at it, combine the Java side as well (so that it can
be deployed to Android for client-independent logic as well). That
will serve also a model how anyone benefits from such a meta-model for
common grounds of recognizing the design layer.

Kalle

On 20 marras, 10:23, michael-hamburg <reg.google....@future-invent.de>
wrote:

[Philip Jander]

Hallo Michael,

> Any comments welcome.

I am missing some things here and there is at least one surplus:
- For me, the concepts of entity and value object are more fundamental
than those of aggregate and aggregate root. From a modeling perspective,
that is. On the implementation/technical side, aggregates become
important due to their role in defining consistency. While it is
possible that you find a model where every entity is an aggregate of its
own, I would consider that highly unlikely. By starting with the concept
of the aggregate as the fundamental one, I feel that there is guidance
towards developing a disfunctional domain model.
The aggregate root is just the entity by which the aggregate is defined.
I find that the strong notion within DDD of only accessing entities and
value objects through their aggregate's root becomes somewhat trivial in
cqrs style architectures as all those other aggregates are practically
inaccessible anyways, if one command only affects one aggregate.

- A DDD model is about behaviour. Not so much about properties. If you
buy into Greg's TDA style, there just are no properties, only private
fields. And my experience is that while some of those fields may in fact
correspond to parts of the read model, this is by far not always the
case (see one example below). While in CQRS (sans event sourcing), you
must define state on entities and value objects in order to have them
persistable, behaviour - i.e. methods - would still come first for me.

- What I do not understand is why a "Query" should realise interface
"AggregateRoot" (section Aggregates(Overview)). Queries should be
completely decoupled from the write model, that's the whole point of
cqrs after all. Instead a projection from the write store to the read
store should update the read side when required, independent of whether
that write store is a state store or event store. Maybe I am
misunderstanding the class diagram here?

More in general: if you try to define a meta model, I believe you will
need to distinguish between CQRS-state backed and CQRS-event sourced.
With the latter, my nouns are "use case", "command", "entity", "value
object", "method", "aggregate", "event", "projector", "projection"/"read
model" and "query". With the former, I am actually using a comparable
meta model for code generation and in retrospect it has failed quite
miserably, as it concentrated too much on properties. So I consider this
a mistake by now :)

Cheers
Phil

Appendix: Example on non-correspondence between write and readmodel:
Take an entity "user account", which has a method Deactivate. As the
result of a legit call to Deactivate, the entity emits an
AccountWasDeactivated event. Upon digesting its own event, the entity
sets a private bool _unaccessible to true. The readmodel reacts to the
event by 1) deleting the user from an "available logins" database and by
2) inserting a new document/row into the "historical users" database
with a "DeactivatedAsOf" field containing the event's time stamp.
So the entity's private field _unaccessible doesn't exist in the read
model, and the read model's timstamp data doesn't exist on the entity.

[michael-hamburg]

Hi Philip,

Thank you for the feedback.

> I am missing some things here and there is at least one surplus:
> - For me, the concepts of entity and value object are more fundamental
> than those of aggregate and aggregate root. From a modeling perspective,

> that is. .....

> With the latter, my nouns are "use case", "command", "entity", "value
> object", "method", "aggregate", "event", "projector", "projection"/"read
> model" and "query".

Hmm... Maybe it's a too technical perspective I created the model
from... I have to think about that.

> - A DDD model is about behaviour. Not so much about properties. If you
> buy into Greg's TDA style, there just are no properties, only private
> fields.

Maybe "Property" is just the wrong term. For me a" property" is just a
field that may, or may not, have setters & getters. I think I'll
change it into "Attribute" .

> - What I do not understand is why a "Query" should realise interface
> "AggregateRoot" (section Aggregates(Overview)).

As you can see in the model a query may also be used in a method. This
means there is a reference from the "Command" (that is part of the
aggregate "Aggregate"... Hmm... I like talking "meta"... ;-) to a
Query. As only aggregate roots can be referenced from the outside, a
query has to be an aggregate root itself.

> Queries should be completely decoupled from the write model, that's the whole point of cqrs after all.

You can always call a query from the write side. That's what's
expressed with the diagram - A method has a reference to 0..N queries.
In a concrete case you'd pass a query to a method as an argument (Java
has no such concept but you can replace it with an interface that has
only a single method and pass that interface to the method).

Cheers,
Michael

[Philip Jander]

Hi Michael,

>> - A DDD model is about behaviour. Not so much about properties. If you
>> buy into Greg's TDA style, there just are no properties, only private
>> fields.
> Maybe "Property" is just the wrong term. For me a" property" is just a

> field that may, or may not, have setters& getters. I think I'll

> change it into "Attribute" .

I guess "attribute" is the technically correct term. Although, since
attribute has a different meaning in the .net CLR, property is in fact
used often instead, so that's really a matter of taste.
My point had a deeper connotation, although I admittedly didn't express
that: in the TDA style/event sourcing style, I do not *design*
fields/attributes *at all*. They are an implementation detail used in
the digesting of events by an entity. Literally internal state.
The thing coming closest to them (the visible data buckets, so to speak)
are command parameters, event properties and read model
fields/properties. And the important thing is that these three must be
independent of each other. There may be some correspondence, but it is
not to be expected by default.

I tried metamodel+model-driven code generation. And although it is nice
in some respects, my largest single mistake was to first define
attributes on entities+value objects and then try to automagically
derive command parameters, event properties and readmodel tables *from*
them. This utterly failed as it drove my design into the data-centric
world. Since them I am doing the exact opposite: Command parameters
derive from use case description, readmodels derive from UI design, and
event properties derive partially from the business description making
up the ubiquitous language and partially from readmodel requirements
(aka enrichment). Thereafter, the domain contains whatever is required
to fulfill the requirements (pun intended). Works way better. Now
admittedly, this leads to some manual mapping code that *seems* to call
for automation. But if I look closely, there are a lot of egde cases
where automation would be inappropriate. So in the end, the manual
mapping is nothing compared to the work required to correct the
automatic mapping coming out of a MDG approach, even if one provides
sufficient hooks. Plus, that manual code is exactly expressing how e.g.
the read model is derived from the events. So there is value in coding
this explicitly.

>
>> - What I do not understand is why a "Query" should realise interface
>> "AggregateRoot" (section Aggregates(Overview)).
> As you can see in the model a query may also be used in a method. This
> means there is a reference from the "Command" (that is part of the
> aggregate "Aggregate"... Hmm... I like talking "meta"... ;-) to a
> Query. As only aggregate roots can be referenced from the outside, a
> query has to be an aggregate root itself.

I am sure you are outsmarting yourself here. Query and Command are
*outside* of the domain model. Commands (or rather command handlers
handling commands) call into the domain model for fulfillment. Queries
only query the readmodel, they are nowhere near the domain model at all
(by definition of cqrs!).

This rule: ">As only aggregate roots can be referenced from the outside"
comes from large domain models in a non-cqrs world. It is a golden rule
to decouple parts of the domain model (i.e. aggregates) by prohibiting
pointers between them. So this rule applies only to the *inside* of the
domain model.

I fear that part of your meta-model is what is called a
"Kategoriefehler" in German.

>
>> Queries should be completely decoupled from the write model, that's the whole point of cqrs after all.
> You can always call a query from the write side. That's what's
> expressed with the diagram - A method has a reference to 0..N queries.
> In a concrete case you'd pass a query to a method as an argument (Java
> has no such concept but you can replace it with an interface that has
> only a single method and pass that interface to the method).

I strongly disagree with that. You must not call a query from the write
side. At least not a query of the same BC. Whatever state you need to
fulfill a specific method call / command is in fact (internal) state of
that part of the domain model. I can allow for querying other BCs (see
current "What to do with supporting data?" thread started by Mikael
�stberg), but even for data coming from different BCs, I would prefer to
inject it as command parameters, if possible. This is all about decoupling.
But for the same BC, having your model generate data, which is projected
into the read model and then re-queried into the domain is a no-no for
me. Remember that the readmodel caters for the UI/view requirements!

The more I think about this, a CQRS meta model should contain the
"domain" as a black box. And the meta-model for the inside of the domain
is amply laid out in Eric's "blue" book. Mixing both seems like a
mistake to me.

Cheers
Phil

[michael-hamburg]

Hi Phil,

> The more I think about this, a CQRS meta model should
> contain the "domain" as a black box. And the meta-model
> for the inside of the domain is amply laid out in Eric's
> "blue" book. Mixing both seems like a mistake to me.

I think this is the most important point - That's exactly why I didn't
include terms like "Value Object" or "Entity" - Because I'm not really
modelling the DDD "entity model" but only the "outside" of it! Maybe
the only fault in the diagram is to model the Aggregate's attributes.

But one question remains: Isn't a command finally only another
representation of an aggregate's method? I always created command
handlers 1:1 to the aggregate root's methods: One method = One command
handler). If this is the case, you can think of the aggregate root as
a container for a set of "remote methods" you can call. The command
handler acts as a kind of "proxy" that can do things you wouldn't like
to include in your domain model. For example execute the mentioned
queries and then call the aggregate's method.

> You must not call a query from the write side.

Why not? I don't even think it's possible to get along without doing
this. Whenever you need data that's not inside your aggregate, you
need to perform a query! A query may be a kind service (check credit
card), data that enriches the event (think about determining a name
that is not part of the write side) or it may be a validation check.
Especially the last one is something that cannot be done on the client
side for security reasons. If you pass me a reference (ID) to another
aggregate, I have at least to check if it exists! This will be done by
executing a query.

Cheers,
Michael

[Philip Jander]

Hi Michael,

>
>> The more I think about this, a CQRS meta model should
>> contain the "domain" as a black box. And the meta-model
>> for the inside of the domain is amply laid out in Eric's
>> "blue" book. Mixing both seems like a mistake to me.
> I think this is the most important point - That's exactly why I didn't
> include terms like "Value Object" or "Entity" - Because I'm not really
> modelling the DDD "entity model" but only the "outside" of it! Maybe
> the only fault in the diagram is to model the Aggregate's attributes.

I guess I agree here. I was initially lead towards thinking about
entities by Kalle's mentioning of entity trees. If we accept "Aggregate"
as "part of a domain model responsible for fulfilling the command", that
makes sense. But then, attributes don't really make any sense here.

> But one question remains: Isn't a command finally only another
> representation of an aggregate's method? I always created command
> handlers 1:1 to the aggregate root's methods: One method = One command
> handler).

I would turn that around: for every command, there should be some method
somewhere on the aggregate (read: in some entity of the aggregate) which
is called by the command handler. This does not need to be on the
aggregate root because the AR would assume responsibility for actions
which are in fact within the purview of one of its constituent entities.
Value objects cannot handle command calls as there is nothing they could
possibly do once they exist, being immutable by definition. Also, not
every method has a corresponding command. If that were the case, the
"aggregate" would be nothing more than a collection of transaction
scripts. The nature of a domain model requires action to be distributed
among the different parts of the aggregate, delegating responsibility to
where it belongs.

> The command
> handler acts as a kind of "proxy" that can do things you wouldn't like
> to include in your domain model. For example execute the mentioned
> queries and then call the aggregate's method.

I agree, as long as you don't query data resulting from previous
operations within the same context. If you do that, you do not have a
cqrs based system, as your domain state is now contained in the read
model. Your read model has in fact become a write model, incurring all
the problems of shared models that cqrs tries to alleviate in the first
place.

> > You must not call a query from the write side.
> Why not?

see above.

> If you pass me a reference (ID) to another
> aggregate, I have at least to check if it exists! This will be done by
> executing a query.

But not against the non-authoritative, possibly not-yet-consistent,
optimised-for-reads readmodel.
What you need here is a part of the write side responsible for checking
whatever is required here. I guess you read the threads about
"uniqueness" problems some time back in this list, where some of these
things were discussed.
Coupling the domain to the readmodel brings quite a number of problems
with it. I agree it looks simple and tempting, especially since some
data may be "already there". I can only state that I have tried that,
too, and ended up in a tangled mess. So now my rule is, if it is
required in the domain, it belongs either into state or command
parameter (with the exception of cross-BC, where orchestration by
command handlers is acceptable).
Violating that rule leads to - if nothing else - impossibility to test
the domain without the readmodel present. And this is really a good
indication that something went wrong. Note that injecting data from a
readmodel via client and command is different as input data may be
freely defined for test cases. It is just the active calls out into the
readmodel from the same domain that really hurt by increasing coupling
and breaking the encapsulation of the domain model.

Cheers,
Phil

[michael-hamburg]

Hi Phil,

Nice discussion! :-)

I refactored the model and the diagram now looks much more
appropriate:
http://www.metacqrs.org/metacqrs-ddd.html

I renamed the "Aggregate" into "AggregateProxy" as I feel that is much
more expressive.
The "DDD domain" is hidden inside the proxy and not part of the CQRS
model.

I still included queries in the command. A queries in this case is
just a function or service call that is needed during the execution of
the command. Of course it will not be serialized and transported along
with the command data, but it's a logical part of executing the
command.

Cheers,
Michael

[Kalle Launiala]

Just for the side note, as we have had quite a bit of experience in
design vs. specification difference (and mostly our approach is
therefore bit misunderstood).

You can have models for specs (that is use-cases and specification-
level domain model if you will) AND for design level stuff.

For design level stuff you DON'T put the use-cases, but something
similar (we had logical operations and operation-trees); that is the
behavioral model.

I'll reply properly when I have time to read the entire thread.

Just keep the specifications and design separate, because that's what
they logically are. They look-alike a lot, but your motivates to drive
the software design needs to be free from the use-case maker's
decisions.

Kalle

On 20 marras, 20:25, michael-hamburg <reg.google....@future-invent.de>
wrote:

[Kalle Launiala]

Ok I'll summarize as I recognize Philip's points that bad-made
compromises lead into.

1. The meta-model needs to be logically pure. No compromises. Be
prepared also to update and fix your model in case the logical
inconsistency is ever found in the model.

Philip mentioned that the model + generation went ill as he tried to
derive command parameters from the entity properties/fields. This is
one example why the logical model needs to be pure; your entities and
command parameters are separate logically.

2. Generators need to bend over and around serving the manual code,
not the other way around.

This might mean that you sometimes get 1:1 look-alike structures, but
be ready to change them if your meta-model's design gets detected to
have shortcomings. If you're ending with *more* manual work than
without the generators, you should look into improving your generators
to do it right.

In the discussion I understand that you might want to compromise
"queries being aggregate roots". I had the same looking structure in
our model + automation but it was logically the other way around; AR
could be flagged to create 1:1 query to represent the AR state.
However it was still technically independent read-model query; it just
had 1:1 mapping code generated. This was "opt-in" flag that allowed
1:1 generation.

The read-model cannot serve as a business logic decision making tool
for domain model, but you can have domain model mapped to read-model
(for the client-side queries) no problem (although it's still bound by
same rules for denormalizing; it is not immediately refreshed by the
ar changes).

I don't want by any means to discourage you to take on the meta-model
of CQRS; on the contrary. I will try to support you doing the same
model (or another variant based on this thread) in parallel and
demonstrate the T4 generators that automate the model.

Kalle

[michael-hamburg]

Hi Kalle,

Tnx for the comments.

I think the current meta model ("Version 2") is the right direction.

I'll add some more details to the diagrams over the next days - That
will make it easier to discuss it.

Cheers,
Michael

[Philip Jander]

Hi Michael,

one question upfront: are you targetting event sourcing or non evet
sourced implementations of cqrs, or are you trying to remain agnostic of
that?

> I refactored the model and the diagram now looks much more
> appropriate:
> http://www.metacqrs.org/metacqrs-ddd.html

I am getting a much better picture of what you intend now. Besides it
starts looking familar :)

I am just noting down remarks as I am looking through the diagram, not
all remarks really considered in depth:

- I think the "AggregateProxy" is in fact a "command handler" in cqrs
terms. This way, you will get rid of DDD terms entirely, which isn't a
bad thing given that a lot of people consider both concepts to be
orthogonal. Now you can define a mapping "command handler" : "method
slot on aggregate" as just a specific driver, connection your cqrs model
to ddd.

- I am wary about the "Command"-"Event" link. Obviously it would be
great to determine externally what events are the result / can be the
result of a given command. But therein lies a danger of restricting the
implementation within the actual "domain black box" by defining this
correspondence upfront. Personally, I have avoided mapping this so far.
I would be interested in any experience once you try to generate an
application from that model.

- As written earlier, I really doubt the Command->Query link. But I
guess it doesn't hurt to start out with it. I just wouldn't be surprised
if it leads to trouble in implementations (not of the code gen but
maintenance and testability of the actual final implementation).

- I am missing one area: projections from the write model to the read
model. Since it looks like you are trying to cover the whole area from
commands to query results (possibly even in a traceable manner), maybe
there is some way to include this. While your meta model so far doesn't
really commit to either event storage or state storage, projections in a
meta model work only for event sourcing. I would recommand against
trying to define an event->readmodel mapping on an attribute level. But
since there is likely a 1:1 correspondence between "readmodel" and
"query result", you might want to think about adding a 1:* relationship
between "Result" and "Event" labelled "influenced by". This would
express which events can possibly lead to a change of one "Result". This
meta information is highly attractive for efficient readmodel projection
of the event stream as an application can be provided with information
on which events to load and which to ignore. Again, all of this applies
to event sourcing, only.

- From a practical POV, I always end up with some access control
metadata in the meta model. Maybe including a concept of claims to both
commands and queries adds to your model. For me, the ususal sets of
required/sufficient/inhibiting claims work very well.

- Finally, the whole model should be boxed and titled "business
context", just to make the limits explicit.

- Since you went this far, what about including the concept of a view
model? You can very likely define "View model" references "Query" as
well as "View model" references "Command" without much trouble. Caveat:
assuming composite applications, the viewmodel will likely sit outside
of the BCs referencing several of them. But then, maybe this is out of
scope here and should be part of a systems/application model.

Cheers
Phil

[Kalle Launiala]

I really like where this thread is heading for. I wanted to give you
heads up just to prepare you to see something unorthodox soon.

I will prepare metamodel using XML schema. It is different looking
than the UML-model, but it allows certain benefits over the UML (while
also certain shortcomings if not bundled with visualizer).

The viewmodel is relatively straightforward to derive from the CQRS
model (that's why I like the design so universal to apply everywhere);
you just have to let go the systems/applications boundaries there.

Consider the software/application/system design to be happening from
the higher level of abstraction. The generators don't mind, if you
just strictly bound them to focus on their field of responsibility;
such as generating view-model proxies that *are not* identical to the
commands, but *can be derived* from the command information 1:1 if
that's desirable.

Kalle

[michael-hamburg]

Hi Phil,

> one question upfront: are you targetting event sourcing or non evet
> sourced implementations of cqrs, or are you trying to remain agnostic of
> that?

I think the model is agnostic of that.

> - I think the "AggregateProxy" is in fact a "command handler" in cqrs terms.

Yep. I already changed that when I was thinking about it again this
morning.

> I am wary about the "Command"-"Event" link.

I think it's necessary as I want to add BDD test descriptions later
on. Tests are defined by 1) Given "events" 2) When "command" 3) Then
"events".

> As written earlier, I really doubt the Command->Query link.

I renamed it into "function" to have a clear distinction between a
service call used during the command execution and the "query side".

> - I am missing one area: projections from the write model to the read model.

I'll add that soon - Here We're going to have terms like "Query" and
"Denormalizer" - The latter will be connected to one or more events.

> - From a practical POV, I always end up with some access control metadata in the meta model.

What exactly you're thinking of?

> - Finally, the whole model should be boxed and titled "business
> context", just to make the limits explicit.

> - Since you went this far, what about including the concept of a view
> model? You can very likely define "View model" references "Query" as
> well as "View model" references "Command" without much trouble. Caveat:
> assuming composite applications, the viewmodel will likely sit outside
> of the BCs referencing several of them. But then, maybe this is out of
> scope here and should be part of a systems/application model.

Hmmm... Not sure about this. I thought about it, but maybe this is not
really the scope. I'll keep that in mind ;-)

Cheers,
Michael

[michael-hamburg]

Hi Kalle,

> I will prepare metamodel using XML schema.

I tried this first but I finally wanted to use CQRS for defining and
working with the CQRS model. I created a small prototype (http://
www.metacqrs.org/metacqrs.swf) but it doesn't feel 100% right yet.
That's why I started creating a CQRS meta model to get things more
clear before I continue working on the prototype.

Cheers,
Michael

[michael-hamburg]

Sorry, here is the correct link:

http://www.metacqrs.org/metacqrs.swf.html

[Philip Jander]

Hi Michael,

>> - I am missing one area: projections from the write model to the read model.
> I'll add that soon - Here We're going to have terms like "Query" and
> "Denormalizer" - The latter will be connected to one or more events.

I'd really propose "projection" instead. Query should be reserved for
client->server queries. I always felt that "denormalizer" is wrong, as
there are not that much write models in 3NF anyway, and denormalizing
from 1NF to another 1NF... . Mathematically, the readmodel generation is
always a projection*, either of the write model (CQRS) or of the event
stream (CQRS+ES). So why not call it by that name?

>
>> - From a practical POV, I always end up with some access control metadata in the meta model.
> What exactly you're thinking of?

I have a "Claim" (as in claims-based auth) on my meta model, and Command
as well as Query reference Claim 1:* with either modality "required",
"sufficient" or "inhibiting". This meta data is used client-side to
predict whether a user is authorized for a specific operation and
disable/enable that functionality. Server-side, the server accepting the
command annotates the command envelope with the actual user claims
before enqueueing it. Lateron, a command handler performs the actual
authorization and can also forward some of the information into the
domain, if required - sometimes business rules depend on some ownership
relations between the commanding user and some entity. Due to queueing
and distribution, the actual user session may not be available at the
time the command is processed. Therefore, I started noting the user's
claims in the command envelope. So far that works quite well. But it is
really a non-critical (=easy) part of my meta model.

Cheers
Phil

* defined in a very very weird space

[Kalle Launiala]

Michael,

I looked at the video. It's not the same approach (of purely raised
level of abstraction) that I use.

The approach is the easiest explained with the demonstration of what
our model looks like and how it's used to realize the generation in a
controlled way. I try to finish this by the end of the week.

Kalle

On 21 marras, 21:59, michael-hamburg <reg.google....@future-invent.de>
wrote:

> Hi Kalle,
>
> > I will prepare metamodel using XML schema.
>
> I tried this first but I finally wanted to use CQRS for defining and

> working with the CQRS model. I created a small prototype (http://www.metacqrs.org/metacqrs.swf) but it doesn't feel 100% right yet.

[Greg Young]

I use the term projection. It fits.

[michael-hamburg]

> I use the term projection. It fits.

@Greg & Phil:

But how should we name the unit that makes the projection? Projector?
(Sounds more like a beamer ;-)

[Greg Young]

I call the thing with the handlers a projection

[michael-hamburg]

Hi,

I now added BDD style test stuff (write side) and also added the read
side:
http://www.metacqrs.org/metacqrs-ddd.html

@Phil: Could you give an example what's exactly inside a "Claim"? I
never worked with Claims based authorization before... (Only "Role
based").

I'm also thinking about defining an service level agreement (SLA) for
a command.

Cheers,
Michael

[Kalle Launiala]

@Greg & Phil

Did I reinterpret correctly;

Projection = Denormalized Read Model Data (+ its handlers)
Query = Actual query; many queries can target to one projection
(index, filtered whatever)

Kalle

On 22 marras, 20:48, Greg Young <gregoryyou...@gmail.com> wrote:
> I call the thing with the handlers a projection

> On Nov 22, 2011 6:57 PM, "michael-hamburg" <reg.google....@future-invent.de>

[michael-hamburg]

Hi,

> Did I reinterpret correctly;
> Projection = Denormalized Read Model Data (+ its handlers)
> Query = Actual query; many queries can target to one projection
> (index, filtered whatever)

I think a projection in this case is the process of converting N
incoming events (from one or more aggregates) into one query. In other
words: The "Projector" (alias "Denormalizer") listens to several
events and uses the data of the events to populate the query.

Cheers,
Michael

[michael-hamburg]

Hi all,

What about modeling synchronous and asynchronous command handling?

I'm thinking about marking a command in the meta model as sync or
async.

In the sync case errors are simply reported back to the client (as
shown in the diagram: http://www.metacqrs.org/metacqrs-ddd.html). How
do we handle errors in asynchronous mode? Is it a valid option to send
events in case of such an error?

Cheers,
Michael

[Greg Young]

I feel less and less that async command handling actually exists. In
all the cases I can find a use for an asynchronous command its
actually an upstream event.

--
Le doute n'est pas une condition agréable, mais la certitude est absurde.

[Henric Edwards]

Depending on type of error I'd guess either log and notify or display in relevant view that enables a corresponding corrective action? Or quite possible all of the above?

-Henric

[Philipp Meier]

Am Donnerstag, 24. November 2011 08:10:02 UTC+1 schrieb Greg Young:

I feel less and less that async command handling actually exists. In
all the cases I can find a use for an asynchronous command its
actually an upstream event.

What about a command requesting a long running recalculation of sales prices? Whould you consider this an event?

-billy.

[Greg Young]

By async command I mean a command that returns an ack representing
that it was accepted not that it was processed. Also called one way
commands. Your example is not really an "async command" it finishes
synchronously saying "the command was validated and accepted" ... no
different that "TransferFunds" in a bank.

--

[Nuno Lopes]

Hi Greg,

By async command I mean a command that returns an ack representing
that it was accepted not that it was processed. 

What do you mean by "upstream event" in this context?

Nuno

[Greg Young]

An event that happened somewhere else (like say the client)

--

[michael-hamburg]

Hi Greg,

> On 24 Nov., 08:10, Greg Young <gregoryyou...@gmail.com> wrote:
> I feel less and less that async command handling actually exists. In
> all the cases I can find a use for an asynchronous command its
> actually an upstream event.

What about a kind of "Upload" command? Example: Upload of a large
Excel file (>100.000 lines) with new agreements. Some lines may lead
to an error because the agreement is a duplicate. It should be async
because it may take a long time to process.

Cheers,
Michael

[michael-hamburg]

OK, already answered ;-)

On 24 Nov., 09:22, Greg Young <gregoryyou...@gmail.com> wrote:
> Your example is not really an "async command" it finishes
> synchronously saying "the command was validated and accepted" ... no
> different that "TransferFunds" in a bank.

On 24 Nov., 10:21, michael-hamburg <reg.google....@future-invent.de>
wrote:

[Greg Young]

Generally the command would have a link to the document not the document itself.

--

[michael-hamburg]

Who will actually do the async work? (Parsing the Excel and sending
commands to the approriate aggregate to add the valid agreements and
send notifications about error lines)

[Greg Young]

you can do on either side ...

--

[Nuno Lopes]

Hi Greg,

Let me see if I get my head around this:

Your example is not really an "async command" it finishes
synchronously saying "the command was validated and accepted" ... no
different that "TransferFunds" in a bank.

If command validation is out of equation for acceptance in which case it could not be accepted? If it can't never be refused why is "accepted" a variable in this equation? Instead

Your example is not really an "async command" it finishes

synchronously saying "the command was validated" ... no

different that "TransferFunds" in a bank.

So the TransferFunds is a sync command since the caller waits for validation correct?

So you are feeling that are no one way commands, only events. Am I following you correctly?

Cheers,

Nuno

[Greg Young]

Take for example the canonical example used of one-way commands an
ATM... We drop a queue in front of the domain model and the
"RemoveFundsFromAccount" command's "ACK" means "it was accepted and
will be processed eventually" ... I do not find this to be a command
at all but an event "MoneyGivenToClientAtATM" ... it can't fail. If it
were to fail it would be a business concern (another event) as such
its not really a command. All the cases I see of "one way" or "async"
commands tend to fit this pattern.

--

[Nuno Lopes]

Hi,

Take for example the canonical example used of one-way commands an
ATM... We drop a queue in front of the domain model and the
"RemoveFundsFromAccount" command's "ACK" means "it was accepted and
will be processed eventually" ... I do not find this to be a command
at all but an event "MoneyGivenToClientAtATM" ... it can't fail. If it
were to fail it would be a business concern (another event) as such
its not really a command. All the cases I see of "one way" or "async"
commands tend to fit this pattern.

I see. So the decision of giving the money to the client is done synchronously at the ATM not the server. It may happen that there aren't sufficient funds still.

Wouldn't the ATM be a separate BC?

Cheers,

Nuno

[andho]

I think I get what you're saying, but let me confirm in an example that I understand.

If I was to create a CancelOrder command, after which the app shows the user that "Cancellation will be processed", it actually should have been a RequestCancellationOfOrder Command, which does not guarantee the order will be cancelled.

[michael-hamburg]

Hi,

I think it's actually an "CancellationOfOrderRequested" event that is
raised by the client.

Cheers,
Michael

[Daniel Pittman]

What I took away was that, in Greg's experience, if you need the
decision made by the central system you need a two way (RPC) call, not
a one way command message. Which makes some sense...

Daniel
--
♲ Made with 100 percent post-consumer electrons

[Kalle Launiala]

Published first draft of the metamodel with XML schema model:

http://abstractiondev.wordpress.com/case-study-cqrs/cqrs-metamodel/

The annotations are lightly documented, because I'd like to get input
on the model; especially on the Projection => Query relation. The
technical names are meant to be self-explaining, but all the comments
on the model are very welcome.

Including also the git repository info that the abstraction is being
built in.

Kalle

On 22 marras, 23:26, michael-hamburg <reg.google....@future-invent.de>
wrote:

[Greg Young]

Maybe I am missing something really obvious ... what is a "Domain
Type" that "owns" all of this stuff?

--

[Kalle Launiala]

It's meant to be the context boundary for certain area of a system.
Usually system is within a single domain entirely, but group of
separate systems can be also managed through describing a set of
domains.

In the model there is a plural Domains => Domain (1-n) in them, but I
left that one out in the diagram for now.

Naming may be ill so if you got my explanation and have better naming
for that, please do suggest better alternative.

Btw a side note, I've modeled AR containing commands, but events being
in parallel to ARs. Is it also so that single AR handles single event
or can there be multiple ARs handling the same event?

This model does not bind single event to any single AR now. I'll
remodel that if its usually the case.

Projections are of course not AR specific.

Kalle

[Greg Young]

I guess I don't see how the bounded context is represented.
I also don't get why an aggregate has commands? What about Command Handlers?
Events are certainly not disconnected as shown since the stream is
partitioned normally by event id

[Kalle Launiala]

@Greg

I think you're looking it as platform-object level; not as a meta-
model level.

This is not an object model. This is a meta-model.

It means that in logical model each domain can have 1-n ARs. Each AR
has 1-n creation commands and 1-n commands that certain AR type
handles.

Events are independent entities that are not "owned" by any AR
currently (but this was one thing that I wasn't sure of).

Kalle

[Kalle Launiala]

If you think the schema XML content it gets easier to understand:

<Domain name="Bank">
<AggregateRoots>
<AggregateRoot name="Customer">
<CreationCommand name="CreateCustomer">
... (parameters and such)
</CreationCommand>
<Command name="SetCustomerCreditLimit">
....
</Command>
<Command name="SetCustomerName">
....
</Command>
</AggregateRoot>
</AggregateRoots>
<Events>
<Event name="CustomerCreated">
... (parameters and stuff)
</Event>
<Event name="CustomerNameSet">
... (parameters and stuff)
</Event>
</Events>
...
</Domain>

This way the metamodel allows defining your whole domain without
constraining it into anything platform specific. Now the generators
can realize any implementation of such thing up to the human written
code lines.

If that sounds like "why waste my time when I can just code"; when you
take documentation autogeneration, versioned events and such things,
you can separate the technical challenges and approaches of the
logical model that says how things should be.

It becomes very clear to communicate your design through this kind of
a model; this is why I'd very much like to understand how people think
CQRS should be modeled.

Kalle

[Greg Young]

Aggregates don't have commands. They don't even respond to commands.
One command can be sent to different kinds of aggregates (What if I
have a command that operates on a role?). There can be a command that
involves multiple aggregates to execute it (this is very common).

Why are you specializing create commands? In all but the most naive
circumstances this doesn't work as being shown. Its not a createXXX
with all the data for a XXX there. Normally another object assists in
the creation of an aggregate.

> Events are independent entities that are not "owned" by any AR
> currently (but this was one thing that I wasn't sure of).

Event streams are partitioned by aggregateid so there must be some
relationship there no?

I am curious have you implemented a CQRS+ES system without layers of
abstraction built on top before trying to put layers of abstraction on
top of things?

[Kalle Launiala]

@Greg

Thanks for explaining this. I got implication earlier in this thread
that certain command always is handled by certain ARs.

For create commands; I didn't specify XXX being all the data.
Basically CreateCommand(s) are separated in design to separate the
ones that cause creation of a new aggregate instance from those that
do not. They still have completely custom parameters each, such as any
other command as well. They help constrain the responsibility and
isolate the design structures (at least some of them) that allow new
AR instances to be created from those that only modify them.

I have no real CQRS project experience, hence I need assistance in
creating the meta-model. Now that Michael started to work up a meta-
model I'm simply working to get my model provide the same
functionality model that he ends up with the supporting audience here.

I'll adjust the model accordingly; thanks a lot for your input

Once I get model solid, I'll explain in better way how one can
automate the CQRS+ES without any framework or library requirement; yet
still requiring minimal amount of manual coding. All the structures
being provided also coder-maintainable templates. I'd guess something
like your "minimalistic" approach would work as one example of
generator structure output.

Kalle

[michael-hamburg]

Hi Greg,

> One command can be sent to different kinds of aggregates (What if I
> have a command that operates on a role?). There can be a command that
> involves multiple aggregates to execute it (this is very common).

The command handler then operates on several aggregates? I thought
there is a 1:1 relationship from a command handler to an aggregate.
But there is always only exactly one command handler that handles the
command!?

Cheers,
Michael

[Joseph Daigle]

I don't think anyone has ever said that commands are strictly 1-1 with a handler within a bounded context. One of the benifits iof using measages is, for instance, you could build a processing pipeline. It could be that all commands are first send to a handler that validates the command. The next handler might be responsible for journaling or something audit-related. Eventually it will be handled by something specific to that command.

[michael-hamburg]

I don't think it makes much sense to have more than one handler for a
command. A handler is the one that finally executes the command.
Things like validating or auditing is more like a filter. Filters may
be chained because they don't make any changes to the system. A
command handler does change the system's state.

[Kalle Launiala]

Michael,

Are you thinking in a way of "responsibility and isolation"; that is
there is single command-handler who'se responsible for "dealing out"
the remaining to various aspects and handlers?

This is in no way contradictionary of "freedom" that you can do what
ever you want with the command, but it provides a controlled entry
point for command entering to the handling-chain.

Kalle

On 26 marras, 00:20, michael-hamburg <reg.google....@future-invent.de>

[Kalle Launiala]

I take it that events are processed through message pipeline, and
hence treated separately.

While some approaches serialize/store commands as well, my
understanding was that their execution is preferred as "immediate" and
the pipelined behaviour belongs to events.

Kalle

[Kalle Launiala]

Updated the model to anticipate single command handler (that has all
the freedom of course to initiate any chain within), AR outside scope
of events and global handlers but ALSO including definition of AR
specific event handling (in the command handling mapping => event
handler procedures).

I havent opened in this model the AR specific definitions for event
handling, where of course the event handling is supposed to be defined
as well. The event handlers specified in the model are AR independent
global handlers.

http://abstractiondev.wordpress.com/case-study-cqrs/cqrs-metamodel/

Kalle

[michael-hamburg]

Hi all,

I added the Saga model and more comments:

http://www.metacqrs.org/metacqrs-ddd.html

Any suggestions are welcome.

Cheers,
Michael

[Rinat Abdullin]

Michael,

Here's the original definition of term "saga": http://www.amundsen.com/downloads/sagas.pdf It deals with deadlocks and long-lived database transactions.

The term you are probably searching for, would be something like  "Domain Process".

Best regards,

Rinat

[Philip Jander]

> The term you are probably searching for, would be something like
> "Domain Process".
>

Actually, the term is "midget"...

[Greg Young]

:)

[Rinat Abdullin]

"stateless blond" in Soviet Russia

[Nuno Lopes]

"stateless blond" in Soviet Russia

:)

[Philip Jander]

Am 28.11.2011 04:37, schrieb Rinat Abdullin:
> "stateless blond" in Soviet Russia

nice double entendre :)

Phil

[Kalle Launiala]

About Sagas:

Are there any variations (I mean anything with sensible basis, not ill-
based) of how Sagas interact with the system?

This is what I gathered from the model:

- Input from events (for whichever state they are and whatever they
require to perform next)
- Output (for forward) through commands
- Output (for rollback/compensation) as commands

Kalle

On 27 marras, 11:18, michael-hamburg <reg.google....@future-invent.de>
wrote:

[Greg Young]

Thats one kind but misses the intent

[michael-hamburg]

Hi Greg,

Could you describe this a bit more in depth?

Cheers,
Michael

On 29 Nov., 02:23, Greg Young <gregoryyou...@gmail.com> wrote:
> Thats one kind but misses the intent

> On Nov 28, 2011 9:43 PM, "Kalle Launiala" <kalle.launi...@gmail.com> wrote:
>
>
>
> > About Sagas:
>
> > Are there any variations (I mean anything with sensible basis, not ill-
> > based) of how Sagas interact with the system?
>
> > This is what I gathered from the model:
>
> > - Input from events (for whichever state they are and whatever they
> > require to perform next)
> > - Output (for forward) through commands
> > - Output (for rollback/compensation) as commands
>
> > Kalle
>
> > On 27 marras, 11:18, michael-hamburg <reg.google....@future-invent.de>
> > wrote:
> > > Hi all,
>
> > > I added the Saga model and more comments:
>
> > >http://www.metacqrs.org/metacqrs-ddd.html
>
> > > Any suggestions are welcome.
>
> > > Cheers,

> > > Michael- Zitierten Text ausblenden -
>
> - Zitierten Text anzeigen -

[Greg Young]

More than one type of saga (document based ones are very interesting
as well)... Also this kind of misses the intent of what a saga is ...

--

[Nuno Lopes]

(document based ones are very interesting
as well)

:)

Cheers,

Nuno

[Kalle Launiala]

If considering overall role of a Saga (if I interpreted it properly;
allowing long-lasting logical transaction with appropriate
compensation actions to allow logical rollback) the "best-practice"
for Saga to react with the system.

Is it common to have Saga with its own data structure and/or rely on
aggregate roots data? Of course alongside with command/event data
depending if Sagas input from either/or.

Could someone point out some reference on using Sagas in context of
CQRS + ES?`

Kalle

[Rinat Abdullin]

Kalle,

Definition of domain processes ("sagas") strongly depends on the definition of Bounded Contexts. Domain processes help to keep different BCs decoupled, while integrating them between each other and time.

Please note, that this simplistic definition does not say anything about "commands" or "events", "ARs" or even "data structures". These are less relevant than BC in this case. 

Best,

Rinat

[Rinat Abdullin]

Document-based ones (or just stateless) are really dangerous, since they tend to confuse the heck of people, when you push them to technological extremes (code to data :)

[Rinat Abdullin]

Document-based ones (or just stateless) are really dangerous, since they tend to confuse the heck of people, when you push them to technological extremes  :)

[Kalle Launiala]

Rinat,

Thanks a lot for this brief summary!

Gives me perspective of what to look and enough keywords to go for the
info.

Kalle

On 29 marras, 16:12, Rinat Abdullin <rinat.abdul...@gmail.com> wrote:
> Kalle,
>
> Definition of domain processes ("sagas") strongly depends on the definition
> of Bounded Contexts. Domain processes help to keep different BCs decoupled,
> while integrating them between each other and time.
>
> Please note, that this simplistic definition does not say anything about
> "commands" or "events", "ARs" or even "data structures". These are less
> relevant than BC in this case.
>
> Best,
> Rinat
>

[Nuno Lopes]

Hi Rinat,

Document-based ones (or just stateless) are really dangerous, since they tend to confuse the heck of people, when you push them to technological extremes  :)

Can you provide an example over what you consider a document based saga and how they are dangerous?

Nuno

[michael-hamburg]

Hi Rinat,

The problem of CQRS is that the discussions about it are often very
mystic. No clear definitions, no code beyond simplistic examples that
really show what makes up such a system. It looks like there are
several "schools" that have very different views on the subject.
That's why I tried to create a meta model - To make things concrete.
So maybe you can make a suggestion in form of a piece of code, a
diagram or something that shows exactly what a Saga is.

Cheers,
Michael

On 29 Nov., 15:12, Rinat Abdullin <rinat.abdul...@gmail.com> wrote:
> Kalle,
>
> Definition of domain processes ("sagas") strongly depends on the definition
> of Bounded Contexts. Domain processes help to keep different BCs decoupled,
> while integrating them between each other and time.
>
> Please note, that this simplistic definition does not say anything about
> "commands" or "events", "ARs" or even "data structures". These are less
> relevant than BC in this case.
>
> Best,
> Rinat
>

[michael-hamburg]

Hi Greg,

I'm not sure if you want to keep the secret until your book is
released... ;-)
But maybe you can share your knowledge with us unaware people?

What is the intent of a Saga?

Cheers,
Michael

[Jimmy Bogard]

See:

http://www.cs.cornell.edu/andru/cs711/2002fa/reading/sagas.pdf

http://citeseer.ist.psu.edu/chrysanthis92acta.html

and http://www.udidahan.com/2007/12/17/no-more-workflow-for-nservicebus-please-welcome-the-saga/

In short - a saga is a means of representing a long-running business process/transaction as a series of short-lived transactions. Typically, glued together with a small set of persistent saga information, and orchestrated through asynchronous messaging.

[michael-hamburg]

Hi Jimmy,

So a good way of describing a Saga then would be a business process
model?
http://en.wikipedia.org/wiki/Business_Process_Model_and_Notation

Cheers,
Michael

On 29 Nov., 23:34, Jimmy Bogard <jimmy.bog...@gmail.com> wrote:
> See:
>
> http://www.cs.cornell.edu/andru/cs711/2002fa/reading/sagas.pdfhttp://citeseer.ist.psu.edu/chrysanthis92acta.html
> andhttp://www.udidahan.com/2007/12/17/no-more-workflow-for-nservicebus-p...

[Jimmy Bogard]

Kinda. BPM is very state-machine-like, but sagas are intentionally not. Sagas are designed with the realities of async messaging, that messages arrive out-of-order, may never arrive, may take a long time to complete, etc. A saga has state, but similar to aggregates, only as much to determine how to react to incoming messages.

Often I'll build sagas that have no notion of what "state" they're in - they're just part of an orchestration of events. But trying to force a state machine on top of this model just doesn't work.

[Rinat Abdullin]

Document based domain process ("saga") matches the original intent (to express relationships of Bounded Contexts in a way that it can be clearly related to the domain model, while allowing BCs to stay decoupled and clear), however it does not necessarily match the simplified technical definition that people like to use ("event goes in and command comes out"). 

Deviations might include cases where:

1) there are no command and event messages, but just a document that is getting filled as it goes through the process (state is determined by the properties, it does not state in the actual "saga")

2) stateless process is loaded directly into the message dispatcher of an aggregate. So we would have a technically mixed entity that accepts commands and sends out events and commands.

If someone is trying to find and build sagas based on the technical definition (which can vary from sub-project to sub-project), the he is likely to fail. 

I'm certain that since I've done this (failing part) on more than one occasion :)

Best,

Rinat

[Nuno Lopes]

Hi Rinat,

I'm trying to understand what you are saying.

You wrote:

Document-based ones (or just stateless) are really dangerous, since they tend to confuse the heck of people, when you push them to technological extremes  :)

Yet bellow you seem to to favor them.

Can you clarify?

I think formulating "natural" Sagas as document based ones is misleading at least.

Assuming that a document-based Saga is say a PurchaseOrder, they aren't stateless, neither they are dangerous. People are dangerous :)

I would like to invite the group to take a leap in perception from all these patterns that come to us from the technical space, to the "natural" domain patterns in the business context. We talk about commands, queries, relational tables, documents, CQRS, Commands, Events, Saga's, Queues, Event Streams almost as a fact of nature. 

IMHO, in fact the system design challenges does not really rely in the domain neither can be solved by applying these technical patterns. But it lays in our perception of the problem domain as technical people. We often try to fit the domain constructs to these technical pattern without fully understanding them. The result s quite often a overly complicated approximation of the natural domain, that is hard to understand and evolve.

In fact, these technical constructs can constrain ones brainwaves, leaving a sense of false assurance.

Let's us pose the question to our technical minds:

"What is a Purchase Order?".

Is it a document?

Is it a list of products to buy along with its cost and some buyer information?

For different people the concept is revealed differently. For instance, for a record manager it might be a record. For a document manager it might be a document, for a bookkeeper .... For a buyer it might be a list of things to purchase.

But there is a deeper meaning to it. One that does not change from people to people, neither from context.

A PurchaseOrder is in fact neither of those things at its core. Those things are just projections of actually a purchase order is. A PurchaseOrder is ......

A BUSINESS OPERATION! A ONE THAT IS LONG RUNNING! 

If it was not a long running business operation it could be a COMMAND (an atomic business operation). In abstract we could do (if a supplier was an Aggregate):

aSupplier.Handle(aPurchaseOrder).

Bu that is not what happens.

Why such operation even exists in an business? We are so used in making purchase orders that the answer, is, "yeah I got that, I do that all the time". But he fact is that most of us simply don't know why, yet the answer is quite revealing.

1) This operation has several outcomes for the same input. It is not deterministic. For instance such operation might not be concluded with a an effective purchase (payment + delivery), ending up being cancelled. It might happen that only a part of the products of an PurchaseOrder is indeed purchased.

2) Both the seller and the buyer are two distinct business agents that act asynchronously and disconnected (communication channels aren't open continuously).

Looks familiar? Indeed, stuff, like order lines, products and prices, customer information etc, are just "ARGUMENTS" of these business operations. 

Ok a Purchase Order is such a chronic example, that by exhaustion it may not mean much to many people.

Take an Insurance Contract. Do you think that an Insurance Contract is a document for the purpose of automating the insurance process? 

An Insurance Contract is another business transaction! A very long one that sets the constraints for other business operations during a period of time (time out?).

Take the act of sending an email. Do you think this is a Command as per CQRS? Not necessarily ... It depends. If a successful send does not depend on successful delivery it may be a command, but if does then it is another example of a long running operation. We click send and the email might never be sent or delivered. Should I assume that it was sent? What about delivery? Is it a long running business operation or a command? 

What about a doctors appointment? 

So when do we introduce LONG RUNNING OPERATION into an algorithm? It does not matter if its is introduced by a business person, a programmer or some IT architect, it dos not cease to be a long running operation. It is a natural pattern :)

For a concept to be a long running business operation it must firstly be some recognizable instant (moment) in time or duration (interval) of time, an activity that we can give a name. However, this is not sufficient. It is a concept that we need to work with or remember in our field of endeavour (our problem domain). If the lifecycle of the operation is not relevant to the domain that we are working in, then the concept is either not worth representing explicitly in our system or the concept, in our domain, is not a long running business operation (it can be an item/product, or the description of some item, or the projection of an operation and so on).

Finally before you start designing a system increase performance by leveraging on async processing some more food for thought. 

Look at the natural laws around you. A small business is not run as a larger business. There is a extra cost in managing a large business, some might say the cost of entropy created by parallelism of multiple long running operations. For instance, one of them is the Employment Contracts. Scaling down is costly. More costly then scaling up (ok clouds can help :)). You should use these laws of nature to help you decide between an async and a synch algorithm.

Want an example?

Say you have a Supermarket. In this Supermarket one can buy a product online and in the shop! The business processes (algorithms) for a purchase are distinct. For one, we don't need a PurchaseOrder when we buy in the Shop. Why? Because all agents are locked to the task (there are only two, the cashier and the buyer and are in CONTINUOUS communication).

The scale is just different, hence a different algorithm is necessary.

Yet we can derive a lot of lessons from the purchase in a supermarket back to our systems :)

In a supermarket you go around, select the products, put them in the shopping cart. When you decide to buy you go in the direction of the cashiers. There are multiple. Each forming a *QUEUE* of agents like yourself. before I pay many things can happen. For instance, I might receive an urgent call, get fade up of waiting or simply decide not to buy anything, ending up abandoning the Shopping Cart right there (someone will for sure notice that the SC is abandoned and finely collect it and put the the products back in the shelves. It might be the case, that after I arrive and declare the goods in the SC to the cachier (checkout), I realize that I don't have enough money to pay and resolve to only declare some goods. Etc etc.

Now notice this, if the all supermarket is considered to be a warehouse open to the public, every time I take a product from the shelve I'm effectively reserving the product to me. This operation has immediate effect. Does it really need to be async and the be managed like a long running business operation? It depends. You can argue, well, that is because in a supermarket you have a limited number of products and the customer aren't really that much in comparison to a online shop with millions of clients on a black friday. Do you know that products in a supermarket are put on shelves? If you have say 500 hundred items, distributed in shelves of 10, you can make 50 reservations at the same time in a split second. If you have 100.000 items, we can have 1000 reservations in a split second (that is not much at all). This is a competition model for you :) Now if reservation fails one can for instance ask the user if he wants to give/make a Purchase Order for the product (long running business operation), or offer an alternative.

You can do the same in our online system. When a person puts something in the shopping cart we could automatically make a reservation. If the shopping cart is abandoned (say for 10 minutes), it can be collected and the products can be put back up for reservation.

As for processing Checkouts. In a supermarket we have a limited number of POS's. So the business uses queues/buffers, open more queues/buffers, monitor the queues and check the constraints. This is drum buffer rope at its best. There is no point in making something extremely performant and async only to be funneled to something extremely slow. You will just get  a lot of abandoned shopping carts (SAGA TIMEOUTS) of you do!

So we can use this lesson's when making our domain models. You can make an aggregates called POS, give it an ID. This Aggregate handles the registration of a PurchaseOrder, processes the transaction and issues a Shipping Order. All in and synchronous way. 

You can say, look payment depends on an external activity so it may take long. Well, let the client pay for the goods before reaching the checkout. The checkout only verifies if the good was payed accordingly and let you walk away giving the subject as closed. You may even realize that each PurchaseOrder can work on behalf of a POS. So each person will have its own personal POS assigned dynamically per ShoppingCart. In a digital world we can do that as  the cost of setup is minimal, only constrained by the power of our Hardware. 

Note nevertheless that going parallel not guarantee maximum velocity. Passing on information between Aggregates or BC per event may not maximize the system throughput. In fact, quite often batching events together towards a common destination (port) helps quite a lot!!! Again theory of constraints.

I also have been hearing stories that a single regular quad core server, with 32GB can process over 6 million transactions in a serial way (LMAX). 

So before you start going async with events all over the place, there is a fare amount of stuff you need to understand and experiment with.

Again, I'm putting Event Sourcing in a different category. Indeed, I think that event sourcing is one of the "natural" behavioral domain patterns much like a long running business operation, aggregates and values objects. CQRS is also a natural domain pattern, yet is more of a representational/projection pattern multiplexing one representation with several. Tables and Documents are more inline with memory / visualization patterns, also natural. Lists, Queues, Trees and so on, are organizational/structural patterns supporting decisions.

All I'm inviting you all is to look at domain concepts  first as operations (yes, that included the shopping cart) and challenge this precept all the time. I find it harder to arrive the simplest design yet flexible if I think simply as Entities and Value objects conceptually countered by commands and events. More difficult even if we put into the mix stuff like purely abstract constructs like Sagas where with no machine affinity.

Cheers,

Nuno

[Rinat Abdullin]

Heh :) That's why I mentioned that technical definition of domain process ("saga") is a misleading one, it "breaks" on non-classical implementations of the same thing. Starting from DDD book and going via BC will lead to better understanding.

I favor non-classical technical implementations of domain processes simply because my environment benefits from that (cloud, elastically scalable and volatile distributed systems, more than 60 messages per second at peaks and still growing). Purely technical reasons.

However, despite the implementation details, at the level of domain models definition of domain process in my systems still stays the same. And that's what really matters.

Hope, this helps.

Best,

Rinat

[Nuno Lopes]

Heh :) That's why I mentioned that technical definition of domain process ("saga") is a misleading one, it "breaks" on non-classical implementations of the same thing. Starting from DDD book and going via BC will lead to better understanding.

I was just trying to understand your point and learn from it. I guess we are arriving to the same conclusions.

Hope this helps,

Nuno

[michael-hamburg]

Hey Nuno,

I think things get mixed up here. As far is I see it CQRS is simply
something very different than DDD. It's about transforming a DDD model
into a technical structure. What you described with your examples is
pure DDD. You are right - This is the first, the most important and
the most difficult step when designing a new system. But the next step
is to "wrap" the CQRS stuff around this DDD model. And this is pure
technical (CQRS) structure.

What I'm trying is to grab the Ubiquitous Language for a CQRS system.
I'm NOT trying to define DDD.

The question is if a Saga belongs to the CQRS "domain" or if it's part
of DDD. Is there a definition of long running business processes in
DDD? I cannot remember there is something in the book...

Cheers,
Michael

On 30 Nov., 11:06, Nuno Lopes <nbplo...@gmail.com> wrote:
> > Heh :) That's why I mentioned that technical definition of domain process ("saga") is a misleading one, it "breaks" on non-classical implementations of the same thing. Starting from DDD book and going via BC will lead to better understanding.
>
> I was just trying to understand your point and learn from it. I guess we are arriving to the same conclusions.
>
> Hope this helps,
>
> Nuno
>
> On Nov 30, 2011, at 9:25 AM, Rinat Abdullin wrote:
>
>
>
> > Heh :) That's why I mentioned that technical definition of domain process ("saga") is a misleading one, it "breaks" on non-classical implementations of the same thing. Starting from DDD book and going via BC will lead to better understanding.
>
> > I favor non-classical technical implementations of domain processes simply because my environment benefits from that (cloud, elastically scalable and volatile distributed systems, more than 60 messages per second at peaks and still growing). Purely technical reasons.
>
> > However, despite the implementation details, at the level of domain models definition of domain process in my systems still stays the same. And that's what really matters.
>
> > Hope, this helps.
>
> > Best,
> > Rinat
>

> ...
>
> Erfahren Sie mehr »- Zitierten Text ausblenden -

[Nuno Lopes]

Hi,

It's about transforming a DDD model

into a technical structure. 

IMHO DDD is about designing a technical solution for a domain problem in such way that there is a strong affinity between the concepts applied in the end solution and the concepts domain (Domain Affinity). To take advantage of the technical solution some reengineering of the domain but IMHO its is usually minimal. Has the business grows in scale, business processes need to be reengineered as a result the technical solution needs also to be reengineered. It is a mutual learning process only possible if the affinity is strong.

This is the difference between for instance DDD and SOA IMHO. In SOA as long as we have a service with a domain name it is fine. It is above the concept of Identity which is at the core of DDD IMHO.

In DDD we see a lot of technical concepts being discussed from a conceptual perspective because several technical implementations are possible. What constraints a technical implementation is the technical infrastructure of choice. The technical infrastructure is very important to an end to end solution, and it might make a huge difference. That is why it is important to know and understand several implementations taking advantage the best the infrastructure has to offer.

As in CQRS in most domains there the need to project the same concept in a varied number of ways depending on the context where those concepts are observed. So I see CQRS as an important DDD construct.

If you search for CQRS in DDD forum you will for sure find Greg stating that CQRS is DDD (or part of DDD).

In fact, every time we see Greg's talking, is not just CQRS but DDD :) CQRS is just a catalyst. IMHO, things start to fall apart when one puts CQRS at the core of ones design (principles?), more specifically, Commands and Queries. 

Cheers,

Nuno

[Michael]

In case that someone reads this old discussion... I moved the UML to a new location:

http://www.fuin.org/metacqrs/metacqrs-ddd.html