Introducing DDD/ES/CQRS: how to deal with Eventual consistency in master/detail forms ?

[Joseph Pachod]

Hi all

I've been busy recently on some DDD/ES/CQRS presentation (in French). At the end of last week I did a "crash test" of the presentation, and the result was very good.

Yet, afterward, eventual consistency started to be a big issue, especially the list/entry update example :

I've a list, I open the details of one item, I change something and returns on the list: most likely the change isn't displayed yet... Really confusing for the user. Baddd.

There I would suggest to do some "optimistic display" in the ui: sending the command, wait for the ack "taken by the model" and then consider the outcome as successful and display accordingly. Inform later the user that it did fail.

Would you suggest something better or more convincing ? It seems indeed quite something to put to work.

If you wish so, I could provide the detail of the part I'm planning to put in the talk, but for me the above point is the most important (pops up easily and quickly in the mind, yet the solution is bit, hum, complex)

Hints & advices welcome for the real talk (in two days :))

best
++

[Bryan Watts]

Distribution often comes part and parcel with these concepts, but it is not an inherent trait. I have had great success showing the concepts within a single db transaction, trading the murky complexity of a network with a crisp alignment to existing preconceptions. It provides a foundation from which to build an understanding of the effects of eventual consistency, in a context that doesn't require immediate buy-in from learners (which often leads to a derailing of the core concepts and a meaty reason to be skeptical).

--
You received this message because you are subscribed to the Google Groups "DDD/CQRS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dddcqrs+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Greg Young]

This is one way of handling it. There are many ux strategies that can be applied to minimize people noticing eventual consistency

--

[Greg Young]

Btw if you have an ack why would you inform them later that it failed?

On 23 Feb 2015 01:35, "Joseph Pachod" <joseph...@gmail.com> wrote:

--

[João Bragança]

'Synchronous' command handling + PRG. The time it takes the browser to do the redirect is usually more than enough time for the projection to update in the background.

--

You received this message because you are subscribed to the Google Groups "DDD/CQRS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dddcqrs+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Sent from my regular computer

http://twitter.com/thefringeninja

http://www.thefringeninja.com/

[Tom Janssens]

You can:

a) Hope the projection update is fast enough.

b) Go synchronous.

c) Fake it on the client side.

d) Inform the user that the request will/is being processed. (Showing a queue somewhere f.e.)

e) a combo of c & d (for example using a different background color for "unconfirmed" changes.

AFAIK "e" is usually best solution from a UX perspective, unless processing takes more then 3 seconds, then I'd rather opt for "d".

The "3 seconds" comes from an article about expected UI responsiveness, but I can't recall the source ATM.

Op maandag 23 februari 2015 00:35:53 UTC+1 schreef joseph:

[Tom Janssens]

Apparently, it's not 3 seconds, but 1 second (as I started looking for the source): http://ux.stackexchange.com/questions/2/what-is-an-acceptable-response-time-for-my-ajax-ui

Op maandag 23 februari 2015 09:02:37 UTC+1 schreef Tom Janssens:

[Joseph Pachod]

Thanks all for your answers =)

@Bryan
Well, I present the following: Command issued/Command Bus/Event issued/Event bus/View update, I don't know of a transactional mechanism for it all.

Actually already asking for a transactional event bus is quite something AFAIK.

On top, the talk is conceptual rather than implementation oriented, so in general eventual consistency is put forward when discussing DDD/CQRS/ES.

In the end though, I put skipping eventual consistency for some specific use cases as a possible mitigation issue, however the master/detail kind of UI is quite common and shouldn't require each time to drop eventual consistency.

@Greg
I'm also presenting after various UX strategies, actually Tom Janssens sums them up. Yet if there's some better way to handle this use case/question I welcome it ;)

Regarding the ack, I was thinking of an "command acknowledge ack", not an "command successful ack", which feels "dirtier/clumsier". However I present this option as well in my "mitigation options" part.

@João
The talk being rather conceptual, I would prefer to avoid saying "go synchronous", esp since I'm speaking of "Command issued/Command Bus/Event issued/Event bus/View update", where I don't know of a synchronous implementation anywhere (and I won't suggest distributed transaction as an option ;) )

@Tom
I'm basically presenting these options as well later.

For a, it doesn't work if the user returns on the list upon edit validation, or at least it feels really unlikely to work out fine.
For b, as said before, I'm presenting "Command issued/Command Bus/Event issued/Event bus/View update", where I don't know of a synchronous impl. In fact in the talk I suggest to skip CQRS/ES if willing to go synchronous.
I present c and d as well, plus one extra being: block on the client side for the required view update, which require as well extra care on the view side.

Well, I conclude at the end of the talk by something like "monolith or distributed, choose your poison", having shown before the pro & cons of each. I guess eventual consistency nicely illustrates this issue ;)

Best
Joseph

--

[João Bragança]

> The talk being rather conceptual, I would prefer to avoid saying "go synchronous", esp since I'm speaking of "Command issued/Command Bus/Event issued/Event bus/View update", where I don't know of a synchronous implementation anywhere (and I won't suggest distributed transaction as an option ;) )

There seems to be a bit of confusion on this list about what synchronous and asynchronous are. Synchronous just means control is handed back to the caller after the method runs. Kind of like the difference between async void and async Task in c# 5.

https://github.com/gregoryyoung/m-r/blob/master/CQRSGui/Controllers/HomeController.cs#L40 <- This is completely synchronous command handling with async projection updating.

Having an asynchronous command bus makes things unnecessarily complicated. If you need non blocking IO, you can wrap your commands in an envelope with a task completion source on it. You get the benefit of both worlds here.

[Greg Young]

Regarding the ack, I was thinking of an "command acknowledge ack", not an "command successful ack", which feels "dirtier/clumsier". However I present this option as well in my "mitigation options" part.

There is another thread recently you should read your improved way doesnt exist (it would be an event not a command) and even then has a ton of tradeoffs where it becomes much more complex. It should not be the default.

[Joseph Pachod]

@João

The "eventually consistent" issue popped up when speaking of a Command, which upon being accepted generates an Event which in turn trigger the update of the view being used.

I don't know of a stack allowing of blocking on the call to send the Command up to the view being updated, do you ?

@Greg

Is it the Growing CQRS Applications discussion ?

I put in the talk current form that this ack should be optional, but I could also just skip it: it was just for the sake of listing options.

In the end, how to tackle this eventual consistency at best is still a bit foggy to me. Personally I would go for letting the user knows work is in progress, but even such an UI is quite something implementation wise IMHO.

Thanks again

++

[Greg Young]

"In the end, how to tackle this eventual consistency at best is still
a bit foggy to me. Personally I would go for letting the user knows
work is in progress, but even such an UI is quite something
implementation wise IMHO."

return sequence of event with your ack ... then when querying say I
must query greater than X. You would never see eventual consistency
but run the risk of queries not working if denormalizers were being
upgraded etc (this can be mitigated with multiple read models etc).

There are other options as well but this would simulate consistent
reads to an arbitrary read model.

> --
> You received this message because you are subscribed to the Google Groups
> "DDD/CQRS" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to dddcqrs+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--

Studying for the Turing test

[João Bragança]

> I don't know of a stack allowing of blocking on the call to send the Command up to the view being updated, do you ?

any underlying database that supports DTC will allow you to do this, not that you should! Our current stack is doing this :(

> In the end, how to tackle this eventual consistency at best is still a bit foggy to me

POST /order/753947D1-6513-4E30-837D-D01E1DDFFF45/checkout HTTP/1.1

Host: myawesomecart.com

** snip **

HTTP/1.1 303 See Other

Location: /orders

GET /orders HTTP/1.1

Host: myawesomecart.com

By the time the browser receives the response to POST and makes the subsequent GET call 200-500 ms have passed. this is more than enough time for your projections to update. I am pretty sure this is exactly how greg's example code works.

--
You received this message because you are subscribed to the Google Groups "DDD/CQRS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dddcqrs+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

[uwe schaefer]

On Monday, February 23, 2015 at 12:35:53 AM UTC+1, joseph wrote:

Hi Joseph,

 

Yet, afterward, eventual consistency started to be a big issue, especially the list/entry update example :

I've a list, I open the details of one item, I change something and returns on the list: most likely the change isn't displayed yet... Really confusing for the user. Baddd.

ahh.. the good old sync UI on top of async system.

what about: the LAST event emitted from the command handler returns its
ID to the UI Layer.

the UI Layer then includes this id in its query with the semantics of:
answer that question after you processed the event with the ID given.

good part 1: you have your sync interface back
good part 2: you can time out and notify the user that his change is not
yet visible, but will be soon

bad part: every model must recieve the event, even if only to know its
ID (so that it knows when to anser the pending questions)

pseudocode:

FactId lastIdEmitted = cmdSink.execute(myCommand)
Future<ItemList> listAfterMyCommand = listModel.query(lasIdEmitted);
...
ItemList il = listAfterMyCommand.get(2,SECONDS)

it has its downsides, but is the best strategy i can come up with. 

 

There I would suggest to do some "optimistic display" in the ui: sending the command, wait for the ack "taken by the model" and then consider the outcome as successful and display accordingly. Inform later the user that it did fail.

then you either create a private "throw away" model, or you query the

writemodel which is both less than optimal, imho. 

best wishes for the presentation,

cu uwe

[Kijana Woodard]

This all depends on how you are doing cqrs.

If your read side is an orm over the same rdbms as the write side, you're done.

What do you mean by "back to lists".

On "add new" go to details and buy time.

On modify state such that it no longer "qualifies for the list", it depends. With raven db, you can remove it after it's returned in the index [messes up paging], display a placeholder or notice that the doc is "changing state", or do a "wait on write" so that the index is up to date.

Greg already gave the answer if event sourcing.

--

[Joseph Pachod]

Thanks all again for your answers.

@Kijana

Well, the plan is to use some stream processing framework, not a transactional one/rdbms.

@Greg

Well, in the end, after speaking with a friend of mine, we ended for a similar solution: put an UUID in the command, wait for this UUID to show up in the targeted view or some error view. On top, limit the waiting time at something like 200 ms, and if it takes longer display a message to the user 'your order is in progress, you'll be notified when done". Sounds pretty slick no ?

@João

While the update might take place before the redirect is done, I wouldn't bet on it and thus plan for fallback plan. However, the above seems pretty good in this field :)

[Jo Geraerts]

Hello,

What about letting your read side emit an event again (that is published to the event bus, but not the event store) and let your UI listen for this?

Kr,

Jo

Op maandag 23 februari 2015 00:35:53 UTC+1 schreef joseph:

Hi all

[Joseph Pachod]

@Jo
Well, indeed, why not. Is it common to have an non stored event bus as well as the read side emitting events ? Somehow it surprises me, but I don't see strong reasons not to do so.

Thanks for the ideas :)

--

[Joseph Pachod]

@Jo
Actually I do have a concern with the view emitting an event to say "updated" : what if there is more than one instance of the view? Should the client somehow be able to make sure it looks into the updated view instance and no other? This doesn't sound good.

On a bigger picture, I guess the client should do a query similar to "If command of UUID commandId is in, then give me the state". This is the required action when waiting for post command update, but actually may help in another use case as well : how to push changes to the client.

For this use case, I would to the following : an "agent" (for lack of proper a word) subscribes to the wanted events. When one event arrives, this agent then queries regularly the view with "if command of uuid lastSeendCommand is in, then give me the data". And then send overt the data to the user interface. As such the view's work isn't duplicated and still the user interface is updated.

In fact, the only drawback is sending to much data to the user interface, indeed all would be sent and not only the update. To avoid this, I guess the query should actually be "If command of UUID Y is in, then give me the changes since command X".

I doubt am I clear about what I'm trying to express, so don't hesitate to ask for more details where needed ;)

cheers

joseph

[Alexandre Potvin Latreille]

I tend to agree with the following IDDD quote:

Discussing this with Eric Evans revealed a very simple and sound guideline. When examining the use case (or story), ask whether it’s the job of the user executing the use case to make the data consistent. If it is, try to make it transactionally consistent, but only by adhering to the other rules of Aggregates. If it is another user’s job, or the job of the system, allow it to be eventually consistent. That bit of wisdom not only provides a convenient tie breaker, but it helps us gain a deeper understanding of our domain. It exposes the real system invariants: the ones that must be kept transactionally consistent. That understanding is much more valuable than defaulting to a technical leaning.
This is a great tip to add to the Aggregate Rules of Thumb. Since there are other forces to consider, it may not always lead to the final choice between transactional and eventual consistency but will usually provide deeper insight into the model. 

 

[Joseph Pachod]

Hi Alexandre

In practice, regarding the current use case, what would it mean for you ?

best

--

[Jo Geraerts]

Op zondag 1 maart 2015 23:48:37 UTC+1 schreef joseph:

@Jo
Actually I do have a concern with the view emitting an event to say "updated" : what if there is more than one instance of the view? Should the client somehow be able to make sure it looks into the updated view instance and no other? This doesn't sound good. 

What do you mean with one instance of the view? You could only emit an event for the specific viewmodel that is being used for the master list.

Kr,

Jo