Udi Dahan's Advanced Distributed System Design with SOA & DDD Course : Thoughts and Revelations
jodomofo
Thoughts and Revelations
Just got back from Udi's five day course and am sorting it all out and
reflecting. It was a intense, fun and tiring five days in San
Francisco and now one day and one badly needed long hot shower later,
I feel I have reached some clarity on his approach to SOA and DDD.
I went into this course with a solid year of consuming much of the
available online material from Greg and Udi, and several reads of the
Blue Book. Having all that under my belt really helped in
comprehending the vast amount of material Udi presented.
As for SOA, Udi's approach is vastly different then most conventional
approaches and centers around the idea of autonomous business
components that are capable of fulfilling particular business
requirements from top to bottom in a complete vertical slice. A
Service's responsibility is blown out to include the data storage
needs(the database), all the way up to the visual representation needs
of the UI. Each service owns all the data it needs to fulfill the
business requirement, and is the sole owner of this data. The concept
of single ownership of data and no data duplication across services is
a fundamental concept in Udi's approach to SOA.
All was going well as we headed into day four of the course and dived
deep into Saga's. By days end I felt I was solidifying my
understanding of a Saga as a business process that spans an
indeterminate amount of time. After class that evening I was trying to
come to terms with how a Saga interacted with the domain objects and
particularly aggregate roots. Does a Saga delegate to one or more
aggregates in order to perform the work that is needed?
By this time it was feeling more then a little unusual that we were
heading into the the final day of the course and the terms Aggregate,
Entity, and Value Object were not even mentioned. What was going on
here, did Udi just not have time to dive into these implementation
details, or was there something missing in my overall understanding?
As day five moved closer to its finality and the term aggregate root
had still yet to come into play, I took the opportunity to ask Udi
directly how a Saga would utilize the aggregates in the domain? The
answer to this was surprising and perplexing. A Saga and a aggregate
were essentially synonymous with each other, and the Saga was the sole
object used to perform business operations; there is no delegation to
anything else. Huh? Everything I learned about DDD and consistency
boundaries centered around the Aggregate Root object, and the fact
that an aggregate is an Entity, which has identity based around a
thread of continuity that spans time. For instance a Customer is a
person with some sort of identifier that makes it unique regardless of
its particular attributes. The customer has an identity that
withstands, regardless of its state changes. Without identifying these
Entities how could we model a system?
The revelation that came one day after the course ended, came by way
of further analysis of the primary concern identified by all of the
material I have read over the last year, consistency boundaries.
Identifying these boundaries is the primary goal when modeling a
system, and by insuring integrity of operations within these
boundaries we can add value towards the scalability, maintainability
and integrity of a system.
Over the entire course Udi kept stressing the value of finding the
smallest possible consistency boundaries that would ensure the
business requirements dictated by the domain experts would be
fulfilled. When you beak a business operation down to its finest grain
of elements you find that what you have is a process that utilizes
these elements in a particular context, and that as long as you
provide consistency of these elements over the course of this process,
you will have fulfilled the business requirements. What we are saying
is that a business requirement can be modeled as a process. Thinking
about the purpose of domain objects, which is to 'do things', I
believe the concept of domain objects and processes fit perfectly.
Looking at the Saga pattern, which manages a business process, I was
able to understand how it is not necessary to delegate to a
AggregateRoot(a Entity) because the business process is being modeled
explicitly by the Saga.
The paradigm shift here comes by way of the single responsibility
principle. When modeling entities(Customer, Order, etc) we are
implicitly setting ourselves up to break SRP. If the purpose of a
Customer is perform business operations, and we put more then one
operation on it, it is very likely that we will have a object that is
not fully cohesive; meaning every operation is probably not touching
all the state owned by that entity. Hence it is managing more then it
needs to for a particular business requirement, and the consistency
boundary has not been broken down to its finest possible number of
elements. The shortest path to solving this problem is to model the
business process explicitly, which will lead you towards a object that
is fully cohesive, and that maintains a consistency boundary no bigger
then what is required. By modeling processes rather then entities the
objective of operating within the smallest possible boundary is
achieved.
With entity modeling you will inevitably find yourself in situation
where one business process will span more then one aggregate root.
Using DDD you might solve this with a Domain Service. If you are doing
this, it is strong evidence that you are not explicitly modeling the
consistency boundaries required by the business, and your model it
wrong. By modeling 'the processes' from the start you will never have
a consistency boundary that spans more then one aggregate root, it
just logically does not make sense.
I feel the primary goal of Udi's approach is to achieve the highest
possible level of maintainability and scalability for large scale
systems, and by searching for the smallest possible consistency
boundaries, you allow for the maximum amount of loose coupling and
independent deployment and scaling opportunities. As it stands in
regard to my current projects and goals, the need/desire to shift my
models away from entities and towards processes is something that will
take much more contemplation, but I feel I have reached a decent level
of understanding of this approach and feel it does make sense. Being
that it is new to me, it feels uncomfortable and sort of closer to
writing transaction script rather then rich Entity/Value Object based
models that I have been striving for.
jodomofo
Sharas
Sharas
jodomofo
concept to understand, but a I hard one to see the value of in many
cases, such as reporting where the data is definitly read only.
After reflecting on this rule, I feel the reason for it might be as
follows:
If you have duplication of data in which the data is stored in a means
that transactional boundaries will be crossed(distributed systems),
you can not ensure the integrity of the data. So in a distributed
system, perhaps its necessary to do this if you want to ensure
integrity.
Sarunas Mazylis
Daniel Pittman
I attended the same course; this is my answer to your question.
> @jodomofo: So what is the reason for each ABC to be the sole holder of the
> data? Why can't other ABCs store reference data? It is read only isn't it?
As I understand it, each BC is contained within a service; inside one
service a BC is only going to deal with some of the data inside that
service. (eg: bulk vs regular customers). Given that, inside a
service sharing the same data means you have some problem with your
business analysis - your BCs are not actually independent.
I would guess that you are asking about sharing the data between BCs
in different services, though, right?
Part of this is as jodomofo says, that you can't manage data integrity.
I took away, though, that the more important reason for this is this rule:
A service is the technical authority for a part of the business.
eg: Marketing is the technical authority for prices.
If you share that data with another service - they have a copy of it -
you no longer have that technical authority. Marketing can't control
how Sales use the price data they have a copy of, Sales have no hope
of knowing if their data is up to date[1].
Giving Sales a copy of that data is bad enough because it smears
responsibility for that information around multiple services, but it
gets worse: it encourages bad design in Sales, and even worse, for
Sales to start exposing price information. Now you have *two* places
that might give you a price - and who do you trust? Sales?
Marketing? Both? Neither?
I think that this problem is why Udi so strongly encouraged visual
composition rather than copying data: instead of thinking that sales
need access to the price information, visually compose them next to
each other so the sales data sits beside the price data, each pulled
from their own service.
Finally, when it comes to things like interacting with the outside
world for, eg, payment, Udi put emphasis on the IT/operations team
owning that, and being the *only* department that can query multiple
business services to get what they need.
So, instead of Sales collecting price data (owned by marketing), card
data (owned by billing) and the invoice details (owned by sales), the
payment service owned by IT/ops will collect the minimum that it
needs, when instructed, to make the external payment.
This makes integration between services a separate consideration to
the services themselves, which I think is an overall win.
Finally, to answer your latest message where you say:
> If data is read only, why do you need to ensure it's integrity? It doesn't change.
One other thing that Udi was very emphatic about through the course,
and I think rightly so, was that time is always a factor in this
design.
If that data never changes, as in, is a constant, then you probably
can have a copy in each service, but they don't need to exchange it.
It is literally unchangeable.
If, instead, you mean "never changes" in the sense of "isn't changed
by the service that has a copy", that seems like a narrow view of the
problem space to me. Once it is possible to change that you have to
go back to the question of freshness, and what business value the data
gives.
If you don't care about integrity (eg: you could replace every value
with the word "fred" instead) then why have a copy of it? If you
intend to use the data, or display it to someone, or whatever, then
you have taken on the responsibility of the other service.
eg: If Sales is showing price information, they have taken over
responsibility from Marketing for making sure the user knows how up to
date that is ... and, worse, they have taken ownership of the business
part of that. Now they have to implement all the freshness policies -
heck, all the policies - of Marketing to get that done.
If you really need that, I think that Sales and Marketing are not
separate services, and you would do better getting rid of that notion
entirely. :)
> Anyway, I'm not gonna interrogate you:) But that bit of sole ownership is really hard to get as you say. Especially in some scenarios where reference data can be stored in a ABC which is most interested in keeping it unchanged, like in an example Udi gives with user name and invoice.
One of the questions that Udi asked was: do you really sound like the
technical authority for something if you say that you can tell someone
what the authoritative, absolute value is right now, but you have no
idea what it was like yesterday?
In the case of the name on an invoice, you don't actually want "the
name on the invoice", you want "the name of customer 12 on the 30th
October, 2011, at 6:45PM" - and the service that owns the "customer
name" will be responsible for making that available even if the name
changes later.
Daniel
Footnotes:
[1] For example, if the marketing server crashes it could take a while
for it to get back, so the fact the price changed is unavailable until
that recovers.
--
♲ Made with 100 percent post-consumer electrons
jodomofo
that says all data can only be owned by one service, and you have
services that are all autonomous, with one service not knowing what
the other is doing, how could you know that data the should be read
only outside its owning service, is actually being used that way?
Perhaps the rule is to ensure that all data is consistent according to
the rules of the service that owns it.
jodomofo
very well.
On Oct 30, 5:50 pm, Daniel Pittman <dan...@rimspace.net> wrote:
jodomofo
the data.
Daniel Pittman
I should add, Sharas, that I covered a very, very small part of what
the course did in that description. I can really recommend the whole
five days as being excellent, and well worth the time. There are lots
of other parts that also play into this picture that make it all gel
together.
Daniel
Sarunas Mazylis
I attended the same course; this is my answer to your question.
> @jodomofo: So what is the reason for each ABC to be the sole holder of the
> data? Why can't other ABCs store reference data? It is read only isn't it?
service a BC is only going to deal with some of the data inside that
service. (eg: bulk vs regular customers). Given that, inside a
service sharing the same data means you have some problem with your
business analysis - your BCs are not actually independent.
I would guess that you are asking about sharing the data between BCs
in different services, though, right?
Part of this is as jodomofo says, that you can't manage data integrity.
I took away, though, that the more important reason for this is this rule:
A service is the technical authority for a part of the business.
eg: Marketing is the technical authority for prices.
If you share that data with another service - they have a copy of it -
you no longer have that technical authority. Marketing can't control
how Sales use the price data they have a copy of, Sales have no hope
of knowing if their data is up to date[1].
Giving Sales a copy of that data is bad enough because it smears
responsibility for that information around multiple services, but it
gets worse: it encourages bad design in Sales, and even worse, for
Sales to start exposing price information. Now you have *two* places
that might give you a price - and who do you trust? Sales?
Marketing? Both? Neither?
In the case of the name on an invoice, you don't actually want "the
name on the invoice", you want "the name of customer 12 on the 30th
October, 2011, at 6:45PM" - and the service that owns the "customer
name" will be responsible for making that available even if the name
changes later.
Daniel
Footnotes:
[1] For example, if the marketing server crashes it could take a while
for it to get back, so the fact the price changed is unavailable until
that recovers.
Sharas
Daniel Pittman
@Daniel: thanks for your answer.
I attended the same course; this is my answer to your question.
As I understand it, each BC is contained within a service; inside one
> @jodomofo: So what is the reason for each ABC to be the sole holder of the
> data? Why can't other ABCs store reference data? It is read only isn't it?
service a BC is only going to deal with some of the data inside that
service. (eg: bulk vs regular customers). Given that, inside a
service sharing the same data means you have some problem with your
business analysis - your BCs are not actually independent.
I would guess that you are asking about sharing the data between BCs
in different services, though, right?
Part of this is as jodomofo says, that you can't manage data integrity.
I took away, though, that the more important reason for this is this rule:
A service is the technical authority for a part of the business.
eg: Marketing is the technical authority for prices.
If you share that data with another service - they have a copy of it -
you no longer have that technical authority. Marketing can't control
how Sales use the price data they have a copy of, Sales have no hope
of knowing if their data is up to date[1].Why not set expiry date on prices and let sales know that those prices are valid for a specific interval, so sales does not have to worry about changes of prices for that period which increases autonomy of sales. Good example described in here with catalog prices.
Giving Sales a copy of that data is bad enough because it smears
responsibility for that information around multiple services, but it
gets worse: it encourages bad design in Sales, and even worse, for
Sales to start exposing price information. Now you have *two* places
that might give you a price - and who do you trust? Sales?
Marketing? Both? Neither?Marketing is entry point for prices which are broadcasted with an expiration date. Sales can display prices for that period but does not modify them. That again increases autonomy of those services.
In the case of the name on an invoice, you don't actually want "thename on the invoice", you want "the name of customer 12 on the 30thOctober, 2011, at 6:45PM" - and the service that owns the "customer
name" will be responsible for making that available even if the name
changes later.I understand that's how Udi SOA is done, but actually, in this case I need a snapshot of users name with invoice, not a knowledge that I need to get it from somewhere by date and id.
Footnotes:
[1] For example, if the marketing server crashes it could take a while
for it to get back, so the fact the price changed is unavailable until
that recovers.
That's the problem with Udis approach, to make this actually work you need to keep those ABCs highly available because they are temporaly coupled, but implicitly by a composite application.
Nuno Lopes
(allocate another couple of clones of the marketing service to give higher availability).
Daniel Pittman
>> (allocate another couple of clones of the marketing service to give higher availability).
>
> How do you mantain the boundaries consistent across clones?
So, this is strictly a problem inside the service, right? Obviously,
if you implement the service so it can't scale out, only scale up,
then you can wind up with terrible problems here. OTOH, you can
potentially also put in the effort as you need it: if your scale
issues come from a lack of CPU, scale out workers against a shared SQL
database. If they come from the database, change the internal store,
or scale up the DB server, or whatever.
(Also, query is less likely to hit your domain model storage database;
because you can heavily denormalize in the view model, the client is
pretty low impact, and can be pushed to a separate database if
required.)
However, if you are designing around the DDD model where you only give
atomic action inside an aggregate root, this is easier than it sounds.
You have the one place that needs to be implemented safely around
that, so it can be atomic within the AR.
This actually gets somewhat easier in the model Udi proposed, where
your saga *is* the domain model: you need to account for a race around
the events that start a saga, if it starts from more than one place,
but otherwise you just respond to events as they come in and manage
local state inside that structure.
It doesn't make handling concurrency easy, but it does make it fairly
manageable. (Also, it means that when someone messes up here you wind
up with one broken part of one service, not everything broken
everywhere, which also helps contain the damage.)
Does that make sense?
Daniel
Thanks to everyone in the thread: this is still being a big help in
forcing me to think through and explain my understanding of what Udi
proposed. Please don't mistake me for any sort of expert with real
world experience of this stuff, though. :)
Rinat Abdullin
> potentially also put in the effort as you need it: if your scale
> issues come from a lack of CPU, scale out workers against a shared SQL
> database. If they come from the database, change the internal store,
> or scale up the DB server, or whatever.
Daniel Pittman
>> OTOH, you can
>> potentially also put in the effort as you need it: if your scale
>> issues come from a lack of CPU, scale out workers against a shared SQL
>> database. If they come from the database, change the internal store,
>> or scale up the DB server, or whatever.
>
> How do you scale out view models here?
On the read side, or the update side?
On the read side, just run more instances that react to events from
the bus, do their calculations, and store the data in their table.
(If you have more than one view, or a view and an augmented view, and
they are not separable by UI composition, Udi suggests just having two
tables.)
Alternately, use database replication for read, or whatever else
floats your boat. Given that the read side is, essentially, select *
from $view, this is cheap. You can also look to, eg, NoSQL database
here in many cases, because you often have view models defined around
"get data for this set of IDs", which those tools are excellent at.
On the update side? You don't have to run each view model on the same
hardware, so if one is too costly, split it off. Give it dedicated
hardware, storage, or whatever you need to make it perform adequately.
(Keep in mind that all this is inside a single BC, and each view
model updater is focused on a single, very small widget shown on
screen - eg, the price of a book, or the title of a book, rather than
an entire page.)
This allows you to focus the money the business gives you to improve
performance on explicitly the part of the system that is hurting
performance.
Oh, and remember that your view model is explicitly stale data; if it
is marginally more stale, you have to assess if there is a business
cost to that, right?
Daniel
Greg Young
can just multiplex a queue to my event handlers?
--
Le doute n'est pas une condition agréable, mais la certitude est absurde.
Daniel Pittman
My recollection is that it came up in the course, but that might have
been the general discussion since it doesn't show up in my notes.
Daniel
Neylor Ohmaly Rodrigues e Silva
Ben Kloosterman
Alexey Raga
Ben Kloosterman
I would disagree with Bennie when he says " if you use the pricing service but a new system uses it so heavily it makes it runs slow. ". I cannot justify maybe breaking encapsulation and spreading logic because of possible performance issues.
To me even if it happens (a system uses it so heavily it makes it runs slow) then it is much easier to scale this service out and have 2, 3, 30 copies of it running than producing and maintaining more copies of data + more logic around these copies just because something may be slow at some point.
Kijana Woodard
I find data to be a leaky abstraction. Logic tends to clump to it. Now the authority service can't change without coordinating.
If you mean the authority maintains a copy of the data closer to the heavy client, sure.
--
You received this message because you are subscribed to the Google Groups "DDD/CQRS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dddcqrs+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Sebastian Good
Ben Kloosterman
Given that SOA guides us away from having a given 3rd normal form entity exist in any one service, it is unlikely that the building blocks of your CQRS design will be those kinds of entities. Most 3rd normal form one-to-many and many-to-many relationships simply do not exist when doing SOA and CQRS properly.
