Event Sourcing, rich events and inter-aggregate madness

[Dawid Ciecierski]

After several hours spent on Google and some A4's later I finally give up and have to reach to you guys for help. This is not a made-up example, it's from a project we're in the process of building. Apologies if there's something horribly wrong with the way we do things - it's our first CQRS+ES project, with all the excitement and hair-pulling that it entails.

The underlying theme is that the system is highly configurable, and it's this flexibility that seems to be the root of our problems. I will focus on just one relationship out of many. There is a Customer in our domain. A Customer has State, and can transition to another state, but only if that transition is allowed. Both States and transitions between them can be changed. We modeled this as:

Customer (aggregate root)

Guid Id
State CurrentState
void ChangeState(State nextState)
void OnCustomerStateChanged(CustomerStateChangedEvent @event)

State (aggregate root)

Guid Id
Set<StateTransition> AllowedTransitions
bool IsAllowed(State nextState)

StateTransition (entity)

State Input
State Output

We thought it would be good to hold a reference to State in Customer rather than StateId because that way Customer can veryfy if it can transition to some requested nextState. This encapsulates some of its logic in the entity and prevents us from dealing with anaemic domain entities.

(So when someone asks the Customer to transition to another state, Customer asks its CurrentState if the requested state IsAllowed. To answer that question, CurrentState queries its AllowedTransitions to see if any has Output of the same Id as the requested nextState. If it is the case, then Customer publishes CustomerStateChangedEvent, and applies it via OnCustomerStateChanged where actual assignment to CurrentState takes place.)

However, holding on to a reference to State in Customer puts additional demand on our domain events. It seems that OnCustomerStateChanged shoud receive State, rather than StateId. What's worse, we might end up storing that State, with its AllowedTransitions (and other properties not shown), in the event store! This does not feel right, as AFAIK it's good practice to keep domain events thin, ie. just ints and strings. There was some discussion on this group on fat events, but the "keep events thin" seems to have prevailed.

Is there any recommended way out? Overall, it seems that domain logic in one aggregate root depends on another aggregate root, but it feels correct from the "avoid anaemic domains" point of view. Perhaps this is a good candidate for a domain service?

Regards,
Dawid

[Marijn Huizendveld]

Hi Dawid,

What is the relationship between Customer and State? Who would be considered the Aggregate Root?

I encountered a similar problem, in my case it turned out that the Aggregate (in your case State) was actually a Value Object and that the Aggregate Root (in your example Customer?) had all the info to construct the its state. 

Is that of any help?

Cheers,

Marijn

[Dawid Ciecierski]

Wow, that was quick! In my case, as I wrote, we considered both State and Customer to be aggregate roots. State exists independently of Customers - admins may change it to allow new transitions, for example. It's like a graph where states are the nodes and state transitions are the arrows.

It therefore seems that State has some concrete identity, a different lifetime to Customer and is not that closely related to Customer. Because of that we considered to be an AR rather than an entity or value object.

Did you have any logic in the aggregate that turned out to be a value object? Where did the related business logic move?

Regards,
Dawid

[Marijn Huizendveld]

In my case it had some logic. That was moved though. I also had a few cases where another Aggregate Root turned out to be not an Aggregate Root and was part of the other Aggregate.

Perhaps you can pass the `State` instance as argument to those methods that need it? 

[ahjohannessen]

I would use double dispatch rather than holding onto the State.

[Dawid Ciecierski]

And so you'd have the following:

public class ChangeCustomerStateCommandHandler : Handles<ChangeCustomerStateCommand>
{
    public void Handle(ChangeCustomerStateCommand command)
    {
        var nextState = _stateRepository.FindById(command.StateId); // injected
        var customer = _customerRepository.FindById(command.CustomerId); // injected
        customer.ChangeStateTo(nextState);
    }
}

public class Customer : BaseAggregateRoot
{
    [...]
    private Guid _stateId;

    public void ChangeStateTo(State nextState)
    {
        if (!nextState.CanBeNextStateFor(this)) throw new SomeDomainException();
        Apply(new CustomerStateChangedEvent(customerId: this.Id, fromStateId: _stateId, toStateId: nextState.Id);
    }

    private void OnCustomerStateChanged(CustomerStateChangedEvent @event)
    {
        this._stateId = @event.ToStateId;
    }
}

Could be a neat trick if I'm applying double dispatch the way you imagined! That way Customer is still responsible for doing the checking, as well as is allowed to refuse a state transition due to its own reasons (eg. being a deleted customer).

Ps. When posting on this group, is it recommended to delete older posts from my message or should the whole thread be included?

[Eric Therond]

Hi Dawid,

The first thing that comes to my mind is that there might be a problem with your aggregate boundaries. They should be defined to ensure consistency within an aggregate. But obviously, here, your Customer aggregate alone isn't able to ensure anything, it has to rely on the State to check that the requested state transition is valid.

But at the same time, I understand that States have their own life cycles, can be defined by admins, etc.. so yeah, they're probably Aggregates in their own right.

I can think of two solutions here :

- Make Aggregate boundaries clearer and go for eventual consistency between aggregates : you give up on strict transactional consistency, Customer won't call State anymore to check if the request transition is valid.. Instead, it might use some internal data to do so, data which might be updated from time to time using information coming asynchronously from State

- Or *much* simpler, go with double dispatch like suggested before in this thread ; that works perfectly well, as long as your Customer and State live on the same machine.

Cheers,
Eric

2012/7/26 Dawid Ciecierski <dawid.ci...@gmail.com>

[ahjohannessen]

Dawid, seems to me that you are on the right track with your code above. Not sure if you have proper boundaries, but there is nothing wrong with AR collaboration using double dispatch :)

I am new to AR/ES too, and my conclusion is that you can take the advice people give here (usually given by bright people). However, don't be afraid of trying things out and *fail* a couple of times before the distillation emerges, and btw don't procrastinate too much because of someone giving you a hard time with some academic model purity masturbation. That is not to say that you shouldn't listen, people here have good intentions and often it is a question of establishing proper frame of reference for your problem.

[Eric Therond]

A big +1 to that piece of advice.

I feel like trying and actually getting into some real issues was the only way that truly helped me get my head around much of DDD/CQRS/ES "principles" or other things described as "good practices".

Eric

2012/7/26 ahjohannessen <ahjoha...@gmail.com>

[Dawid Ciecierski]

I guess part of the confusion stems from the fact that we're using NHibernate and are mixing in ORM concepts with pure domain modelling. By that I mean we're allowed the convenience of having State instead of StateId not because State is what we really have on that entity, but solely because NHibernate does the hard work behind the scenes (slapping that StateId column on Customer and fetching things on the fly with lazy loading and creating proxies).

In other words, we might be used to the convenience of using an ORM, are transposing that convenience onto our (supposedly persistence-agnostic!) domain model, and expecting the domain to feature that convenience too even thou in reality it was just an ORM-incurred delusion / magic when fetching stuff from a database for us.

Re-thinking all that I read in the light of what you both are saying, I came to the following conclusion (please correct me if it's wrong): it is never the case that an entity in one aggregate has a property of type belonging in a different aggregate. If a relationship between an entity in one aggregate ("us") and an aggregate root in another aggregate ("them") exists, it is always recorded as a key. Since we're not supposed to have services injected into us, what follows from these constraints is that we can never examine them (other aggregate root) "at will". They have to always be given to us by the thing trying to use us. Either that or we switch to using eventual consistency.

(In this particular case the business cannot accept eventual consistency, so double dispatch seems to be the reasonable way out.)

A tangential question then (actually two!), if you still have the patience:

1. How is one supposed to check for referrential integrity? Eg. a customer holds a reference to StateId = 5, and admin tries to execute RemoveStateCommand(stateId: 5). Do we ask CustomerRepository in RemoveStateCommandHandler that no Customer holds a reference to that StateId? Sounds good?
2. When doing pure event sourcing, would it be correct to say that if a need arises to ever go back to a certain point in time, we'd need to replay events not just for one aggregate root, but the whole domain to keep those consistent? Same story with a "2011" Customer referring to something that existed in 2011 but does not now.

Perhaphs the problems above consistency problems point to improper boundaries, just as Alex mentioned? But on the other hand I guess it's not possible to have 100% independent and fully consistent AR, otherwise we could never have Users or Customers as AR!

Cheers,
Dawid

Ps. Two new messages arrived while I was writing this post... now that's what I call community! Thank you for your feedback, it helps immensely to hear advice from others rather than go in circles trying to make sense of all that's posted on the interwebs as "good practices".

Ps2. "Academic model purity masturbation" - hit the nail on the head with this one :-) Coding is already taking place, but we got stuck trying to solve the problem described in the original post.

[Eric Therond]

Yep, I agree with your conclusion about aggregates references in other aggregates…
Personally I just don't do this anymore. I find it cleaner - and ultimately simpler - to load explicitly aggregates at the command handling level via repositories, just like in the code you provided ; no magic involved, and you don't end up loading a crazy object graph without knowing it (I also try to avoid relational dbs and ORMs when there's no justification for using them, and now I feel better in my life, but that's another story :)

About the "referential integrity" question : you can either do that, or not do any RemoveStateCommand but rather something like MakeStateUnavailable, which won't delete the state but just mark it  as unavailable for future use. I think you can find a lot of stuff in the ML archives about how to approach "referential integrity" when doing DDD/CQRS.

Eric

2012/7/26 Dawid Ciecierski <dawid.ci...@gmail.com>

I guess part of the confusion stems from the fact that we're using NHibernate and are mixing in ORM concepts with pure domain modelling. By that I mean we're allowed the convenience of having State instead of StateId not because State is what we really have on that entity, but solely because NHibernate does the hard work behind the scenes (slapping that StateId column on Customer and fetching things on the fly with lazy loading and creating proxies).

[Dawid Ciecierski]

Glad you feel better in your life :-) I'm beginning to see the light as well, this whole CQRS+ES+DDD thing is really making me reconsider the way complex software ought to be done and what place an RDBMS has in all this. Awesome to be around when such things are happening.

Thanks for confirming my understanding of what you both were saying. The idea of "DisableStateCommand" sounds good as well, and fits right in with ES principles (try not to delete stuff you might need later... or "earlier"!).

I think I now know how to progress with our project... at least until the next conceptual hurdle :-) Till then! I'm off to read up on referential integrity.

And thanks again for lending a helping hand.

Best regards,
Dawid

[@yreynhout]

The first thing that struck me as odd is that everybody went along with the model you came up with. Could you give some practical examples and motivations why you need this 'highly configurable' design? What is it exactly that you are trying to accomplish with this swiss army knife? Are you in a multi-tenant or multi-customer (your customer, not the one you're modeling) situation, and is you're solution trying to satisfy everybody in a generic way? Am I to understand that someone goes out of their way and describes the customer specific (again not the one you modelled) statemachine using your software? Not judging here, just trying to understand. To be honest I don't like these generic models very much. They tend to hide the real business model (but then again, YMMV).

HTHM,

Yves.

[Dawid Ciecierski]

Sure, will be glad to explain. The project will be offered in a SaaS model as an app to manage prospective customers, and each client most likely will have different workflows (ie. states and related changes) they might need to support. And so when client A (say a telemarketing business) may have New > Telephoned > Prospective > Deal, client B (direct b2b sales) may want their operators to have New > Scheduled appointment > Meeting > Negotiating contract > Deal.

What's more, clients (call them clients to distinguish from the Customer entites) should be able to specify conditions that will constrain when a particular state transtion may take place. Eg. it may be that transtion New > Test may only be allowed if Customer.Email matches a specific regex, or Negotiating contract > Meeting with management can only be navigated by a manager (and not a lowly representative). (I did not talk about these conditions as they seem to be secondary to the original problem of "what referece to X do I keep and how do I access it".)

It's these different state workflows that led us to conceive the model you saw in the first post. There's meant to be other configurable workflows that Customers can participate in, but we hope to build the basics first and see if we need bigger guns (eg. sagas) to support those other workflows.

So yes, we are in a multi-client situation, and hope to satisfy everybody (as many as possible, anyway) in a generic way.

Regards,
Dawid

[Greg Young]

I have had very bad experiences with these types of systems. Generally they end up being badly designed workflow tools (saga frameworks, etc). Often what I want to do is not capable of being done as they have not seen all of people's needs (or if mature is massively complex as the have tried to shoehorn them in). Also it being all driven throu custom uis can have a huge learning curve.

I tend to go a different direction today in using a tool that exists for that part (modeling in ways to help it such as raising events). I then recommend making some of the most common things available out of the box and offering either guidance and/or services in changing the workflows in that tool. For some very simple cases I might even provide 4-5 different implementations that cover 80-90% of scenarios but leave it open for change in this way instead of trying to bake it into the software I distribute.

Greg

--
Le doute n'est pas une condition agréable, mais la certitude est absurde.

[Dawid Ciecierski]

I understand where you're coming from, and do realise this is not the ideal situation. Baking in 80-90% of common scenarios would be fine in most other projects we do, but in this case the business sees customizability as something that will give them a key edge over their competitors. It's a pretty specific business niche they're trying to fill in. Drawing from experience from a simple, less configurable version version of just such a system in production, and having gathered some potential customer feedback, it seems they're going in the right direction.

All in all, I am hoping that DDD can help us achieve what the business wants without making too much of an abomination. The current version is more of a CRUD app that's grown way beyond maintainability from a simple data/ORM-centric system to pretty much a big ball of mud. Hence, wanting to avoid mistakes of the past, we're approaching this much more complex system with a different methodology.

Thanks for sharing your experiences thou.
Dawid

[@yreynhout]

+1 on doing the workflow in a tool or at least decoupled/external.

[Dawid Ciecierski]

So you're suggesting that we shouldn't keep a reference to State in Customer at all, rather have something external keep track of State definitions, transitions, and what state Customers are in? And so when needing to change state to something else, I would be asking that external system to do it for me (passing a Customer to that function), rather than use double dispatch on Customer.

Whould that work with state transitions being dependent on Customer properties? I think I used the example of an email matching a particular pattern, or currently logged in user being the same as the one assigned to take care of the particular Customer.

The more I think about it the more it sounds like a compelling idea possibly simplifying the system as a whole in terms of responsibilities. But if both domains are inherently dependent on each other, isn't it a little artificial to move them to separate systems?

Regards,
Dawid

[Dawid Ciecierski]

(Just in case posterity comes across this thread, I just found an older one on this group that deals with where I came from: taking what ORMs accustomed me to doing and trying to teach DDD systems the same tricks - essentially breaking all consistency boundaries DDD has to offer. Well worth reading.)

[belitre]

Yes, I think they're seggesting you to isolate "workflow concerns" in a Generic Subdomain. So Customer properties that affect the workflow (state changes) have to be provided to the Generic Subdomain. But they can be abstracted, so the Generic Subdomain just know about ICondition, IState, IRule, IProjection, ITrigger, IActivity, whatever... 

That Generic Subdomain could be as simple as providing the minimum workflow semantics you need, can be an existing tool as suggested, or could be a custom one using some known paradigm (EPCs, BPMN, Petri Nets, ACM, State Machines, etc.)... This is working for us. And with Event Sourcing this can be achieved in really interesting ways using Events and Functions over Events.

[@yreynhout]

Customer.Register(..., email, ...); => CustomerRegistered { CustomerId = <insert guid here>, Email = "bl...@company.com" }

Customer.AssignCareTaker(CareTaker careTaker); => CareTakerAssignedToCustomer { CustomerId = <insert guid here>, CareTakerId = <insert guid here> }

How hard would it be for something external to subscribe itself to those events?

Orthogonal to all this is that you might need to think about a way to steer your UI with the dynamic, external workflow (i.e. which transitions (buttons enabled) am I allowed to make at this point?).

[Dennis Traub]

The first thing that struck me is that we are probably talking about two separate Bonded Contexts here. Customer management on one hand, and workflow modelling on the other hand. The State may be an Aggregate Root in the workflow BC that's being mapped to a mere Value Object as part of the Customer aggregate in the Customer BC.

[Dawid Ciecierski]

Apologies for staying quiet for the past few days - it's only because I was furiously coding your suggestions in. I went down the separate BC + subsystems path as many of you were kind to suggest. Particular bits communicate via a MassTransit message bus using shared message interfaces (like IEvent, ICommand, ICommandHandler, IQueryResponse, etc). So far I have to say this is working fine, and - perhaps even more importantly - leaves me a lot more confident in the robustness of the overall solution.

Just today we were having to plug in another module that will be responsible for managing and actually sending emails (either in batches via an external partner or as single messages depending on load) and it was incredible to be able to add a new project, add a reference to Domain.Interface and start communicating with others! So much better than the old "hmm where do I add this thing and is the likelyhood of all the other bits breaking high or very high..?" way.

So, taking my hat off to you guys for honest and wise advice, it so far it's served us very well ideed. Our only concern now it throughput - most communication is done via messaging (query side also, except queries are request/reply as opposed to fire-and-forget), and so we're going to consciously monitor page render times as we have more test data in.

Will get back to you once we have more bits of the system in place and working!

Best regards,
Dawid