Domain services in aggregates which query/lookup and try to enforce invariants.

[Tomas Roos]

I see more and more people who use domain services within their aggregates to lookup for instance if a username is already registered or not.

But this doesn't really make sense to me.

I would say that this really weakens the enforced invariant. Or am i missing something? 

The view will be eventually consistent against the events which gets dispatched.

I'm making a very simple example here.

   public class ExampleAggregate

   {
        public void AddUser(IUserIndexService index, string login, string password)
        {
            if (index.IsLoginRegistered(login))
                throw ...

            Apply(new UserAdded(Guid.NewGuid(), login, password));
        }

        //VERSUS
        
         List<string> usernames = new List<string>();

        public void AddUser(IUserIndexService index, string login, string password)
        {
            if (usernames.Contains(login))
                throw ...

            Apply(new UserAdded(Guid.NewGuid(), login, password));
        }
    }
}

Whats your opinions on this? 
Is it a way to get out of modeling of real aggregates?

Tomas

[Colin Yates]

One problem with the stateful List<String> approach is memory requirements.  The domain service would (hopefully!) delegate to the persistent store to do the range query.  If you hydrate aggregates on demand as well then every time you need to access that aggregate you have to replay every single one of those events.

One question is why is the uniqueness a consistency rule?  A far more scalable approach is to allow collisions and compensate using an out-of-bound compensator (see http://codebetter.com/gregyoung/2010/08/12/eventual-consistency-and-set-validation/).

I too struggle with the problem of how to use eventually consistent reads as part of the command handler but every problem use case I can think of usually comes down to my consistency boundaries not being correct and two aggregates actually elipsing a third aggregate.  On a related note - if a set of something is used in a command handler (i.e. uniqueness) then I think access to, and updating of that set must be part of the command handler (i.e. serialised and versioned, with complete disregard to the one aggregate update per transaction).  In your username uniqueness, it cannot be eventually consistent otherwise there is a window of time (the time it takes for it to become consistent) where multiple instances of the same username will be incorrectly accepted as unique.

<still-newbie-disclaimer/>

[Bennie Kloosteman]

In my case its read model .. for 2 reasons

1) There are a lot of tools which do everything i need so why reinvent
the wheel .
2) Its a poor fit for CQRS . I dont really call logged in or not
business logic ,its just infrastructure.

Now if i was building a library / service which just did authentication ...

Ben

[Martijn van den Broek]

Using data outside of the aggregate's consistency boundary for its invariants is not a good idea, since consistency cannot be guaranteed. It could be a hint of wrong aggregate boundaries. 

Your example shows a very common set invariant (uniqueness): http://codebetter.com/gregyoung/2010/08/12/eventual-consistency-and-set-validation/

Op donderdag 10 januari 2013 10:16:10 UTC+1 schreef Tomas Roos het volgende:

[Tomas Roos]

On Thursday, January 10, 2013 10:43:53 AM UTC+1, Colin Yates wrote:

One problem with the stateful List<String> approach is memory requirements.  The domain service would (hopefully!) delegate to the persistent store to do the range query.  If you hydrate aggregates on demand as well then every time you need to access that aggregate you have to replay every single one of those events.

I don't really buy this argument since we can use snapshots + replay only remaining events.

 

One question is why is the uniqueness a consistency rule?  A far more scalable approach is to allow collisions and compensate using an out-of-bound compensator (see http://codebetter.com/gregyoung/2010/08/12/eventual-consistency-and-set-validation/).

If I can avoid it I really think that introducing inconsistent data into a system is a win. But it may just be preconceptions.

[Tomas Roos]

On Thursday, January 10, 2013 11:29:48 AM UTC+1, Bennie Kloosteman wrote:

In my case its read model .. for 2 reasons

1) There are a lot of tools which do everything i need so why reinvent
the wheel .
2) Its a poor fit for CQRS . I dont really call logged in or not
business logic  ,its just infrastructure.

Whats a poor fit for CQRS, aggregates?

[Tomas Roos]

Word. Thought this was exactly what I wrote in the topic :) But I do agree.

[Colin Yates]

Snapshotting means you don't have to replay the events, but my argument is more about the memory requirements of storing the data in the aggregate.

Sure, we are only talking Strings, but if you have 1 million unique users then either you have 1 million strings sitting in memory constantly (assuming a memory image) or every access to that aggregate requires loading 1 million usernames (snapshotted or not).

[Colin Yates]

I think you did too, I just wanted to untangle it a bit :)

[Vytautas Mackonis]

I think the List<string> flaw is more about throughput and less about memory. It takes a lot of users for the array of strings to be that big to cause memory issues. However, if all users live inside one aggregate, you will lock all of them each time you make a change to any single one. Theoretically you can have all your system inside one aggregate, in practice you want to make the aggregates as small as possible to maximize throughput.

[Tomas Roos]

I would say that the creation is done through lets name it ManagerAR.

Which sends out events which then creates a aggregate UserAR which each of the users can have for their sake and invoke commands on without locking or having concurrency conflicts with other users.

But if they need to change for instance the username again, they would need to send that command to the ManagerAR.

So you have eventual consistency between AR's.

Tomas

[Dennis Traub]

On Thu, Jan 10, 2013 at 12:42 PM, Tomas Roos <ptoma...@gmail.com> wrote:

On Thursday, January 10, 2013 10:43:53 AM UTC+1, Colin Yates wrote:

 

One question is why is the uniqueness a consistency rule?  A far more scalable approach is to allow collisions and compensate using an out-of-bound compensator (see http://codebetter.com/gregyoung/2010/08/12/eventual-consistency-and-set-validation/).

If I can avoid it I really think that introducing inconsistent data into a system is a win. But it may just be preconceptions.

Tomas,

Please remember that "eventually consistent" is by no means synonymous to "inconsistent". And introducing eventual consistency is not a bad thing. Instead it offers great benefits in many cases because it helps get away from tight coupling.

Dennis

[Mervin Yan]

What about to inject the IUserIndexService into the CommandHandler...

[Vytautas Mackonis]

On Thu, Jan 10, 2013 at 2:48 PM, Tomas Roos <ptoma...@gmail.com> wrote:

I would say that the creation is done through lets name it ManagerAR.

Which sends out events which then creates a aggregate UserAR which each of the users can have for their sake and invoke commands on without locking or having concurrency conflicts with other users.

But if they need to change for instance the username again, they would need to send that command to the ManagerAR.

So you have eventual consistency between AR's.

By doing that you only replace one kind of eventual consistency with another kind. In the case of a service that queries a read model you don't know whether the username is unique but you can tell for sure whether the user with id X exists or not. In your proposed case you can ensure the username is unique but what if you get a DoSomethingWithUser(userId) command right before the other aggregate is notified of user creation?

Also, the complexity of the system that does what you propose is higher. Why do that extra work when you will (probably) never have a duplicate username created during that small window? Unless, of course, this is a real business invariant and we need to enforce it.

[Nuno Lopes]

Hi,

Using data outside an Aggregates consistency boundary over which it checks internal invariants its quite common. It does not matter if the input is supplied in args or supplied by a service that it calls internally. They are input.

It can only protect its invariants by rejecting invalid input!

What you have coded a that given two inputs,  a list of logins and a login, it will reject a a login that is already in the list. The aggregate as no control over the given list.

It does not seam to be what you want.

What you want is to protect an invariant over this list not over an element.

Cheers,

Nuno

[Tomas Roos]

But if we accept a new user which had the same username even if it is for 1 minute before a cleanup / compensator process is running. We are inconsistent for a while.

Agree?

[Tomas Roos]

I would exp back off and try again to parse that command.

The other one yes can be argumented. That it probably won't happen in 99% of the time.

But then I still need to have a backup plan when it happens. :)

I just feel that using lookup services isn't really enforcing true invariants. It's a bit looser enforcing. Not as good. But works in most times.

[Kasey Speakman]

Architecture is about making the best choices for the situation. In this case if uniqueness is a requirement, I would use a separate service, because set validation is just not in the CQRS wheel house. If the application is not distributed, the "service" can just be an object (which uses locking around a HashSet) on the command handler.

But in many cases, this level of consistency is just not needed. If we're not talking about social media, but companies (based on the examples given), I wouldn't even worry about it. Assuming your UI checks a read model for uniqueness before sending the command, what is the likelihood that 2 managers are in-processing 2 people with the same name (assuming username is based on that) at the same precise moment in time? Even in a large company, that chance is low. If it did happen, the cost of support scratching their heads for an hour over it once in a while is relatively low compared to time spent programming/maintaining/optimizing a uniqueness guarantee.

[Tomas Roos]

So what you're saying is to be able to be consistent in the decision in all times we need to have the list internally in the aggregate.

Because the other is just an input as any argument which can be validated but not maybe always consistent?

[Kasey Speakman]

What makes one invariant truer than another? Is it truer when the aggregate controls all the code vs. when it delegates some specialized work to an external service?

[Tomas Roos]

Right, it was a bad word choice :)

What i'm referring to is the level of consistency i guess.

If its being persisted with the aggregate it would say that it is a consistent boundary. If though it is a lookup to a view which is eventually consisten from the actuall decision then it won't be *fully consistent* if there is a delay before the view is updated and again i agree that the likelyhood is very low probably.

All, I hope i dont sound to stubborn really appreciate the feedback. I'm here for a opinion not to prove i'm right.

Tomas

[Kasey Speakman]

No, not in the aggregate. The aggregate is loaded from state, and there are many instances of the same aggregate type, so that could potentially load many many copies of the list, which are out of sync with each other. I said to put the list on the handler, which I presume is instantiated only once (non-distributed app). You pass the list (aka username service) in to the aggregate via method call when the message is handled. The list then becomes the single point of uniqueness checking. Just make sure you're using locks when you access the list.

When your handler is instantiated, it loads the list from the read model. Then when a new user is created successfully, the new username is in memory. Eventually the denormalizers will update this read model so it is consistent with what is in memory.

At least, that's what I would do if my requirement was no username collisions ever and my app was not distributed.

[Vytautas Mackonis]

As Nuno mentioned, the aggregate itself will still be consistent because the user list is not part of it. However, the user list may get a duplicate user as a result. Now the real question is, how bad it is for the business to have a tiny risk of a duplicate appearing. If it is a show-stopper, you should make the user list fully consistent, as this is a true invariant. If the business impact is high, but they can still live with it, you may consider automating the recovery (like letting the user choose a new username upon next login). But in most cases its best just to let the support handle it *if* it ever happens.

[Tomas Roos]

Login i the code was more of the username. It was a bad word choice. 

I Agree that authentication is not done through an aggregate.

On Thursday, January 10, 2013 11:29:48 AM UTC+1, Bennie Kloosteman wrote:

[@yreynhout]

Hi Tomas,

I'll be honest, bringing up "user management" was not your best move. Some "flameworthy" thoughts popped up in my head. Luckily, I managed to suppress most of them ;-) #angermanagement #payingoff

The bad news is ... you are ill. The disease you suffer from is called "relational cataract", for years inflicted upon us by these "RDBMS vendors", causing us to think the world is as immediately consistent as their "database". It's not. Life as we know it on earth is async and - if this economy thing ever takes a turn for the better - eventually consistent.

I'm sorry I had to be the one to break it to you.

Get well soon,

Yves.

[Tomas Roos]

Well I really thought that consistency in a AR was crucial, since its often talk around the consistency boundary of an AR. 

I think you understood it a bit wrong actually. 

I have no issue with a view being eventually consistent for a read.

But from the decision making of a invariant I've never been thinking in the terms that its "good enough"

Cheers! :)

Tomas

[Colin Yates]

From my view "on the fence" I think the question being raised is; is the set based constraint a real business constraint or one assumed by us people "sickened" :) by too many years working with relational databases where set based consistency can be as simple as a unique index.

What would the business do if there were two users with the same username?  Would the cost to the business to resolve that be more or less than the cost to the business for you writing a solution to enforce consistency (with all the performance trade off that entails).

[João Bragança]

It's a question of *how* consistent and in what context vs cost to implement. It is far cheaper  / performant to have a specialized database that enforces set uniqueness than to try and do it on the command side.

[@yreynhout]

Oh, but I understood you quite well :-)

Simple advice? Don't roll your own user management using CQRS/ES/DDD unless you are in the business of user management. Make it a separate bounded context, take something off the shelve, ... I'll bet it's cheaper. Your prescious time is best invested elsewhere, no?

As for the balance of how much code goes into a domain service vs the aggregates it calls, yes, that is a very good topic to discuss (on a separate thread). We, as a community, definitely need to "share" more practical examples and model in various ways to get a feel for the trade-offs of each approach taken, so others can hopefully learn from them and advance quicker.

Regards,

Yves.

[Colin Yates]

+1 for sharing!

[Tomas Roos]

ExampleAggregate..... ;-)

Need to take concepts next time no one can relate to!

Anyway we can "close" this post now.

Thanks for the feedback folks!

[Johanna Belanger]

Wait, I have something to say! =)

Even though user management is probably not worth your precious time, there may be other set invariants which are. In my domain, for instance, we have meaningful product codes which must be unique. But I'll continue using the user management example. Sorry. =)P

The first thing to recognize is that user name uniqueness is NOT an invariant of the user aggregate. It is an invariant of the set of user names and must be enforced by the implementation of that set. (As Nuno pointed out.)

Let's look again at Tomas' example of a service injected into the aggregate:

public class ExampleAggregate

   {

        public void AddUser(IUserIndexService index, string login, string password)

        {

            if (index.IsLoginRegistered(login))

                throw ...

            Apply(new UserAdded(Guid.NewGuid(), login, password));

        }

   }

This solution does not guarantee uniqueness unless you somehow lock your service until your UserAdded event is processed by the service and the set is updated.

What if we try this instead:

public class ExampleAggregate

   {

       public void AddUser(IUserIndexService index, string login, string password)

       {

           try

  {

              index.RegisterLogin(login);

           }

  catch

           {

     throw ...

           }

       }

   }

In this case, to enforce the invariant, the service must either be on the same partition as the aggregate or must be called inside a distributed transaction. With this design, the set is the real aggregate root and all User aggregates are restricted to the same partition as the set. Yuk.

So I would agree with Tomas', injecting a service is not a very clean solution, given there is another solution that is just as simple. Here it is.

UserNameIndexAggregate (Tomas' has an excellent point!)

This aggregate does not have to be implemented as an event-sourced in-memory set!!!!!!! =)

For example, it could be a relational table with a facade that receives commands and emits events.  Simple.

Given that we have made that decision, now we need to find out at what point the user aggregate needs to know whether the user name request succeeded or failed.

The most likely case, it doesn't need to know right away. So we create the user aggregate, temporarily assign it the user name which may or may not be unique, and carry on. Later if we receive a UserNameRequestFailed or DuplicateUserNameDetected event, then we require the user to change their name. Or support gets notified of the failure event and they fix it. Whatever the business wants to pay for.

If it must have its user name request accepted before anything else can be done with it, then we must wait to create the User aggregate until we get that UserNameAssigned event. If that's a real business requirement, then the "We'll let you know when your user has been created" screen won't bother them. =)

Or, if the business requires it,we can use the Reservation pattern.

Thoughts?

Johanna

[Kasey Speakman]

The UserNameIndexAggregate actually sounds a lot like a domain service with DTC, with the relational table being your coordination method, and just dressed up as an aggregate. Also, it seems to me that the aggregate plus the process manager for DuplicateUserNameDetected or UserNameAssigned is actually more complicated than just doing a service.

[Tomas Roos]

Thanks for spreading some light and refreshing thoughts! :-) 

Interesting to see that compensating would be you're preferred way.

There is hope after all.

[Johanna Belanger]

Hi Kasey,

There is no distributed transaction. Immediate consistency is only maintained within the set of user names, not with the user aggregates. The user name in the aggregate is "eventually unique". The reason to call it an aggregate is that it is a consistency boundary and it is responsible for maintaining an invariant.

If you make an immediately consistent call to a domain service from your aggregate, you have to do 1 of 3 things:

1) Your domain service must be on the same partition as all of your User aggregates OR

2) You make an RPC style call to your domain service, which will affect availability OR

3) You use a distributed transaction

How is your domain service implemented? A relational table? The aggregate implementation will probably have the same complexity as the domain service implementation. You have complete flexibility in the aggregate implementation as long as it maintains the invariant and the set consistency. So the only added complexity is basically an event handler which does whatever the business wants done when there is a collision (sends an email, updates the User aggregate, updates a read model). If you know you'll never need the partitionability/availability, then fine. But if you do, it's a pretty small price to pay.

Regards,

Johanna

[Kasey Speakman]

I probably didn't understand exactly what you were proposing. From "it could be a relational table with a facade that receives commands and emits events", I was assuming the "aggregate" would use a relational table for set validation... i.e. try an INSERT, it failed, so name reservation failed. Produce event to that effect, registration process issues compensating command (e.g. send an email notifying someone to fix it). If registration passes, process manager sends CreateUser command.

The exact same process could be done using a service in one very simple class. Then the CreateUser command could be issued directly to the user aggregate first and pass or fail based on the result of the service. And you wouldn't need the process manager, the extra messages, nor the set aggregate. Also the set aggregate would be a bit different than your normal aggregate being that it is essentially a singleton (in its partition). I wonder if there would be an architectural cost to accomodate its different nature? But it's functionally doing the same thing as the domain service I described.

That is if I understood your design right.

[Johanna Belanger]

I guess the core of the issue for me is that the set of user names should be modeled as an aggregate because it has an invariant (uniqueness) that it needs to maintain.

If you value consistency and availability over partitionablity, call your UserNameIndex directly from your user aggregate.

If you value consistency and partitionablity over availability, call your UserNameIndex via RPC.

If you value partitionability and availability over consistency, communicate between your aggregates with commands, events, and handlers.

[Johanna Belanger]

Yes, you understood me, with the exception that the CreateUser command only needs to be delayed if the business doesn't want to do compensation. 

You are right, depending on your CAP priorities, the commands/events/event handler may or may not be the right solution. See my last post.

Either way, IMHO it is an aggregate because it is a consistency boundary, maintains an invariant, and manipulates state. Maybe we could also correctly call it a service, the definition of that word is not precise! =)

I have wondered about your singleton point as well, but I haven't come to any conclusions. Hmm.

Interesting discussion, thanks!

[Kasey Speakman]

I don't understand how the aggregate calling INSERT on a table is any less partitioned than another class that is instantiated per partition and calls the same INSERT command. e.g.

public class UserNameService : IUserNameService

{

    public bool TryReserveName(string username)

    {

        bool wasReserved = false;

        try

        {

            InsertUserName(username); // run INSERT statement

            wasReserved = true;

        }

        catch (Exception ex)

        {

            // maybe verify duplicate key exception

        }

        return wasReserved;

    }

}

public class UserHandler : Handles<CreateUser>

{

    // can be in as many partitions as you want

    readonly IUserNameService userNameService = new UserNameService();

    public void Handle(CreateUser message)

    {

        ... // load aggregate

        userAggregate.CreateUser(userNameService, userName, ...);

        ... // save events

    }

}

public class UserAggregate

{

    public void CreateUser(IUserNameService userNameService, string userName, ...)

    {

        if (!userNameService.TryReserveName(userName))

            thrown new DomainException("This username is already taken. Please choose another.");

        ...

        Apply(new UserCreated(userName, ...));

    }

}

There's the complete implementation with the service. It basically has all the same consistency/availability/partitionability properties as what I understand your aggregate would. But it's less code.

Ultimately, I might still choose to implement a UserRegistrationProcessManager if the requirements got complex enough. But it depends on how much value you'll get out of the extra code.

[Kasey Speakman]

I agree, it's been an enlightening discussion. Your point about bringing the registration into a process manager is well taken, and it might be just what the business needs for the long term. I was just explaining how I would do it, not needing the business value. :)

[@yreynhout]

I see two sides to this debate (and believe me it pops up every now and then): one is about what technical ways there are to provide uniqueness across a set of things that need to be *gasp* unique and the other, more fundamental, is the business angle: why it needs to be unique, under what conditions that uniqueness might be violated, if something could be done to prevent it from a flow perspective, what the risk is if it is violated, how much it's going to cost, what the cheaper solution is, etc ... That second part is very 'your context' specific, hence why in discussions like these, we go for lowest common denominator, something we all know, something technical.

Just a reflection.

[Kasey Speakman]

And I have a technical note. :) One problem with the code I posted is that ordering matters, in that: Calling TryReserveName better be the last thing you do. If you do it first, but then the CreateUser command fails on some other invariant, the name is still reserved and can't be used by subsequent attempts to register the same user. I guess you could implement an Unregister method for those cases. But the point is that the arms-length property of the process manager has definite advantages against service approach. It's all about the trade-offs.

[Johanna Belanger]

Ah I see what I am missing! Your are saying that since in my example I was using a relational table anyway, I could use the database's DTC to get partitionability more simply. Right you are. I chose a relational table because relational is good at memory-efficient set manipulation, and ignored the rest of its strengths.

The availability of your User aggregate is still equal to the availability of your database server, so you can either use technical strategies to increase the availability of your database or you can trade consistency for it.

[Johanna Belanger]

Agreed, it always helps to be reminded not to dive for technical power/complexity before you find out if it's really necessary. However, sometimes the business really does need the uniqueness, and for me it's beneficial that the discussion cover the technical options and tradeoffs too. I know I have blind spots on the technical side as well as the business side.

[Dennis Traub]

1. Why does everybody think that usernames must be unique? That may have been a technical constraint some 50 years ago. But I have yet to see a case where this is an actual business constraint.

2. What is a "Process Manager"?

3. I agree that the user list itself might be the aggregate here.

4. but we don't need to call the class "UserIndexAggregate". This term probably isn't part of the UL. It might be something around the lines of "RegisteredUsers". There's no need to suffix an aggregate's class name with "-Aggregate", just like nobody calls every class CustomerClass, ProductClass or OrderHistoryViewModelClass. Aggregate is a concept.

5. in the described use case, if we have RegisteredUsers as an explicit concept do we even need a separate user aggregate?

[Johanna Belanger]

Good point.

[Kasey Speakman]

Dennis, I wouldn't read too much into the actual domain used as an example. It was for illustration purposes. The real topic being set constraints. Really, for a company, your username is probably based on your real name. And most internet sites use your email address which is already unique. :)

Process Manager

4. I know. It was probably named that way for illustration, and I just tried to stay consistent with the original example given.

5. Maybe not?

[Johanna Belanger]

1) Yeah, you are right, we really need to ditch the user name example. May I propose this one from my domain: Our product codes are calculated from a subset of the product's properties. However, it is somewhat common for 2 products to have identical calculated codes. When this happens, the final character of the code is a sequential number that differentiates one product from the other. The product codes must be unique before the product is released into the wild. If 2 products are released with the same code, there is currently no business mechanism for detecting or correcting this. Any time after the product is released, products are sent out based on this code and data is connected to this code. The current solution (which I am in the process of replacing) uses a relational database table to enforce the uniqueness constraint. Before we had a database, the product designer would manually check to ensure no other product had this code.

2) Also known as Saga (incorrectly), Workflow, Task, etc.

4) Yeah I did that by accident. I agree classes should NEVER be suffixed with -Aggregate.

5) Maybe, maybe not. Depends on the behavior. =)

[Tom Janssens]

Think about it in the following way: how would we do it with only paper forms:

A potential member sends an application form

A librarian verifies the member does not exist yet in his cabinet, refuses the application form if it does.

If not, the librarian puts the application form in his cabinet

If we have multiple libraries and you can only register in at most one library, there are 2 options:

- send a copy of the application to all the other libraries, and wait for confirmation from everyone that the member does not exist yet in their cabinet (i.e. a command addressed at an AR type, not an instance)

- have a single "master cabinet" somewhere, that contains all the application forms (i.e. an indexing service)

This brings in the notion that we make a distinction between a potential member and a verified member... Maybe they have different rights etc, maybe they do not....

This approach is about making things explicit (i.e. the fact whether the applicant is verified or not).

Once you "get" CQRS, you usually understand that consistency always happens in a context, consistency does not exist on it's own. I like to refer to Einstein's relativity: time and place can not exist separately, and neither can consistency and context. (that context can be an AR, a database, an index, a transaction...).

This is also the reason that showing real-time data is a contradictio in terminus, because you switch your consistency context from a database to a screen for example, and that conversion takes time (even if it is only a few milliseconds).

Op donderdag 10 januari 2013 22:18:24 UTC+1 schreef Tomas Roos het volgende:

[Johanna Belanger]

"Once you "get" CQRS, you usually understand that consistency always happens in a context, consistency does not exist on it's own. I like to refer to Einstein's relativity: time and place can not exist separately, and neither can consistency and context. (that context can be an AR, a database, an index, a transaction...)."

Yes, and at some level this concept is always expressed within the business process. Once we find the consistency boundaries in the business process, we are freed to choose an implementation that supports it, rather than struggling to maintain an illusion of universal consistency.

[Henrik Feldt]

Agree with Johanna at large and with Tomas; one could just do it with an aggregate that checks set membership.

[C] TryRegisterUsername -> [A] UsernameIndex -> [E] UsernameRegistered

[R] UserCreator -> [C] CreateUserWithName -> [E] UserCreated

C = Command

A = Aggregate

E = Event

R = Reactor

In UsernameIndex, we keep check for set existence. The event UsernameRegistered has the usual Version field that is consistent-checked in the event store.

No race conditions, invariants around the username index. 1M events is not that much. You don't need to keep a lock around the index aggregate at all; just let the event store handle the consistency. Do some snapshotting.

CreateUserWithName can never fail because we've already checked the username uniqueness and have crossed that checkpoint.

***

Then it's requirements' time - do we need multiple availability zones to function independently? Well, in that case we could make it a requirement that W=ROUNDUP(N/2) on the event store in general. A network partition means that one of the two clusters (the one with the lesser number of nodes) would serve as read-only until connectivity is restored.

***

What about having a requirement for deletion? In that case we need to keep a negative set of deleted usernames, each UsernameDeleted having a version (clock) so that we can still register new users with that same username in the future, but this requirement would warrant even further analysis. For example, it could maybe be possible to model with Conflict-free Replicated Data Types' Set-construct.

[Tomas Roos]

+1

[Nuno Lopes]

Hi,

> So what you're saying is to be able to be consistent in the decision in all times we need to have the list internally in the aggregate.
>
> Because the other is just an input as any argument which can be validated but not maybe always consistent?

Consistent with what? Input is input. For instance, you tell me your name I don't know if it actually your name, I just trust that it actually is. The same thing goes with you age. In fact this is not really about consistency, but about Soundness.

Let me focus in your example. Your challenge is not really about CQRS but about using proper OO analysis and design.

Analysis is about understanding the domain for proper automation, design in this case is about "inventing" the algorithm for the automation. One cannot do the later properly without doing the first properly!

Inline with your example, I suppose you may want to automate user credentials registration and authentication. If not, let's just stay with this business context for a moment.

During the analysis we came to find out that a UserCredential is

1) composed by a username and password.
2) username s unique across all users
3) the user may want to change the password and username.
4) the credential might be suspended
5) Authenticate
…..

Quite often I see people go on and design a user a domain object:

UserCredentials
Give(userid, username, password)
ModifyName(newUserName)
ModifyPassword(newUserPassword)
Suspend()
Authenticate(userPassword)
……

Then we put a unique constraint on Username and so on and so on. We find out that in our system there could be potentially millions of users and huston we have a problem since any of of these operation will be serialised. That is, they will block each other. So then we go on and make the system eventually consistent, cool word yeah? Since we want to authenticate the user as fast as possible we start publishing credential and let authentication occour outside the Aggregates boundaries and so on. I'm, not against this, but more on that later.

An aggregate is a cluster of entities where on is an Aggregate Root. The aggregate root protects the invariants across all objects in the cluster. What this means is that protects invariants across all activities and use cases performed by the cluster, all methods!!!!!

Within this we start looking for invariants that cross cut all activities listed above. Let's take the invariant, "usernames need to be unique across all user credential". You ask the question why? That is what you should do. The domain expert will tell that UserCredentials is a serious business. The username identifies the a person against the system. Anyone that see the username will see YOU!. It's like a photograph. It's like your name in your passport, your photography. So when you present the passport, the agent looks at the photo and sees you there is great chances that is actually you. They need to be unique.

But at which point such this is guaranteed? "Well when we give a new credential to the user" he might say. We ask the user to tell us what is the user name he wants to be identified with and he give us that. We then verify if is unique. If it is we can give him/she we can emit a new Credential. Same as a password, or Identity Card.

What about authentication? Authentication is the system authenticating that your what you say you are, that why we need that password. It's your digital signature. When a user comes in we ask for his credentials and we then cross that data with or internal systems. If all is checks we give him Clearance to work with the system. Same as getting into a Country.

Ok ok. we got that. What about Suspend. Well Suspend …. well you got the picture.

So we know that the unique username is an invariant to be enforced for UsernameCredentialsIssued. So we get a new object.

NameCredentialRegistry
IssueNameCredential(credentialid, username) -> NameCredentialIssued(credentialid, username).
ReissueNameCredential(credentialid, oldUserName, newUsername) ->NameCredentialReissued(credentialid, oldUserName, newUserName).
LookUpUserNameCredential(username) : credentiaId;

This Aggregate enforces the invariant.

So we apply the same heuristic with the rest given the invariants and how they intercept the business activities that we are automating.

PasswordCredential
Issue(credentialId, password) - PasswordCredentialIssued)=
Reissue(oldPassword, newUserPassword) - PasswordCredentialReissued(credentialid, oldpasswordhash, newpasswordhash)

UserCredentials
Register(userid, credentialId)
Authenticate(username, password) -> UserCrendentialsAuthenticated(credentialid…)
Suspend()
Handle(NameCredentialReissued);
Handle(PasswordCredentialReissued);
Handle(NameCredentialIssued);
Handle(PasswordCredentialIssued);

There is nothing eventually consistent here IMHO.

Furthermore, now we have SRP. Each entity performs a single business activity. Say for instance that PasswordCredential is Reissued at the same time the User Credentials are being authenticated.The user will be authenticated either way with both an officially issued username and password. It might not be the newest one, but that does not mean there are contradictions between facts around any of this concepts. Why, because none of them state facts about the same thing! For instance, UserCredentials don't state what is the latest issued password credential. It just states that a user is authenticated against officially issued passwords and user name credentials.

Cheers,

Nuno

[Tomas Roos]

Thanks for the full scope of this example.

It makes a lot of sense.

Tomas

[Kasey Speakman]

Thanks for the example. This seems to be a good model.

[Nuno Lopes]

Thank you. 

But my advice is to follow the invariants tracks along the business activities at hand. And use them to validate if you domain concepts. You we find new ones even though we might not have the proper name for them. Look up in the domain expert dictionary ;)

That is all what is required to start getting some insight. There is no magic.

This does not solve the challenge of having a massive amount username credentials being issued for new UserCredentials. Say the system becomes very popular and it needs to handle say 50000 new users per second.  But provides the structure for it. First new user registrations do not block user authentications and vice versa. Which is good. The same goes for password expirations (leave that to think about).

This can be handled in different ways. You might not have just one Registry aggregate for all of them. For for instance you can have registries per geographical location. Per alphabetical order. and so. You would need a router for commands to issuer new NameCredentials. 

There is a long way to go before one might actually need an eventually consistent data set at the transaction level. For views that is another matter.

You see, this  would only become inconsistent when within the system one has for example two UserCredentials with the same CREDENTIALID (basically the same credential) stating different facts floating around in the system. Say one is Authenticates (UserAthenticated) and the other does not. Depending on the domain, again algorithms, this might be an issue or not. But we are reaching the extreme here IMHO. In those cases where such thing is needed, we can still make it consistent. Model and reengineer business processes in order to allow more the one UserCredential floating around the system per person. Two with distinct credentialids and all.

Cheers,

Nuno

[Johanna Belanger]

Hi Nuno,

>But my advice is to follow the invariants tracks along the business activities at hand.

+1!

Nice analysis of the domain. Good points about the separation of responsibilities preventing unnecessary blocking. There were a few bits I didn't understand:

1)

>NameCredentialRegistry

>   LookUpUserNameCredential(username) : credentiaId; 

How is this method used?

2)

>UserCredentials 
>       Register(userid, credentialId) 

What does this method do? What is userid?

3)

>UserCredentials

>   Authenticate(username, password) -> UserCrendentialsAuthenticated(credentialid…) 

As I understand it, this method authenticates the username and password against the username and password it got from the NameIssued and PasswordIssued events. Is that correct?

I know this model wasn't your main point, but I have a couple of thoughts and I want to make sure I understand your example first.

Thanks,
Johanna

[Ramin]

Not sure I undertand all the fuss. If you need to enforce unique usernames, simply create the Account, then create a username for that Account in the UniqueUserNames Aggregate. Just send the commands one after the other, prefereably the second one only after the first one succeeded. The handler of the second command should query for the AccountID it is supposed to write a username for and reject the command if that ID is not found.

So what can happen?

1) Both commands succeed - every one is happy,

2) First one succeeds, second one fails. So you have an Account without a username. You can either hide it from the UI, or show it as an Account without a username and let the user try to enter the UserName again, or a different one,

3) First command fails - second one won't be sent, or the command handler for it cannot find the AccountID and it will also fail. Try again.

That's it, no other possibilites here. The username exists just inside the UniqueUserNames Aggregate, *not* in the Account (if it were to exist in both, which one is correct?). Simply have an event handler build a view of the combination (Join), so you can show the account together with the username to the user in the UI. The UniqueUserNames Aggregate will hold on to all username-AccountId combinations, but it will only be accessed when adding, removing or chaning usernames. That doesn't happen to often in most domains. To authenticate a username/password entered by a user, you query a denormalized view model of the UniqueUserNames, something like SELECT AccountID FROM Authentication.UniqueUserNames WHERE username=@username. You can then use that AccountID to retrieve the Account and validate the password.

If I missed something, please correct me! By the way, I am not suggesting to wite an authentication domain like this with CQRS/ES, just trying to show an example of how I think it *could* be done.

Here are the Aggregates, the events and further methods (eg. to change a password or username) should be straight forward:

public class Account: AggregateRoot

{

  private string Password; // hold on to it (in a secure way) for authentication. If authentication is done on Read side, Password is not needed here.

  public Account(Guid id, string password)

  {

      ApplyChange(new AccountAdded(id, password));

  }

  

  private void Apply(AccoundAdded @event)

  {

      Password = @event.Password;

  }

}

The Aggregate holding on to the user names is global/singleton. Simply include a UniqueUserNamesAdded { ID = Guid.Empty ) domain event in the event store when installing. The singleton UniqueUserNames Aggregate can then be retrieved by new Repository<UniqueUserNames>.Get(Guid.Empty); No further instances of UniqueUserNames can or needs to be created. There are probably more elegant ways - any ideas?

public class UniqueUserNames: AggregateRoot

{

  private IDictionary<Guid, string> _names = new Dictionary<Guid, string>();

  

  public void AddUserName(Guid id, string name)

  {

    if (_names.ContainsKey(id)) throw new DuplicateAccountIdException(); // this really should not happen, because accounts with unique ids are always created before the name is added...

    if (_names.Values.Contains(name)) throw new DuplicateUserNameException();

    ApplyChange(new UniqueUserNameAdded(id, name)); // (the event is straight forward, no need to show its declaration in this example)

  }

  

  public void RemoveUserName(Guid id) // if an account gets removed, you want to remove the username first if you want to allow for a new account to reuse already used usernames

  {

    ApplyChange(new UniqueUserNameRemoved(id)); // (the event is straight forward, no need to show its declaration in this example)

  }

  

  private void Apply(UniqueUserNameAdded @event)

  {

    _names.Add(@event.AccountId, @event.Username);

  }

  

  private void Apply(UniqueUserNameRemoved @event)

  {

    _names.Remove(@event.AccountId);

  }

}

enjoy

[Nuno Lopes]

The method is a standard method on the Registry Pattern. The method is basically used internally to check if a username is already used. 

The method Register is used, well to create a userCredential. 

Basically UI calls the following commands

IssueUserNameCredential(credentialId, username)

IssuePasswordCredential(credentialId, password)

RegisterUserCredential(credentialId, userid)

The handlers of these commands call the respective methods. CredentialId works as a correlation value between these multiple activities. This of course is reflected on the events, that I like to call facts. We can process everything async

Userid is a third correlation value that you can use as you see fit. For instance to correlate the credential to a business party. Typically in CQRS these are Ids are of type GUID since they are "unique" across tiers and can be easily generated without blocking anything. 

[Kasey Speakman]

It doesn't seem like the UI should have to know to send all these commands to complete registration. The UI should only have to send the RegisterUser command. (Or maybe RequestUserRegistration).

[Nuno Lopes]

You may wrap it up in some service if you will to simplify the API. Don't see the advantage. User registration may require more stuff the issuing credentials.

Notice. The focus is not registration, but issuing user credentials.  

Registration is a tem that is over used. Register this, register that, and may disguise the lack of understanding of the business at stake. 

Now signing up a user, opening a customer account may require more then issuing these kind of credentials. 

[Johanna Belanger]

Alright, that makes sense.

It seems to me that unless you are transmitting and handling your events(facts) synchronously, you do still have eventual consistency. If a user tries to authenticate after their credentials have been issued and after they have registered but before the User aggregate has handled the NameCredentialIssued and PasswordCredentialIssued events, authentication will fail. 

Is that likely to happen or a serious problem in this context? Probably not. It's a tiny bit of eventual consistency, with nice benefits if you can use them.

[Johanna Belanger]

Another technical note, in this style of implementation, User (Aggregate) is coupled to the implementation of UserNameService, even though it calls it through an interface. The coupling is not at the code level, but at the architectural level. There is an implicit assumption that UserNameService is handling consistency,concurrency,partitionability,availability. Introducing events and event handlers breaks this coupling.

[Kasey Speakman]

If you recall, my example was in the context of what I would do if it was low business value. If my business has nothing to do with authentication, then I am already depending on an external service for this. In that case the "architectural coupling" is probably something I can live for the time savings, and it is also now explicit. (Although maybe there are better ways to express the dependency?) As far as the external service's CAP, if it's not in my purview (e.g. OpenID), then I have to assume (or verify that) the external service meets the business's SLAs. Otherwise, obviously I would have to do so myself.

[Johanna Belanger]

>If you recall, my example was in the context of what I would do if it was low business value

Sorry, I didn't quite catch on to that. I thought we were pretending that this was an important context. ;) 

Maybe I am drifting too far from pragmatism and from my level of experience, but there is just something here that bugs me. Let's say I design my aggregate using this injected service style. Down the road, we realize we need to partition by this aggregate type. No problem, it's an aggregate right? It's a consistency boundary by definition. Partition away. 

Wait! No can do. We have to check the internals of our aggregate to make sure it's really an aggregate. We have to notice that it depends on this service. We have to check the CAP properties of this service to make sure we can partition here. 

So was our aggregate really an aggregate? Apparently not.

[Kasey Speakman]

Well, eventually we started pretending it was important.

Assuming you had an external service that you had to use for specialized capability, how would you model the dependency so you didn't have to consider it when changing partitioning? Your scenario assumes you didn't consider it up front. I'm not claiming I have the answer. :) Just wondering.

[Johanna Belanger]

Continuing with our favorite example then:

In the design where we have a User aggregate and a UserNameIndex aggregate, we can partition the Users without considering the UserNameIndex at all. We broke the architectural dependency by accepting eventual consistency between the two and removing the direct communication.

I'll give other types of service dependencies some more thought though...

In my example, I did consider partitioning up front, in that I modeled a User aggregate. Was the service inside of the aggregate? How could we tell? Well, was all access to the service through the aggregate root? No, because the same (singleton) service is accessed through the aggregate roots of all User aggregates, not just one. By modeling a User aggregate, my design was communicating that my consistency boundary lay around a single User and its subordinate objects. And without looking inside the aggregate, I couldn't tell this was a lie. My real consistency boundary lay around the service and all of the User aggregates.

So here are my current thoughts:

-If your business requires immediate consistency between 2 components, model them inside of the same aggregate. Find the root of that aggregate, and make it the gateway for all access to its subordinate components. If you must partition within the aggregate, use an implementation that provides distributed transactions or something. Now your architectural properties are explicit within your aggregate, and unimportant outside of your aggregate.

-If your business does not require immediate consistency between 2 components, model them in separate aggregates. Accept the small bit of extra code this requires.

-If the context you are working in is so unimportant that you can't afford the extra code, don't use the aggregate pattern. 

What do you think?

[Kasey Speakman]

I don't think I expressed my last point well enough.

I also considered partitioning up front when designing access to the service example (by calling SQL and letting it worry about concurrency issues). But your previous reply assumed that partitioning wasn't considered for the service and suddenly was a stopper. If partitioning wasn't on the table when the project started, then it seems reasonable to re-examine external dependencies among other things when partitioning becomes to a priority. You could even model unpartitionable aggregates (probably poor models, but hey some of us need more practice ;) )

I also posed the question: what would you do if you had to depend on an external service? But honestly I can't think of a good example where you couldn't at least model a domain interface to the external dependency (maybe not easily, but doable)... so maybe it's just an academic question. Perhaps services are just quick and dirty. But it might be that's all you have time for. Obviously not me since I write on forums at all hours. ;)

[@yreynhout]

I have yet to find a uniqueness constraint where the underlying reason is not some form of identification, or a way to alleviate confusion, or a lack of process being put in place (leave alone we already know what it is), or "saving" on operational cost - the ostrich way - because we don't want to deal with the real issue (in which case I can only wonder about the commitment to the core domain). You can talk all the tech you want here, entertaining reading, but I'd still put my money on the talk with the domain expert where we work out probability, risk of this particular duplication happening, what's a "good enough" solution, how we're able to deal with this from an operational pov, who knows, maybe we'll uncover a new insight that makes the requirement moot. Rarely is the outcome full consistency.

Different opinions, not conflicting opinions.

[Nuno Lopes]

Hi. 

No I don't. At least not according to definition of consistency. You see, the password credential in the context of an authentication maybe different then the one issued in the sense that it might not be the latest but it is only inconsistent.

This is quite common. For instance the price of a product in a catalogue might be different  then the price of the same product in the context of an order. Say the price in the catalogue changed while handling the order. 

If it is expected by the business process to be the same at all times, in such case we would have an business invariant across them. 

Now the case you pose where a password credential is issued yet the user credential is not able to authenticate the user against it in time may not be a desirable effect in all situations but might be in a few. 

On normal to hard load ones infrastructure is in place the issued password credentials are propagated in in time for such a scenario. it might happen when the system is heavily loaded in which case business needs to decide what is more desirable. Not issue the credential at all, hence probably refusing to register the user at all, this in context of sign up say a customer. Or allowing it, yet probably the user is not able to login immediately as it needs to be autheticated. 

This is a business decision not a technical one. So it's about business requirements. Is up to us not to assume them. 

[Nuno Lopes]

You see, the password credential in the context of an authentication maybe different then the one issued in the sense that it might not be the latest but it is NOT inconsistent.

Writing on the iPhone has its hurdles, adding to my mild dyslexia. 

[Johanna Belanger]

Alright, I give! 

No, seriously, I think I would be ok with calling a service from within an aggregate where the code savings was substantial. For example, if the context in which I was working wasn't already using event-based communication, I probably wouldn't introduce events just to avoid calling a service from an aggregate. I still think it's kind of dirty (I know you've called it quick and dirty too), because the aggregate is only maintaining its consistency boundary by means of magic inside the domain service. But you've convinced me that in practice it probably wouldn't be truly harmful.

But if my context was already using events, I would use a 2nd aggregate. I think it expresses the real consistency boundaries of the domain more clearly, and is more flexible for future change because it is less "architecturally coupled". Those benefits would be worth the extra little classes to me.

Like Yves said, "Different opinions, not conflicting opinions."

>Obviously not me since I write on forums at all hours. ;)

I know, right?! =)

[Johanna Belanger]

>maybe different then the one issued in the sense that it might not be the latest but it is NOT inconsistent.

Right. It is not "inconsistent". It is "eventually consistent". Inconsistent means wrong. Eventually consistent means "may not be the latest."

[Nuno Lopes]

Hi,

Right. It is not "inconsistent". It is "eventually consistent". Inconsistent means wrong. Eventually consistent means "may not be the latest."

No. Mathematically no! In mathematics there is no right or wrong. Either it works or it does not.

You need to understand that algorithmically PasswordCredential and UserCredential are different different entities, different object types. They do not represent the same thing so they cannot be consistent or inconsistent logically! UserCredential and PasswordCredential can have different values as they represent different things in process.

It is true that there is a relationship of cause and effect between PasswordCredentialIssued and UserCredentialIssued but one does not necessarily depend on the other algorithmically.

UserCredentials 
        Register(userid, credentialId) 

        Authenticate(username, password) -> UserCrendentialsAuthenticated(credentialid…) 

        Suspend() 
        Handle(NameCredentialReissued); 
        Handle(PasswordCredentialReissued); 
        Handle(NameCredentialIssued); 
        Handle(PasswordCredentialIssued); 

For instance you could  do this:

NameCredentialRegistry 

        IssueNameCredential(nameCredentialId, userCredentialId, username) -> NameCredentialIssued(nameCredentialId, userCredentialId, username). 

        ReissueNameCredential(nameCredentialId, oldUserName, newUsername) ->NameCredentialReissued(nameCredentialId, userNameCredentialId, username. 

        LookUpUserNameCredential(username) : credentiaId; 

UserCredentials 

        Issue(userCredentialId, userid, username, password)  -> UserCredentialIssued(userCredentialId, userid, username, password)

ReIssue(userCredentialId, userid, username, password)  -> UserCredentialIssued(userCredentialId, userid, username, password)

        Authenticate(username, password) -> UserCrendentialsAuthenticated(userCredentialId…) 

        Suspend() 

The relationship of cause and effect is not imposed by an invariant that cross cut them so would be external to these Aggregates. Say implemented as a DomainService. It's important to understand this from the point of view of correctness.

This kind of stuff happens all the time in logic. The product price in an order is not the same as the product price in a catalogue. The products in the a ProductShippingOrder are not the same as the products in a ProductPurchaseOrder. We might have a relationship of cause in effect between cancelling a ProductPurchaseOrder and a ProductShippingOrder, they might be related but the reason why one might be cancelled and the other is not (say because it was already shipped) it's not because they are eventually consistent. The answer lies in the fact they represent different things. They are different. They expose different facts. Their lifecycle is different!.

What i'm trying to explain is that the crux of the matter is about business invariants. If the business does require NameCredential, PasswordCredential and UserCredential to be one and the same they need to be in the same aggregate! There is no way around it. Issuing a NameCredential should block Authentication (they cannot happen at the same time). That is what the business would expect if that was the case of the business process..

But you have several evidences that they aren't the same:

- A user can be authenticated regardless of other users credentials. At the same time or not.

- A user password can be changed regardless of other users credentials and its username. At the same time or not.

- A user name can be changed regardless of other users credentials and its password At the same time or not.

What is funnily is that some of us could be tempted to put all together and then have authentication to be non blocking (non serialisable reads). That would cause eventual consistency. In fact this happens all the time even in the so called fully consistent models as a matter of Sideffects. You could be authenticating a user against a user credential that already changed and no one would pose that as a problem. But now this thing is explicit and consistent is non eventual we totally changed our perception to the opposite.

Reality is not what it seams to be! 

Cheers,

Nuno

[Nuno Lopes]

Hopefully this helps.

You have eventual consistency when:

1) We have replicas.

2) We have projections/views. 

Usually we expect projections and replicas not show different values from the actually entity (Immediate Consistency as opposed to Eventual Consistency). But in a lot of scenarios its the only option due to availability constraints (CAP). Non distributed database, aka traditional centralised database systems such you very own Microsoft SQL Server pr Oracle Server provide immediate consistency. Some of them also provide synchronous replication. They also provide Strict Consistency facilities.

On Jan 13, 2013, at 7:42 AM, Johanna Belanger <johanna....@gmail.com> wrote:

[Johanna Belanger]

OK, here is a set of invariants that to a naive business person seem reasonable:

-User names must be unique

-If a user has been issued a UserNameCredential and a PasswordCredential, when he/she logs in with that UserName and Password, he/she will be authenticated.

Up to some load, a system can protect both of these invariants. This is immediate consistency. Credentials are issued and log-ins are authenticated within the same consistency boundary (or as we like to call it, aggregate.)

Given increasing load and limited budget, at some point the system will no longer perform acceptably/will become unavailable, and if we want to continue increasing load and limiting budget, we will have to relax one of those invariants.

Either:

-User names must EVENTUALLY be unique.

OR

-If a user has been issued a UserNameCredential and a PasswordCredential, when he/she logs in with that UserName and Password, he/she

will EVENTUALLY be authenticated.

Which invariant to relax and how to relax it are business decisions. That EVENTUALLY is what I mean when I say eventual consistency. But it seems you have a different definition that I am not grasping?

[Johanna Belanger]

By the way, I didn't miss this one:

NameCredentialRegistry 

        IssueNameCredential(nameCredentialId, userCredentialId, username) -> NameCredentialIssued(nameCredentialId, userCredentialId, username). 

        ReissueNameCredential(nameCredentialId, oldUserName, newUsername) ->NameCredentialReissued(nameCredentialId, userNameCredentialId, username. 

        LookUpUserNameCredential(username) : credentiaId; 

UserCredentials 

        Issue(userCredentialId, userid, username, password)  -> UserCredentialIssued(userCredentialId, userid, username, password)

ReIssue(userCredentialId, userid, username, password)  -> UserCredentialIssued(userCredentialId, userid, username, password)

        Authenticate(username, password) -> UserCrendentialsAuthenticated(userCredentialId…) 

        Suspend() 

I can think of two workflows that use this design:

1)The username and password passed to UserCredentials.Issue come directly from the user. There may be a period of time during which two users might have the same username until the NameCredentialRegistry rejects one and a compensating action is performed.

2)A process manager listens for the NameCredentialIssued event (and PasswordCredentialIssued event) and then calls UserCredentials.Issue with the username and password from those events.

I think you meant the 2nd one. This one I agree does not have eventual consistency. The user has to wait for the whole process to complete before he/she can authenticate, just like in the single aggregate solution, but now we can partition between aggregates without DTC. This is actually the solution that works best in my Product Coding domain.

So maybe this is the point you were trying to make? That there is a solution that is not atomic across the whole domain, but is also not eventually consistent?

[Nuno Lopes]

Eventual consistency means that for a given accepted update and a given replica, eventually, either the update reaches the replica, or the replica retires from service.

That is not the case of the model I suggested. There is no replica. 

What there is is a relationship of cause and effect between aggragates., UsernameCredential, PasswordCredential and UserCredentials. None of them is a replica of the other. 

[Nuno Lopes]

Yes. That is exactly the point I'm trying to make. If you follow the invariants you arrive to a solution where there is no eventual consistency as you have no replicas, partial or full,  and the relationships of cause and effect may not be atomic. 

On Monday, January 14, 2013, Johanna Belanger wrote:

[Johanna Belanger]

Excellent point. =)

[Nuno Lopes]

Hi Johanna glad to help. I few more notes that might help you get more insights about how each pattern play together and their roles. At least as I see it at the moment.

You have stated 2 options. You understand now the second is not eventually consistent. 

Yet in the first there might something that sounds like eventual consistency, but it is not. That might be where you get stuck.

SOUNDNESS (mark this word).

The proper term to explain the phenomena of the first is IMHO, "eventually sound" :) (look of soundness).

Passwords in user credentials are unique

X is a user credential

Therefore X.password is unique

Since there might be some passwords that are not unique within the context of user credentials albeit internally user information is consistent the logic is not sound since not all passwords in UserCredential are unique! (look for Soundness) The good thing of eventually soundness is that we can quickly make it sound by adding default behaviour and still keep it consistent (look for Default Logic).

"Default logic can express facts like “by default, something is true”; by contrast, standard logic can only express that something is true or that something is false. This is a problem because reasoning often involves facts that are true in the majority of cases but not always."

This is something most developers don't get when they talk with domain experts.

So given this how do we proceed with option 1)?

You wanted the user not to be given username credentials if its username is not unique. Ok. That is assured. But also you wanted the user not to be authenticated if passwords in user password credentials are not unique as we are not enforcing such thing in this context! That is the all point of uniqueness in the context of authentication.

This can be easily be done on read!!!!!!

Authenticating a user requires to look up a UserCredentials given a username. If usernames are unique in this context that means that such lookup would return only one UserCredential. But if they aren't it would return more then 1.

So in a Domain Service: AuthenticateUser(username, password) would do something like this:

UserCredentials[] listOfUserCredentials = UserCredentialsRepository.LookupByUserName(username);

// DEFAULT LOGIC

default = listOfUserCredentials != null 

If (default && listOfUserCredentials.Count() != 1) {

Bus.Publish(new UnsoundUserCredentials(….)

exit

} 

if (default) 

listOfUserCredentials[0].Authenticate(username, password); 

Something will resolve the unsoundness in which case the user will not be authenticated. In other words it will act to compensate. This works in case the PasswordCredentialIssued never get to be handled in time, or never at all.

The benefit of this approach is that its pretty fast and we don't need to block anything during authentication and keep it sound. User experience is also not hurt at all, at least from the point of view of logic.

And we still have not entered I'm the realms of an eventually consistent system at its pretty, pretty fast!!!!!! Furthermore we can scale horizontally well, but probably not elastically.

We will get into that now!

EVENTUALLY CONSISTENT CREDENTIALS

The problem now, is that we might be collecting too much "garbage" as UserCredentials. The probability is minimal during normal operation, so MAJORITY is guaranteed, either way the user experience is not hurt. But in case the node processing PasswordCredentials being down we still want the user to be registered and authenticated. We need to access the levels or garbage that we might collect given such a failure if its ok. We certainly don't what the exception to be the norm.

The question is how do assure the word "MAJORITY" further within the realms of our system so leading to less "compensations". (check the semantics of default logic).

To this we can use an eventually consistent projection of passwords issued, and eventually consistent set of password credentials. Before calling it to issue any credentials, including UserCredentials  we look up on this list to check if one already exists or not. If it does not then we issue all of the credentials in parallel. 

The question is where we make this check?

From encapsulation point of view the first tier (The Presentation) is probably not ideal. But from availability and performance is. It's easier to scale the first tier then it is the the inner tiers (business and all) as it is less subjugated to invariants. They will be assured more strongly anyway down the chain.

So I would check this in the first tier.  By doing this, we are assuring "MAJORITY" within the system to a greater degree. Furthermore, we are helping the inner layers not to get bogged down with conflicts that can be solved without locking the entire set of password credentials. 

As you can see every single step adds some development and maintenance cost. What is needed depends on the business context. If you have the luxury of designing a domain model from scratch, check does invariants and use the domain Luke. They can help go a long way in terms of availability and scalability without eventual consistency. But when you need it, it is well structured enough to proceed with confidence. More importantly, you can opt out of it if necessary.

Hope it helps.

Nuno

[Johanna Belanger]

Hi Nuno,

Sorry for the slow reply; very busy week here! Interesting information. Let me present you with 2 more cases. Please tell me whether you see eventual consistency or eventual soundness in each case and why. (Sorry I don't have time right now to do more than cursory reading of the logic stuff. So feel free to refuse to continue this converation. =))

3) NameCredentialRegistry 

        IssueNameCredential(credentialid, username) -> NameCredentialIssued(credentialid, username) 

   UserCredentials 

        Register(userid, credentialId) 

        Authenticate(username, password) -> UserCredentialsAuthenticated(credentialid…) 

        Handle(NameCredentialIssued); 

        Handle(PasswordCredentialIssued); 

   

where Authenticate fails until NameCredentialIssued and PasswordCredentialIssued are handled.

4) NameCredentialRegistry

ReissueNameCredential(credentialid, username) -> NameCredentialReissued(credentialid, username) 

   UserCredentials

Authenticate(username, password) -> UserCredentialsAuthenticated(credentialid…) 

Handle(NameCredentialReissued);

Handle(PasswordCredentialReissued);

where Authenticate with the original credentials succeeds and Authenticate with the new credentials fails until NameCredentialReissued and    PasswordCredentialReissued are handled. (The two changes being independent of each other, of course.)

I see the username and password state held by UserCredentials in both of these cases as an eventually consistent partial replica of the official state in the NameCredentialRegistry (system of record). Neither of these cases break the uniqueness invariant. It's only a matter of time before the two aggregates agree, and no compensating mechanism is necessary.

Whereas in case 1 (UserCredentials accepts the username without input from NameCredentialRegistry), the uniqueness invariant is temporarily broken and requires compensation. I am glad to have a name for this (eventual soundness).

And case 2(Registration is delayed until NameCredential and PasswordCredential are issued) clearly has neither eventual consistency nor eventual soundness.

Thanks,

Johanna

[Nuno Lopes]

Hi,

where Authenticate with the original credentials succeeds and Authenticate with the new credentials fails until  NameCredentialReissued and    PasswordCredentialReissued are handled. (The two changes being independent of each other, of course.)

You can do it in the authenticate method. Or if you prefer use a "saga". Usually I prefer the saga when the logic gets more complex.

I see the username and password state held by UserCredentials in both of these cases as an eventually consistent partial replica of the official state in the NameCredentialRegistry (system of record). 

Look the problem here is what I call the "pop culture" around the CAP Theorem.

You only have an eventual consistent system when given one question the system answers it with two facts where one contradicts the other. This can happen when two sets of information that are supposed to be equal but aren't at all times and the system uses both to compute then fact. That is the theory is not sound.

That is not the case at hand!!! Just because we use compensation does not mean it is incosistent.

 

In this case.

1) In the context of NameCredentials you can only answer questions such as  "Was (name) credential issued"?

2) In the context of UserCredentials one can only answer questions such as "Was (name,password) credential issued?" or "Was (name, password) authenticated?

The CAP theory proof has a narrower context then what we usually talk about. That is all there is to say about consistency and eventual consistency.

The challenge we face in our mind is that we want the rule: "If (name) credential and (password) credential are issued, a (name, password) credential is issued, for any (name, password)."

Because this rule crosses aggregates we implement this with "sagas" or "message handlers", unless we use transactions across sagas.

Because messages are processed async this approach can never be "Sound" with respect to this rule, but can produce consistent results. That is our programs behaviour (theory) is not sound if we keep the rule as it is.

Say, there is a partition in the system. The program gives me the answer true when i question "are name and passwords issued?" yet when i question is (username, password) issued it returns false at times, hence not able to authenticate. Yet typically we can say that it does.

"If (name) credential and (password) credential are issued,  typically (name, password) credential is issued, for any (name, password)."

There are several ways to enforce this. The best depends in a lot of factors, including the business factor.

One way is to assume nothing by default That is a "user credentials" is issued only when we have both facts. The benefit here is that we may be able to compute premisses in parallel. User authentication is performed only after all premisses are true. The separation of aggregates guarantee that the nodes issuing username and password credentials can be down but we can till authenticate current users. Further more, each pretty fast since new user don't block authentications and vice versa. This without eventual consistency of any kind.

Another way is to assume by default that (name, password) credential is issued unless the premisses are proven false. That is the case of the second approach. We can do this, because we may already verified in an eventually consistent projection issued name credentials and there is no evidence that Not(NameCredentialIssued) or Not(NameCredentialIssued) in the first tier. We also know if this evidences come in after we can compensate within the domain model

These are some algorithms we have at our disposal while keeping the domain model fully consistent.

Cheers,

Nuno

[Johanna Belanger]

Thanks for this, Nuno. 

If I understand you correctly, you are saying a write-side domain model should never have eventual consistency because no two aggregates should compute the same fact.

And that means the CAP theory doesn't apply within the write-side domain model.

Instead, sometimes an aggregate will compute its facts using incomplete (or out-of-date) premises, and default logic can be used to determine the behavior in such a case.

Interesting, I will have to look into that further when I get the time.

Regards,

Johanna