Creating one instance of an Akka actor per entity (domain-driven-design)

[Andrew Easter]

I'm currently doing some prototyping using Akka to implement the Domain Driven Design (DDD), CQRS and Event Sourcing patterns. I'm not sure whether I'm barking up the wrong tree, but my general preference would for the system to work something like this:

• Some command comes in to modify a given Aggregate Root (AR) entity
• If an Actor representing the AR is not already running in the system, create it _atomically_ (i.e. make sure you never have more than one Actor representing a specific entity)
• Actors should be removed after a period of idleness. This will avoid keeping actors around for entities with low interaction rates, but ensure 'hot' entities are kept in memory for performance (starting a new actor requires loading events from an event store in order to restore the memory image representing the entity state)

I'm wondering if there are any best practices with regard to creating actors atomically whilst avoiding any performance overload during the lookup process - i.e. if an actor is already running, you'd ideally not incur any sort of lookup related performance penalty. This applies in a clustered environment as well as when running in a single JVM. 

I have seen documentation on the Cluster Singleton Pattern, but my interpretation of this was that each ClusterSingletonManager would need to know up front what actors needed to be singletons. In my suggested design, you only create the Aggregate Root actors on the fly and thus do not know up front.

Would be interested in opinions on this.

[Patrik Nordwall]

Hi Andrew,

On Mon, Oct 7, 2013 at 2:54 PM, Andrew Easter <andrew...@gmail.com> wrote:

I'm currently doing some prototyping using Akka to implement the Domain Driven Design (DDD), CQRS and Event Sourcing patterns. I'm not sure whether I'm barking up the wrong tree, but my general preference would for the system to work something like this:

• Some command comes in to modify a given Aggregate Root (AR) entity
• If an Actor representing the AR is not already running in the system, create it _atomically_ (i.e. make sure you never have more than one Actor representing a specific entity)
• Actors should be removed after a period of idleness. This will avoid keeping actors around for entities with low interaction rates, but ensure 'hot' entities are kept in memory for performance (starting a new actor requires loading events from an event store in order to restore the memory image representing the entity state)

That is a sane design. You probably also want to look at akka-persistence.

Actors don't consume much more than memory when they are idle, but that can be a good enough reason to prune them.

 

I'm wondering if there are any best practices with regard to creating actors atomically whilst avoiding any performance overload during the lookup process - i.e. if an actor is already running, you'd ideally not incur any sort of lookup related performance penalty.

That is best handled by a parent actor. The entities can stop themselves when idle (e.g. by using ReceiveTimeout), and that will automatically remove them from the parent in a safe way.

 

This applies in a clustered environment as well as when running in a single JVM. 

That can become more complicated.

 

I have seen documentation on the Cluster Singleton Pattern, but my interpretation of this was that each ClusterSingletonManager would need to know up front what actors needed to be singletons. In my suggested design, you only create the Aggregate Root actors on the fly and thus do not know up front.

The singleton actor can be the parent (mentioned above) that creates the child entity actors on demand. The disadvantage with this usage of the singleton is that the node hosting the active singleton (the oldest node) will host all entities, and standby nodes will be empty until failover. This can be somewhat mitigated by using cluster roles.

You could also use the singleton as a central coordinator for deciding where to create the entity, but that is easier said than done.

You could also use consistent hashing to decide on which node to host the entity without a central coordinator, but that has less strict characteristics, since some entities will "move" when the node ring is changed and there will be an overlap where an entity may exist on several nodes if nothing is done to prevent that.

Regards,

Patrik

 

Would be interested in opinions on this.

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/groups/opt_out.

--

Patrik Nordwall
Typesafe -  Reactive apps on the JVM
Twitter: @patriknw

[Andrew Easter]

Thanks for your ideas.

I really haven't thought through this in much detail, but could you somehow imagine a sort of 'multi-master' approach whereby you have multiple clusters and use consistent hashing to identify which cluster to use?

In this way, each cluster would have a singleton managing creation of entities. We'd maintain the "one actor per entity" requirement and at least be spreading the work out.

It kind of seems brute force and would require that clients be configured to know explicitly about each cluster. And I'm sure there are plenty of implementation challenges that I can't predict by just playing through this scenario in my head.

[Patrik Nordwall]

You can use cluster node roles for that, one singleton per role. Consistent hashing for picking the responsible role is a good idea.

--
>>>>>>>>>>      Read the docs: http://akka.io/docs/
>>>>>>>>>>      Check the FAQ: http://akka.io/faq/
>>>>>>>>>>      Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/groups/opt_out.

[Andrew Easter]

Hmmmmm...just rethinking one of your ideas.

Am I right in thinking that using node roles (as you mentioned) would allow for the 'multi master' approach without separate clusters? The idea being to create as many node roles as I want 'masters' and have multiple cluster singletons, each associated with a unique role?

I suppose consistent hashing could be used to determine which singleton to use for a particular entity id.

[Patrik Nordwall]

On Tue, Oct 8, 2013 at 9:45 AM, Andrew Easter <andrew...@gmail.com> wrote:

Hmmmmm...just rethinking one of your ideas.

Am I right in thinking that using node roles (as you mentioned) would allow for the 'multi master' approach without separate clusters? The idea being to create as many node roles as I want 'masters' and have multiple cluster singletons, each associated with a unique role?

Yes, exactly.

 

I suppose consistent hashing could be used to determine which singleton to use for a particular entity id.

yes, for picking the responsible role, and then pick the node with that role that is oldest, and there is where the active singleton is running, or perhaps better use DistributedPubSub and let the active singleton actor register itself with the role as path or topic and send the messages through the mediator to that path/topic.

Apart from the documentation there are two activator tutorials that should be a good start for you.

http://akka.io/downloads/

- Akka Clustered PubSub

- Distributed Workers with Akka

Regards,

Patrik

 

--
>>>>>>>>>>      Read the docs: http://akka.io/docs/
>>>>>>>>>>      Check the FAQ: http://akka.io/faq/
>>>>>>>>>>      Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/groups/opt_out.

[Andrew Easter]

Thanks again - seem there was a bit of crossover in our messages - I didn't see your response before writing my message beginning "Hmmmmm..."!

Cheers,

Andrew

[Justin du coeur]

[Just got back from my honeymoon, so catching up on back email.]

A general observation: the description of the problem here sounds more or less structurally identical to what I'm going to be wrestling with for Querki in the next 4 months or so, as I scale up from a single node to a cluster.  The details of the endpoint actors are different, but the high concept -- wanting at most one Actor for each named conceptual entity, starting that on-demand, and timing it out after an idle period -- is identical.  And I get the impression that we're not the only ones using Akka for this pattern.

So we may want to explore this particular problem in more depth in the coming months, with an eye towards coming up with some best-practices recommendations, or at least an understanding of the tradeoffs between the various options.  (And maybe some generic framework code for folks to use.)

For reference, my theorizing had wound up going in a somewhat different direction, basically starting with Glokka and expanding that into a cluster-wide distributed registration system; an outline can be found here.  It's going to be another month or two before I start actually committing code, so any additional thoughts on the subject (including tradeoffs between the distributed-registrar approach I've been pondering vs. the discussion in this thread) would be welcomed.  I'm not especially wedded to my architecture yet, and would love to have this lead to a proper OSS module, possibly even part of the standard Akka Patterns if it seems generally useful enough...

[Akka Team]

Hi Justin,

the use case you so aptly summarize is indeed one which should become more and more commonplace. While I am currently too busy with other cool things to contribute directly I'd be delighted to take part in discussions on this topic. Patrik already outlined some promising strategies and it will be interesting to see where this ends up.

Regards,

Roland

--

Akka Team

Typesafe - The software stack for applications that scale

Blog: letitcrash.com
Twitter: @akkateam

[Vaughn Vernon]

This is all what I have been working on for months, in addition to other Akka grid features.

Vaughn

On Wednesday, October 23, 2013 9:32:49 AM UTC-6, Akka Team wrote:

Hi Justin,

the use case you so aptly summarize is indeed one which should become more and more commonplace. While I am currently too busy with other cool things to contribute directly I'd be delighted to take part in discussions on this topic. Patrik already outlined some promising strategies and it will be interesting to see where this ends up.

Regards,

Roland

On Wednesday, October 23, 2013, Justin du coeur wrote:

[Just got back from my honeymoon, so catching up on back email.]

A general observation: the description of the problem here sounds more or less structurally identical to what I'm going to be wrestling with for Querki in the next 4 months or so, as I scale up from a single node to a cluster.  The details of the endpoint actors are different, but the high concept -- wanting at most one Actor for each named conceptual entity, starting that on-demand, and timing it out after an idle period -- is identical.  And I get the impression that we're not the only ones using Akka for this pattern.

So we may want to explore this particular problem in more depth in the coming months, with an eye towards coming up with some best-practices recommendations, or at least an understanding of the tradeoffs between the various options.  (And maybe some generic framework code for folks to use.)

For reference, my theorizing had wound up going in a somewhat different direction, basically starting with Glokka and expanding that into a cluster-wide distributed registration system; an outline can be found here.  It's going to be another month or two before I start actually committing code, so any additional thoughts on the subject (including tradeoffs between the distributed-registrar approach I've been pondering vs. the discussion in this thread) would be welcomed.  I'm not especially wedded to my architecture yet, and would love to have this lead to a proper OSS module, possibly even part of the standard Akka Patterns if it seems generally useful enough...

On Mon, Oct 7, 2013 at 8:54 AM, Andrew Easter <andrew...@gmail.com> wrote:

I'm currently doing some prototyping using Akka to implement the Domain Driven Design (DDD), CQRS and Event Sourcing patterns. I'm not sure whether I'm barking up the wrong tree, but my general preference would for the system to work something like this:

• Some command comes in to modify a given Aggregate Root (AR) entity
• If an Actor representing the AR is not already running in the system, create it _atomically_ (i.e. make sure you never have more than one Actor representing a specific entity)
• Actors should be removed after a period of idleness. This will avoid keeping actors around for entities with low interaction rates, but ensure 'hot' entities are kept in memory for performance (starting a new actor requires loading events from an event store in order to restore the memory image representing the entity state)

I'm wondering if there are any best practices with regard to creating actors atomically whilst avoiding any performance overload during the lookup process - i.e. if an actor is already running, you'd ideally not incur any sort of lookup related performance penalty. This applies in a clustered environment as well as when running in a single JVM. 

I have seen documentation on the Cluster Singleton Pattern, but my interpretation of this was that each ClusterSingletonManager would need to know up front what actors needed to be singletons. In my suggested design, you only create the Aggregate Root actors on the fly and thus do not know up front.

Would be interested in opinions on this.

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.

To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscribe@googlegroups.com.

To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/groups/opt_out.

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.

To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscribe@googlegroups.com.

To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/groups/opt_out.

[Andrew Easter]

Justin,

Are you including event sourcing in the scope of this?

Like Roland says, I definitely foresee this overall approach becoming much more commonplace, especially with akka-persistence on the horizon.

I think defining some best practices and patterns around using actors as unique persistent entities in a clustered environment would be extremely valuable.

I still need to actually have a crack at prototyping some of Patrik's ideas but am interested in exploring alternatives.

This is a really exciting area of research as it has the potential to offer a very compelling alternative to existing techniques for building an application domain layer. It feels like a really natural way of implementing domain driven design principles - it just makes sense.

Andrew

[Vaughn Vernon]

I guess my posts are still being censored by the moderators. Just in case my posts are at some point approved, that would just peachy. I've spent many months promoting this very approach, developing solutions for it. I have also made numerous presentations, ncluding several community events, including one just last week:

    http://vimeo.com/63802840
    http://skillsmatter.com/podcast/design-architecture/vaughn-vernon
    http://www.meetup.com/ddd-denver/events/138251862/

Just in case this post makes it's way past protection...

To make your aggregates naturally transient, you will have to work around some basic problems. For example, when an Actor is stopped for some reason (just unloaded due to lack of use), the ActorRef will stay around for some time. So when you get ready to reload the Actor in question, the preexisting ActorRef will prevent the Actor from being created again.

Vaughn

[Roland Kuhn]

Hi Vaughn,

first off: we don’t censor anything on the Akka mailing lists, we only sometimes (very rarely) get spam which we then reject. I have not seen any message from you in the moderation process in the past weeks, if messages were indeed sent then we need to figure out where they went.

24 okt 2013 kl. 06:41 skrev Vaughn Vernon:

I guess my posts are still being censored by the moderators. Just in case my posts are at some point approved, that would just peachy. I've spent many months promoting this very approach, developing solutions for it. I have also made numerous presentations, ncluding several community events, including one just last week:

    http://vimeo.com/63802840
    http://skillsmatter.com/podcast/design-architecture/vaughn-vernon
    http://www.meetup.com/ddd-denver/events/138251862/

Just in case this post makes it's way past protection...

To make your aggregates naturally transient, you will have to work around some basic problems. For example, when an Actor is stopped for some reason (just unloaded due to lack of use), the ActorRef will stay around for some time. So when you get ready to reload the Actor in question, the preexisting ActorRef will prevent the Actor from being created again.

This is not true: stopped is stopped, existing ActorRefs effectively turn into deadLetters. But for such a system I would recommend not ever exposing the ActorRef of the actual aggregate to the rest of the system, but only go through its parent. Otherwise you will have all kinds of race conditions around starting/stopping aggregates.

Regards,

Roland

Vaughn


On Wednesday, October 23, 2013 9:42:24 PM UTC-6, Andrew Easter wrote:

Justin,

Are you including event sourcing in the scope of this?

Like Roland says, I definitely foresee this overall approach becoming much more commonplace, especially with akka-persistence on the horizon.

I think defining some best practices and patterns around using actors as unique persistent entities in a clustered environment would be extremely valuable.

I still need to actually have a crack at prototyping some of Patrik's ideas but am interested in exploring alternatives.

This is a really exciting area of research as it has the potential to offer a very compelling alternative to existing techniques for building an application domain layer. It feels like a really natural way of implementing domain driven design principles - it just makes sense.

Andrew

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/groups/opt_out.

Dr. Roland Kuhn
Akka Tech Lead
Typesafe – Reactive apps on the JVM.
twitter: @rolandkuhn

[Björn Antonsson]

Hi,

There were a bunch of messages in the "queue" this morning that I just let through after a brief skim. That's probably it.

B/

-- 

Björn Antonsson

Typesafe – Reactive Apps on the JVM

twitter: @bantonsson

[Vaughn Vernon]

Hi Roland,

Any time I post a message it takes at least hours before it is actually visible on the group. As soon as I post I see a message that says that if my post is approved by a moderator it will become visible. I am pretty sure it will happen again just after I click the Post button. That to me means censorship, but I have no idea who has time to read and approve every message to this group. The message previous to the one you are replying to didn't show up for about 10 hours after I posted it. In fact, at least one other message that was posted around 6 hours after my message, was visible to the group immediately after it was posted, and 4 hours before before mine that was posted 6 hours prior. Perhaps this is not the way you intend for this to work, but it is the way it has worked for me since my first post.

Also, as you know I have spent at least a few months pouring over the Akka Scala code. An observation I have made is that the name of an Actor is not usable again after it has been stopped, at least for some time. So when I try to dynamically reload the Actor I have to play tricks with it's name so I can get it to load again. I probably should have asked you about this, but I didn't want to bother you thinking that for some reason it was supposed to work this way (e.g. stopped means stopped with the possible chance of restarting in the near future, so the Actor and ActorRef linger for some time before they are actually ejected from the Children collection). In fact one way I work around this is indeed to handle messages that are received by deadLetters by registering a listener, dynamically reloading the aggregate-actor, and then dispatching the message to the newly loaded aggregate-actor.

The fact that the name is not immediately available is a bit puzzling to me, because the same name must be reused during a supervision recovery.

Regarding race conditions, I don't think this would necessarily be a problem. It's not the responsibility of any given client to stop an aggregate-actor. It just happens during a LRU clean up, and I want the sort of "page fault" response to sending an ActorRef a message that is received by deadLetters that causes the actor to reload.

Vaughn

[Justin du coeur]

On Wed, Oct 23, 2013 at 11:42 PM, Andrew Easter <andrew...@gmail.com> wrote:

Are you including event sourcing in the scope of this?

Like Roland says, I definitely foresee this overall approach becoming much more commonplace, especially with akka-persistence on the horizon.

I haven't looked much into event sourcing or akka-persistence in this context yet (I only realized the existence of akka-persistence a short while ago).  I'm not sure whether it's appropriate for my particular problem or not -- I'm building what amounts to a database, and the details of the persistence strategy matter a lot -- but I'm not ruling it out yet.  Some aspects of what I'm hearing about akka-persistence, such as the emphasis on journaling, are things that I'd already been planning on building myself.  So I'm going to need to understand this stuff better.

(I get the impression that I should maybe follow the akka-dev list, for a better sense of what's coming down the pike?)

In general, there's an interesting philosophical issue here that I'm going to need to ponder.  I've been thinking of my Spaces as fundamentally *database* structures, that happen to use an Actor as their front end mediating communication.  So it never occurred to me to even think about akka-persistence as relevant, since I don't really think of the Actor per se as the thing I'm persisting.  But it's possible that the difference isn't actually meaningful.  I'll need a better understanding of akka-persistence to know -- is there a good overview of the plan?

[Justin du coeur]

On Thu, Oct 24, 2013 at 2:56 AM, Roland Kuhn <goo...@rkuhn.info> wrote:

This is not true: stopped is stopped, existing ActorRefs effectively turn into deadLetters. But for such a system I would recommend not ever exposing the ActorRef of the actual aggregate to the rest of the system, but only go through its parent. Otherwise you will have all kinds of race conditions around starting/stopping aggregates.

Right -- my hypothetical architecture was designed around that assumption.  In order to have reliable guarantees about the "client" Actors, all messages get routed through their supervisors; those supervisors, in turn, are members of a coordinated Registry system.  Each node has a pool of LocalRegistrars, each of which in turn manages and routes to a pool of the Actors we actually care about.  All nicely decentralized except for the CentralRegistrar, and that isn't consulted too terribly often.

(I actually rather like the architecture I outlined -- my only serious concern with it is that failure recovery may get persnickety.  But that's enough for me to be looking seriously at the alternatives.  And I don't know how many wheels I'm reinventing here...)

[Roland Kuhn]

Hi Vaughn,

24 okt 2013 kl. 15:19 skrev Vaughn Vernon:

Hi Roland,

Any time I post a message it takes at least hours before it is actually visible on the group. As soon as I post I see a message that says that if my post is approved by a moderator it will become visible. I am pretty sure it will happen again just after I click the Post button. That to me means censorship, but I have no idea who has time to read and approve every message to this group. The message previous to the one you are replying to didn't show up for about 10 hours after I posted it. In fact, at least one other message that was posted around 6 hours after my message, was visible to the group immediately after it was posted, and 4 hours before before mine that was posted 6 hours prior. Perhaps this is not the way you intend for this to work, but it is the way it has worked for me since my first post.

Without moderation this list would be full of spam, and that is just not an option. I apologize for the delays, especially during the European night hours, but with all due respect I must object to the allegation of censorship. It is troubling that you think this badly of us.

Your observation of another post not being moderated is due to the fact that the moderation mechanism learns after a while who is trustworthy and who is not, but that takes a few posts (I don’t know the exact algorithm).

Also, as you know I have spent at least a few months pouring over the Akka Scala code. An observation I have made is that the name of an Actor is not usable again after it has been stopped, at least for some time. So when I try to dynamically reload the Actor I have to play tricks with it's name so I can get it to load again. I probably should have asked you about this, but I didn't want to bother you thinking that for some reason it was supposed to work this way (e.g. stopped means stopped with the possible chance of restarting in the near future, so the Actor and ActorRef linger for some time before they are actually ejected from the Children collection). In fact one way I work around this is indeed to handle messages that are received by deadLetters by registering a listener, dynamically reloading the aggregate-actor, and then dispatching the message to the newly loaded aggregate-actor.

http://doc.akka.io/docs/akka/2.2.3/general/addressing.html#Reusing_Actor_Paths

http://doc.akka.io/docs/akka/2.2.3/scala/actors.html#Stopping_actors (see the Note)

http://doc.akka.io/docs/akka/2.2.3/scala/actors.html#Actor_Lifecycle (actor path is free <=> Terminated)

In short, you can reuse the name as soon as the parent knows that the child has terminated, and we do offer an API (context.watch()) which allows detection of this event. This is not overly aggressively documented because reusing an actor name is not good practice—use restarts instead (where needed).

The fact that the name is not immediately available is a bit puzzling to me, because the same name must be reused during a supervision recovery.

No, please read up on those links above: restart does NOT stop and the start again the actor. That is what Erlang does, and it leads to races (scroll down to “Don't drink too much kool-aid”; thank us later ;-) ).

Regarding race conditions, I don't think this would necessarily be a problem. It's not the responsibility of any given client to stop an aggregate-actor. It just happens during a LRU clean up, and I want the sort of "page fault" response to sending an ActorRef a message that is received by deadLetters that causes the actor to reload.

If you expose the aggregate-actor directly to clients then they will continue sending to it after it is dead, leading to no replies, timeouts, etc. It is much better to always send requests to the aggregate via its parent, so that creation/expiry/requests are handled consistently.

Regards,

Roland

[Vaughn Vernon]

Thanks Roland. I will drop the subtopic, especially since something seemed to learn recently that my posts are okay :) And I don't think badly of any of you. If I didn't have very high regard for you guys I wouldn't be working on this stuff in the first place. So I apologize that my terminology was taken another way.

I have read those docs already, but I may have missed a finer point. Actually I think I did read that note about not being able to reuse an actor's name until a Terminated message is received. That's why I took the route of tweaking the name a little so I could almost reuse the same name. (Basically I am encoding the actor's name in a few different ways so that the grid add-ons can deal with reloading actors just in time.)

I have several concerns with sending through a parent. For one thing, depending on the policy used as a parent, the aggregates would have less memory to consume in a single VM. My goal is to maintain as many aggregates in memory as possible. If, for example, there was a 1:1 parent to child then it obviously halves the memory for aggregates. I am pretty sure that's not what you are suggesting, however. Otherwise, the parent of the aggregate becomes roughly like an application service, and I am purposely eliminating those form he architecture. Besides, we could still fall into the same problems you describe of losing messages if the parent goes away.

I thought for sure that as soon as an actor is stopped that any message sent to it would go to deadLetters. Why would this ever fail to be true? Why would a message ever be completely lost, as in "poof" it's gone?

Actually what I have wanted to discuss with you is some way to create a user-defined ActorRef, much like LocalActorRef or InternalActorRef. I think being able to create, for example, a GridActorRef could be a cure for most of the issues I have previously had to address. Essentially the GridActorRef could track termination and auto-reloading. In that case clients could use Akka in the simplest way and not have to deal with special routing policies.

Vaughn

[Akka Team]

Hi Vaughn,

On Thu, Oct 24, 2013 at 10:18 PM, Vaughn Vernon <vve...@shiftmethod.com> wrote:

Thanks Roland. I will drop the subtopic, especially since something seemed to learn recently that my posts are okay :) And I don't think badly of any of you. If I didn't have very high regard for you guys I wouldn't be working on this stuff in the first place. So I apologize that my terminology was taken another way.

That is good to hear, apology accepted.

 

I have read those docs already, but I may have missed a finer point. Actually I think I did read that note about not being able to reuse an actor's name until a Terminated message is received. That's why I took the route of tweaking the name a little so I could almost reuse the same name. (Basically I am encoding the actor's name in a few different ways so that the grid add-ons can deal with reloading actors just in time.)

I have several concerns with sending through a parent. For one thing, depending on the policy used as a parent, the aggregates would have less memory to consume in a single VM. My goal is to maintain as many aggregates in memory as possible. If, for example, there was a 1:1 parent to child then it obviously halves the memory for aggregates. I am pretty sure that's not what you are suggesting, however. Otherwise, the parent of the aggregate becomes roughly like an application service, and I am purposely eliminating those form he architecture. Besides, we could still fall into the same problems you describe of losing messages if the parent goes away.

The idea would be to have one parent per node, which should be enough since the messaging bottleneck will be the network interface. The parent would not go away, because it is the node (for all intents and purposes), the node shutting down would be the only reason why the parent goes away. Routing to these parents could be done using a cluster-aware consistent hashing router (of Group terminology in 2.3-M1 speak). This means no memory overhead and no application service (unless I understand that word wrongly).

 

I thought for sure that as soon as an actor is stopped that any message sent to it would go to deadLetters. Why would this ever fail to be true? Why would a message ever be completely lost, as in "poof" it's gone?

Yes, messages can be completely lost as in “poof”. Especially during actor termination. The reason is that the mailbox is replaced by a reference to deadLetters to free up memory even if someone holds on to the ActorRef, and this replacement cannot be done without a certain race condition that allows a message to go nowhere at all.

 

Actually what I have wanted to discuss with you is some way to create a user-defined ActorRef, much like LocalActorRef or InternalActorRef. I think being able to create, for example, a GridActorRef could be a cure for most of the issues I have previously had to address. Essentially the GridActorRef could track termination and auto-reloading. In that case clients could use Akka in the simplest way and not have to deal with special routing policies.

That would be what we have been calling ClusterActorRef, playing together with the cluster naming service. But that is a hard problem—at least if you want it to scale and not have a SPOF—wherefore it is not implemented yet. But I do believe that using the mechanism outlined above this problem can be solved without the need for a special kind of ActorRef, which if true would almost by definition mean that the special ActorRef cannot be fundamentally better (since we are currently SPOF-free and perfectly scalable).

[Vaughn Vernon]

Okay, if I have a client actor C, and an aggregate actor A, and the parent of all aggregates on node N, how does C send a message to A?

    C -> N -> A
       or
    C -> A

I think you mean the first example. How would you write the code to do this? Let's say that N becomes nodeRef and A becomes backlogItemRef, and there is a second aggregate sprintRef. Would a client telling a backlogItem to commit to a sprint be done like this?

     nodeRef ! Commit(backlogItemId, sprintId)

Then how does the Node parent actor dispatch to the backlogItem? Does it have to understand how to map a Commit message to a CommitTo message that gets sent to the backlogItem?

Vaughn

[Andrew Easter]

Justin,

Sorry if I end up reciting things obvious to you here, but I'm going to explain my general opinions about combining domain driven design, event sourcing and the actor model. Vaughn, I'm probably going to be covering plenty of stuff you've already formulated and shared with others! However, I wanted to share my own raw thoughts that have not specifically influenced by anyone else's thinking.

As Vaughn has clearly worked out quicker than the rest of us :-) it seems like the actor model just fits very well with domain driven design. It feels quite natural to me to consider an actor analogous to an aggregate root (in DDD speak). So, rather than just seeing an actor as a mediator sitting in front of a database, you see the actor conceptually as the entity. By incorporating event-sourcing, the actor works on the basis of processing a command (representing some action) and outputting one or many events that represent the result of processing that command. At all times the actor encapsulates its current state, which can simply be seen as the result of a series of ordered events applied to it.

Initially, it's actually quite helpful to ignore persistence requirements altogether and just rely on the state within the actor - in this way you don't allow persistence concerns to influence the way you design your domain - I often find that domain models end up messy because, despite attempts to avoid doing so, persistence concerns end up, in one form or another, leaking into the business logic. Any approach to building a domain layer that can genuinely remove the need to consider persistence technologies is an attractive proposition in my mind!  

By following an event-sourcing approach, persistence becomes no more complex than just persisting the events - that the actor produces - in an event store (journal). Given that events represent an immutable history of state changes, the journal can be append only and that's pretty exciting from a performance perspective. This is something that can quite clearly be bolted on once you've already put together a working in-memory only version of your domain - the actors themselves can be completely agnostic to the existence of the persistence mechanism. This is where akka-persistence comes in - it works on the following basis:

1) Actor receives command

2) Actor processes command (applies business logic) and produces events that represent results of processing

3) Events are persisted to journal 

4) Events are applied to actor so it can update it's internal state

akka-persistence pretty much deals with everything other than the command handling step, which obviously represents your custom business logic - akka is not quite good enough (yet) to do that bit ;-)

Step 4) is quite important - it's key that internal state is only applied once you're sure events have been persisted to the journal. Also, by separating this step, it serves the additional purpose of allowing events in the journal to be replayed onto a "green" instance of the actor such to bring it's internal state back up to current. And this is exactly what would be required for what we've discussed previously with regard to actors being reloaded into the cluster having been cleared out (due to idleness). akka-persistence also has support for "snapshots" which allow you to take periodic snapshots of current state to be used as a performance optimisation when replaying events.

So, to sum up, the actor model just "fits" with domain driven design and event sourcing. It seems to deal with a majority of the pain points experienced when building a domain layer using more traditional CRUD techniques. It's pretty exciting to me how natural this design feels without any need to consider using an ORM ;-)

Hope I've not written a complete load of un-intelligible trash :-)

[Martin Krasser]

On 25.10.13 06:45, Andrew Easter wrote:

Justin,

Sorry if I end up reciting things obvious to you here, but I'm going to explain my general opinions about combining domain driven design, event sourcing and the actor model. Vaughn, I'm probably going to be covering plenty of stuff you've already formulated and shared with others! However, I wanted to share my own raw thoughts that have not specifically influenced by anyone else's thinking.

As Vaughn has clearly worked out quicker than the rest of us :-) it seems like the actor model just fits very well with domain driven design. It feels quite natural to me to consider an actor analogous to an aggregate root (in DDD speak). So, rather than just seeing an actor as a mediator sitting in front of a database, you see the actor conceptually as the entity. By incorporating event-sourcing, the actor works on the basis of processing a command (representing some action) and outputting one or many events that represent the result of processing that command. At all times the actor encapsulates its current state, which can simply be seen as the result of a series of ordered events applied to it.

Initially, it's actually quite helpful to ignore persistence requirements altogether and just rely on the state within the actor - in this way you don't allow persistence concerns to influence the way you design your domain - I often find that domain models end up messy because, despite attempts to avoid doing so, persistence concerns end up, in one form or another, leaking into the business logic. Any approach to building a domain layer that can genuinely remove the need to consider persistence technologies is an attractive proposition in my mind!  

By following an event-sourcing approach, persistence becomes no more complex than just persisting the events - that the actor produces - in an event store (journal). Given that events represent an immutable history of state changes, the journal can be append only and that's pretty exciting from a performance perspective. This is something that can quite clearly be bolted on once you've already put together a working in-memory only version of your domain - the actors themselves can be completely agnostic to the existence of the persistence mechanism. This is where akka-persistence comes in - it works on the following basis:

1) Actor receives command

2) Actor processes command (applies business logic) and produces events that represent results of processing

3) Events are persisted to journal 

4) Events are applied to actor so it can update it's internal state

akka-persistence pretty much deals with everything other than the command handling step, which obviously represents your custom business logic - akka is not quite good enough (yet) to do that bit ;-)

Step 4) is quite important - it's key that internal state is only applied once you're sure events have been persisted to the journal. Also, by separating this step, it serves the additional purpose of allowing events in the journal to be replayed onto a "green" instance of the actor such to bring it's internal state back up to current. And this is exactly what would be required for what we've discussed previously with regard to actors being reloaded into the cluster having been cleared out (due to idleness). akka-persistence also has support for "snapshots" which allow you to take periodic snapshots of current state to be used as a performance optimisation when replaying events.

So, to sum up, the actor model just "fits" with domain driven design and event sourcing. It seems to deal with a majority of the pain points experienced when building a domain layer using more traditional CRUD techniques. It's pretty exciting to me how natural this design feels without any need to consider using an ORM ;-)

Hope I've not written a complete load of un-intelligible trash :-)

Not at all, very good summary, really like it. Thanks for sharing!

On Thursday, 24 October 2013 06:25:52 UTC-7, Justin du coeur wrote:

On Wed, Oct 23, 2013 at 11:42 PM, Andrew Easter <andrew...@gmail.com> wrote:

Are you including event sourcing in the scope of this?

Like Roland says, I definitely foresee this overall approach becoming much more commonplace, especially with akka-persistence on the horizon.

I haven't looked much into event sourcing or akka-persistence in this context yet (I only realized the existence of akka-persistence a short while ago).  I'm not sure whether it's appropriate for my particular problem or not -- I'm building what amounts to a database, and the details of the persistence strategy matter a lot -- but I'm not ruling it out yet.  Some aspects of what I'm hearing about akka-persistence, such as the emphasis on journaling, are things that I'd already been planning on building myself.  So I'm going to need to understand this stuff better.

(I get the impression that I should maybe follow the akka-dev list, for a better sense of what's coming down the pike?)

In general, there's an interesting philosophical issue here that I'm going to need to ponder.  I've been thinking of my Spaces as fundamentally *database* structures, that happen to use an Actor as their front end mediating communication.  So it never occurred to me to even think about akka-persistence as relevant, since I don't really think of the Actor per se as the thing I'm persisting.  But it's possible that the difference isn't actually meaningful.  I'll need a better understanding of akka-persistence to know -- is there a good overview of the plan?

--

>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/groups/opt_out.

-- 
Martin Krasser

blog:    http://krasserm.blogspot.com
code:    http://github.com/krasserm
twitter: http://twitter.com/mrt1nz

[Andrew Easter]

Unless our thinking has diverged, a single nodeRef is the parent of all instances (or subset of instances) of a particular aggregate root type. In my mind, a nodeRef would not need to map the Commit message to a CommitTo message, it need only act somewhat like a router - surely the backlogItemRef can just be sent the original Commit message?

On receipt of a command, the nodeRef either forwards the same Commit message to a new actor (created on demand and cached) that represents the backlogItemRef or to an existing actor that it's already created and cached.

Is there something you particularly don't like about the idea of the nodeRef having to forward on the message?

As I believe Patrik alluded to earlier in this thread, Distributed PubSub could be used to help solve the problem of distributing nodeRefs around the cluster, and thus commands wouldn't necessarily be sent as messages directly to the nodeRef. In this case clients would send commands destined for backlogItemRefs to a backlogItem topic to which a nodeRef would be subscribed.

[Patrik Nordwall]

If we look for a name of "parent of all instances of a particular aggregate root type" it would be Repository in DDD terminology. It doesn't have to be one per node, but typically one per aggregate root type per node.

In it's simplest form it would look like this:

  case a @ AddOrderItem(orderId, item) =>

      val order = context.child(orderId).getOrElse {

        context.actorOf(Order.props(orderId), orderId)

      }

      order forward a

And since then entities are children and you only hold on to them in the ordinary children container you don't even have to watch for termination for cleanup.

The entities themselves can set a receiveTimeout and stop when idle to cleanup.

/Patrik

[Vaughn Vernon]

IMO, this works well if you are willing to think in terms of using a grid/fabric product such as Coherence or GemFire. Essentially the parent is a named cache/region that is forwarding entry processors/functions to the cached entities. You can now balance and distribute a given cache using consistent hashing.

From a DDD perspective this is disappointing. It's not an explicit message send from one client/aggregate to an aggregate. You always have to work out in your mind a routing that's happening. And, FWIW, I don't like thinking of the aggregate-type parent as a repository; that makes it even worse for me. It's not that it is not a workable solution. It's just that it falls short of the goals I have for making everything explicit. And I want the grid/fabric out of the way, and allow the delivery of a entry processor/function, as it were, to go directly from whatever client to the aggregate.

Vaughn

[Vaughn Vernon]

Okay, this is what I've been talking about: https://vaughnvernon.co/?p=770

Vaughn

[Andrew Easter]

Vaughn,

Whilst I totally understand the case you arI'm still not really sure that I fully share your concerns with regard to lack of explicitness in some of proposals in this thread. We've not really talked about CQRS explicitly in this thread, but I was under the impression (maybe wrongly) that the typically async nature of command dispatch fits quite nicely with the idea of just firing off a command and trusting it will be processed by the correct aggregate.

[Andrew Easter]

Sorry, pressed send too early!!!

========================

Vaughn,

Whilst I totally understand the case you are making, I'm still not really sure that I fully share your concerns with regard to lack of explicitness in some of the proposals in this thread. We've not really talked about CQRS explicitly in this thread, but I was under the impression (maybe wrongly) that the typically async nature of command dispatch fits quite nicely with the idea of just firing off a command and trusting it will be processed by the correct aggregate. On the other hand, I suppose one could reasonably argue that the lack of explicitness in that mechanism is not something one should be content with.

I do get the argument that this approach exposes the presence of application services to the client, but I guess I don't really see this as such a smell as you do!

Thanks for an interesting read - it does seem like you are on a good path with your prototype.

Andrew

[Andrew Easter]

Would also like to suggest a possible "tweak" to your example.

I don't know whether you are deliberately avoiding mentioning CQRS and event sourcing in your example, but I think it's worth considering how your example might change to accommodate those patterns. After all, I originally created this thread to discuss those topics as well as DDD. 

One thing that stands out to me in your example is that the aggregateOf() method takes in a Props for creating the underlying Actor. I'm wondering whether it would actually make more sense for the Props to, instead, be passed to the registerAggregateType() method. The only reason it seems it must be passed to the aggregateOf() method is to pre-initialise the aggregate with state - for me this is incompatible with the principles of CQRS and event sourcing where it would make more sense for all state changes to be the result of processing a command. In my opinion, you needn't pass state during creation, but simply send commands/messages to the actor once it has been created. In this way, the Props you'd send to registerAggregateType() need only define the type of the aggregate actor and nothing more. 

Do you see what I'm getting at?

[Patrik Nordwall]

On Fri, Oct 25, 2013 at 11:20 PM, Vaughn Vernon <vve...@shiftmethod.com> wrote:

Okay, this is what I've been talking about: https://vaughnvernon.co/?p=770

Vaughn, I see where you want to go and I think it is the right path, but there is one thing in your code that is completely forbidden. You expose and use the context (via AggregateCacheWorker) outside of the actor. You must find another way of doing that.

/Patrik

 

Vaughn

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/groups/opt_out.

[Vaughn Vernon]

I agree. That's the main reason that II said in the post: "NOTE: If you think the above code is a little hacky, you would be correct..." I did that because I wanted to post the blog and had little.

I will fix all that with an upcoming post, where I also what to address a bunch of other DDD-Akka stuff that I have been working on here and there over the past few months.

Vaughn

[Vaughn Vernon]

If you haven't yet watched this presentation, I suggest doing so: http://skillsmatter.com/podcast/design-architecture/vaughn-vernon

It all includes the use of CQRS and what I'd call "message sourcing" as opposed to "event sourcing."  I gave a much more complete presentation at the recent NYC DDD eXchange, but I think that was not recorded, or at least not posted. However, you should understand that both CQRS and Event Sourcing (or Message Sourcing) are orthogonal to DDD. I discussion of explicitness has nothing to do with incorporating CQRS/ES. Yet, these are both mostly necessary patterns when using Actor Model. I'd say that you need to look at Martin's work on akka-persistence, (and/or Eventsourced) as it is the most complete expression of this in Scala and Akka to date. While I basically agree I think there are a few differences in our approaches. Once I get the whole thing put together perhaps Martin would like to consider what I have to say. I imagine that he's got too much to deal with now anyway, so the timing will probably be better in a month or two.

Vaughn

[Andrew Easter]

Vaughn,

I've created a gist to express better my ideas for tweaking the code. It also takes into account Patrik's comments about exposing the context outside the actor.

https://gist.github.com/DrewEaster/7175926

To summarise:

• Registering an aggregate type now takes in a factory for creating actor Props for that type - this means aggregate actors are created without state (other than an id). Also means aggregateOf(...) just takes in a type and id (no Props).
• I've removed the need for the AggregateCacheWorker
• Changed the message protocol for Order to be more CQRS/ES "compatible" 

Can you see any flaw here? The absence of an AggregateCacheWorker simplifies the code and doesn't appear to lose any of the design intentions. Or have I missed the reasoning for the AggregateCacheWorker somewhere?

Andrew

[Andrew Easter]

BTW, thanks for the link - I'll definitely watch your presentation :-)

And, FWIW, I've been actively following akka-persistence recently - it's pretty much how I've found myself involved in this conversation.

Andrew

On Saturday, 26 October 2013 16:28:38 UTC-7, Vaughn Vernon wrote:

[Vaughn Vernon]

Here you go: https://vaughnvernon.co/?p=780

I'll get to the other important parts of the Akka grid vision ASAP,

Vaughn

[Martin Krasser]

On 27.10.13 01:28, Vaughn Vernon wrote:

If you haven't yet watched this presentation, I suggest doing so: http://skillsmatter.com/podcast/design-architecture/vaughn-vernon

It all includes the use of CQRS and what I'd call "message sourcing" as opposed to "event sourcing."  I gave a much more complete presentation at the recent NYC DDD eXchange, but I think that was not recorded, or at least not posted. However, you should understand that both CQRS and Event Sourcing (or Message Sourcing) are orthogonal to DDD. I discussion of explicitness has nothing to do with incorporating CQRS/ES. Yet, these are both mostly necessary patterns when using Actor Model. I'd say that you need to look at Martin's work on akka-persistence, (and/or Eventsourced) as it is the most complete expression of this in Scala and Akka to date. While I basically agree I think there are a few differences in our approaches. Once I get the whole thing put together perhaps Martin would like to consider what I have to say.

Of course I want - thanks for the link :-) It's on my list.

I imagine that he's got too much to deal with now anyway, so the timing will probably be better in a month or two.

Vaughn


On Friday, October 25, 2013 8:55:36 PM UTC-6, Andrew Easter wrote:

Sorry, pressed send too early!!!

========================

Vaughn,

Whilst I totally understand the case you are making, I'm still not really sure that I fully share your concerns with regard to lack of explicitness in some of the proposals in this thread. We've not really talked about CQRS explicitly in this thread, but I was under the impression (maybe wrongly) that the typically async nature of command dispatch fits quite nicely with the idea of just firing off a command and trusting it will be processed by the correct aggregate. On the other hand, I suppose one could reasonably argue that the lack of explicitness in that mechanism is not something one should be content with.

I do get the argument that this approach exposes the presence of application services to the client, but I guess I don't really see this as such a smell as you do!

Thanks for an interesting read - it does seem like you are on a good path with your prototype.

Andrew

On Friday, 25 October 2013 19:49:57 UTC-7, Andrew Easter wrote:

Vaughn,

Whilst I totally understand the case you arI'm still not really sure that I fully share your concerns with regard to lack of explicitness in some of proposals in this thread. We've not really talked about CQRS explicitly in this thread, but I was under the impression (maybe wrongly) that the typically async nature of command dispatch fits quite nicely with the idea of just firing off a command and trusting it will be processed by the correct aggregate.

On Friday, 25 October 2013 14:20:28 UTC-7, Vaughn Vernon wrote:

Okay, this is what I've been talking about: https://vaughnvernon.co/?p=770

Vaughn

--

>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/groups/opt_out.

-- 
Martin Krasser

blog:    http://krasserm.blogspot.com

[Andrew Easter]

Vaughn,

Please don't take this the wrong way, but is it merely coincidental that some of your changes appear similar to my suggestions? Whilst self promotion is not particularly important to me, it's still nice to be acknowledged where one's input has been valued :-)

Anyhow...a couple of comments on your changes:

• Do you have some future requirement in mind given your decision to register aggregate ids with an AggregateCache using a message (RegisterAggregateId)? I'm struggling to understand the reasoning behind wanting to restrict what aggregate ids an AggregateCache supports. At least in the scope of this simple prototype, it seems to be a superfluous feature/design.
  
• I'm not sure I'm a massive fan of passing around class names as Strings - it doesn't feel very clean/typesafe.

I've made some tweaks to my gist:

https://gist.github.com/DrewEaster/7175926

Of particular note:

• AggregateType (trait) is now a metadata provider - implementations define the concrete Aggregate types that exist in the domain.
• The DomainModel now takes in a collection of AggregateTypes on creation rather than supporting ad-hoc registration after construction.
• Using Prop.create() to create Props from a class - new Props(Class) is deprecated.

I think these changes provide a cleaner design than passing round class names as strings. It also makes construction of the DomainModel quite pretty, e.g:

val model = DomainModel("my-domain") {
    Order,
    Customer,
    Payment
}

I'm quite interested to see your Akka grid vision. Supporting this type of design (e.g. single actor per aggregate instance) in a clustered environment is definitely not a trivial problem to solve.

Andrew

[Justin du coeur]

On Fri, Oct 25, 2013 at 12:45 AM, Andrew Easter <andrew...@gmail.com> wrote:

As Vaughn has clearly worked out quicker than the rest of us :-) it seems like the actor model just fits very well with domain driven design. It feels quite natural to me to consider an actor analogous to an aggregate root (in DDD speak). So, rather than just seeing an actor as a mediator sitting in front of a database, you see the actor conceptually as the entity. By incorporating event-sourcing, the actor works on the basis of processing a command (representing some action) and outputting one or many events that represent the result of processing that command. At all times the actor encapsulates its current state, which can simply be seen as the result of a series of ordered events applied to it.

Oh, I get all that -- that's why this whole conversation brought me up short.  I wouldn't even be participating in this discussion if it wasn't clear that this approach has a lot to recommend it, and that akka-persistence and event-sourcing are potentially a very appealing way to handle my needs.  I rather wish I had cottoned to this line of development earlier: if I'd realized this was coming down the pike, I might have factored it into my plans from the start.

 

Initially, it's actually quite helpful to ignore persistence requirements altogether and just rely on the state within the actor - in this way you don't allow persistence concerns to influence the way you design your domain - I often find that domain models end up messy because, despite attempts to avoid doing so, persistence concerns end up, in one form or another, leaking into the business logic. Any approach to building a domain layer that can genuinely remove the need to consider persistence technologies is an attractive proposition in my mind!  

Actually, that's a bit of a non-sequiteur in my case, because the word "domain" applies poorly.

The thing is, what I'm building (Querki) isn't really an application in the conventional sense, it's a *platform*.  Specifically, it's a combination of three very idiosyncratic pieces -- a wiki platform, a programming language and a database -- each of which is formulated specifically around making it relatively easy to build simple runtime "applications" on top of them.  The objective is to create something that's simple enough for ordinary folks to use routinely, but still deep enough for power users to build moderately interesting (if rather constrained) apps.

So the thing is, my primary Actors aren't domain objects: they are, in a very literal sense, databases.  Very small, very strange databases, but databases nonetheless.  They don't contain any real domain logic -- the domain logic is what you build *in* the system I'm building.

That being the case, I *can't* ignore persistence: persistence is much of the point.  It may well be that akka-persistence will turn out to be good enough for my needs, but I can't take that question casually: the fine details of features and performance have wide-ranging implications for the system I'm building.  Before I jump on the bandwagon, I'm going to have to take an *extremely* acid look at all sorts of design issues, do a bunch of performance comparisons, and make sure that it all makes sense.

In practice, I suspect I'm going to adopt a wait-and-see attitude to akka-persistence for purposes of Querki.  I believe that right now isn't the time for me to start using it, for several reasons:

-- Querki is very nearly at Alpha already, and I really need to get it out the door.

-- Rewriting to use akka-persistence would be a fairly major project.

-- Akka 2.3 sounds like it is a ways from full release yet, and I prefer not to depend on pre-release code.

-- As mentioned in my previous email, it doesn't sound like the persistence layer has all of the introspection requirements I expect to need.  (That is, I will want to be able to examine and display the history, and to build a *copy* of a previous state.)  So the first release of akka-persistence *may* not suffice for me.  (I'm honestly not sure yet; I'll have to take a much deeper look to decide whether it could be made to work.)

That said, none of this is insuperable.  I think it's entirely plausible that akka-persistence will eventually be well-suited to my requirements.  So my main takeaway for the time being is that I need to factor my persistence layer away from the core code (which was already in the plans anyway, simply as a matter of good practice), so that replacing it down the line becomes a less onerous project.  And when I write my own history mechanism (in a couple of months), I should keep in mind that I am *essentially* writing an eventsourced layer, and do that with an eye towards possibly dropping in akka-persistence down the line -- basically, I should try to line up my API with that of akka-persistence, possibly with enhancements if I need them...

[Zoop]

On Monday, October 7, 2013 2:17:02 PM UTC-4, Patrik Nordwall wrote:

Hi Andrew,

On Mon, Oct 7, 2013 at 2:54 PM, Andrew Easter <andrew...@gmail.com> wrote:

I'm currently doing some prototyping using Akka to implement the Domain Driven Design (DDD), CQRS and Event Sourcing patterns. I'm not sure whether I'm barking up the wrong tree, but my general preference would for the system to work something like this:

• Some command comes in to modify a given Aggregate Root (AR) entity
• If an Actor representing the AR is not already running in the system, create it _atomically_ (i.e. make sure you never have more than one Actor representing a specific entity)
• Actors should be removed after a period of idleness. This will avoid keeping actors around for entities with low interaction rates, but ensure 'hot' entities are kept in memory for performance (starting a new actor requires loading events from an event store in order to restore the memory image representing the entity state)

That is a sane design. You probably also want to look at akka-persistence.

Actors don't consume much more than memory when they are idle, but that can be a good enough reason to prune them.

 

I'm wondering if there are any best practices with regard to creating actors atomically whilst avoiding any performance overload during the lookup process - i.e. if an actor is already running, you'd ideally not incur any sort of lookup related performance penalty.

That is best handled by a parent actor. The entities can stop themselves when idle (e.g. by using ReceiveTimeout), and that will automatically remove them from the parent in a safe way.

 

Hi Patrick,

At the risk of hijacking the thread, could I ask if you have a pointer to example code that does this.  I'm new to Akka and I'm not certain what would be a good, efficient implementation for this.

Thanks!

[Patrik Nordwall]

I'm not sure what code you are looking for. Vaughn and Andrew are exploring some ideas, and their blog posts include some code.

setReceiveTimeout is described in the Actors section of the documentation.

/Patrik

 

Thanks!

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/groups/opt_out.

--

[Andrew Easter]

I'm intending to work on a more complete example that incorporates akka-persistence, timeouts etc.

Might turn out to be a pretty useful Typesafe Activator template in time :-)

Andrew

[Patrik Nordwall]

On Mon, Oct 28, 2013 at 5:56 PM, Andrew Easter <andrew...@gmail.com> wrote:

I'm intending to work on a more complete example that incorporates akka-persistence, timeouts etc.

Thank you Andrew.

 

Might turn out to be a pretty useful Typesafe Activator template in time :-)

Yes, that would be excellent. Let us know if you need any assistance, reviewing or whatever.

/Patrik

 

Andrew

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/groups/opt_out.

[Vaughn Vernon]

Hi Andrew,

No offense taken at all. And no I didn't even look at your gist. As soon as I read that you suggested using a factory I stopped reading. I hope you are not offended. That is definitely not the way I was headed, and I wasn't about to change my direction. I am not sure how you concluded that I somehow used your ideas.

As I said in my post I purposely left the example in a wrong state. I was running out of time as I went to see the Colorado Avalanche play on Friday night (awesome win BTW). Sorry I didn't have time to read your gist, nor the desire I must admit.

I'd also suggest one minor change to your blog post, which I did read this morning. I haven't even for one second thought that I was collaborating with you. If you could see the body of work that I have accomplished, in part over the past year while creating my own Actor Model toolkit, you'd probably understand where I am coming from. Only a small portion of my DDD Actor Model prototype is available here, which as I said I started just about 1 year ago:

https://github.com/VaughnVernon/ActorModel

Anyway, as I stated I will be blogging on this ASAP.

It's not that I would mind collaborating with you at some point. That would be great, but I do have my own ideas and direction far underway on this. I have been quite open with all my efforts. With dozens and dozens of favorites and retweets I haven't asked anyone to give me credit for what I have accomplished. In fact as you can read in my first blog post, all the credit for the AggregateCache mechanism goes to Roland and Patrick, where it belongs. It was the change in direction that I needed to move away from ActorRef and deadLetters-based reload that I already had working for the past two months, although I didn't realize that the actor mailbox to deadLetters change over could lose messages.

I do very much appreciate you starting this thread. Otherwise I would not have gotten Roland's and Patrick's feedback on my faulty approach to actor reload until later. I hope we can work together in the future.

Best,

Vaughn

[Andrew Easter]

Vaughn,

Clearly, I used the wrong terminology by referring to the term "factory" to describe what I was getting at. My point was EXACTLY what your subsequent changes represent, i.e. not passing in Props on creation of an instance of an aggregate (you just pass the class of the actor). My gist also made clear the concept of ensuring all state changes take place via message passing. Further, my changes demonstrated the removal of the AggregateCacheWorker. 

So, if you'd actually read the gist, you'd have seen how I concluded that your updates had been influenced by my ideas. I did update my gist after your follow up, and think there are some useful improvements in there if you can muster the desire to read it.

I accept I jumped the gun by assuming we were "collaborating". To me, and probably to others reading the thread, it appeared we were trading some ideas. Sorry for misinterpreting that - I will update my blog post accordingly so it's not misleading.

I'll probably continue to explore my own path with this stuff, but would like to thank you for changing my mind about explicit message passing between clients and aggregates. As you can tell from earlier in the thread, I wasn't so convinced in the beginning :-)

I don't take offense easily, so no need to worry about that. It certainly doesn't offend me to read "Sorry I didn't have time to read your gist, nor the desire I must admit." - I just doubt you'll win many admirers with that sort of pretentious attitude.

Andrew

[Roland Kuhn]

Andrew & Vaughn,

thanks a lot for this very fruitful discourse, everyone expanded their knowledge (myself included), and at least in that sense we sure are collaborating—that is what open source is all about and for Akka it works great! Of course all progress is driven by passion, and passion can stir strong emotions, but please let’s keep these discussions on a strictly technical level.

Regards,

Roland

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/groups/opt_out.

[Vaughn Vernon]

Hi Andrew,

LOL. I like your determination.

I will be speaking in London with SkillsMatter on this very subject in a few weeks (it's an "In the Brain of" event connected with my http://idddworkshop.com class). Can you attend? If you think you can I am sure I could get SkillsMatter to allow us to have an open discussion comparing our two approaches as part of my presentation: http://skillsmatter.com/event/agile-scrum/reactive-ddd-with-scala-and-akka

Vaughn

[Andrew Easter]

Vaughn,

Whilst I'm normally based in London, I'm in San Jose until Xmas. So, sadly, I won't be able to make it. Thanks for the suggestion, though.

Andrew

[jdon com]

Akka Actor maybe is not borned for Aggregate Root,   Actor's start and stop are stupid method, maybe a POJO is better fit for AR entity.

see java framework supports DDD+ Actor :https://github.com/banq/jdonframework

Jdon make normal pojo object act as Aggregate Root, no more process like start or stop.

在 2013年10月7日星期一UTC+8下午8时54分15秒，Andrew Easter写道：

• Some command comes in to modify a given Aggregate Root (AR) entity
• If an Actor representing the AR is not already running in the system, create it _atomically_ (i.e. make sure you never have more than one Actor representing a specific entity)
• Actors should be removed after a period of idleness. This will avoid keeping actors around for entities with low interaction rates, but ensure 'hot' entities are kept in memory for performance (starting a new actor requires loading events from an event store in order to restore the memory image representing the entity state)

I'm wondering if there are any best practices with regard to creating actors atomically whilst avoiding any performance overload during the lookup process - i.e. if an actor is already running, you'd ideally not incur any sort of lookup related performance penalty. This applies in a clustered environment as well as when running in a single JVM. 

I have seen documentation on the Cluster Singleton Pattern, but my interpretation of this was that each ClusterSingletonManager would need to know up front what actors needed to be singletons. In my suggested design, you only create the Aggregate Root actors on the fly and thus do not know up front.

Would be interested in opinions on this.

[Patrik Nordwall]

Hi Jdon, 

If you only want to promote your framework I have to ask you to do so somewhere else. Your statement is clearly uninformed.

Regards,

Patrik

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/groups/opt_out.

[Patrik Nordwall]

Andrew, Vaughn, and others, you are probably interested in my post on akka-dev: Cluster sharding for akka-persistence

Cheers,

Patrik

[Andrew Easter]

You bet I'm interested! Will take some time to digest this and give feedback.

Andrew