Using ReceiveTimeout in a parent/child relationship

[Andrew Easter]

Hi,

As some of you are probably aware, I'm playing around with Akka and domain driven design. In another thread, we discussed the idea of a single actor per aggregate instance. We also discussed the idea of timing out an actor when it's not being used. 

I'm aware I can use ReceiveTimeout on a child. My thinking up to now, has been to then let the parent coordinator (which supervises all instances of a given aggregate) know via a StopMe message. The parent would then call context.stop(childRef) in order to stop the child. According to the docs, context.stop will immediately free up the actor name for further use. So, should the next message in the parent's mailbox be for the same aggregate instance, it will create a new actor and forward the message on to that - all seems good. However, maybe I'm mistaken, but I can see a possible race condition here. 

Calling context.stop() would allow the existing child to finish processing the current message it's working on (despite ReceiveTimeout, it's feasible it could have received another message) and thus, in theory at least, for a very short time you could have two actors, representing the same aggregate instance, alive at the same time. With akka-persistence (EventsourcedProcessor) in the mix, one of a few implications is that, the "dying" instance writes an event to the journal after the new instance has finished recovering events from the journal - the new instance could/would, therefore, be in an invalid state. 

Another issue in the above example is that, if there are further messages in the "dying" child's mailbox when calling context.stop(), these messages will be sent to dead letters when, in fact, you'd want them to be processed by the new child _before_ any "new" messages destined for it.

I'm sorry if I've explained this badly - is it at all clear what I'm getting at? If I'm not talking nonsense, can anyone suggest a better pattern that can:

1) Enforce the unique actor per aggregate instance guarantee

2) Ensure messages (commands) are not lost or delivered out of order

This is probably an edge case because, in most systems, an actor reaching its configured ReceiveTimeout will probably mean there really is a low chance of it receiving further messages within the window where the race condition exists.

Andrew

 

[Roland Kuhn]

Hi Andrew,

5 nov 2013 kl. 02:19 skrev Andrew Easter <andrew...@gmail.com>:

Hi,

As some of you are probably aware, I'm playing around with Akka and domain driven design. In another thread, we discussed the idea of a single actor per aggregate instance. We also discussed the idea of timing out an actor when it's not being used. 

I'm aware I can use ReceiveTimeout on a child. My thinking up to now, has been to then let the parent coordinator (which supervises all instances of a given aggregate) know via a StopMe message. The parent would then call context.stop(childRef) in order to stop the child. According to the docs, context.stop will immediately free up the actor name for further use.

No, it will not (and it cannot, as you note further down). You will need to watch() and wait for the Terminated, stashing all messages for that aggregate in the meantime.

So, should the next message in the parent's mailbox be for the same aggregate instance, it will create a new actor and forward the message on to that - all seems good. However, maybe I'm mistaken, but I can see a possible race condition here. 

Calling context.stop() would allow the existing child to finish processing the current message it's working on (despite ReceiveTimeout, it's feasible it could have received another message) and thus, in theory at least, for a very short time you could have two actors, representing the same aggregate instance, alive at the same time. With akka-persistence (EventsourcedProcessor) in the mix, one of a few implications is that, the "dying" instance writes an event to the journal after the new instance has finished recovering events from the journal - the new instance could/would, therefore, be in an invalid state. 

Another issue in the above example is that, if there are further messages in the "dying" child's mailbox when calling context.stop(), these messages will be sent to dead letters when, in fact, you'd want them to be processed by the new child _before_ any "new" messages destined for it.

I'm sorry if I've explained this badly - is it at all clear what I'm getting at? If I'm not talking nonsense, can anyone suggest a better pattern that can:

1) Enforce the unique actor per aggregate instance guarantee

2) Ensure messages (commands) are not lost or delivered out of order

This is probably an edge case because, in most systems, an actor reaching its configured ReceiveTimeout will probably mean there really is a low chance of it receiving further messages within the window where the race condition exists.

Think about it as if you were to tell some people how to do it.

C=Client

S=Supervisor

A=Aggregate

C->S: send this to A

S: create and watch A

S->A: here’s some work

...

A->S: I’m idling

(possibly) C->S->A: some more work

S->A: Okay, please go to sleep

(*)

A: stop myself

A->S: I’m done (i.e. Terminated())

After this S may create a new A and send it further work, possibly from the stash. This stashing could in principle be avoided by tracking outstanding work and simply not responding to “I’m idling” if there currently is some. At (*) S will need to stash fresh work for A in any case.

So, to be more precise: think of people who can only send mail to each other.

Regards,

Roland

Andrew

 

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/groups/opt_out.

Dr. Roland Kuhn
Akka Tech Lead
Typesafe – Reactive apps on the JVM.
twitter: @rolandkuhn

[Andrew Easter]

Thanks, Roland. I'm pleased you mentioned the stashing approach because, after sending my post, I began to flirt with that idea in my mind (wasn't sure whether I was barking up the wrong tree). I reasoned about it by thinking of the actors as human beings and how they might satisfy the desired behaviour in a real life situation. I know that many of the team have suggested thinking through problems in this way - it really works!

With regard to "context.stop will immediately free up the actor name for further use." - DOH! Turns out I was looking at some out of date 2.0 API docs. Idiot.

Thanks again.

Andrew

[Roland Kuhn]

Hi Andrew,

no worries. That must have been 2.0-M1, brings back fond memories of Davy Jones and his Locker (which implemented that particular feature). So sad that it didn't work out…

Regards,

Roland

Regards,

Dr. Roland Kuhn
Akka Tech Lead
Typesafe – Reactive apps on the JVM

twitter: @rolandkuhn