Stash with a priority mailbox

[Andrew Easter]

My use case is to support stash along with a priority mailbox.

Stash requires deque semantics, but out the box, there is no akka mailbox implementation backed by a deque that also offers priority.

I realise that there is no PriorityBlockingDeque in Java. I can't seem to determine if there's any particular reason why not.

I would have thought it quite a common use case.

In my case, an actor enters a state (using become) whereby it begins to stash messages. On receipt of a particular message, the behaviour is reverted, and unstashAll() is called. It makes absolute sense for the message triggering the unstashAll() to take priority over any message that would be stashed. Without priority, there would be a completely unnecessary degradation in performance, especially in the case where an actor is being sent lots of messages.

Am I missing something obvious or will I have to implement this myself using a custom mailbox? Wanted to check first before pressing on :-)

[√iktor Ҡlang]

Hi Andrew,

You'll need to add a custom mailbox.

Cheers,
V

--
>>>>>>>>>>      Read the docs: http://akka.io/docs/
>>>>>>>>>>      Check the FAQ: http://akka.io/faq/
>>>>>>>>>>      Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/groups/opt_out.

[Patrik Nordwall]

Hi Andrew,

On Tue, Oct 8, 2013 at 12:51 PM, √iktor Ҡlang <viktor...@gmail.com> wrote:

Hi Andrew,

You'll need to add a custom mailbox.

Cheers,
V

On Oct 8, 2013 10:11 AM, "Andrew Easter" <andrew...@gmail.com> wrote:

My use case is to support stash along with a priority mailbox.

Stash requires deque semantics, but out the box, there is no akka mailbox implementation backed by a deque that also offers priority.

I realise that there is no PriorityBlockingDeque in Java. I can't seem to determine if there's any particular reason why not.

I would have thought it quite a common use case.

In my case, an actor enters a state (using become) whereby it begins to stash messages. On receipt of a particular message, the behaviour is reverted, and unstashAll() is called. It makes absolute sense for the message triggering the unstashAll() to take priority over any message that would be stashed. Without priority, there would be a completely unnecessary degradation in performance, especially in the case where an actor is being sent lots of messages.

I don't understand what you are trying to do. You already received the message that triggered unstashAll. Why would you need to give that message priority? Why not just continue processing it?

Regards,

Patrik

 

Am I missing something obvious or will I have to implement this myself using a custom mailbox? Wanted to check first before pressing on :-)

--
>>>>>>>>>>      Read the docs: http://akka.io/docs/
>>>>>>>>>>      Check the FAQ: http://akka.io/faq/
>>>>>>>>>>      Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/groups/opt_out.

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/groups/opt_out.

--

Patrik Nordwall
Typesafe -  Reactive apps on the JVM
Twitter: @patriknw

[Andrew Easter]

I don't think I did a great job of explaining the situation!

Let me try to break it down into a sequence of events:

1) Actor receives message representing a 'Command' in CQRS land

2) Actor creates an 'Event' representing the result of processing the command

3) Actor alters receive behavior using 'become'. The new behavior stashes further commands/messages - except for EventPersisted and FailedToPersistEvent (see below)

4) The event is sent asynchronously to an event store

5) In callback from writing to event store, actor sends 'self' a message to say either EventPersisted or FailedToPersistEvent

6) Actor handles EventPersisted or FailedToPersistEvent, reverts state using 'become' and unstashes all stashed messages

That is the general idea, anyway. My concern is that, the EventPersisted/FailedToPersistEvent messages do not have any priority over the messages that would be stashed. I know we are probably talking fractions of a second here, but it could be an issue if a large number of messages were sent to the actor between steps 4) and 5). The EventPersisted/FailedToPersistEvent messages _could_ be sat behind a bunch of other messages in the mailbox queue. The messages ahead of them would only be stashed anyway so why bother doing that when you know that the EventPersisted/FailedToPersistEvent messages would clear the stash? It seems to make more sense to allow them to be processed first.

I'm happy to accept that this is quite an edge case - most systems won't have that level of contention on a single actor. However, for example, what about if you did want a bounded mailbox? Surely the EventPersisted/FailedToPersistEvent messages should be processed regardless of whether the mailbox is full or not? This in itself is something to bear in mind - ensuring actor recognises when it's stuck in a state waiting for EventPersisted/FailedToPersistEvent!

I think the best answer might be stop trying to implement this myself and use akka-persistence ;-)

[Patrik Nordwall]

On Fri, Oct 11, 2013 at 12:53 PM, Andrew Easter <andrew...@gmail.com> wrote:

I don't think I did a great job of explaining the situation!

Let me try to break it down into a sequence of events:

1) Actor receives message representing a 'Command' in CQRS land

2) Actor creates an 'Event' representing the result of processing the command

3) Actor alters receive behavior using 'become'. The new behavior stashes further commands/messages - except for EventPersisted and FailedToPersistEvent (see below)

4) The event is sent asynchronously to an event store

5) In callback from writing to event store, actor sends 'self' a message to say either EventPersisted or FailedToPersistEvent

6) Actor handles EventPersisted or FailedToPersistEvent, reverts state using 'become' and unstashes all stashed messages

That is the general idea, anyway. My concern is that, the EventPersisted/FailedToPersistEvent messages do not have any priority over the messages that would be stashed. I know we are probably talking fractions of a second here, but it could be an issue if a large number of messages were sent to the actor between steps 4) and 5). The EventPersisted/FailedToPersistEvent messages _could_ be sat behind a bunch of other messages in the mailbox queue. The messages ahead of them would only be stashed anyway so why bother doing that when you know that the EventPersisted/FailedToPersistEvent messages would clear the stash? It seems to make more sense to allow them to be processed first.

Thanks for the description, now I understand.

I think this might be an premature optimization that I'm not sure that it will give better performance. You will not have less memory consumption. You will have a (much) slower mailbox by introducing the priority.

 

I'm happy to accept that this is quite an edge case - most systems won't have that level of contention on a single actor. However, for example, what about if you did want a bounded mailbox? Surely the EventPersisted/FailedToPersistEvent messages should be processed regardless of whether the mailbox is full or not? This in itself is something to bear in mind - ensuring actor recognises when it's stuck in a state waiting for EventPersisted/FailedToPersistEvent!

I think the best answer might be stop trying to implement this myself and use akka-persistence ;-)

Yes, I guess you have seen Martin's refinement of EventsourcedProcessor in akka-dev.

/Patrik

[Andrew Easter]

Thanks again for your feedback/input into my approach. I did think it smelled a bit of a premature optimization and, as you advise, it may have negligible (or no) benefits in the majority of cases. 

I had seen Martin's refinement and do foresee using that instead. However, it's been a useful exercise to try to solve this problem from scratch as it's given me a better understanding of CQRS/ES and of Akka :-)

[Patrik Nordwall]

11 okt 2013 kl. 14:28 skrev Andrew Easter <andrew...@gmail.com>:

Thanks again for your feedback/input into my approach. I did think it smelled a bit of a premature optimization and, as you advise, it may have negligible (or no) benefits in the majority of cases. 

I had seen Martin's refinement and do foresee using that instead. However, it's been a useful exercise to try to solve this problem from scratch as it's given me a better understanding of CQRS/ES and of Akka :-)

Great, please continue and give us feedback on akka-persistence. We want it to become awesome.

Thanks,

Patrik

[Jason Goodwin]

I think there are some cases where this makes sense.

If I have an actor that can restart when receiving some sort of message for state change - eg lets say active/passive store sends a message on a pubsub channel saying a failover has started (eg redis sentinel), then you probably want to unbecome and stash messages until the failover is complete. You would want to prioritize that state change message above all others in the hopes of reducing the number of failed messages.

I guess if you're timing out and retrying it's not a really strong motivation but that seems like a decent design decision to me.

If anyone has an example of a priority deque mailbox I would be interested in seeing how that is implemented. I'm assuming it's actually fairly simple to do - just mixing in a deque mailbox on a priority mailbox.

[Patrik Nordwall]

Yes, there are cases where it makes sense. I don't know of an implementation. It would be an interesting contribution.

/Patrik

--

>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/groups/opt_out.