Journal recovery with Akka FSM

[Stephen Stewart]

Hi all,

We've come across an interesting problem and wondered if anyone could shed some light on the best way to handle it ....

We have a system which has many FSM Actors, each of which moves through a couple of different states : Created, Active, Inactive, Finished. Inside each state there are a variety of Events which the Actor will respond to - many of these Events require communication with child Actors of the FSM Actor and so we make use of the ask() syntax to return a Future with the responses we need. Based on these responses we may need to alter the Actor data. We've had to make use of a pattern of sending the Futures to the FSM itself and then defining a wrapper case class to hold the original event, the result and the original sender that the result should be sent to (derived from this discussion on the Akka mailing list - https://groups.google.com/forum/#!topic/akka-user/Ze_nkZe-lXU). So we transition from our current state to an AwaitingResult state, and once the event is processed in the AwaitingResult state we then move back to the state we came from.

This works (up to a point) but on journal recovery we find that we receive a lot of messages in our AwaitingResult state which would originally have been received in our other states - presumably as the journal is trying to feed the messages as quickly as it can to the FSM. It seems strange that this would be the only way for an Akka FSM to handle Futures - are we missing something?

Thanks,

Stephen

[Stephen Stewart]

I've created this gist to hopefully explain things slightly better - https://gist.github.com/woodenboat/6048218

[Martin Krasser]

Hi Stephen,

sorry for the late reply. I have very limited internet connection and time these days (on vacation). I'll try to take a closer look at your example this week.

--
You received this message because you are subscribed to the Google Groups "Eventsourced User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eventsourced...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 
Martin Krasser

blog:    http://krasserm.blogspot.com
code:    http://github.com/krasserm
twitter: http://twitter.com/mrt1nz

[Martin Krasser]

Hi Stephen,

here's a modified example that demonstrates how to persist FSMs that deal with "awaiting" states. Please note that this is only one out of several options how to solve the problem. The most important things to consider when using services (child actors) that return futures are:

- that service must be considered as "external service" (more on that in this blog post). Therefore you should use a channel when communicating with that service
- any reply message from the service (or returned future) must be journaled (i.e. self ! Message(result) instead of self ! result): This ensures that you keep *past* results in the journal which are required to consistently recover the FSM. In your example, the asynchronous/future results got recomputed and are sent to the FSM concurrently to the replayed messages. This was leading to the problem of receiving several messages in the "awaiting" state.
- in my example, I let the service (worker) confirm the receipt of the message (see Confirm trait). A further delayed confirmation could also be done by the FSM when receiving the result.

Furthermore, the example ensures that non-blocking recovery successfully completes before processing any new messages, using a simple buffer. Also recovery is delayed to receiving  the first new message. In your example, processing of new messages can occur while recovery is still running, which is probably not what you want.

Hope that helps.

Cheers,
Martin