Akka FSM, persistence and timeouts

[Eugene Dzhurinsky]

Hello all!

I have to create the simple workflow execution engine, that supports timeouts in its states. The sample made-up definition and the sequence of the states could be as following:

1) Send the confirmation e-mail to the customer once

2) Await the confirmation event for 24 hours

3) If there is a timeout error - then send another e-mail to the customer with the reminder and wait for 48 hours

4) If there is the confirmation event - then complete the registration and send welcome e-mail

5) if there's another timeout from step 3 - then cancel the registration process and cleanup the database

I think that I could easily create 10K-100K of instances of the actors in the JVM - actors seems to be cheap, however I'm not sure

- how the timeouts for the states will be processed, could I jam into some bottleneck with the scheduled events processing? What is the recommended thread pool size for 10K of actors, ballpark value?

- how do I persist the states of the workflow actors and recover them with the correct timeouts, if for some reason actor dies on step 2 after 10 hours of waiting, and upon the restart I need to adjust the timeout to be not 24 hours, but 14 hours. I know that I can start FSM with any state and state data, perhaps I could also set the custom step timeout upon init of the FSM.

I think that I could create some sort of the database that may keep the state and state data of each of workflows actor, but may be I am reinventing the wheel and there's already some akka-scheduler contrib module I overlooked?

Thanks in advance!

[Tal Pressman]

Hi,

One possible solution using PersistentFSM would be to save the persist with the timestamp when it happened, and then when applying the event you could set a timer (using setTimer or system.scheduler) that would trigger the state change. This way, when the events are processed during recovery, the timer would be set properly.

Regarding the number of threads, there are a couple things:

1. Number of threads is unrelated to the number of actors. Threads are in charge of actually processing the messages, so what you have to consider is how many messages you have to process, and how long this processing is per message.
2. Assuming you don't do any blocking during message processing adding more threads will not improve your performance noticeably, as it will only cause more context switches. If you do end up in a place where you can't keep up with message processing, you would have to scale your system (up or out).

Hope this helps,

Tal

[Eugene Dzhurinsky]

Tal, thanks for the response!

As far as I know, when an actor is recovered through Akka Persistence - the whole set of events is being re-send to the actor, so it can recover the state. In my case, some states will have side-effects, like "send an e-mail" or "update the database". Obviously I don't want those events to be "replayed" by the new actor, I just need to

- set the state data as it was seen when actor was "terminated"

- set the timeouts (calculate the current time and recent state change time to set the appropriate timeout)

- set the state to start from.

So I need not only record the state data and the current state, but also the time when that state was changed.

Do I need to write some custom persistence model for that or I still can re-use Akka Persistence with some settings regarding the message replay policy?

As for the number of threads my concern is: I have 10000 of actors, and all of them have certain timeouts. Once they recover the state after the actor system restart - there might be some bottleneck in processing of the new incoming messages from the external system (e.g to start a new actor that serves another workflow and init it's states) and process the timeout events.

Thanks!

[Evgeny Shepelyuk]

Hello

We're trying to implement something similar (i.e. when Actor recovered, we want side-effects to not happen again).

During recovery, events are not applied immediately to internal state, but we're waiting until recovery is completed and then decide what to do next, based on latest replied event.

In fact, it's identical to cleaning journal upto latest event after each successful event storage.

понеділок, 26 вересня 2016 р. 17:55:27 UTC+3 користувач Eugene Dzhurinsky написав:

[Eugene Dzhurinsky]

Zhenya, thanks for the reply!

Actually at this point I see that there're no benefits on using of Akka Persistence, I could easily use some K/V storage to store the key as the workflow ID and value - the state of the actors, so upon recovery I could read from that storage and recreate the FSM actors with their states/timeouts set.