Stateful actors an restarts
Derek Wyatt
This is going to be long... sorry.
Stateful actors are a problem, but they are also a necessary evil. We can try to cobble together a ton of stateless (or one-state) actors but the memory cost of doing so can be prohibitive. In my current case, holding state for 20 things costs me 80 bytes whereas creating 20 new actors is 12000 bytes (yeah, three orders of magnitude). Multiply that by millions, and you’ve got a problem.
When a stateful actor dies, its supervisor restarts it in its initialized state using the factory method that "actorOf" was given in the first place. This is a problem - a stateful actor has a lifecycle and it has, most likely, moved beyond its initialized state by the time it died. Messages in the mailbox are geared toward that new state.
How to solve this problem?
The Onion Layer. The classic method for solving this problem is to use the onion layer pattern - in short, don't let the stateful actor die. Anything you're going to do that's "risky" should be done in another actor that can die and have no state to lose.
The onion pattern works... but it's a pain. There are two main problems with it that I can see: 1) It can be very complicated to read and thus to reason about, and 2) the notion of "risky" must be understood and is fragile in the face of change.
On 1) – easy reasoning:
Actor {
// mutable state
self.faultHandler = OneForOneStrategy(List(classOf[Throwable]), 3, 1000)
def receive {
case Message1 =>
val parentActor = self
val a = actorOf(new Actor {
self.id = "something easier for debugging"
self.lifeCycle = Permanent
override def preStart = self ! 'go
receive {
case 'go =>
// set up some stuff
self ! DoSomethingRisky
case DoSomethingRisky =>
.. blah blah ..
if (good) parentActor ! RiskyThingSucceeded
else parentActor ! RiskyThingFailed
self ! PoisonPill
}
})
self.startLink(a)
case RiskyThingSucceeded =>
// mutate state
case RiskyThingFailed =>
// do something else
}
}
That's a lot of code to look at. Note that we've had to come up with some intra-actor protocol so that the state mutation can still be serialized in the parent actor. Note also that there's some inherent debugging annoyance here, not to mention the memory and performance overhead. While these issues aren't awful, they're not exactly arguments in favour of the pattern.
Now, we can also make use of the STM in order to eliminate the intra-actor protocol, and while that's an improvement, it doesn't help performance and it doesn't necessarily help clarity a ton.
And let’s not forget that all we’ve really done here is process “Message1” – generally we have actors process more than one message.
How much clearer is this?
Actor {
// mutable state
def receive {
case Message1 =>
// set up some stuff
// do 'risky' thing
if (good) mutate state
else do something else
}
}
(Note that "risky thing failed" doesn't mean crashed, just that it didn't work; normal behaviour.)
But, that's a very bad model because if Actor fails then its mutable state is lost.
On 2) - fragility:
What's risky? Today I can have a function that's behaving quite well that I can call from my actor. Later, someone modifies a call 4 levels deep into that function that I have no control over (and indeed don't even know is being called) that now throws an exception when the moon is full. I don't find out, until it happens on a customer site.
Summary
More than once, I've run up against "trivial" state that I'm trying to maintain, such as success / failure counts, message sizes, stats, or in my latest case, a list of strings.
The hoops one has to jump through to protect this trivial data is staggering. The Actor model is not my friend in these cases. If it weren't for the fact (and it is a fact) that it's an amazing friend for a thousand other things, this pain would make me question the paradigm when it claims to be “easy”.
We need something simpler. My suggest hasn't changed:
trait ActorRef ... {
...
def alternateRestartFactory: Option[() => Actor] = None
...
protected[akka] def restart(...) {
...
// pseudo
val freshActor = alternateRestartFactory().getOrElse(newActor)
...
}
}
This allows me to:
override def alternateRestartFactory: = {
if (mutableStateisOK)
Some(() => new MyActor(mutableState))
else
None
}
· If something happens due to fragility, I’m still covered.
· The business logic is left to doing business logic, not onion things
· What’s really happened is that the “old” actor should be wiped from history. The memory of its existence is not a feature – it’s a pain in the ass. But we can’t slide a new history into the ActorRef, such that restarting it would restart it the way we’d like, so we use this strategy to achieve the exact same thing.
The bottom line is that Akka should make simple things simple. (and complicated things simple :D)
Thanks,
Derek
This transmission (including any attachments) may contain confidential information, privileged material (including material protected by the solicitor-client or other applicable privileges), or constitute non-public information. Any use of this information by anyone other than the intended recipient is prohibited. If you have received this transmission in error, please immediately reply to the sender and delete this information from your system. Use, dissemination, distribution, or reproduction of this transmission by unintended recipients is not authorized and may be unlawful.
Gerolf Seitz
--
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To post to this group, send email to akka...@googlegroups.com.
To unsubscribe from this group, send email to akka-user+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.
--
Derek Wyatt
One could argue that returning the AnyRef leaks the abstraction. Your idea is very similar to mine, but if I had to argue it, I'd go for hiding the abstraction inside the factory method.
________________________________________
From: akka...@googlegroups.com [akka...@googlegroups.com] On Behalf Of Gerolf Seitz [gerolf...@gmail.com]
Sent: Tuesday, June 14, 2011 10:25 AM
To: akka...@googlegroups.com
Subject: Re: [akka-user] Stateful actors an restarts
Maybe it's just too much of a brute force approach, but wouldn't it be the easiest to have preRestart (which is called on the failed actor) return some arbitrary state (AnyRef), and supply that value as a parameter on postRestart (which is called on the fresh actor)?
Since the state doesn't have to be stored as a field anywhere, it's not a memory hit either.
I'm not sure if there are problems with this approach wrt. remote actors, so I might be missing a few pieces of the bigger pictures.
On Tue, Jun 14, 2011 at 1:37 PM, Derek Wyatt <dwy...@rim.com<mailto:dwy...@rim.com>> wrote:
This is going to be long... sorry.
Stateful actors are a problem, but they are also a necessary evil. We can try to cobble together a ton of stateless (or one-state) actors but the memory cost of doing so can be prohibitive. In my current case, holding state for 20 things costs me 80 bytes whereas creating 20 new actors is 12000 bytes (yeah, three orders of magnitude). Multiply that by millions, and you’ve got a problem.
When a stateful actor dies, its supervisor restarts it in its initialized state using the factory method that "actorOf" was given in the first place. This is a problem - a stateful actor has a lifecycle and it has, most likely, moved beyond its initialized state by the time it died. Messages in the mailbox are geared toward that new state.
How to solve this problem?
The Onion Layer. The classic method for solving this problem is to use the onion layer pattern - in short, don't let the stateful actor die. Anything you're going to do that's "risky" should be done in another actor that can die and have no state to lose.
The onion pattern works... but it's a pain. There are two main problems with it that I can see: 1) It can be very complicated to read and thus to reason about, and 2) the notion of "risky" must be understood and is fragile in the face of change.
On 1) – easy reasoning:
Actor {
// mutable state
self.faultHandler = OneForOneStrategy(List(classOf[Throwable]), 3, 1000)
def receive {
case Message1 =>
val parentActor = self
val a = actorOf(new Actor {
self.id<http://self.id> = "something easier for debugging"
On 2) - fragility:
Summary
This allows me to:
To post to this group, send email to akka...@googlegroups.com<mailto:akka...@googlegroups.com>.
To unsubscribe from this group, send email to akka-user+...@googlegroups.com<mailto:akka-user%2Bunsu...@googlegroups.com>.
For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.
--
Gerolf Seitz
twitter: @gersei
code: github.com/gseitz<http://github.com/gseitz>
blog: www.gerolfseitz.com<http://www.gerolfseitz.com>