Problem with parallel creation of top-level actors

[Mathias]

HAkkers,

we might have discovered a problem with top-level actor creation, but I'd like to get your opinion on things before filing an issue:

When you quickly create a large number of top-level actors it appears as if the blocking `Await.result` calls in the `actorOf` implementations of `ActorSystemImpl` can hog _all_ available threads in the underlying dispatcher thereby starving the guardian of the execution resources required to actually perform the actor creation.

Check out this example:

object Main extends App {
implicit val system = ActorSystem()
val n = 100
println("Creating " + n + " actors in parallel ...")
Future.traverse((0 to n).toList) { i =>
Future(system.actorOf(Props(ctx => { case _ => })))
} onComplete { result =>
println(result.fold("Error: " + _, _ => "Finished ok"))
system.shutdown()
}
}

On my machine, on Akka 2.0.1 with the default config, this example will block for 20 seconds before coming back with an future timeout error.
When you reduce n to 10 everything works as expected.
The problem only appears with top-level actors, moving creation on level down from the ActorSystem into an ActorContext prevents the deadlock.

We can't seem to find any mechanism in the current code preventing the described starvation, so it might be a design choice or an omission.

Looking forward to your feedback on this!

Cheers,
Mathias

---
mat...@spray.cc
http://www.spray.cc

[Roland Kuhn]

Hi Mathias,

you are right, this is a limitation. To give you a bit of background, it arises from the hierarchical nature of actor systems in Akka 2.0, which requires actor creation to be done by the parent&supervisor for everything to work out fine. The design choice was to make system.actorOf() synchronous—which requires blocking—in order to retain consistent actor creation semantics when compared to context.actorOf().

Since the guardian actor is not configurable and runs on the default dispatcher, there is nothing which can be done on this front; this is intentional, as creating top-level actors should be rarely done (preferably at application start-up, and then with appropriate fan-out, i.e. hanging your hierarchy off of very few top-level nodes). You can basically create your own application guardian and then adopt the same principle, i.e. send a message containing Props and get back the ActorRef, with the semantics you desire.

In summary, this limitation is meant to encourage “properly” hierarchical actor systems, which is also a good idea considering that restarting/stopping many actors below the same supervisor goes through a bottle-neck (which is the supervisor’s mailbox).

Regards,

Roland

> --
> You received this message because you are subscribed to the Google Groups "Akka User List" group.
> To post to this group, send email to akka...@googlegroups.com.
> To unsubscribe from this group, send email to akka-user+...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.
>

Roland Kuhn
Typesafe – The software stack for applications that scale.
twitter: @rolandkuhn

[Mathias]

Roland,

I figured that it might be an intentional design choice but was thrown off by the (apparent) lack of documentation on this rather important limitation.

I think the section "Actor Best Practices" in the documentation chapter "Actor Systems" should point out clearly that the top-level is not meant to be the "working-level" in the actor hierarchy and that calls to `system.actorOf` are blocking (with the resulting limitations) as opposed to `context.actorOf`, which is non-blocking. (Just saw that the scaladocs on ActorRefFactory already contain the mentioning of `system.actorOf` being blocking, but would would still like to suggest a more prominent discussion of this point in the sphinx docs.)

Thanks for clearing this up for us!

Cheers,

Mathias

> To unsubscribe from this group, send email to akka-user+unsubscribe@googlegroups.com.

[Roland Kuhn]

Hi Mathias,

fixed in release-2.0 and master, thanks for reporting!

Regards,

Roland

To view this discussion on the web visit https://groups.google.com/d/msg/akka-user/-/BEqH9HrmezwJ.

To post to this group, send email to akka...@googlegroups.com.

To unsubscribe from this group, send email to akka-user+...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.

[Thomas Lockney]

When you say "fixed in release-2.0 and master," what specifically are you referring to? I ask because I just lost far more time than I'd like to admit due to this issue and I only just now found this discussion. I reviewed the 2.0.1 docs and have yet to find anywhere mentioning this rather big gotcha. I know it may seem obvious, but... it's not. 

Thanks,

Thomas

[Peter Vlugter]

Hey Thomas,

These warnings were added after the 2.0.1 release. You can find them in the latest snapshot docs. Here's the commit on master (found with `git log -Sblocking`):

https://github.com/akka/akka/commit/0c4b2a11ae6cd2edf7dfbd3bacf51abcd37fa5d1

Cheers,
Peter

>> > To unsubscribe from this group, send email to akka-user+...@googlegroups.com.

>> > For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.
>> >
>>
>> Roland Kuhn
>> Typesafe – The software stack for applications that scale.
>> twitter: @rolandkuhn
>>
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups "Akka User List" group.
>> To view this discussion on the web visit https://groups.google.com/d/msg/akka-user/-/BEqH9HrmezwJ.
>> To post to this group, send email to akka...@googlegroups.com.

>> To unsubscribe from this group, send email to akka-user+...@googlegroups.com.

>> For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.
>
> Roland Kuhn
> Typesafe – The software stack for applications that scale.
> twitter: @rolandkuhn
>
>
>
> --
> You received this message because you are subscribed to the Google Groups "Akka User List" group.

> To view this discussion on the web visit https://groups.google.com/d/msg/akka-user/-/7eSt3JbFxtgJ.

> To post to this group, send email to akka...@googlegroups.com.

> To unsubscribe from this group, send email to akka-user+...@googlegroups.com.

[Thomas Lockney]

On Tue, May 22, 2012 at 7:55 PM, Peter Vlugter <pvlu...@gmail.com> wrote:
> Hey Thomas,
>
> These warnings were added after the 2.0.1 release. You can find them in the latest snapshot docs. Here's the commit on master (found with `git log -Sblocking`):
>
> https://github.com/akka/akka/commit/0c4b2a11ae6cd2edf7dfbd3bacf51abcd37fa5d1

Doh! Guess I should have checked the actual tags/commits. Just
glancing at it, I assumed 2.0.1 came after release-2.0.

[Roland Kuhn]

Hi Thomas,

sorry for the confusion, «master» and «release-2.0» are the currently active branches relevant for that question (didn’t exist on «release-1.3»), and we drop point releases along those as we go. I should have been less precise and more obvious in that mail you replied to.


Regards,

Roland Kuhn
Typesafe — The software stack for applications that scale

twitter: @rolandkuhn

[Thomas Lockney]

No worries - I was being a bit daft and assuming this has somehow
gotten lost in the shuffle without checking the actual commits. That
said, is it impossible to "backport" this doc change? I wonder how
many others have gotten mysteriously bitten by this issue and not
spoken up. Then again, maybe most people are smart enough to never do
anything crazy like creating tons of actors from the top level. ;~)

ps. Funny thing, it was actually via Mathias' awesome Spray library
that I hit this -- I was creating a simple HTTP load testing system
and thought "oh, I'll just keep this simple and create a whole bunch
of HttpClient actors" -- I had a top level actor, but was passing
those down to the individual "simulator" actors for each client device
I was trying to simulate. Lesson learned.

--
http://about.me/tlockney

[Johannes Rudolph]

Hi,

even if this will be resolved in the next version, we've hit the problem again: With the current implementation (< 2.1) Agents suffer from the same problem because they are always created directly under the ActorSystem so that concurrent creation of too many Agents will first lock and then eventually lead to timeouts.

Maybe the context should be configurable for Agents as well so that they can be created under actors as well.

Johannes

[√iktor Ҡlang]

On Wed, Jul 25, 2012 at 8:37 PM, Johannes Rudolph <johannes...@googlemail.com> wrote:

Hi,

even if this will be resolved in the next version, we've hit the problem again: With the current implementation (< 2.1) Agents suffer from the same problem because they are always created directly under the ActorSystem so that concurrent creation of too many Agents will first lock and then eventually lead to timeouts.

Maybe the context should be configurable for Agents as well so that they can be created under actors as well.

Yes, indeed. This will also be covered in: http://www.assembla.com/spaces/akka/tickets/2344

Any changes that can be made with binary compatibility preserve is candidate to go into 2.0.3

Since you see to have quite some experience using Agents, care to take a stab at it?

Cheers,

√

 

Johannes

On Monday, April 23, 2012 12:30:40 PM UTC+2, rkuhn wrote:

fixed in release-2.0 and master, thanks for reporting!

--

You received this message because you are subscribed to the Google Groups "Akka User List" group.

To view this discussion on the web visit https://groups.google.com/d/msg/akka-user/-/IZjXrJuv5UoJ.

To post to this group, send email to akka...@googlegroups.com.
To unsubscribe from this group, send email to akka-user+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.

--
Viktor Klang

Akka Tech Lead

Typesafe - The software stack for applications that scale

Twitter: @viktorklang

[√iktor Ҡlang]

What about something like this?

https://github.com/akka/akka/pull/583