Multiple ActorSystems ('weight', uses, etc?)

[Glenn / devalias]

Heya,

So, most/pretty much everything I read says ActorSystems are heavyweight, and you should only use one unless you have a 'really good reason' to use more than one since they are 'heavy'.

I was wondering

• Just how heavy is an ActorSystem? 
• Conceivably how many could you run in a JVM and how much memory/threads/etc would it require? 
• Is there a commonly known 'weight' for an ActorSystem?

I've also been wondering about what would be a 'really good reason' to have multiple ActorSystems. The one topic that particularly came to mind was an overall 'network' of 'nodes' (though not necessarily in the clustered sense) where each node needs to maintain it's own sandboxed ActorSystem to prevent 'crosstalk' from another ActorSystem (except by well defined channels using remoting, receptionist, 'protected mode', and all the other fun things Akka provides in that regard)

Conceivably this model COULD be put into a conventional model (single ActorSystem + potentially a few more/cluster setup for remote deployment/etc), but then it seems like you would have to manually re-implement a lot of protections to attempt to isolate each 'node' from affecting one another. (eg context.parent.parent.parent, context.select(../../../../../../../*) or context.system.shutdown, etc)

Aside from the 'heaviness' of running these multiple ActorSystems, it raises a few more questions.

• In the 'normal actor' model, we can make use of remote deployment to easily farm these 'nodes' out to remote servers, possibly even lazily/on demand (not starting them till required), though it would seem there is no way to remotely 'deploy' an ActorSystem from within Akka (since it would need to 'deploy' into an ActorSystem)
  
• Can an ActorSystem be created 'inside' another ActorSystem (eg. an Actor creates a new ActorSystem) If so, how does this effect the lifecycle of the 'inner' ActorSystem? Will it die when the outer ActorSystem dies?
• Is clustering designed to allow arbitrary potentially untrusted ActorSystems to communicate? I would think it doesn't really fit the model, since it's possible that a rogue system in a cluster could break/interfere with gossip/etc.
• If clustering isn't the answer, is there a 'best practice' for disparate potentially untrusted ActorSystems to communicate/interact?

Lot's of ponderings and mind dumping here, so hopefully it makes enough sense to be followed.

Looking forward to your thoughts/insights! :)

- Glenn / devalias

[Heiko Seeberger]

Glenn,

Find my answers below.

Heiko

On 22 Aug 2014, at 05:03, Glenn / devalias <glenn...@wlpc.com.au> wrote:

Heya,

So, most/pretty much everything I read says ActorSystems are heavyweight, and you should only use one unless you have a 'really good reason' to use more than one since they are 'heavy’.

Where did you find that last statement? I’m asking because in order to use Akka actors you need at least one ActorSystem.

I was wondering

• Just how heavy is an ActorSystem? 

The heaviest part of an actor system is the dispatcher: Each actor system has at least one. The dispatcher essentially is the engine that makes the actors run. In order to do so it needs threads, usually obtained from a thread pool. The default dispatcher by default (default to the power of two) uses a fork-join thread pool with at least 8 threads.

There are other shared facilities, some in user space and some internal, e.g. the guardian actors, the event stream, the scheduler, etc. All of these need to be created and started.

• Conceivably how many could you run in a JVM and how much memory/threads/etc would it require? 

I don’t think that heaviness in terms of memory is an issue here. What really matters are the thread pools. As I said, per default an actor system creates a fork-join pool with at least 8 threads. If your application needs to bulkhead certain actors from others, e.g. because some of the actors are doing blocking stuff, you need multiple dispatchers, i.e. multiple thread pools.

• Is there a commonly known 'weight' for an ActorSystem?

I've also been wondering about what would be a 'really good reason' to have multiple ActorSystems.

One good reason that comes to my mind are tests which are executed in parallel. In my project that’s usually the case, I would say that the tests typically create more actor systems at one time than I have cores available, i.e. some 10.

Apart from that, I don’t think having multiple local actor systems, i.e. in one JVM or on one machine, is a common pattern. If you really want to scale up most efficiently, one actor system with one thread pool configured to the numbers of cores (assuming a blocking factor of zero) should give you the best results. If you then need to further scale, scale out, i.e. create actor systems on remote nodes.

The one topic that particularly came to mind was an overall 'network' of 'nodes' (though not necessarily in the clustered sense) where each node needs to maintain it's own sandboxed ActorSystem to prevent 'crosstalk' from another ActorSystem (except by well defined channels using remoting, receptionist, 'protected mode', and all the other fun things Akka provides in that regard)

Conceivably this model COULD be put into a conventional model (single ActorSystem + potentially a few more/cluster setup for remote deployment/etc), but then it seems like you would have to manually re-implement a lot of protections to attempt to isolate each 'node' from affecting one another. (eg context.parent.parent.parent, context.select(../../../../../../../*) or context.system.shutdown, etc)

Aside from the 'heaviness' of running these multiple ActorSystems, it raises a few more questions.

• In the 'normal actor' model, we can make use of remote deployment to easily farm these 'nodes' out to remote servers, possibly even lazily/on demand (not starting them till required), though it would seem there is no way to remotely 'deploy' an ActorSystem from within Akka (since it would need to 'deploy' into an ActorSystem)
  
• Can an ActorSystem be created 'inside' another ActorSystem (eg. an Actor creates a new ActorSystem) If so, how does this effect the lifecycle of the 'inner' ActorSystem? Will it die when the outer ActorSystem dies?

No

• Is clustering designed to allow arbitrary potentially untrusted ActorSystems to communicate? I would think it doesn't really fit the model, since it's possible that a rogue system in a cluster could break/interfere with gossip/etc.

There are security features: http://doc.akka.io/docs/akka/2.3.5/scala/remoting.html#Remote_Security

• If clustering isn't the answer, is there a 'best practice' for disparate potentially untrusted ActorSystems to communicate/interact?

Lot's of ponderings and mind dumping here, so hopefully it makes enough sense to be followed.

Looking forward to your thoughts/insights! :)

- Glenn / devalias

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[Glenn / devalias]

Keeping my responses in the thread for ease.

On Friday, 22 August 2014 15:32:10 UTC+10, Heiko Seeberger wrote:

Glenn,

Find my answers below.

Heiko

On 22 Aug 2014, at 05:03, Glenn / devalias <glenn...@wlpc.com.au> wrote:

Heya,

So, most/pretty much everything I read says ActorSystems are heavyweight, and you should only use one unless you have a 'really good reason' to use more than one since they are 'heavy’.

Where did you find that last statement? I’m asking because in order to use Akka actors you need at least one ActorSystem.

Probably bad phrasing/lack of grammar on my part. You should only use 1 ActorSystem, unless you have a really good reason to use multiple ActorSystem's, per logical application. That's from the doco/most blog posts/etc i've read across the net.

 

I was wondering

• Just how heavy is an ActorSystem? 

The heaviest part of an actor system is the dispatcher: Each actor system has at least one. The dispatcher essentially is the engine that makes the actors run. In order to do so it needs threads, usually obtained from a thread pool. The default dispatcher by default (default to the power of two) uses a fork-join thread pool with at least 8 threads.

There are other shared facilities, some in user space and some internal, e.g. the guardian actors, the event stream, the scheduler, etc. All of these need to be created and started.

• Conceivably how many could you run in a JVM and how much memory/threads/etc would it require? 

I don’t think that heaviness in terms of memory is an issue here. What really matters are the thread pools. As I said, per default an actor system creates a fork-join pool with at least 8 threads. If your application needs to bulkhead certain actors from others, e.g. because some of the actors are doing blocking stuff, you need multiple dispatchers, i.e. multiple thread pools.

Makes sense 

• Is there a commonly known 'weight' for an ActorSystem?

I've also been wondering about what would be a 'really good reason' to have multiple ActorSystems.

One good reason that comes to my mind are tests which are executed in parallel. In my project that’s usually the case, I would say that the tests typically create more actor systems at one time than I have cores available, i.e. some 10.

Apart from that, I don’t think having multiple local actor systems, i.e. in one JVM or on one machine, is a common pattern. If you really want to scale up most efficiently, one actor system with one thread pool configured to the numbers of cores (assuming a blocking factor of zero) should give you the best results. If you then need to further scale, scale out, i.e. create actor systems on remote nodes.

One place I have seen it mentioned as a (seemingly) common pattern is on the remoting security that you linked below, for isolating a 'protected internal' local ActorSystem and only allowing access over a couple of well defined, well secured channels (the external ActorSystem)

The one topic that particularly came to mind was an overall 'network' of 'nodes' (though not necessarily in the clustered sense) where each node needs to maintain it's own sandboxed ActorSystem to prevent 'crosstalk' from another ActorSystem (except by well defined channels using remoting, receptionist, 'protected mode', and all the other fun things Akka provides in that regard)

Conceivably this model COULD be put into a conventional model (single ActorSystem + potentially a few more/cluster setup for remote deployment/etc), but then it seems like you would have to manually re-implement a lot of protections to attempt to isolate each 'node' from affecting one another. (eg context.parent.parent.parent, context.select(../../../../../../../*) or context.system.shutdown, etc)

Aside from the 'heaviness' of running these multiple ActorSystems, it raises a few more questions.

• In the 'normal actor' model, we can make use of remote deployment to easily farm these 'nodes' out to remote servers, possibly even lazily/on demand (not starting them till required), though it would seem there is no way to remotely 'deploy' an ActorSystem from within Akka (since it would need to 'deploy' into an ActorSystem)
  
• Can an ActorSystem be created 'inside' another ActorSystem (eg. an Actor creates a new ActorSystem) If so, how does this effect the lifecycle of the 'inner' ActorSystem? Will it die when the outer ActorSystem dies?

No

Knocked up a quick little test last night, and I could certainly create a new ActorSystem from an actor within another ActorSystem. I haven't looked particularly deeply at it/what the ramifications are of it, but it looks as though it just created another 'high level'/'outside' ActorSystem, rather than any sort of nesting/etc (which makes sense) 

• Is clustering designed to allow arbitrary potentially untrusted ActorSystems to communicate? I would think it doesn't really fit the model, since it's possible that a rogue system in a cluster could break/interfere with gossip/etc.

There are security features: http://doc.akka.io/docs/akka/2.3.5/scala/remoting.html#Remote_Security

Those security features are specifically related to remoting, though i'm not sure how well they would fit in with the cluster model. For clustering to work, all nodes need to be able to talk, and to converge. If there was a rogue node then it could forge messages, or refuse to acknowledge others, and basically cause havoc within the cluster. For that reason I assume clustering is designed only for 'trusted' nodes (though I could be wrong here)

Those security features you mentioned are a lot of the reason that i've been pondering the multiple ActorSystems model in general. Particularly because you can then isolate/sandbox off a collection of actors, only providing access to their 'receptionist' actor from the outside. By setting untrusted mode you prevent external akka systems from 'deep selecting' actors beyond the receptionist, so you force all communication to go through there (at least until they get a message back with an ActorRef they can use to talk directly)

You get the 'sandboxed' effect by having each collection of related actors in their own ActorSystem. In this way, if a rogue actor within that system was to try and cause havoc (PoisonPill to an essential upper actor, system.shutdown, etc), the most damage it could do is to kill it's own ActorSystem (and because each of these ActorSystems uses the security features of untrusted mode, no sandboxed collection of Actors can negatively interfere with another sandboxed collection of Actors)

[Prakhyat Mallikarjun]

Hi Glen,

Was going through your post did not understand what you meant by 'protected mode'?

-Prakhyat M M

[Glenn Grant]

Hey Prakhyat,

By that if was referring to the 'untrusted' mode in remoting security (or whatever it's actually called)

- G

Sent from my iPhone

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---

You received this message because you are subscribed to a topic in the Google Groups "Akka User List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/akka-user/Dh55Mjv_vOU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to akka-user+...@googlegroups.com.

[Akka Team]

Hi Glenn,

One other reason to use a separate ActorSystem would be if you actually launch it in a second JVM, even though both systems are local. This might be useful in cases when you want to delegate dangerous jobs to the secondary system so even if it dies with OOM you still have the primary alive. Not a very common use case, but can be useful.

-Endre

You received this message because you are subscribed to the Google Groups "Akka User List" group.

To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.

To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Akka Team

Typesafe - The software stack for applications that scale

Blog: letitcrash.com
Twitter: @akkateam