Akka Cluster: Routed partition ==> Router/Supervisor is SPOF?

[Sebastien Diot]

I just read through the Akka user Manual (Java version). I am trying to see if Akka can solve the problem I have. My current impression is that it doesn't and so I want to check my assumptions.

I want to have relatively heavy stateful actors, maybe 0.5MB state per actor. And I want them to be replicated in my cluster, such that each actor is replicated twice, and the death of any single host will not affect the clustered application. My assumption is that if the actors in a partition use the same code, start with the same state, loaded from the DB, and process the same events in the same order (don't know yet how to do that, but I assume it is possible), then they keep in sync. By adding a regular check (before saving to DB), I can discover corruption using some form of quorum algorithm.

The Akka cluster doc says partitions are rooted, which means the instances in a partition all receive their messages from a router, which, as far as I can tell, must be their supervisor too. Assuming this is correct, then it must mean that even if a partition is spread over the cluster, the supervisor is physically located in one host, and when that host goes down, all instances in the partition become orphaned and are killed, making the supervisor a SPOF for the partition.

Am I missing something here?

[Roland Kuhn]

Hi Sebastien,

you are raising an excellent point, and I cannot give a definitive answer now because the precise semantics of clustered actors have not yet been defined. Looking at it from the bottom up, I could well imagine the parent reference to be clustered as well, meaning that node failure and actor failure could both be handled in the same way. But there might be other constraints I overlook: we'll see when we flesh out the full system. You can be sure that we will realize the most powerful semantics which are still sane ;-)

Regards,

Roland Kuhn

Typesafe — The software stack for applications that scale

twitter: @rolandkuhn

--
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To view this discussion on the web visit https://groups.google.com/d/msg/akka-user/-/fPTaANGIZC4J.
To post to this group, send email to akka...@googlegroups.com.
To unsubscribe from this group, send email to akka-user+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.

[Roland Kuhn]

Hi Honnix,

you can already share routees between routers by creating them with a routee list instead of a number of routees to be created. The routers would not know about each other, of course, but you could route to the same targets in a scalable or redundant fashion. A similar scheme will be put in place in the cluster-aware version, but there we can actually change the routees on the fly if you add/remove cluster nodes.

Regards,

Roland

30 dec 2012 kl. 22:41 skrev Honnix Liang:

Hi Roland,

I ran into the same question discussed here.

Say I have a router named "MyRouter" which is configured exactly the same on two different nodes (physically or different JVMs), wouldn't it be good that there is some way I can configure the two routers knowing each other, which means two routers share the same group of routees.

Cheers,

Honnix

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---

You received this message because you are subscribed to the Google Groups "Akka User List" group.

To post to this group, send email to akka...@googlegroups.com.
To unsubscribe from this group, send email to akka-user+...@googlegroups.com.

Visit this group at http://groups.google.com/group/akka-user?hl=en.
 
 

Dr. Roland Kuhn
Akka Tech Lead
Typesafe – The software stack for applications that scale.
twitter: @rolandkuhn

[Evan Chan]

I have a similar problem.   For now, what I'm doing is creating a custom router that encapsulates the type of inner router desired, and can in addition take messages that add or remove routees.   These custom routers are distributed and any addition or removal of routees is broadcast to all the custom routers in the cluster.

A slightly easier way to handle this would be to have a cluster router which routes to a fixed coordinator actor on each node, which is then responsible for forwarding messages to the correct local routees.

-Evan

--
--
Evan Chan
Senior Software Engineer | 
e...@ooyala.com | (650) 996-4600
www.ooyala.com | blog | @ooyala

[Hongxin Liang]

Yes Samuel, this is also my concern.

What I need is a configurable router that can be aware of other routers with exactly the same configuration across the whole cluster, and all these routers share the same group of routees which are totally managed by the routers. Then from the client side, I can just submit any job to my local router and it will take care of distribution.

Of course I can write a custom router to achieve this as described by Evan, but I really think this semantics makes sense and it could be better a off-the-shelf feature provided by Akka.

/ Honnix

On W2, Wednesday, January 9, 2013 at 3:32 AM, Samuel Gendler wrote:

On Tuesday, January 8, 2013 9:04:48 AM UTC-8, rkuhn wrote:

Hi Honnix,

you can already share routees between routers by creating them with a routee list instead of a number of routees to be created. The routers would not know about each other, of course, but you could route to the same targets in a scalable or redundant fashion. A similar scheme will be put in place in the cluster-aware version, but there we can actually change the routees on the fly if you add/remove cluster nodes.

If I read the clustering documentation correctly, if you configure a cluster-aware router to use named actors that have already been created, then you are limited to only a single actor per node.  You can only have multiple actors per node if you allow the router to manage the routees, which means you can't route to the same routees from anywhere in the cluster unless you are comfortable with being limited to only a single routee per node (or you cascade route, with the cluster routing to one actor per node and then that actor in turn routing to to one of many actors on that node, but then you lose the abilty to transition an actor with all of its pending messages from one node to another).

To quote the clustering documentation:

"When using a router with routees looked up on the cluster member nodes, i.e. the routees are already running, [...] nr-of-instances defines total number of routees in the cluster, but there will not be more than one per node. "

"When using a router with routees created and deployed on the cluster member nodes [...] nr-of-instances defines total number of routees in the cluster, but the number of routees per node, max-nr-of-instances-per-node, will not be exceeded. Setting nr-of-instances to a high value will result in creating and deploying additional routees when new nodes join the cluster."

So it seems that we currently have to choose between having many actors per node (and distributing them around the cluster as needed) or routing to a single pool of actors from all nodes in the cluster.  I'm really hoping I'm wrong on this point, as that limitation really hampers my ability to use akka to solve the problem I'm trying to solve.  Any clarification you can offer would be much appreciated.

[Patrik Nordwall]

It's great that you report what you need. Thanks.

I think the ticket for this exists: https://www.assembla.com/spaces/akka/tickets/2676#/activity/ticket:

The background to the choice of "there will not be more than one per node" for routers that lookup routees is to keep the configuration simple. You define the path of the routees, and there can only be one such path per node (ActorSystem instance).

The routee can create children that perform the actual work.

Regards,

Patrik

--

Patrik Nordwall
Typesafe -  The software stack for applications that scale
Twitter: @patriknw

[Patrik Nordwall]

Hi all,

This is a very old thread, but I wonder if the cluster router questions/requests that was brought up here are solved by these two pull requests:

- #3669 Possibility to define routing logic in DistributedPubSub

- #3627 Cluster router group with multiple paths per node

/Patrik

Patrik Nordwall
Typesafe -  Reactive apps on the JVM
Twitter: @patriknw

[√iktor Ҡlang]

Awesomely done, Patrik!

I've wanted this done for soooo long :)

To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.

To post to this group, send email to akka...@googlegroups.com.

Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/groups/opt_out.

--

Viktor Klang
Director of Engineering

Typesafe

Twitter: @viktorklang

[Patrik Nordwall]

On Wed, Oct 16, 2013 at 2:02 PM, √iktor Ҡlang <viktor...@gmail.com> wrote:

Awesomely done, Patrik!

I've wanted this done for soooo long :)

Thanks, the context of that is not only those two pull requests, but also the big one #3549 Simplify and enhance routers

The documentation is online: http://doc.akka.io/docs/akka/snapshot/index.html

/Patrik

[√iktor Ҡlang]

That's the one I was mainly thinking of :)