Akka Cluster manual joining

[João Alves]

Hello,

I'm trying to use Akka Cluster but I wasn't able to join nodes manually. I'm getting the nodes via DNS service discovering, but when I got the address, they don't join themselves.

I'm using the function Cluster(context.system).join(AddressFromURIString(addr)), but nothing occurs. No MemberUp event or something.

Am I doing something wrong?

[Patrik Nordwall]

We need more information to be able to help. Any log output?

Note that the node that you join to must already be a cluster member, which means that for bootstrapping some node must join itself.

Regards,

Patrik

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/groups/opt_out.

--

Patrik Nordwall
Typesafe -  Reactive apps on the JVM
Twitter: @patriknw

[João Alves]

Okay. But it raises a question: If I'm building a blind cluster (i.e. no node knows how many nodes are in the system nor their addresses), how am I supposed to do that? 

If I join every initialized node with itself I get this message when I'm trying to join it to another node:

[INFO] [10/28/2013 22:24:21.953] [inesc-aa4all-gateway-akka.actor.default-dispatcher-4] [Cluster(akka://inesc-aa4all-gateway)] Cluster Node [akka.tcp://inesc-aa4a...@127.0.1.1:43229] - Trying to join [akka.tcp://inesc-aa4a...@127.0.1.1:43379] when already part of a cluster, ignoring

This happens because the node running on port 43229 is already in a cluster with itself, I suppose. So, what is the best way to accomplish what I'm doing?

[√iktor Ҡlang]

Every club needs to have a founding member.

Cheers,

√

Viktor Klang
Director of Engineering

Typesafe

Twitter: @viktorklang

[João Alves]

Ok, I see your point. It's solved. Thank you!

On Monday, October 28, 2013 10:39:03 PM UTC, √ wrote:

Every club needs to have a founding member.

On Mon, Oct 28, 2013 at 11:30 PM, João Alves <joaoq...@gmail.com> wrote:

Okay. But it raises a question: If I'm building a blind cluster (i.e. no node knows how many nodes are in the system nor their addresses), how am I supposed to do that? 

If I join every initialized node with itself I get this message when I'm trying to join it to another node:

[INFO] [10/28/2013 22:24:21.953] [inesc-aa4all-gateway-akka.actor.default-dispatcher-4] [Cluster(akka://inesc-aa4all-gateway)] Cluster Node [akka.tcp://inesc-aa4all-gat...@127.0.1.1:43229] - Trying to join [akka.tcp://inesc-aa4all-gat...@127.0.1.1:43379] when already part of a cluster, ignoring

[Patrik Nordwall]

The reason is to make sure that you can't form islands of isolated nodes.

You can also use the Cluster(system).joinSeedNodes and programmatically construct the list of seed nodes as described in the documentation: http://doc.akka.io/docs/akka/2.3-M1/scala/cluster-usage.html#Joining_to_Seed_Nodes

Then the first seed node is special, for the very same reason.

/Patrik

[João Alves]

Yes, I get the point but I think it should be possible to "merge" clusters. I mean:

Node 1 is connected with Node 2, 3 and 4.

Node 5 is in a cluster with itself and is discovered by one of the above nodes. It should be possible to merge clusters. It just makes sense to me.

On Tuesday, October 29, 2013 6:24:35 AM UTC, Patrik Nordwall wrote:

The reason is to make sure that you can't form islands of isolated nodes.

You can also use the Cluster(system).joinSeedNodes and programmatically construct the list of seed nodes as described in the documentation: http://doc.akka.io/docs/akka/2.3-M1/scala/cluster-usage.html#Joining_to_Seed_Nodes

Then the first seed node is special, for the very same reason.

/Patrik

On Tue, Oct 29, 2013 at 3:26 AM, João Alves <joaoq...@gmail.com> wrote:

Ok, I see your point. It's solved. Thank you!

On Monday, October 28, 2013 10:39:03 PM UTC, √ wrote:

Every club needs to have a founding member.

On Mon, Oct 28, 2013 at 11:30 PM, João Alves <joaoq...@gmail.com> wrote:

Okay. But it raises a question: If I'm building a blind cluster (i.e. no node knows how many nodes are in the system nor their addresses), how am I supposed to do that? 

If I join every initialized node with itself I get this message when I'm trying to join it to another node:

[INFO] [10/28/2013 22:24:21.953] [inesc-aa4all-gateway-akka.actor.default-dispatcher-4] [Cluster(akka://inesc-aa4all-gateway)] Cluster Node [akka.tcp://inesc-aa4all-gatewa...@127.0.1.1:43229] - Trying to join [akka.tcp://inesc-aa4all-gatewa...@127.0.1.1:43379] when already part of a cluster, ignoring

--

[√iktor Ҡlang]

On Tue, Oct 29, 2013 at 4:17 PM, João Alves <joaoq...@gmail.com> wrote:

Yes, I get the point but I think it should be possible to "merge" clusters. I mean:

Node 1 is connected with Node 2, 3 and 4.

Node 5 is in a cluster with itself and is discovered by one of the above nodes. It should be possible to merge clusters. It just makes sense to me.

Can you outline how it would work?

Cheers,

√

[João Alves]

Node 1 starts and clusters with itself (29 Oct 2013, 15:28:05).

Node 2 starts and clusters with itself (29 Oct 2013, 15:28:07).

Node 3 starts and clusters with itself (29 Oct 2013, 15:28:08).

Node 4 starts and clusters with itself (29 Oct 2013, 15:28:08).

Node 5 starts and clusters with itself (29 Oct 2013, 15:28:10).

• Node 1 discovers Node 2. The first created/larger cluster is maintained.  In this case, the size of both clusters is 1, so the first created is maintained. Node 2 disconnects from the cluster with itself and joins Node 1. 
• Node 3 discovers Node 4. Since node 3 and 4 have the same size and timestamp, both nodes should send a (multi-cast?) message and the first to arrive is maintained, the rest of the nodes should join it.
• Node 3 or 4 discover Node 5. The cluster {3,4} is larger than the {5}, so Node 5 joins {3,4}.
  

Are there any problem that I'm not seeing? 

[√iktor Ҡlang]

On Tue, Oct 29, 2013 at 4:38 PM, João Alves <joaoq...@gmail.com> wrote:

Node 1 starts and clusters with itself (29 Oct 2013, 15:28:05).

Node 2 starts and clusters with itself (29 Oct 2013, 15:28:07).

Node 3 starts and clusters with itself (29 Oct 2013, 15:28:08).

Node 4 starts and clusters with itself (29 Oct 2013, 15:28:08).

Node 5 starts and clusters with itself (29 Oct 2013, 15:28:10).

• Node 1 discovers Node 2. The first created/larger cluster is maintained.  In this case, the size of both clusters is 1, so the first created is maintained. Node 2 disconnects from the cluster with itself and joins Node 1. 

How does Node1 discover Node2? Who becomes the leader? What if Node2 "discovers" Node1 at the same time, who becomes the leader?

 

• Node 3 discovers Node 4. Since node 3 and 4 have the same size and timestamp, both nodes should send a (multi-cast?) message and the first to arrive is maintained, the rest of the nodes should join it.

One cannot assume anything about walltime in a distributed system. How does Node3 discover Node4? Multicast—generally does not work in the "cloud" and is too narrow for general solutions, (akka cluster does not use multicast).

[Roland Kuhn]

Hi João,

merging clusters in a general sense may work, but Akka cluster is not a generic cluster, it is specifically built for hosting Actors and thus supporting certain related features. There are many problems with joining such clusters, e.g.

• what if one of the nodes of cluster1 has been previously kicked out of cluster2, firing DeathWatch?
• what should happen for cluster singletons? cluster1 and cluster2 may each have one instance running, and we must ensure that there is no overlap
• cluster-aware routers which remote-deploy their routees: now you have twice the number of actors on each node, plus quite some confusion as to where the router heads live
• starting up the system is a mess if your user code needs to find out when it is fine to start your services (i.e. when “enough” of the cluster has already been joined together)
• you’ll be able to find a few more easily by walking in the neighborhood of these

For this reason we only allow a pristine ActorSystem to join a cluster, and only once. Everything else would have very complex behavior, which is both difficult to code and difficult to understand.

Regards,

Roland

Dr. Roland Kuhn
Akka Tech Lead
Typesafe – Reactive apps on the JVM.
twitter: @rolandkuhn

[João Alves]

@Viktor

Yes, I'm aware that Akka Cluster does not use multi-cast. Let me be clear here: the discovery part is up to the programmer. You can just assume that is a blackbox and you just get the available nodes from there. It does not influence the problem here.

Questions such as "who becomes the leader?" could be solved with leader election algorithms, right? Anyway, I think the Akka Cluster documentation should be more clear about this.

@Roland

Thank you for the explanation and the questions raised. Now I understand why is Akka Cluster built this way. I'll work out my solution other way.

Cheers

[√iktor Ҡlang]

On Tue, Oct 29, 2013 at 5:18 PM, João Alves <joaoq...@gmail.com> wrote:

@Viktor

Yes, I'm aware that Akka Cluster does not use multi-cast. Let me be clear here: the discovery part is up to the programmer. You can just assume that is a blackbox and you just get the available nodes from there. It does not influence the problem here.

Questions such as "who becomes the leader?" could be solved with leader election algorithms, right? Anyway, I think the Akka Cluster documentation should be more clear about this.

Akka does not use Leader Election.

Cheers,

√