Akka Cluster seed node problem

[Rohit Kumar]

Hi,

I am working with akka distributed worker template available on typesafe. I am using it to write a backend job which takes data from siebel using soap calls and inserts in mongo. This job is supposed to run once a week for few hours.

Based on the cluster-usage and other documentation on AKKA website, I imported akka-cluster.jar and configured the application configuration file with SEED nodes (akka.cluster.seed-nodes). But when I start the first node (MASTER NODE) with the configuration I mentioned (seed nodes etc), I start getting errors on the server console saying failed to join the seed node which is obvious (as it is the first node and there is nothing to join). Now I start the second node with akka.cluster.seed-nodes configured with the ip-address and port of the process where master node is running. I once again get the errors on the server console.

Now what I do next is - take the first join address of the master actor  from the MASTER NODE and set it dynamically in the slave node in the code (construct an Address object and pass it to the actors on the slave node). THIS WORKS!!! If I take the same join address and configure it in the application configuration akka.cluster.seed-nodes, it throws me error and slave doesn't join the cluster.

So I have following questions :-

1. How to configure the akka.cluster.seed-node configuration in application. I could never make it work/count in the configuration.

2. Is there any way to pre-configure the seed nodes in the configuration. As per me trying it out, it looks like the configuration is dynamic i.e. to take the join address of actor on the master node from the logs and configure the slave's seed-node configuration with that address ?

[Patrik Nordwall]

I'm not sure I understand your problem. You should used something like this:

Node 1:

akka.remote.netty.tcp.hostname = 127.0.0.1

akka.remote.netty.tcp.port = 2551

akka.cluster.seed-nodes = ["akka.tcp://Cluste...@127.0.0.1:2551", "akka.tcp://Cluste...@127.0.0.1:2552"]

Node 2:

akka.remote.netty.tcp.hostname = 127.0.0.1

akka.remote.netty.tcp.port = 2552

akka.cluster.seed-nodes = ["akka.tcp://Cluste...@127.0.0.1:2551", "akka.tcp://Cluste...@127.0.0.1:2552"]

Node 3:

akka.remote.netty.tcp.hostname = 127.0.0.1

akka.remote.netty.tcp.port = 2553

akka.cluster.seed-nodes = ["akka.tcp://Cluste...@127.0.0.1:2551", "akka.tcp://Cluste...@127.0.0.1:2552"]

When you run on multiple machines you should change hostname to the real host name or ip address instead of 127.0.0.1.

Read more here: http://doc.akka.io/docs/akka/2.3.3/scala/cluster-usage.html#Joining_to_Seed_Nodes

/Patrik

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Patrik Nordwall
Typesafe -  Reactive apps on the JVM
Twitter: @patriknw

[Rohit Kumar Gupta]

Patrik,

Thanks for the reply. I changed the configuration the same way you described. But i am getting the below error now. I started only master node and I am getting this error. Please note that I am using play framework and akka for the past 1 year. This is the first time I am using akka clustering. I have already gone through the link you mentioned. The error is well known but I don't know the solution for this error.

Any help will be appreciated.

Thanks,

Rohit

2014-06-30 15:44:52,667 INFO  [play-internal-execution-context-1] application play.Logger$ALogger info: The join address is - akka.tcp://Wor...@127.0.0.1:2551

2014-06-30 15:44:57,840 ERROR [New I/O worker #1] application play.api.LoggerLike$class error: 

! @6il7jdmec - Internal server error, for (GET) [/fase/v1/job/product?isMaster=true] ->

play.api.UnexpectedException: Unexpected exception[ChannelException: Failed to bind to: /127.0.0.1:2551]

at play.core.ReloadableApplication$$anonfun$get$1$$anonfun$apply$1$$anonfun$1.apply(ApplicationProvider.scala:148) ~[play_2.10.jar:2.2.3]

at play.core.ReloadableApplication$$anonfun$get$1$$anonfun$apply$1$$anonfun$1.apply(ApplicationProvider.scala:112) ~[play_2.10.jar:2.2.3]

at scala.Option.map(Option.scala:145) ~[scala-library-2.10.2.jar:na]

at play.core.ReloadableApplication$$anonfun$get$1$$anonfun$apply$1.apply(ApplicationProvider.scala:112) ~[play_2.10.jar:2.2.3]

at play.core.ReloadableApplication$$anonfun$get$1$$anonfun$apply$1.apply(ApplicationProvider.scala:110) ~[play_2.10.jar:2.2.3]

at scala.util.Success.flatMap(Try.scala:200) ~[scala-library-2.10.2.jar:na]

at play.core.ReloadableApplication$$anonfun$get$1.apply(ApplicationProvider.scala:110) ~[play_2.10.jar:2.2.3]

at play.core.ReloadableApplication$$anonfun$get$1.apply(ApplicationProvider.scala:102) ~[play_2.10.jar:2.2.3]

at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) ~[scala-library-2.10.2.jar:na]

at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) ~[scala-library-2.10.2.jar:na]

at scala.concurrent.forkjoin.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1361) ~[scala-library-2.10.2.jar:na]

at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) ~[scala-library-2.10.2.jar:na]

at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) ~[scala-library-2.10.2.jar:na]

at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) ~[scala-library-2.10.2.jar:na]

at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) ~[scala-library-2.10.2.jar:na]

Caused by: org.jboss.netty.channel.ChannelException: Failed to bind to: /127.0.0.1:2551

at org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:272) ~[netty.jar:na]

at akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:391) ~[akka-remote_2.10-2.2.3.jar:na]

at akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:388) ~[akka-remote_2.10-2.2.3.jar:na]

at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:253) ~[scala-library-2.10.2.jar:na]

at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:249) ~[scala-library-2.10.2.jar:na]

at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:29) ~[scala-library-2.10.2.jar:na]

at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67) ~[akka-actor_2.10-2.2.3.jar:2.2.3]

at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82) ~[akka-actor_2.10-2.2.3.jar:2.2.3]

at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59) ~[akka-actor_2.10-2.2.3.jar:2.2.3]

at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59) ~[akka-actor_2.10-2.2.3.jar:2.2.3]

at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72) ~[scala-library-2.10.2.jar:na]

at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58) ~[akka-actor_2.10-2.2.3.jar:2.2.3]

at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:42) ~[akka-actor_2.10-2.2.3.jar:2.2.3]

at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386) ~[akka-actor_2.10-2.2.3.jar:2.2.3]

... 4 common frames omitted

java.net.BindException: Address already in use

at sun.nio.ch.Net.bind(Native Method) ~[na:1.6.0_20]

at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:137) ~[na:1.6.0_20]

at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:77) ~[na:1.6.0_20]

at org.jboss.netty.channel.socket.nio.NioServerBoss$RegisterTask.run(NioServerBoss.java:193) ~[netty.jar:na]

at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:366) ~[netty.jar:na]

at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:290) ~[netty.jar:na]

at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42) ~[netty.jar:na]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.6.0_20]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.6.0_20]

at java.lang.Thread.run(Thread.java:636) [na:1.6.0_20]

 

[√iktor Ҡlang]

It says what's wrong-that the address is already in use.

Cheers,
V

--

[Rohit Kumar Gupta]

Hi,

Only the distributed process deployed in play is using port 2551. Even if I change the port, the error is same.

Any help will be appreciated.

Thanks,

Rohit

[√iktor Ҡlang]

Are you trying to create the same ActorSystem more than one time?

--

Cheers,

√

[Rohit Kumar Gupta]

Here is my Global.java which starts the ActorSystem. This happens once at the application startup. I removed multiple create calls and used to retrieve 'syste' from cache. But still I am getting the same error.

What is that I am doing wrong ?

 

/**

 * Global.java

 *

 * This class allows to handle global settings for the application.

 *

 * Copyright(c) 2014 Equinix, Inc.  All Rights Reserved.

 * This software is the proprietary information of Equinix.

 *

 */

import java.util.HashSet;

import java.util.Set;

import java.util.concurrent.TimeUnit;

import play.Application;

import play.GlobalSettings;

import play.Logger;

import play.libs.Akka;

import scala.concurrent.duration.Duration;

import scala.concurrent.duration.FiniteDuration;

import akka.actor.ActorRef;

import akka.actor.ActorSelection;

import akka.actor.ActorSystem;

import akka.actor.Address;

import akka.actor.PoisonPill;

import akka.actor.Props;

import akka.cluster.Cluster;

import akka.contrib.pattern.ClusterClient;

import akka.contrib.pattern.ClusterSingletonManager;

import akka.contrib.pattern.ClusterSingletonPropsFactory;

import play.cache.Cache;

import com.equinix.fase.job.product.master.Frontend;

import com.equinix.fase.job.product.master.Master;

import com.equinix.fase.job.product.master.WorkProducer;

import com.equinix.fase.job.product.master.WorkResultConsumer;

import com.equinix.fase.job.product.worker.ProductListActor;

import com.equinix.fase.job.product.worker.WorkExecutor;

import com.equinix.fase.job.product.worker.Worker;

import com.typesafe.config.Config;

import com.typesafe.config.ConfigFactory;

public class Global extends GlobalSettings {

private static String systemName = "Workers";

private static FiniteDuration workTimeout = Duration.create(120, "seconds");

@Override

public void onStart(Application application) {

//Address joinAddress = startBackend(new Address("akka.tcp", systemName, "127.0.0.1", 57779), "backend");

Address joinAddress = startBackend(null, "backend");

Cache.set("seedAddress",joinAddress);

Logger.info("The join address is - " + joinAddress);

try {

Thread.sleep(5000);

startBackend(joinAddress, "backend");

startWorker(joinAddress);

} catch (InterruptedException e) {

e.printStackTrace();

}

super.onStart(application);

}

public static Address startBackend(Address joinAddress, String role) {

Config conf = ConfigFactory.parseString(

"akka.cluster.roles=[" + role + "]").withFallback(

ConfigFactory.load());

ActorSystem system = ActorSystem.create(systemName, conf);

Address realJoinAddress = (joinAddress == null) ? Cluster.get(system)

.selfAddress() : joinAddress;

Cluster.get(system).join(realJoinAddress);

system.actorOf(ClusterSingletonManager.defaultProps(

                "active", PoisonPill.getInstance(), role,

                new ClusterSingletonPropsFactory() {

                    @Override

                    public Props create(Object handOverData) {

                        return Master.props(workTimeout);

                    }

                }), "master");

return realJoinAddress;

}

public static void startWorker(Address contactAddress) {

ActorSystem system = ActorSystem.create(systemName);

Set<ActorSelection> initialContacts = new HashSet<ActorSelection>();

initialContacts.add(system.actorSelection(contactAddress

+ "/user/receptionist"));

ActorRef clusterClient = system.actorOf(

ClusterClient.defaultProps(initialContacts), "clusterClient");

system.actorOf(

Worker.props(clusterClient, Props.create(WorkExecutor.class)),

"worker");

}

public static ActorRef startFrontend(Address joinAddress) {

ActorSystem system = ActorSystem.create(systemName);

Cluster.get(system).join(joinAddress);

ActorRef frontend = system.actorOf(Props.create(Frontend.class),

"frontend");

ActorRef producer = system.actorOf(

Props.create(WorkProducer.class, frontend), "producer");

system.actorOf(Props.create(WorkResultConsumer.class), "consumer");

return producer;

}

}

[√iktor Ҡlang]

Too much code for me to follow casually.
Are you calling "ActorSystem.create(systemName, conf);" with the same netty.host & port more than once? (it looks like you are)

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Cheers,

√

[Rohit Kumar Gupta]

ActorSystem system = ActorSystem.create(systemName, conf);

ActorSystem system = ActorSystem.create(systemName);

ActorSystem system = ActorSystem.create(systemName);

I am using create call at 3 places 1. while starting master(backend) 2. while starting worker 3. while starting frontend. Even if I make it only one place and use play.cache to cache it and reuse it across, it gives me the same error.

Any help ?

[√iktor Ҡlang]

Use different port configuration for the 3 different instances.

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Cheers,

√

[Rohit Kumar Gupta]

Here is my configuration.

akka {

    actor {

        provider = "akka.cluster.ClusterActorRefProvider"

    }

    remote {

        #log-remote-lifecycle-events = off

        netty.tcp {

            hostname = "127.0.0.1"

            port = 2551

        }

    }

    

cluster {

        seed-nodes = ["akka.tcp://Wor...@127.0.0.1:2551",

           "akka.tcp://Wor...@127.0.0.1:2552","akka.tcp://Wor...@127.0.0.1:2553"]

            auto-down-unreachable-after = 10s

        }

    

    

    extensions = ["akka.contrib.pattern.ClusterReceptionistExtension"]

} 

How should I change it to configure multiple ports as I am getting the same error as below.

! @6il8mjhhg - Internal server error, for (GET) [/fase/v1/job/product?isMaster=true] ->

2014-06-30 19:28:28,145 INFO  [Workers-akka.actor.default-dispatcher-5] application play.Logger$ALogger info: Message in master is instance of - CleanupTick

[√iktor Ҡlang]

Change the config of akka.remote.netty.tcp.port in a config that you pass to the respective ActorSystem.create

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Cheers,

√

[Rohit Kumar Gupta]

I couldn't get your answer. Could you please explain in detail ?

Thanks,

Rohit 

[√iktor Ҡlang]

I believe I have given enough information for you to solve this problem,

if you require more info, I kindly ask of you to read the documentation: http://doc.akka.io/docs/akka/2.3.3/scala/remoting.html

--

>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Cheers,

√

[Rohit Kumar Gupta]

Thanks for all you help.

Bye.

-Rohit 

[Patrik Nordwall]

hint: ConfigFactory.parseString(...).withFallback(...)

I think that is used in several of our cluster related activator templates.

/Patrik

--

>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

[Rohit Kumar Gupta]

Now all three processes are started (master,worker & frontend) on three different ports. But I am getting the below error.

[INFO] [07/01/2014 00:07:18.695] [Workers-akka.actor.default-dispatcher-4] [akka://Workers/system/cluster/core/daemon/joinSeedNodeProcess] Message [akka.cluster.InternalClusterAction$InitJoinAck] from Actor[akka.tcp://Wor...@172.16.144.66:2552/system/cluster/core/daemon#-450702087] to Actor[akka://Workers/system/cluster/core/daemon/joinSeedNodeProcess#1270182348] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.

[INFO] [07/01/2014 00:07:18.699] [Workers-akka.actor.default-dispatcher-7] [akka://Workers/system/cluster/core/daemon/joinSeedNodeProcess] Message [akka.cluster.InternalClusterAction$InitJoinAck] from Actor[akka.tcp://Wor...@172.16.144.66:2551/system/cluster/core/daemon#-1993872094] to Actor[akka://Workers/system/cluster/core/daemon/joinSeedNodeProcess#1270182348] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.

This is causing the frontend to not submit job to the master and nothing is happening.

[Rohit Kumar Gupta]

Patrik,

Do you have answer for the below problem ? Any help will be greatly appreciated.

Thanks,

Rohit

On Monday, June 30, 2014 9:42:41 PM UTC+5:30, Rohit Kumar Gupta wrote:

Now all three processes are started (master,worker & frontend) on three different ports. But I am getting the below error.

[INFO] [07/01/2014 00:07:18.695] [Workers-akka.actor.default-dispatcher-4] [akka://Workers/system/cluster/core/daemon/joinSeedNodeProcess] Message [akka.cluster.InternalClusterAction$InitJoinAck] from Actor[akka.tcp://Workers@172.16.144.66:2552/system/cluster/core/daemon#-450702087] to Actor[akka://Workers/system/cluster/core/daemon/joinSeedNodeProcess#1270182348] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.

[INFO] [07/01/2014 00:07:18.699] [Workers-akka.actor.default-dispatcher-7] [akka://Workers/system/cluster/core/daemon/joinSeedNodeProcess] Message [akka.cluster.InternalClusterAction$InitJoinAck] from Actor[akka.tcp://Workers@172.16.144.66:2551/system/cluster/core/daemon#-1993872094] to Actor[akka://Workers/system/cluster/core/daemon/joinSeedNodeProcess#1270182348] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.

[Patrik Nordwall]

It is not an ERROR. It is logged at INFO level. Do you see in the log if the nodes have joined successfully? You should see things like 

"Node ... is JOINING"
"Welcome from ..."

""Leader is moving node..."

/Patrik

--

>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[Rohit Kumar Gupta]

Hi Patrik,

I can see all those messages now. It is working :). Thanks for all your help.

Regards,

Rohit

[Patrik Nordwall]

Thanks for the update. Glad to hear that you got it working.

/Patrik

--

[Rohit Kumar Gupta]

Hi Patrik,

The basic cluster things are working (as per the distributed akka workers template). But few things (may be advanced) are not working.

I have akka cluster consisting of two different m/c. Each m/c hosts once master and one worker process running on different ports. 

When I start my frontend process (on one of the nodes), it starts submitting the jobs to the master process. 

First master process is picked. Master process is a singleton in the cluster so only one instance of it can run at any point of time in the cluster.

First master process starts processing the jobs and sends them to workers on both m/c. There is no problem so far. Clustering is working.

But when I pull down this master process (kill it), then ideally other master process running on different m/c should come into picture as this is also running and part of the cluster. 

But that is not happening, and frontend can no longer able to submit the job to the master process.

 It keeps on complaining about the original master process that it is unreachable. Could you tell me hhow to solve it ? Any help will be very much appreciated.

[Rohit Kumar Gupta]

Hi,

Does akka treat loopback address (127.0.0.1) and actual ip address differently. If I run the cluster on the same m/c (different processes) it seems I have to use one or the other for consistency (in the akka remote configuration) else things do not work.

This is causing the configuration file to be different for different m/c when I open the cluster to other boxes.

Any help will be appreciated.

Thanks & Regards,

Rohit

[Martynas Mickevičius]

Hi Rohit,

how do you reference the back-end actor from the front-end? If you hold an ActorRef, then after the back-end machine is stopped, the ActorRef still points to that non existent actor.

To solve this you should use a cluster membership service (which is what Akka Cluster provides) to look-up cluster member you want to send the message to. Otherwise you can use Cluster Sharding which solves this particular issue. With Cluster Sharding you reference actors by a logical identifier and sending message to that identifier will send it to the desired actor on some node in the cluster.

--

>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Martynas Mickevičius

Typesafe – Reactive Apps on the JVM

[Rohit Kumar Gupta]

Hi Martynas,

 Thanks for the reply. 

Both master and workers use distributed pub-sub to send receive messages. There is a distributed pub-sub mediator which acts as a mediator and all the messages are sent to it. Master and workers subscribe to it. So no direct ActorRef is held by anyone. Master uses the mediator and Workers use the clusterClient to send the messages. You can have a look at the akka-distributed-worker template available on Typesafe.  

If you see the below code, logical path have been used to send the message to master. So if one master goes down, another should kick in automatically, as I have two master processes running in the cluster.

Future<Object> f =

      ask(mediator, new Send("/user/master/active", message, false), new Timeout(Duration.create(5, "seconds")));

Any help will be very much appreciated. 

Thanks in advance,

Rohit

[Rahat Hossain]

Hi Rohit,

Just a thought, as you mentioned earlier "Each m/c hosts once master and one worker process running on different ports. "

What about running a test, such as running master in one m/c and running worker in other m/c. 

Thanks 

Rahat

[Rohit Kumar Gupta]

Hi Rahat,

I tried that as well. But I am getting the same error.

Thanks,

Rohit

[Rohit Kumar Gupta]

Hi All,

I have been able to solve the problem. Now the leader election process kicks in if the original master goes down.

Thanks,

Rohit

[Martynas Mickevičius]

Hi Rohit,

great to hear that you have solved the problem! Can you share the solution?

--

>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[Rohit Kumar Gupta]

I added below config to my configuration.

 

# Settings for the DistributedPubSubExtension

akka.contrib.cluster.pub-sub {

  # Actor name of the mediator actor, /user/distributedPubSubMediator

  name = distributedPubSubMediator

 

  # Start the mediator on members tagged with this role.

  # All members are used if undefined or empty.

  role = ""

 

  # How often the DistributedPubSubMediator should send out gossip information

  gossip-interval = 1s

 

  # Removed entries are pruned after this duration

  removed-time-to-live = 120s

}

Also, I added below line to akka.cluster.seed-nodes

auto-down=on

auto-down-unreachable-after = 300s 

akka.extensions were set to  extensions = ["akka.contrib.pattern.DistributedPubSubExtension","akka.contrib.pattern.ClusterReceptionistExtension"]

Regards,

Rohit

[Andre Piwoni]

With akka-cluster 2.3.11 I seem to be able to have configuration below work as well, contrary to the claim here (https://groups.google.com/forum/#!searchin/akka-user/seed$20node/akka-user/Wehd8AcN1pw/Tc0PYBq2DQAJ) that it doesn't work.

Node 1:

akka.remote.netty.tcp.hostname = 127.0.0.1

akka.remote.netty.tcp.port = 2551

akka.cluster.seed-nodes = ["akka.tcp://ClusterSystem@127.0.0.1:2551", "akka.tcp://Cluster...@127.0.0.1:2552"]

Node 2:

akka.remote.netty.tcp.hostname = 127.0.0.1

akka.remote.netty.tcp.port = 2552

akka.cluster.seed-nodes = ["akka.tcp://ClusterSystem@127.0.0.1:2552", "akka.tcp://Cluster...@127.0.0.1:2551"]

Is it a fluke? The reason I tried this because I thought if it works it may be more flexible in cases where I don't want to run first seed node which in Patrick's configuration would prevent cluster from initializing.

So, if you list all known nodes in seed nodes and first one is always the running node (rather weird) this may allow to start any number of nodes in any order and always guarantee cluster is initialized.

Am I missing something here?

Thanks,

Andre Piwoni

On Sunday, June 29, 2014 at 11:05:45 AM UTC-7, Patrik Nordwall wrote:

I'm not sure I understand your problem. You should used something like this:

Node 1:

akka.remote.netty.tcp.hostname = 127.0.0.1

akka.remote.netty.tcp.port = 2551

akka.cluster.seed-nodes = ["akka.tcp://ClusterSystem@127.0.0.1:2551", "akka.tcp://Cluster...@127.0.0.1:2552"]

Node 2:

akka.remote.netty.tcp.hostname = 127.0.0.1

akka.remote.netty.tcp.port = 2552

akka.cluster.seed-nodes = ["akka.tcp://ClusterSystem@127.0.0.1:2551", "akka.tcp://Cluster...@127.0.0.1:2552"]

Node 3:

akka.remote.netty.tcp.hostname = 127.0.0.1

akka.remote.netty.tcp.port = 2553

akka.cluster.seed-nodes = ["akka.tcp://ClusterSystem@127.0.0.1:2551", "akka.tcp://Cluster...@127.0.0.1:2552"]

When you run on multiple machines you should change hostname to the real host name or ip address instead of 127.0.0.1.

Read more here: http://doc.akka.io/docs/akka/2.3.3/scala/cluster-usage.html#Joining_to_Seed_Nodes

/Patrik

On Fri, Jun 27, 2014 at 6:49 PM, Rohit Kumar <rohitkum...@gmail.com> wrote:

Hi,

I am working with akka distributed worker template available on typesafe. I am using it to write a backend job which takes data from siebel using soap calls and inserts in mongo. This job is supposed to run once a week for few hours.

Based on the cluster-usage and other documentation on AKKA website, I imported akka-cluster.jar and configured the application configuration file with SEED nodes (akka.cluster.seed-nodes). But when I start the first node (MASTER NODE) with the configuration I mentioned (seed nodes etc), I start getting errors on the server console saying failed to join the seed node which is obvious (as it is the first node and there is nothing to join). Now I start the second node with akka.cluster.seed-nodes configured with the ip-address and port of the process where master node is running. I once again get the errors on the server console.

Now what I do next is - take the first join address of the master actor  from the MASTER NODE and set it dynamically in the slave node in the code (construct an Address object and pass it to the actors on the slave node). THIS WORKS!!! If I take the same join address and configure it in the application configuration akka.cluster.seed-nodes, it throws me error and slave doesn't join the cluster.

So I have following questions :-

1. How to configure the akka.cluster.seed-node configuration in application. I could never make it work/count in the configuration.

2. Is there any way to pre-configure the seed nodes in the configuration. As per me trying it out, it looks like the configuration is dynamic i.e. to take the join address of actor on the master node from the logs and configure the slave's seed-node configuration with that address ?

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Patrik Nordwall

[Patrik Nordwall]

When starting a new cluster (no existing nodes) you would risk that each node joins itself and thereby creating many separate clusters.

From docs:
The seed nodes can be started in any order and it is not necessary to have all seed nodes running, but the node configured as the first element in the seed-nodes configuration list must be started when initially starting a cluster, otherwise the other seed-nodes will not become initialized and no other node can join the cluster. The reason for the special first seed node is to avoid forming separated islands when starting from an empty cluster.

ons 6 jan. 2016 kl. 20:21 skrev Andre Piwoni <api...@gmail.com>:

With akka-cluster 2.3.11 I seem to be able to have configuration below work as well, contrary to the claim here (https://groups.google.com/forum/#!searchin/akka-user/seed$20node/akka-user/Wehd8AcN1pw/Tc0PYBq2DQAJ) that it doesn't work.

Node 1:

akka.remote.netty.tcp.hostname = 127.0.0.1

akka.remote.netty.tcp.port = 2551

akka.cluster.seed-nodes = ["akka.tcp://Cluste...@127.0.0.1:2551", "akka.tcp://Cluste...@127.0.0.1:2552"]

Node 2:

akka.remote.netty.tcp.hostname = 127.0.0.1

akka.remote.netty.tcp.port = 2552

akka.cluster.seed-nodes = ["akka.tcp://Cluste...@127.0.0.1:2552", "akka.tcp://Cluste...@127.0.0.1:2551"]

Is it a fluke? The reason I tried this because I thought if it works it may be more flexible in cases where I don't want to run first seed node which in Patrick's configuration would prevent cluster from initializing.

So, if you list all known nodes in seed nodes and first one is always the running node (rather weird) this may allow to start any number of nodes in any order and always guarantee cluster is initialized.

Am I missing something here?

Thanks,

Andre Piwoni

On Sunday, June 29, 2014 at 11:05:45 AM UTC-7, Patrik Nordwall wrote:

I'm not sure I understand your problem. You should used something like this:

Node 1:

akka.remote.netty.tcp.hostname = 127.0.0.1

akka.remote.netty.tcp.port = 2551

akka.cluster.seed-nodes = ["akka.tcp://Cluste...@127.0.0.1:2551", "akka.tcp://Cluste...@127.0.0.1:2552"]

Node 2:

akka.remote.netty.tcp.hostname = 127.0.0.1

akka.remote.netty.tcp.port = 2552

akka.cluster.seed-nodes = ["akka.tcp://Cluste...@127.0.0.1:2551", "akka.tcp://Cluste...@127.0.0.1:2552"]

Node 3:

akka.remote.netty.tcp.hostname = 127.0.0.1

akka.remote.netty.tcp.port = 2553

akka.cluster.seed-nodes = ["akka.tcp://Cluste...@127.0.0.1:2551", "akka.tcp://Cluste...@127.0.0.1:2552"]

Visit this group at https://groups.google.com/group/akka-user.

[Andre Piwoni]

I guess this could happen when node 1 starts up and later on node 2 starts up and for some reason cannot connect to running node 1. The more nodes in the cluster, when all of them are listed, the less likelihood of this happening though one of the nodes coming up online later could have issues connecting to all other seed nodes. I see your point. It is probably better to have such node uninitialized and hope that connectivity be restored later on than running it as its own cluster.

Thanks,

Andre