[joinSeedNodeProcess] is not unique Exception at the Cluster Startup

[Ryadh khsib]

Hi all,

We are getting intermittently a "[actor name [joinSeedNodeProcess] is not unique!]" exception at the akka Cluster system start up. 

The production setup is a akka Singleton cluster deployed on 12 tomcats across 3 physical machines (4 nodes on every machine).

Do you have any idea how we could fix this?

Many thanks,

Ryadh

Exception trace:

 ERROR akka.cluster.ClusterCoreSupervisor 2014-03-21 08:59:28,092 [build=null] [System-akka.actor.default-dispatcher-8] - Cluster node [akka.tcp://System@host084:2551] crashed, [actor name [joinSeedNodeProcess] is not unique!] - shutting down...
 [exceptionClass=akka.actor.dungeon.ChildrenContainer$NormalChildrenContainer, exceptionMethod=reserve, exceptionID=74a5d13f, ]
akka.actor.InvalidActorNameException: actor name [joinSeedNodeProcess] is not unique!
	at akka.actor.dungeon.ChildrenContainer$NormalChildrenContainer.reserve(ChildrenContainer.scala:130) ~[akka-actor_2.10-2.2.0.jar:2.2.0]
	at akka.actor.dungeon.Children$class.reserveChild(Children.scala:77) ~[akka-actor_2.10-2.2.0.jar:2.2.0]
	at akka.actor.ActorCell.reserveChild(ActorCell.scala:338) [akka-actor_2.10-2.2.0.jar:2.2.0]
	at akka.actor.dungeon.Children$class.makeChild(Children.scala:186) ~[akka-actor_2.10-2.2.0.jar:2.2.0]
	at akka.actor.dungeon.Children$class.actorOf(Children.scala:38) ~[akka-actor_2.10-2.2.0.jar:2.2.0]
	at akka.actor.ActorCell.actorOf(ActorCell.scala:338) [akka-actor_2.10-2.2.0.jar:2.2.0]
	at akka.cluster.ClusterCoreDaemon.joinSeedNodes(ClusterDaemon.scala:361) ~[akka-cluster_2.10-2.2.0.jar:na]
	at akka.cluster.ClusterCoreDaemon$$anonfun$tryingToJoin$1.applyOrElse(ClusterDaemon.scala:302) ~[akka-cluster_2.10-2.2.0.jar:na]
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498) [akka-actor_2.10-2.2.0.jar:2.2.0]
	at akka.actor.ActorCell.invoke(ActorCell.scala:456) [akka-actor_2.10-2.2.0.jar:2.2.0]
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237) [akka-actor_2.10-2.2.0.jar:2.2.0]
	at akka.dispatch.Mailbox.run(Mailbox.scala:219) [akka-actor_2.10-2.2.0.jar:2.2.0]
	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386) [akka-actor_2.10-2.2.0.jar:2.2.0]
	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [scala-library-2.10.2.jar:na]
	at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [scala-library-2.10.2.jar:na]
	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [scala-library-2.10.2.jar:na]
	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [scala-library-2.10.2.jar:na]

ERROR akka.actor.OneForOneStrategy 2014-03-21 08:59:28,093 [build=null] [System-akka.actor.default-dispatcher-8] - actor name [joinSeedNodeProcess] is not unique!
 [exceptionClass=akka.actor.dungeon.ChildrenContainer$NormalChildrenContainer, exceptionMethod=reserve, exceptionID=74a5d13f, ]
akka.actor.InvalidActorNameException: actor name [joinSeedNodeProcess] is not unique!
	at akka.actor.dungeon.ChildrenContainer$NormalChildrenContainer.reserve(ChildrenContainer.scala:130) ~[akka-actor_2.10-2.2.0.jar:2.2.0]
	at akka.actor.dungeon.Children$class.reserveChild(Children.scala:77) ~[akka-actor_2.10-2.2.0.jar:2.2.0]
	at akka.actor.ActorCell.reserveChild(ActorCell.scala:338) [akka-actor_2.10-2.2.0.jar:2.2.0]
	at akka.actor.dungeon.Children$class.makeChild(Children.scala:186) ~[akka-actor_2.10-2.2.0.jar:2.2.0]
	at akka.actor.dungeon.Children$class.actorOf(Children.scala:38) ~[akka-actor_2.10-2.2.0.jar:2.2.0]
	at akka.actor.ActorCell.actorOf(ActorCell.scala:338) [akka-actor_2.10-2.2.0.jar:2.2.0]
	at akka.cluster.ClusterCoreDaemon.joinSeedNodes(ClusterDaemon.scala:361) ~[akka-cluster_2.10-2.2.0.jar:na]
	at akka.cluster.ClusterCoreDaemon$$anonfun$tryingToJoin$1.applyOrElse(ClusterDaemon.scala:302) ~[akka-cluster_2.10-2.2.0.jar:na]
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498) [akka-actor_2.10-2.2.0.jar:2.2.0]
	at akka.actor.ActorCell.invoke(ActorCell.scala:456) [akka-actor_2.10-2.2.0.jar:2.2.0]
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237) [akka-actor_2.10-2.2.0.jar:2.2.0]
	at akka.dispatch.Mailbox.run(Mailbox.scala:219) [akka-actor_2.10-2.2.0.jar:2.2.0]
	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386) [akka-actor_2.10-2.2.0.jar:2.2.0]
	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [scala-library-2.10.2.jar:na]
	at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [scala-library-2.10.2.jar:na]
	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [scala-library-2.10.2.jar:na]
	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [scala-library-2.10.2.jar:na]

[√iktor Ҡlang]

Always upgrade to at least the latest bugfix release for the version of Akka you are using and see if that resolves your problem.

Did that solve it?

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Cheers,

√

———————

Viktor Klang

Chief Architect - Typesafe

Twitter: @viktorklang

[Patrik Nordwall]

Ryadh, did that solve the problem? Otherwise I would like to see a ticket so that we can fix it.

http://doc.akka.io/docs/akka/current/project/issue-tracking.html

/Patrik

[Ryadh khsib]

Hi Patrik,

I have upgraded to the version 2.3.0 (the current version is the 2.2.0). The fix is going to production next week. Hopefully this will fix it and will stabilize the cluster setup in general. 

I will keep you informed ;)

Thanks,

Ryadh 

[Patrik Nordwall]

Hi Ryadh,

I looked at the implementation and this is bug that can happen if you call joinSeedNodes repeatedly. I don't see a reason why you would need to do that, but it is supported usage and should be fixed. Ticket: https://www.assembla.com/spaces/akka/tickets/3973

Regards,

Patrik

Patrik Nordwall
Typesafe -  Reactive apps on the JVM
Twitter: @patriknw

JOIN US. REGISTER TODAY!

Scala

Days

June 16th-18th,

Berlin

[Ryadh khsib]

Hi Patrik,

I did the upgrade to the version 2.3.0 and the joinSeedNodeProcess issue is fixed. For the current setup, 2 singletons and 1 router clusters are deployed inside the same Akka system. I presume this is causing the repeated calls the joinSeedNodeProcess actor. 

What do recommend for this setup? Should it be a unique cluster for every Akka system?

We faced a new issue after the upgrade. The thread count on Tomcat increase due to 500 New I/O worker threads. Are you aware of any issue with the 2.3.0 that could cause this?

Many thanks!
Ryadh

The thread dump of I/O threads:

Thread 10466: (state = IN_NATIVE)

 - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise)

 - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled frame)

 - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=81 (Compiled frame)

 - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Compiled frame)

 - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame)

 - org.jboss.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=68 (Compiled frame)

 - org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(java.nio.channels.Selector) @bci=1, line=409 (Compiled frame)

 - org.jboss.netty.channel.socket.nio.AbstractNioSelector.run() @bci=49, line=206 (Compiled frame)

 - org.jboss.netty.channel.socket.nio.AbstractNioWorker.run() @bci=1, line=90 (Interpreted frame)

 - org.jboss.netty.channel.socket.nio.NioWorker.run() @bci=1, line=178 (Interpreted frame)

 - org.jboss.netty.util.ThreadRenamingRunnable.run() @bci=55, line=108 (Interpreted frame)

 - org.jboss.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame)

 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=46, line=1110 (Interpreted frame)

 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame)

 - java.lang.Thread.run() @bci=11, line=722 (Interpreted frame)

[Björn Antonsson]

Hi Ryadh,

On 3 April 2014 at 13:30:02, Ryadh khsib (riadh...@gmail.com) wrote:

Hi Patrik,

I did the upgrade to the version 2.3.0 and the joinSeedNodeProcess issue is fixed. For the current setup, 2 singletons and 1 router clusters are deployed inside the same Akka system. I presume this is causing the repeated calls the joinSeedNodeProcess actor. 

What do recommend for this setup? Should it be a unique cluster for every Akka system?

We faced a new issue after the upgrade. The thread count on Tomcat increase due to 500 New I/O worker threads. Are you aware of any issue with the 2.3.0 that could cause this?

It's very difficult to diagnose why you get so many Netty worker threads without having any information about the system you are running on.

• Do the machines have a large number of cores?
• Do you start multiple actor systems inside the same JVM?
• Have you done custom configuration of I/O dispatchers?

B/

-- 

Björn Antonsson

Typesafe – Reactive Apps on the JVM

twitter: @bantonsson

[Ryadh khsib]

[Björn Antonsson]

Hi Ryadh,

This doesn't answer any of my questions, so I'll repeat them with some additions.

• Do the machines have a large number of cores?
• Do you start multiple actor systems inside the same JVM?

• Have you done custom configuration of the akka I/O dispatchers?
• The number of threads that you are seeing is that from a single Tomcat instance?

B/

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[Ryadh khsib]

Hi Bjorn,

My answers are in comments.

Thanks

On Tuesday, April 8, 2014 9:31:00 AM UTC+1, Björn Antonsson wrote:

Hi Ryadh,

This doesn't answer any of my questions, so I'll repeat them with some additions.

• Do the machines have a large number of cores?

Yes, 64 cores per machine. 3 machines, 4 tomcat instances on each machine

• Do you start multiple actor systems inside the same JVM?

 No, only 1 actor system per JVM

• Have you done custom configuration of the akka I/O dispatchers?

 Yes:

akka.actor.provider=akka.cluster.ClusterActorRefProvider

akka.actor.creation-timeout=20s

akka.cluster.seed-nodes=akka.tcp://system@app1:2551,akka.tcp://

system@app2

:2551,akka.tcp://

system@app3

:2551

akka.cluster.roles=orchestrate,eventProcess,extractProcess,extractSingleton

akka.cluster.seed-node-timeout=25s

akka.cluster.failure-detector.threshold=15.0

akka.cluster.failure-detector.acceptable-heartbeat-pause=25s

akka.actor.default-dispatcher.executor=fork-join-executor

akka.actor.default-dispatcher.fork-join-executor.parallelism-min=20

akka.actor.default-dispatcher.fork-join-executor.parallelism-factor=2

akka.actor.default-dispatcher.fork-join-executor.parallelism-max=40

• The number of threads that you are seeing is that from a single Tomcat instance?

Yes 

[Björn Antonsson]

Hi Ryadh,

I've run some tests here, but I don't get that many worker threads. On my machine they are effectively capped by these settings

akka.remote.netty.tcp.client-socket-worker-pool.pool-size-max = 2

akka.remote.netty.tcp.server-socket-worker-pool.pool-size-max = 2

They are set to 2 by default.

I haven't run Akka inside tomcat though, but I guess that shouldn't affect this.

There is one setting that you could try, that would use a special dispatcher for the netty threads.

akka.remote.netty.tcp.use-dispatcher-for-io = "your-dispatcher-name-here"

Hope that helps.

B/