2.3.12. Akka cluster init timeouts

Marek Żebrowski

unread,

Aug 3, 2015, 4:37:23 AM8/3/15

to Akka User List

Quite often, especially on slower machines (test, intergration) akka cluster extension timeouts:

08:11:39.642 [sgActors-akka.actor.default-dispatcher-4] ERROR Cluster(akka://sgActors): Failed to startup Cluster. You can try to increase 'akka.actor.creation-timeout'.

java.util.concurrent.TimeoutException: Futures timed out after [60000 milliseconds]

I already increased timeout from 20s to 60s but it still might be not enough.

I think it is due to some other stuff being too heavy while doing application startup, so I maybe there is a way to block until cluster actor is ready - like some event ? This way of course means application will start longer, but with higher chance of success. I don't need any particular cluster member to be up before continuing, only [cluster] actor to be constructed.

Marek Żebrowski

unread,

Aug 3, 2015, 4:42:38 AM8/3/15

to Akka User List

That failure leads to another failure: actor name not unique

08:11:39.893 [sgActors-akka.actor.default-dispatcher-17] ERROR a.c.ClusterCoreSupervisor: actor name [cluster] is not unique!
akka.actor.InvalidActorNameException: actor name [cluster] is not unique!

Patrik Nordwall

unread,

Aug 5, 2015, 4:57:05 PM8/5/15

to akka...@googlegroups.com

Have you changed the default configuration of the default-dispatcher, or do you do any blocking (long running) tasks on the default-dispatcher?

/Patrik

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Patrik Nordwall
Typesafe - Reactive apps on the JVM
Twitter: @patriknw

Marek Żebrowski

unread,

Aug 6, 2015, 5:59:39 AM8/6/15

to Akka User List

I did not change the default dispatcher. I hope that I have all heavy task run by other dispatchers - I'll double check it.

Marek Żebrowski

unread,

Aug 26, 2015, 9:50:30 AM8/26/15

to Akka User List

Problem still persists.

1. I changed boot procedure to wait until cluster starts. it is done by:

class ClusterWaiter(p: Promise[Boolean]) extends Actor {
  override def receive = {
    case _ =>
      p.complete(Success(true))
      context.stop(self)
  }
}

def startCluster() = {
  //eagerly init cluster dispatcher, the one used to start cluster daemon
  system.dispatchers.lookup("cluster-dispatcher")
  val p = Promise[Boolean]
  val waiter = system.actorOf(actor.Props(classOf[ClusterWaiter], p), "cluster-waiter")
  // sendCurrentClusterState requires internal cluster actor to start
  Cluster(system).sendCurrentClusterState(waiter)
  //wait for some signal from cluster
  Await.ready(p.future, 60 seconds)
}

In that time, nothing else takes CPU or IO, app just waits for cluster to start

2. I changed actor creation timeout to 120 seconds

3. Still I get exceptions on startup:

12:15:43.025 [sgActors-akka.actor.default-dispatcher-3] INFO Cluster(akka://sgActors): Cluster Node [akka.tcp://sgAc...@app1.groupl.es:2552] - Starting up...

12:17:43.105 [main] ERROR n.l.h.p.HTTPProvider: Failed to Boot! Your application may not run properly

2 minutes of idle waiting for cluster start.

12:17:43.186 [sgActors-akka.actor.default-dispatcher-4] ERROR Cluster(akka://sgActors): Failed to startup Cluster. You can try to increase 'akka.actor.creation-timeout'.

java.util.concurrent.TimeoutException: Futures timed out after [120000 milliseconds]

at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) ~[scala-library-2.11.7.jar:na]

at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) ~[scala-library-2.11.7.jar:na]

at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:190) ~[scala-library-2.11.7.jar:na]

at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) ~[scala-library-2.11.7.jar:na]

at scala.concurrent.Await$.result(package.scala:190) ~[scala-library-2.11.7.jar:na]

at akka.cluster.Cluster.liftedTree1$1(Cluster.scala:172) ~[akka-cluster_2.11-2.3.12.jar:na]

at akka.cluster.Cluster.<init>(Cluster.scala:171) ~[akka-cluster_2.11-2.3.12.jar:na]

at akka.cluster.Cluster$.createExtension(Cluster.scala:42) ~[akka-cluster_2.11-2.3.12.jar:na]

at akka.cluster.Cluster$.createExtension(Cluster.scala:37) ~[akka-cluster_2.11-2.3.12.jar:na]

at akka.actor.ActorSystemImpl.registerExtension(ActorSystem.scala:713) ~[akka-actor_2.11-2.3.12.jar:na]

at akka.actor.ExtensionId$class.apply(Extension.scala:79) ~[akka-actor_2.11-2.3.12.jar:na]

at akka.cluster.Cluster$.apply(Cluster.scala:37) ~[akka-cluster_2.11-2.3.12.jar:na]

at akka.cluster.ClusterActorRefProvider.createRemoteWatcher(ClusterActorRefProvider.scala:66) ~[akka-cluster_2.11-2.3.12.jar:na]

at akka.remote.RemoteActorRefProvider.init(RemoteActorRefProvider.scala:186) ~[akka-remote_2.11-2.3.12.jar:na]

at akka.cluster.ClusterActorRefProvider.init(ClusterActorRefProvider.scala:58) ~[akka-cluster_2.11-2.3.12.jar:na]

at akka.actor.ActorSystemImpl.liftedTree2$1(ActorSystem.scala:620) ~[akka-actor_2.11-2.3.12.jar:na]

at akka.actor.ActorSystemImpl._start$lzycompute(ActorSystem.scala:617) ~[akka-actor_2.11-2.3.12.jar:na]

at akka.actor.ActorSystemImpl._start(ActorSystem.scala:617) ~[akka-actor_2.11-2.3.12.jar:na]

at akka.actor.ActorSystemImpl.start(ActorSystem.scala:634) ~[akka-actor_2.11-2.3.12.jar:na]

at akka.actor.ActorSystem$.apply(ActorSystem.scala:142) ~[akka-actor_2.11-2.3.12.jar:na]

at akka.actor.ActorSystem$.apply(ActorSystem.scala:119) ~[akka-actor_2.11-2.3.12.jar:na]

Caused by: java.lang.NullPointerException: null

at akka.cluster.Cluster.shutdown(Cluster.scala:358) ~[akka-cluster_2.11-2.3.12.jar:na]

at akka.cluster.Cluster.liftedTree1$1(Cluster.scala:176) ~[akka-cluster_2.11-2.3.12.jar:na]