[Cluster Sharding] Futures timed out after [20000 milliseconds] when starting?

[kraythe]

When starting our server in a cluster of 9 nodes we are getting this error randomly. This causes our actors that use cluster sharding to restart and basically fubars the server. Any idea what might be causing this? 

Thanks in advance. 

2017-02-16 17:20:12 +0000 - [ERROR] - [OneForOneStrategy] akka://mycom/user/$a -  Futures timed out after [20000 milliseconds]
java.util.concurrent.TimeoutException: Futures timed out after [20000 milliseconds]
   at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
   at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
   at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:190)
   at akka.dispatch.MonitorableThreadFactory$AkkaForkJoinWorkerThread$$anon$3.block(ThreadPoolBuilder.scala:167)
   at scala.concurrent.forkjoin.ForkJoinPool.managedBlock(ForkJoinPool.java:3640)
   at akka.dispatch.MonitorableThreadFactory$AkkaForkJoinWorkerThread.blockOn(ThreadPoolBuilder.scala:165)
   at scala.concurrent.Await$.result(package.scala:190)
   at akka.cluster.sharding.ClusterSharding.start(ClusterSharding.scala:209)
   at akka.cluster.sharding.ClusterSharding.start(ClusterSharding.scala:276)
   at akka.cluster.sharding.ClusterSharding.start(ClusterSharding.scala:314)
   at actors.cache.ShardedCacheActor.startShardRegion(ShardedCacheActor.java:36)
   at actors.system.SystemStartupActor.process(SystemStartupActor.java:94)
   at actors.system.SystemStartupActor.onReceive(SystemStartupActor.java:69)
   at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:165)
   at akka.actor.Actor$class.aroundReceive(Actor.scala:496)
   at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:95)
   at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
   at akka.actor.ActorCell.invoke(ActorCell.scala:495)
   at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
   at akka.dispatch.Mailbox.run(Mailbox.scala:224)
   at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
   at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
   at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
   at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
   at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

[Patrik Nordwall]

The only reason I can imagine is that all threads of the default dispatcher are blocked. Please take a few thread dumps if you can reproduce it. I think it will be clear from those what is going on.

/Patrik

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscribe@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Patrik Nordwall
Akka Tech Lead
Lightbend -  Reactive apps on the JVM
Twitter: @patriknw