Strange timeout exception during cluster start

[nitesh...@gmail.com]

Hello,

I have few clusters running happily with the same code. We are trying to deploy a new cluster with same code. But, keeps getting following exception. The big difference is we have compute coming from openstack based internal cloud. Any hints on how to figure out what is going on will be really helpful. Looked at the code Remoting.scala. But, unable to get any hint. I am  using Akka microkernel to bootstrap and akka version is 2.2.3.


17:46:24.181 [BotCloud-akka.actor.default-dispatcher-3] INFO  akka.event.slf4j.Slf4jLogger - Slf4jLogger started
17:46:24.195 [BotCloud-akka.actor.default-dispatcher-3] DEBUG akka.event.EventStream - logger log1-Slf4jLogger started
17:46:24.195 [BotCloud-akka.actor.default-dispatcher-3] DEBUG akka.event.EventStream - Default Loggers started
17:46:24.286 [BotCloud-akka.actor.default-dispatcher-2] INFO  Remoting - Starting remoting
Exception in thread "main" java.util.concurrent.TimeoutException: Futures timed out after [10000 milliseconds]
    at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:96)
    at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:100)
    at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
    at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
    at scala.concurrent.Await$.result(package.scala:107)
    at akka.remote.Remoting.start(Remoting.scala:173)
    at akka.remote.RemoteActorRefProvider.init(RemoteActorRefProvider.scala:184)
    at akka.cluster.ClusterActorRefProvider.init(ClusterActorRefProvider.scala:52)
    at akka.actor.ActorSystemImpl._start$lzycompute(ActorSystem.scala:579)
    at akka.actor.ActorSystemImpl._start(ActorSystem.scala:577)
    at akka.actor.ActorSystemImpl.start(ActorSystem.scala:588)
    at akka.actor.ActorSystem$.apply(ActorSystem.scala:111)
    at akka.actor.ActorSystem$.apply(ActorSystem.scala:104)
    at akka.actor.ActorSystem$.create(ActorSystem.scala:66)
    at akka.actor.ActorSystem.create(ActorSystem.scala)
    at botcloud.core.NodeSystem.startup(NodeSystem.java:36)
    at akka.kernel.Main$$anonfun$main$1.apply(Main.scala:86)
    at akka.kernel.Main$$anonfun$main$1.apply(Main.scala:84)
    at scala.collection.Iterator$class.foreach(Iterator.scala:727)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
    at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
    at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
    at akka.kernel.Main$.main(Main.scala:84)
    at akka.kernel.Main.main(Main.scala)

Regards,
Nitesh

[√iktor Ҡlang]

Hi Nitesh,

it means that the Listen operation times out (10 seconds default), you can try to increase that timeout and see if it helps, but I have never seen that happen before so if I were you I'd go over the removing configuration both for Akka and for the machine.

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Cheers,

√

[nitesh...@gmail.com]

Hello,

Changed the configuration of startup-timeout = 60 s. Still getting the same issue. The strange thing is the same code is deployed on three separate clusters and works fine.

22:38:57.853 [BotCloud-akka.actor.default-dispatcher-4] INFO  Remoting - Starting remoting
Exception in thread "main" java.util.concurrent.TimeoutException: Futures timed out after [60000 milliseconds]

Appreciate any pointers to help finding out the root cause. Did enable the DEBUG logs. But, no other message is coming out other than timeout.

Regards,
Nitesh

[√iktor Ҡlang]

Have you tried to bind to the same interface port from another program on the same machine?

I think you need to make sure it is an Akka problem rather than a machine problem before we can even start debugging it from an Akka perspective.

[nitesh...@gmail.com]

Hello,

Yes. We can have processes bind to ports and able to telnet/access from outside. Even on the same cluster process which fails on listening to akka, the JMX port binds and is accessible.

Regards,
Nitesh

[√iktor Ҡlang]

On Thu, Apr 10, 2014 at 2:24 AM, <nitesh...@gmail.com> wrote:

Hello,

Yes. We can have processes bind to ports

Not "ports", the _same_ port as the Akka remoting will use.

On the same interface Akka remoting will use.

Just so you aren't comparing apples and oranges.

--

Cheers,

√

[nitesh...@gmail.com]

Hello,

Yes. we did bind to the same port which we are using for Akka. I have found the following offending configuration which stops from cluster remoting to start. The interesting part is this causes problem only for this new cluster. Our other clusters are working fine with same code and config. Also, I adjusted the sizes below to minimum to try. The cluster did not start unless remove the complete configuration.

netty-dispatcher {
             type = "Dispatcher"
             executor = "thread-pool-executor"
             
            thread-pool-executor {
                core-pool-size-min = 5
                core-pool-size-factor = 1.0
                core-pool-size-max = 24
                max-pool-size-min = 5
                max-pool-size-factor  = 1.0
                max-pool-size-max = 24
            }
            throughput = 1
        }
       
   
    akka.remote.netty.tcp.use-dispatcher-for-io = netty-dispatcher

[√iktor Ҡlang]

Interesting!

Please open an Issue: https://github.com/akka/akka/issues?state=open