Rebalance -> NotAliveException (Storm 0.8.1)

[Benjamin Süß]

Hi there,

I've been trying to execute the rebalance command and I am into a NotAliveException (see error message attached below). My topology is working, which means communication between nodes should be OK. I tried to execute rebalance from both the nimbus and a supervisor machine - same result. I am using Storm 0.8.1.

I have searched the log files, but could not find any information in any of them. Any help will be appreciated.

Regards,
Benjamin

[root@WUM56200 bin]# ./storm rebalance
Running: java -client -Dstorm.options= -Dstorm.home=/root/storm-0.8.1 -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -cp /root/storm-0.8.1/storm-0.8.1.jar:/root/storm-0.8.1/lib/asm-4.0.jar:/root/storm-0.8.1/lib/reflectasm-1.07-shaded.jar:/root/storm-0.8.1/lib/clout-0.4.1.jar:/root/storm-0.8.1/lib/curator-framework-1.0.1.jar:/root/storm-0.8.1/lib/joda-time-2.0.jar:/root/storm-0.8.1/lib/commons-lang-2.5.jar:/root/storm-0.8.1/lib/kryo-2.17.jar:/root/storm-0.8.1/lib/libthrift7-0.7.0.jar:/root/storm-0.8.1/lib/commons-io-1.4.jar:/root/storm-0.8.1/lib/slf4j-log4j12-1.5.8.jar:/root/storm-0.8.1/lib/clj-time-0.4.1.jar:/root/storm-0.8.1/lib/compojure-0.6.4.jar:/root/storm-0.8.1/lib/jline-0.9.94.jar:/root/storm-0.8.1/lib/hiccup-0.3.6.jar:/root/storm-0.8.1/lib/commons-logging-1.1.1.jar:/root/storm-0.8.1/lib/tools.logging-0.2.3.jar:/root/storm-0.8.1/lib/tools.cli-0.2.2.jar:/root/storm-0.8.1/lib/servlet-api-2.5-20081211.jar:/root/storm-0.8.1/lib/jetty-6.1.26.jar:/root/storm-0.8.1/lib/guava-13.0.jar:/root/storm-0.8.1/lib/ring-core-0.3.10.jar:/root/storm-0.8.1/lib/carbonite-1.5.0.jar:/root/storm-0.8.1/lib/json-simple-1.1.jar:/root/storm-0.8.1/lib/core.incubator-0.1.0.jar:/root/storm-0.8.1/lib/minlog-1.2.jar:/root/storm-0.8.1/lib/math.numeric-tower-0.0.1.jar:/root/storm-0.8.1/lib/tools.macro-0.1.0.jar:/root/storm-0.8.1/lib/zookeeper-3.3.3.jar:/root/storm-0.8.1/lib/servlet-api-2.5.jar:/root/storm-0.8.1/lib/disruptor-2.10.1.jar:/root/storm-0.8.1/lib/objenesis-1.2.jar:/root/storm-0.8.1/lib/jzmq-2.1.0.jar:/root/storm-0.8.1/lib/commons-fileupload-1.2.1.jar:/root/storm-0.8.1/lib/curator-client-1.0.1.jar:/root/storm-0.8.1/lib/slf4j-api-1.5.8.jar:/root/storm-0.8.1/lib/log4j-1.2.16.jar:/root/storm-0.8.1/lib/clojure-1.4.0.jar:/root/storm-0.8.1/lib/commons-codec-1.4.jar:/root/storm-0.8.1/lib/httpcore-4.1.jar:/root/storm-0.8.1/lib/jgrapht-0.8.3.jar:/root/storm-0.8.1/lib/snakeyaml-1.9.jar:/root/storm-0.8.1/lib/jetty-util-6.1.26.jar:/root/storm-0.8.1/lib/ring-jetty-adapter-0.3.11.jar:/root/storm-0.8.1/lib/junit-3.8.1.jar:/root/storm-0.8.1/lib/ring-servlet-0.3.11.jar:/root/storm-0.8.1/lib/httpclient-4.1.1.jar:/root/storm-0.8.1/lib/commons-exec-1.1.jar:/root/.storm:/root/storm-0.8.1/bin backtype.storm.command.rebalance
0    [main] INFO  backtype.storm.thrift  - Connecting to Nimbus at localhost:6627
Exception in thread "main" NotAliveException(msg: is not alive)
    at backtype.storm.generated.Nimbus$rebalance_result.read(Nimbus.java:5559)
    at org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:78)
    at backtype.storm.generated.Nimbus$Client.recv_rebalance(Nimbus.java:257)
    at backtype.storm.generated.Nimbus$Client.rebalance(Nimbus.java:243)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:616)
    at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
    at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)
    at backtype.storm.command.rebalance$_main.doInvoke(rebalance.clj:24)
    at clojure.lang.RestFn.invoke(RestFn.java:397)
    at clojure.lang.AFn.applyToHelper(AFn.java:159)
    at clojure.lang.RestFn.applyTo(RestFn.java:132)
    at backtype.storm.command.rebalance.main(Unknown Source)

[Nathan Marz]

>> storm help rebalance

Syntax: [storm rebalance topology-name [-w wait-time-secs] [-n new-num-workers] [-e component=parallelism]*]

    Sometimes you may wish to spread out where the workers for a topology 

    are running. For example, let's say you have a 10 node cluster running 

    4 workers per node, and then let's say you add another 10 nodes to 

    the cluster. You may wish to have Storm spread out the workers for the 

    running topology so that each node runs 2 workers. One way to do this 

    is to kill the topology and resubmit it, but Storm provides a "rebalance" 

    command that provides an easier way to do this.

    Rebalance will first deactivate the topology for the duration of the 

    message timeout (overridable with the -w flag) and then redistribute 

    the workers evenly around the cluster. The topology will then return to 

    its previous state of activation (so a deactivated topology will still 

    be deactivated and an activated topology will go back to being activated).

    

    The rebalance command can also be used to change the parallelism of a running topology.

    Use the -n and -e switches to change the number of workers or number of executors of a component

    respectively.

--
Twitter: @nathanmarz
http://nathanmarz.com

[Benjamin Süß]

Oh stupid me...thanks Nathan.