Can't run local cluster, Zookeeper connection loss

[Theo]

Hi,

I ran a few tests with Storm in local mode and everything worked
great. However, now, an hour or so later, when I run the exact same
code I get the following error:

0 [main-EventThread] INFO backtype.storm.zookeeper - Zookeeper
state update: :connected:none
6762 [main-EventThread] INFO backtype.storm.zookeeper - Zookeeper
state update: :disconnected:none
6763 [main] ERROR backtype.storm.daemon.nimbus - Error on
initialization of server service-handler
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /storm
at org.apache.zookeeper.KeeperException.create(KeeperException.java:
90)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:
42)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:809)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:837)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
43)
at java.lang.reflect.Method.invoke(Method.java:601)
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:90)
at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)
at backtype.storm.zookeeper$exists_node_QMARK_.invoke(zookeeper.clj:
60)
at backtype.storm.zookeeper$mkdirs.invoke(zookeeper.clj:67)
at backtype.storm.cluster
$mk_distributed_cluster_state.invoke(cluster.clj:23)
at backtype.storm.cluster$mk_storm_cluster_state.invoke(cluster.clj:
188)
at backtype.storm.daemon.nimbus
$fn__2457$exec_fn__837__auto____2458.invoke(nimbus.clj:388)
at clojure.lang.AFn.applyToHelper(AFn.java:163)
at clojure.lang.AFn.applyTo(AFn.java:151)
at clojure.core$apply.invoke(core.clj:540)
at backtype.storm.daemon.nimbus
$fn__2457$service_handler__2626.doInvoke(nimbus.clj:384)
at clojure.lang.RestFn.invoke(RestFn.java:409)
at backtype.storm.testing$mk_local_storm_cluster.doInvoke(testing.clj:
97)
at clojure.lang.RestFn.invoke(RestFn.java:398)
at backtype.storm.LocalCluster$_init.invoke(LocalCluster.clj:10)
at backtype.storm.LocalCluster.<init>(Unknown Source)
at stormtest.SomeTopology.main(SomeTopology.java:44)
6785 [main] INFO backtype.storm.util - Halting process: ("Error on
initialization")

What can I do to get around it?

[nathanmarz]

This is a strange error. Can you provide me some more information
about how you were running this (e.g., from Eclipse, from the command
line, etc.)?

-Nathan

[Theo]

I run it from the command line, just

storm jar path/to.jar some.package.SomeTopology

it seems that this is the fewest lines needed to trigger the error

class SomeTopology {
public static void main(String[] args) {
LocalCluster cluster = new LocalCluster();
}
}

are there any directories created by storm/zookeeper that could hold
some kind of leftover state? I might have ctrl-c'ed storm earlier, but
I'm not sure, could that have caused it?

T#

[nathanmarz]

Well, it gets a unique temporary directory for Zookeeper every time it
starts up local mode, so that shouldn't be the case. Also, I ctrl-C
local mode all the time, so that's not the issue.

One thing is that you shouldn't do is run Storm in local mode with
"storm jar". That command is for submitting topologies to a remote
cluster. You can try running Storm in local mode with the following
steps to see if it solves your issue:

1. Install Leiningen if you haven't already
2. Run "lein deps && lein compile" to compile storm-starter
3. Run "lein repl" to open up a Clojure REPL
4. Run these commands:

user=> (import 'storm.starter.WordCountTopology)
user=> (WordCountTopology/main nil)

Or you can follow the steps on the wiki for running Storm within
Eclipse: https://github.com/nathanmarz/storm/wiki/Creating-a-new-Storm-project

Let me know if any of that helps.

-Nathan

[Theo]

ok, I must have misunderstood the tutorial. it makes sense now, why
else would I create a main method? weird that it worked a few times
first, but stopped working.

however, just running the topology class with java on the command line
still fails, with more or less the same error.

I've pasted the whole output here: https://gist.github.com/1232610

thanks for the help
T#

[Theo]

I realized what the problem was, finally. I'd accidentally switched
java versions between the runs. I guess the problem is that the
compiled jzmq will only work with the java version that it was
compiled against. I've got two java 1.6's installed, one with headers
and one without, and the one without was still the default. so opening
a new terminal changed the java version.

T#

[bradhouse]

Nathan,

Is that a hard requirement to not use storm.jar for local mode? Is
there something specific about the storm.jar packaging that prevents a
stable local mode? If so, I would suggest you need a jar version of
storm for local mode that is available via a Maven repo. There is no
way I could get a set of non-maven artifacts into our corporate dev
environment.

Also, if is a requirement to use lein deps instead of the storm.jar
for local mode I'd document that somewhere more clearly. I completely
missed that when setting this up. :)

Cheers,
-brad

On Sep 22, 1:47 am, nathanmarz <nathan.m...@gmail.com> wrote:
> One thing is that you shouldn't do is run Storm in local mode with
> "storm jar". That command is for submitting topologies to a remote
> cluster. You can try running Storm in local mode with the following
> steps to see if it solves your issue:

[nathanmarz]

Well, you just need to get the storm jars on the classpath of your
project, whether through "lein deps" or otherwise. storm-starter uses
Leiningen for its build, so that's what it does, but there are a
million ways to setup a Java project. "storm jar" does do this, but
it's really meant for submitting to clusters. I think it's cleaner to
just do java -cp and list the classpath.

-Nathan