trouble configuring a storm cluster to run in remote mode

2,130 views
Skip to first unread message

indraniel

unread,
Nov 17, 2011, 4:22:03 PM11/17/11
to storm-user
After experimenting with the local mode, I am now attempting to run
the
storm-starter test word count topology using a one node test cluster
(on
Ubuntu 10.04) in remote mode, but have been unsuccessful so far.

I get the following error:

java.lang.RuntimeException: java.lang.RuntimeException:
java.io.IOException: Cannot run program "python" (in directory "/tmp/
bd40286b-afc7-4713-a6ac-af2718bfa9fe/supervisor/stormdist/word-
count-1-1321561558/resources"): java.io.IOException: error=2, No such
file or directory
at clojure.lang.LazySeq.sval(LazySeq.java:47)

I am not entirely clear how to debug this issue and was wondering if
anyone could suggest some useful pointers? Am I simply mis-configuring
something, or is this one node cluster setup infeasible?

Thanks in advance!

Sincerely,
Indraniel

Below are my notes and observations so far:

I have been using the following storm configuration (storm.yaml):

# storm.yml config
########################################################
storm.zookeeper.servers:
- "localhost"

nimbus.host: "localhost"

storm.local.dir: "/tmp/storm-data"

# my load path for the native libraries that Storm uses (ZeroMQ and
JZMQ).
java.library.path: "/home/archive/idas/lib"
###########################################################################

As well as the following zookeeper configuration (zoo.cfg):

# zoo.cfg config
##########################################################
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
dataDir=/tmp/zookeeper-data
# the port at which the clients will connect
clientPort=2181
###########################################################################

# xterm commands
##########################################################
In one terminal (on xterm 1), I ran the following commands:

$ storm nimbus
$ storm supervisor
$ storm ui

After the above, on a separate xterm (xterm 2), and in my git cloned
storm-starter directory, I ran the following commands:

$ lein deps
$ lein uberjar

and successfully got a 'storm-starter-0.0.1-SNAPSHOT-standalone.jar'
file.

Then I ran the following:

$ storm jar storm-starter-0.0.1-SNAPSHOT-standalone.jar
storm.starter.WordCountTopology

Which lead to the 'Cannot run program "python" error'. The full
stdout/sterr log is shown towards the bottom of this message.

The python executable is installed and in my $PATH :

$ which python
/usr/bin/python.

$ python -V
Python 2.6.5

Similarly so for the unzip program:

$ which unzip
/usr/bin/unzip

$ unzip -v
UnZip 6.00 of 20 April 2009, by Debian. Original by Info-ZIP.

Latest sources and executables are at ftp://ftp.info-zip.org/pub/infozip/
;
see ftp://ftp.info-zip.org/pub/infozip/UnZip.html for other sites.

Compiled with gcc 4.4.3 for Unix (Linux ELF) on Mar 7 2010.

I have verified that the "multilang/resources" directory is in the
standalone jar as well:

jar -tf storm-starter-0.0.1-SNAPSHOT-standalone.jar | grep resources
resources/splitsentence.py
resources/storm.py
resources/storm.rb

# xterm logs
##############################################################

Input/Outputs from xterm 1:

$ storm nimbus
Running: java -server -Djava.library.path=/home/archive/idas/lib -
Xmx1024m -Dlogfile.name=nimbus.log -
Dlog4j.configuration=storm.log.properties -cp /home/archive/idas/opt/
storm-0.5.4/storm-0.5.4.jar:/home/archive/idas/opt/storm-0.5.4/lib/clj-
time-0.3.0.jar:/home/archive/idas/opt/storm-0.5.4/lib/
clojure-1.2.0.jar:/home/archive/idas/opt/storm-0.5.4/lib/clojure-
contrib-1.2.0.jar:/home/archive/idas/opt/storm-0.5.4/lib/
clout-0.4.1.jar:/home/archive/idas/opt/storm-0.5.4/lib/commons-
codec-1.3.jar:/home/archive/idas/opt/storm-0.5.4/lib/commons-
exec-1.1.jar:/home/archive/idas/opt/storm-0.5.4/lib/commons-
fileupload-1.2.1.jar:/home/archive/idas/opt/storm-0.5.4/lib/commons-
io-1.4.jar:/home/archive/idas/opt/storm-0.5.4/lib/commons-
lang-2.5.jar:/home/archive/idas/opt/storm-0.5.4/lib/commons-
logging-1.1.1.jar:/home/archive/idas/opt/storm-0.5.4/lib/
compojure-0.6.4.jar:/home/archive/idas/opt/storm-0.5.4/lib/
core.incubator-0.1.0.jar:/home/archive/idas/opt/storm-0.5.4/lib/
hiccup-0.3.6.jar:/home/archive/idas/opt/storm-0.5.4/lib/
httpclient-4.0.1.jar:/home/archive/idas/opt/storm-0.5.4/lib/
httpcore-4.0.1.jar:/home/archive/idas/opt/storm-0.5.4/lib/
jetty-6.1.26.jar:/home/archive/idas/opt/storm-0.5.4/lib/jetty-
util-6.1.26.jar:/home/archive/idas/opt/storm-0.5.4/lib/
jline-0.9.94.jar:/home/archive/idas/opt/storm-0.5.4/lib/joda-
time-1.6.jar:/home/archive/idas/opt/storm-0.5.4/lib/json-
simple-1.1.jar:/home/archive/idas/opt/storm-0.5.4/lib/junit-3.8.1.jar:/
home/archive/idas/opt/storm-0.5.4/lib/jvyaml-1.0.0.jar:/home/archive/
idas/opt/storm-0.5.4/lib/jzmq-2.1.0.jar:/home/archive/idas/opt/
storm-0.5.4/lib/libthrift7-0.7.0.jar:/home/archive/idas/opt/
storm-0.5.4/lib/log4j-1.2.16.jar:/home/archive/idas/opt/storm-0.5.4/
lib/ring-core-0.3.10.jar:/home/archive/idas/opt/storm-0.5.4/lib/ring-
jetty-adapter-0.3.11.jar:/home/archive/idas/opt/storm-0.5.4/lib/ring-
servlet-0.3.11.jar:/home/archive/idas/opt/storm-0.5.4/lib/servlet-
api-2.5-20081211.jar:/home/archive/idas/opt/storm-0.5.4/lib/servlet-
api-2.5.jar:/home/archive/idas/opt/storm-0.5.4/lib/slf4j-
api-1.5.8.jar:/home/archive/idas/opt/storm-0.5.4/lib/slf4j-
log4j12-1.5.8.jar:/home/archive/idas/opt/storm-0.5.4/lib/
tools.macro-0.1.0.jar:/home/archive/idas/opt/storm-0.5.4/lib/
zookeeper-3.3.2.jar:/home/archive/idas/opt/storm-0.5.4/log4j:/home/
archive/idas/opt/storm-0.5.4/conf backtype.storm.daemon.nimbus
$ storm supervisor
Running: java -server -Djava.library.path=/home/archive/idas/lib -
Xmx1024m -Dlogfile.name=supervisor.log -
Dlog4j.configuration=storm.log.properties -cp /home/archive/idas/opt/
storm-0.5.4/storm-0.5.4.jar:/home/archive/idas/opt/storm-0.5.4/lib/clj-
time-0.3.0.jar:/home/archive/idas/opt/storm-0.5.4/lib/
clojure-1.2.0.jar:/home/archive/idas/opt/storm-0.5.4/lib/clojure-
contrib-1.2.0.jar:/home/archive/idas/opt/storm-0.5.4/lib/
clout-0.4.1.jar:/home/archive/idas/opt/storm-0.5.4/lib/commons-
codec-1.3.jar:/home/archive/idas/opt/storm-0.5.4/lib/commons-
exec-1.1.jar:/home/archive/idas/opt/storm-0.5.4/lib/commons-
fileupload-1.2.1.jar:/home/archive/idas/opt/storm-0.5.4/lib/commons-
io-1.4.jar:/home/archive/idas/opt/storm-0.5.4/lib/commons-
lang-2.5.jar:/home/archive/idas/opt/storm-0.5.4/lib/commons-
logging-1.1.1.jar:/home/archive/idas/opt/storm-0.5.4/lib/
compojure-0.6.4.jar:/home/archive/idas/opt/storm-0.5.4/lib/
core.incubator-0.1.0.jar:/home/archive/idas/opt/storm-0.5.4/lib/
hiccup-0.3.6.jar:/home/archive/idas/opt/storm-0.5.4/lib/
httpclient-4.0.1.jar:/home/archive/idas/opt/storm-0.5.4/lib/
httpcore-4.0.1.jar:/home/archive/idas/opt/storm-0.5.4/lib/
jetty-6.1.26.jar:/home/archive/idas/opt/storm-0.5.4/lib/jetty-
util-6.1.26.jar:/home/archive/idas/opt/storm-0.5.4/lib/
jline-0.9.94.jar:/home/archive/idas/opt/storm-0.5.4/lib/joda-
time-1.6.jar:/home/archive/idas/opt/storm-0.5.4/lib/json-
simple-1.1.jar:/home/archive/idas/opt/storm-0.5.4/lib/junit-3.8.1.jar:/
home/archive/idas/opt/storm-0.5.4/lib/jvyaml-1.0.0.jar:/home/archive/
idas/opt/storm-0.5.4/lib/jzmq-2.1.0.jar:/home/archive/idas/opt/
storm-0.5.4/lib/libthrift7-0.7.0.jar:/home/archive/idas/opt/
storm-0.5.4/lib/log4j-1.2.16.jar:/home/archive/idas/opt/storm-0.5.4/
lib/ring-core-0.3.10.jar:/home/archive/idas/opt/storm-0.5.4/lib/ring-
jetty-adapter-0.3.11.jar:/home/archive/idas/opt/storm-0.5.4/lib/ring-
servlet-0.3.11.jar:/home/archive/idas/opt/storm-0.5.4/lib/servlet-
api-2.5-20081211.jar:/home/archive/idas/opt/storm-0.5.4/lib/servlet-
api-2.5.jar:/home/archive/idas/opt/storm-0.5.4/lib/slf4j-
api-1.5.8.jar:/home/archive/idas/opt/storm-0.5.4/lib/slf4j-
log4j12-1.5.8.jar:/home/archive/idas/opt/storm-0.5.4/lib/
tools.macro-0.1.0.jar:/home/archive/idas/opt/storm-0.5.4/lib/
zookeeper-3.3.2.jar:/home/archive/idas/opt/storm-0.5.4/log4j:/home/
archive/idas/opt/storm-0.5.4/conf backtype.storm.daemon.supervisor
$ storm ui
Running: java -server -Djava.library.path=/home/archive/idas/lib -
Xmx768m -Dlogfile.name=ui.log -
Dlog4j.configuration=storm.log.properties -cp /home/archive/idas/opt/
storm-0.5.4/storm-0.5.4.jar:/home/archive/idas/opt/storm-0.5.4/lib/clj-
time-0.3.0.jar:/home/archive/idas/opt/storm-0.5.4/lib/
clojure-1.2.0.jar:/home/archive/idas/opt/storm-0.5.4/lib/clojure-
contrib-1.2.0.jar:/home/archive/idas/opt/storm-0.5.4/lib/
clout-0.4.1.jar:/home/archive/idas/opt/storm-0.5.4/lib/commons-
codec-1.3.jar:/home/archive/idas/opt/storm-0.5.4/lib/commons-
exec-1.1.jar:/home/archive/idas/opt/storm-0.5.4/lib/commons-
fileupload-1.2.1.jar:/home/archive/idas/opt/storm-0.5.4/lib/commons-
io-1.4.jar:/home/archive/idas/opt/storm-0.5.4/lib/commons-
lang-2.5.jar:/home/archive/idas/opt/storm-0.5.4/lib/commons-
logging-1.1.1.jar:/home/archive/idas/opt/storm-0.5.4/lib/
compojure-0.6.4.jar:/home/archive/idas/opt/storm-0.5.4/lib/
core.incubator-0.1.0.jar:/home/archive/idas/opt/storm-0.5.4/lib/
hiccup-0.3.6.jar:/home/archive/idas/opt/storm-0.5.4/lib/
httpclient-4.0.1.jar:/home/archive/idas/opt/storm-0.5.4/lib/
httpcore-4.0.1.jar:/home/archive/idas/opt/storm-0.5.4/lib/
jetty-6.1.26.jar:/home/archive/idas/opt/storm-0.5.4/lib/jetty-
util-6.1.26.jar:/home/archive/idas/opt/storm-0.5.4/lib/
jline-0.9.94.jar:/home/archive/idas/opt/storm-0.5.4/lib/joda-
time-1.6.jar:/home/archive/idas/opt/storm-0.5.4/lib/json-
simple-1.1.jar:/home/archive/idas/opt/storm-0.5.4/lib/junit-3.8.1.jar:/
home/archive/idas/opt/storm-0.5.4/lib/jvyaml-1.0.0.jar:/home/archive/
idas/opt/storm-0.5.4/lib/jzmq-2.1.0.jar:/home/archive/idas/opt/
storm-0.5.4/lib/libthrift7-0.7.0.jar:/home/archive/idas/opt/
storm-0.5.4/lib/log4j-1.2.16.jar:/home/archive/idas/opt/storm-0.5.4/
lib/ring-core-0.3.10.jar:/home/archive/idas/opt/storm-0.5.4/lib/ring-
jetty-adapter-0.3.11.jar:/home/archive/idas/opt/storm-0.5.4/lib/ring-
servlet-0.3.11.jar:/home/archive/idas/opt/storm-0.5.4/lib/servlet-
api-2.5-20081211.jar:/home/archive/idas/opt/storm-0.5.4/lib/servlet-
api-2.5.jar:/home/archive/idas/opt/storm-0.5.4/lib/slf4j-
api-1.5.8.jar:/home/archive/idas/opt/storm-0.5.4/lib/slf4j-
log4j12-1.5.8.jar:/home/archive/idas/opt/storm-0.5.4/lib/
tools.macro-0.1.0.jar:/home/archive/idas/opt/storm-0.5.4/lib/
zookeeper-3.3.2.jar:/home/archive/idas/opt/storm-0.5.4/log4j:/home/
archive/idas/opt/storm-0.5.4:/home/archive/idas/opt/storm-0.5.4/conf
backtype.storm.ui.core

Input/Outputs from xterm 2:

$ storm jar storm-starter-0.0.1-SNAPSHOT-standalone.jar
storm.starter.WordCountTopology
Running: export STORM_JAR=storm-starter-0.0.1-SNAPSHOT-standalone.jar;
java -client -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/
lib -cp /home/archive/idas/opt/storm-0.5.4/storm-0.5.4.jar:/home/
archive/idas/opt/storm-0.5.4/lib/clj-time-0.3.0.jar:/home/archive/idas/
opt/storm-0.5.4/lib/clojure-1.2.0.jar:/home/archive/idas/opt/
storm-0.5.4/lib/clojure-contrib-1.2.0.jar:/home/archive/idas/opt/
storm-0.5.4/lib/clout-0.4.1.jar:/home/archive/idas/opt/storm-0.5.4/lib/
commons-codec-1.3.jar:/home/archive/idas/opt/storm-0.5.4/lib/commons-
exec-1.1.jar:/home/archive/idas/opt/storm-0.5.4/lib/commons-
fileupload-1.2.1.jar:/home/archive/idas/opt/storm-0.5.4/lib/commons-
io-1.4.jar:/home/archive/idas/opt/storm-0.5.4/lib/commons-
lang-2.5.jar:/home/archive/idas/opt/storm-0.5.4/lib/commons-
logging-1.1.1.jar:/home/archive/idas/opt/storm-0.5.4/lib/
compojure-0.6.4.jar:/home/archive/idas/opt/storm-0.5.4/lib/
core.incubator-0.1.0.jar:/home/archive/idas/opt/storm-0.5.4/lib/
hiccup-0.3.6.jar:/home/archive/idas/opt/storm-0.5.4/lib/
httpclient-4.0.1.jar:/home/archive/idas/opt/storm-0.5.4/lib/
httpcore-4.0.1.jar:/home/archive/idas/opt/storm-0.5.4/lib/
jetty-6.1.26.jar:/home/archive/idas/opt/storm-0.5.4/lib/jetty-
util-6.1.26.jar:/home/archive/idas/opt/storm-0.5.4/lib/
jline-0.9.94.jar:/home/archive/idas/opt/storm-0.5.4/lib/joda-
time-1.6.jar:/home/archive/idas/opt/storm-0.5.4/lib/json-
simple-1.1.jar:/home/archive/idas/opt/storm-0.5.4/lib/junit-3.8.1.jar:/
home/archive/idas/opt/storm-0.5.4/lib/jvyaml-1.0.0.jar:/home/archive/
idas/opt/storm-0.5.4/lib/jzmq-2.1.0.jar:/home/archive/idas/opt/
storm-0.5.4/lib/libthrift7-0.7.0.jar:/home/archive/idas/opt/
storm-0.5.4/lib/log4j-1.2.16.jar:/home/archive/idas/opt/storm-0.5.4/
lib/ring-core-0.3.10.jar:/home/archive/idas/opt/storm-0.5.4/lib/ring-
jetty-adapter-0.3.11.jar:/home/archive/idas/opt/storm-0.5.4/lib/ring-
servlet-0.3.11.jar:/home/archive/idas/opt/storm-0.5.4/lib/servlet-
api-2.5-20081211.jar:/home/archive/idas/opt/storm-0.5.4/lib/servlet-
api-2.5.jar:/home/archive/idas/opt/storm-0.5.4/lib/slf4j-
api-1.5.8.jar:/home/archive/idas/opt/storm-0.5.4/lib/slf4j-
log4j12-1.5.8.jar:/home/archive/idas/opt/storm-0.5.4/lib/
tools.macro-0.1.0.jar:/home/archive/idas/opt/storm-0.5.4/lib/
zookeeper-3.3.2.jar:storm-starter-0.0.1-SNAPSHOT-standalone.jar:/
gscuser/idas/.storm:/home/archive/idas/opt/storm-0.5.4/bin
storm.starter.WordCountTopology
1 [main-EventThread] INFO backtype.storm.zookeeper - Zookeeper
state update: :connected:none
48 [main-EventThread] INFO backtype.storm.zookeeper - Zookeeper
state update: :connected:none
59 [main-EventThread] INFO backtype.storm.zookeeper - Zookeeper
state update: :connected:none
80 [main-EventThread] INFO backtype.storm.zookeeper - Zookeeper
state update: :connected:none
139 [main-EventThread] INFO backtype.storm.zookeeper - Zookeeper
state update: :connected:none
210 [main] INFO backtype.storm.daemon.nimbus - Received topology
submission for word-count with conf {"topology.serializations" {},
"storm.id" "word-count-1-1321561558", "topology.max.task.parallelism"
3, "topology.debug" true}
289 [main] INFO backtype.storm.daemon.nimbus - Task word-
count-1-1321561558:1 timed out
290 [main] INFO backtype.storm.daemon.nimbus - Task word-
count-1-1321561558:2 timed out
291 [main] INFO backtype.storm.daemon.nimbus - Task word-
count-1-1321561558:3 timed out
292 [main] INFO backtype.storm.daemon.nimbus - Task word-
count-1-1321561558:4 timed out
293 [main] INFO backtype.storm.daemon.nimbus - Task word-
count-1-1321561558:5 timed out
294 [main] INFO backtype.storm.daemon.nimbus - Task word-
count-1-1321561558:6 timed out
295 [main] INFO backtype.storm.daemon.nimbus - Task word-
count-1-1321561558:7 timed out
296 [main] INFO backtype.storm.daemon.nimbus - Task word-
count-1-1321561558:8 timed out
297 [main] INFO backtype.storm.daemon.nimbus - Task word-
count-1-1321561558:9 timed out
297 [main] INFO backtype.storm.daemon.nimbus - Task word-
count-1-1321561558:10 timed out
301 [main] INFO backtype.storm.daemon.nimbus - Reassigning word-
count-1-1321561558 to 1 slots
369 [main] INFO backtype.storm.daemon.nimbus - Reassign ids: [1 2 3
4 5 6 7 8 9 10]
386 [main] INFO backtype.storm.daemon.nimbus - Available slots:
(["c59bdd06-132a-4901-95e3-8e4ff97cd5b0" 1]
["c59bdd06-132a-4901-95e3-8e4ff97cd5b0" 2]
["c59bdd06-132a-4901-95e3-8e4ff97cd5b0" 3] ["c6b906b2-d2f5-4c0d-
a2a5-5853e95e3213" 4] ["c6b906b2-d2f5-4c0d-a2a5-5853e95e3213" 5]
["c6b906b2-d2f5-4c0d-a2a5-5853e95e3213" 6])
446 [main] INFO backtype.storm.daemon.nimbus - Setting new
assignment for storm id word-count-1-1321561558:
#:backtype.storm.daemon.common.Assignment{:master-code-dir "/tmp/
762d440e-7cd0-48fd-a4a6-e1d8d8190884/nimbus/stormdist/word-
count-1-1321561558", :node->host
{"c59bdd06-132a-4901-95e3-8e4ff97cd5b0"
"linus82.gsc.wustl.edu"}, :task->node+port {1
["c59bdd06-132a-4901-95e3-8e4ff97cd5b0" 1], 2
["c59bdd06-132a-4901-95e3-8e4ff97cd5b0" 1], 3
["c59bdd06-132a-4901-95e3-8e4ff97cd5b0" 1], 4
["c59bdd06-132a-4901-95e3-8e4ff97cd5b0" 1], 5
["c59bdd06-132a-4901-95e3-8e4ff97cd5b0" 1], 6
["c59bdd06-132a-4901-95e3-8e4ff97cd5b0" 1], 7
["c59bdd06-132a-4901-95e3-8e4ff97cd5b0" 1], 8
["c59bdd06-132a-4901-95e3-8e4ff97cd5b0" 1], 9
["c59bdd06-132a-4901-95e3-8e4ff97cd5b0" 1], 10
["c59bdd06-132a-4901-95e3-8e4ff97cd5b0" 1]}, :task->start-time-secs {1
1321561558, 2 1321561558, 3 1321561558, 4 1321561558, 5 1321561558, 6
1321561558, 7 1321561558, 8 1321561558, 9 1321561558, 10 1321561558}}
460 [main] INFO backtype.storm.daemon.nimbus - Activating word-
count: word-count-1-1321561558
473 [Thread-11] INFO backtype.storm.daemon.supervisor - Downloading
code for storm id word-count-1-1321561558 from /tmp/762d440e-7cd0-48fd-
a4a6-e1d8d8190884/nimbus/stormdist/word-count-1-1321561558
473 [Thread-6] INFO backtype.storm.daemon.supervisor - Downloading
code for storm id word-count-1-1321561558 from /tmp/762d440e-7cd0-48fd-
a4a6-e1d8d8190884/nimbus/stormdist/word-count-1-1321561558
474 [Thread-11] INFO backtype.storm.daemon.supervisor - Finished
downloading code for storm id word-count-1-1321561558 from /tmp/
762d440e-7cd0-48fd-a4a6-e1d8d8190884/nimbus/stormdist/word-
count-1-1321561558
474 [Thread-6] INFO backtype.storm.daemon.supervisor - Finished
downloading code for storm id word-count-1-1321561558 from /tmp/
762d440e-7cd0-48fd-a4a6-e1d8d8190884/nimbus/stormdist/word-
count-1-1321561558
513 [Thread-7] INFO backtype.storm.daemon.supervisor - Launching
worker with assignment
#:backtype.storm.daemon.supervisor.LocalAssignment{:storm-id "word-
count-1-1321561558", :task-ids (1 2 3 4 5 6 7 8 9 10)} for this
supervisor c59bdd06-132a-4901-95e3-8e4ff97cd5b0 on port 1 with id
b82a83e5-d54d-4183-8699-fdfcc614033a
513 [Thread-7] INFO backtype.storm.daemon.worker - Launching worker
for word-count-1-1321561558 on c59bdd06-132a-4901-95e3-8e4ff97cd5b0:1
with id b82a83e5-d54d-4183-8699-fdfcc614033a
516 [Thread-7-EventThread] INFO backtype.storm.zookeeper -
Zookeeper state update: :connected:none
614 [Thread-7] ERROR backtype.storm.daemon.worker - Error on
initialization of server mk-worker
java.lang.RuntimeException: java.lang.RuntimeException:
java.io.IOException: Cannot run program "python" (in directory "/tmp/
bd40286b-afc7-4713-a6ac-af2718bfa9fe/supervisor/stormdist/word-
count-1-1321561558/resources"): java.io.IOException: error=2, No such
file or directory
at clojure.lang.LazySeq.sval(LazySeq.java:47)
at clojure.lang.LazySeq.seq(LazySeq.java:56)
at clojure.lang.RT.seq(RT.java:450)
at clojure.core$seq.invoke(core.clj:122)
at clojure.core$dorun.invoke(core.clj:2450)
at clojure.core$doall.invoke(core.clj:2465)
at backtype.storm.daemon.worker
$fn__3074$exec_fn__858__auto____3075.invoke(worker.clj:172)
at clojure.lang.AFn.applyToHelper(AFn.java:187)
at clojure.lang.AFn.applyTo(AFn.java:151)
at clojure.core$apply.invoke(core.clj:540)
at backtype.storm.daemon.worker
$fn__3074$mk_worker__3216.doInvoke(worker.clj:78)
at clojure.lang.RestFn.invoke(RestFn.java:513)
at backtype.storm.daemon.supervisor
$fn__3668.invoke(supervisor.clj:398)
at clojure.lang.MultiFn.invoke(MultiFn.java:188)
at backtype.storm.daemon.supervisor
$fn__3452$exec_fn__858__auto____3453$sync_processes__3455$iter__3512__3516$fn__3517.invoke(supervisor.clj:
233)
at clojure.lang.LazySeq.sval(LazySeq.java:42)
at clojure.lang.LazySeq.seq(LazySeq.java:56)
at clojure.lang.RT.seq(RT.java:450)
at clojure.core$seq.invoke(core.clj:122)
at clojure.core$dorun.invoke(core.clj:2450)
at clojure.core$doall.invoke(core.clj:2465)
at backtype.storm.daemon.supervisor
$fn__3452$exec_fn__858__auto____3453$sync_processes__3455.invoke(supervisor.clj:
220)
at backtype.storm.event$event_manager
$fn__1971$fn__1972.invoke(event.clj:25)
at backtype.storm.event$event_manager
$fn__1971.invoke(event.clj:22)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.RuntimeException: java.io.IOException: Cannot run
program "python" (in directory "/tmp/bd40286b-afc7-4713-a6ac-
af2718bfa9fe/supervisor/stormdist/word-count-1-1321561558/resources"):
java.io.IOException: error=2, No such file or directory
at
backtype.storm.task.ShellBolt.initializeSubprocess(ShellBolt.java:83)
at backtype.storm.task.ShellBolt.prepare(ShellBolt.java:89)
at backtype.storm.daemon.task$fn__3031.invoke(task.clj:457)
at clojure.lang.MultiFn.invoke(MultiFn.java:194)
at backtype.storm.daemon.task$mk_task.invoke(task.clj:278)
at backtype.storm.daemon.worker
$fn__3074$exec_fn__858__auto____3075$iter__3120__3124$fn__3125$fn__3126.invoke(worker.clj:
172)
at backtype.storm.daemon.worker
$fn__3074$exec_fn__858__auto____3075$iter__3120__3124$fn__3125.invoke(worker.clj:
172)
at clojure.lang.LazySeq.sval(LazySeq.java:42)
... 25 more
Caused by: java.io.IOException: Cannot run program "python" (in
directory "/tmp/bd40286b-afc7-4713-a6ac-af2718bfa9fe/supervisor/
stormdist/word-count-1-1321561558/resources"): java.io.IOException:
error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
at
backtype.storm.task.ShellBolt.initializeSubprocess(ShellBolt.java:74)
... 32 more
Caused by: java.io.IOException: java.io.IOException: error=2, No such
file or directory
at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
... 33 more
641 [Thread-7] INFO backtype.storm.util - Halting process: ("Error
on initialization")

indraniel

unread,
Nov 17, 2011, 6:47:40 PM11/17/11
to storm-user
On Nov 17, 3:22 pm, indraniel <indran...@gmail.com> wrote:

> I get the following error:
>
> java.lang.RuntimeException: java.lang.RuntimeException:
> java.io.IOException: Cannot run program "python" (in directory "/tmp/
> bd40286b-afc7-4713-a6ac-af2718bfa9fe/supervisor/stormdist/word-
> count-1-1321561558/resources"): java.io.IOException: error=2, No such
> file or directory
>         at clojure.lang.LazySeq.sval(LazySeq.java:47)

I also noticed on the post-mortem running of storm command:

storm jar storm-starter-0.0.1-SNAPSHOT-standalone.jar
storm.starter.WordCountTopology

that the

/tmp/bd40286b-afc7-4713-a6ac-af2718bfa9fe/supervisor/stormdist/word-
count-1-1321561558/resources

directory did not exist. Only the files:

stormcode.ser
stormconf.ser

existed inside:

/tmp/bd40286b-afc7-4713-a6ac-af2718bfa9fe/supervisor/stormdist/word-
count-1-1321561558/

I am not sure how important an observation that is.

-Indraniel

nathanmarz

unread,
Nov 19, 2011, 2:53:15 PM11/19/11
to storm-user
Hey Indraniel,

Getting the multilang stuff working can be frustrating. The behavior
seems to be different depending on what OS you're using.

One question I have for you: are you at least able to run pure Java
topologies on your remote cluster (e.g., ExclamationTopology)?

One thing you can try to get multilang working is upgrading your OS to
Ubuntu 10.10. This is what we use for our clusters and the multilang
works without a hitch.

As you're debugging this, it also might be helpful to look at the code
for how Storm workers launch the multilang scripts:
https://github.com/nathanmarz/storm/blob/master/src/jvm/backtype/storm/task/ShellBolt.java#L69
As you can see, it's pretty simple and it's not clear why this doesn't
work on some setups. My gut feeling is that it has something to do
with the shell environment not getting passed to the subprocess or
something.

-Nathan

indraniel

unread,
Nov 21, 2011, 6:18:11 PM11/21/11
to storm-user
Nathan,

Thanks for the tip. I think I am getting closer to getting my test
one-cluster running.

Upon further investigation it appears, I was having trouble getting
zookeeper appropriately started via my installation froma debian
package. I ended up downloading the latest stable version of
zookeeper (zookeeper-3.3.3) and invoking the 'bin/zkServer.sh start'
in the downloaded zookeeper software directory. Afterwards, I was
able to successfully start up the 'storm nimbus' and 'storm
supervisor' daemons.

Now I am getting a different error when submitting the storm toplogies
to nimbus. Below describes what I did and the error I observed.

Inside the storm-starter directory I ran the following commands:

$ lein clean; # to clean up any prior project builds
Cleaning up.
$ lein deps;
[INFO] snapshot org.twitter4j:twitter4j-core:2.2.5-SNAPSHOT: checking
for updates from twitter4j
[INFO] snapshot org.twitter4j:twitter4j-core:2.2.5-SNAPSHOT: checking
for updates from clojars
[INFO] snapshot org.twitter4j:twitter4j-stream:2.2.5-SNAPSHOT:
checking for updates from twitter4j
[INFO] snapshot org.twitter4j:twitter4j-stream:2.2.5-SNAPSHOT:
checking for updates from clojars
Copying 5 files to /home/archive/idas/src/storm-starter/lib
Copying 35 files to /home/archive/idas/src/storm-starter/lib/dev
$ lein compile;
Compiling 14 source files to /home/archive/idas/src/storm-starter/
classes
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
Compiling storm.starter.clj.word-count
Compilation succeeded.
$ lein uberjar;
Cleaning up.
Compiling 14 source files to /home/archive/idas/src/storm-starter/
classes
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
Copying 5 files to /home/archive/idas/src/storm-starter/lib
Copying 35 files to /home/archive/idas/src/storm-starter/lib/dev
Created /home/archive/idas/src/storm-starter/storm-starter-0.0.1-
SNAPSHOT.jar
Including storm-starter-0.0.1-SNAPSHOT.jar
Including clojure-contrib-1.2.0.jar
Including clojure-1.2.0.jar
Including twitter4j-stream-2.2.5-SNAPSHOT.jar
Including json-simple-1.1.jar
Including twitter4j-core-2.2.5-SNAPSHOT.jar
Created /home/archive/idas/src/storm-starter/storm-starter-0.0.1-
SNAPSHOT-standalone.jar

I then successfully obtained a 'storm-starter-0.0.1-SNAPSHOT-
standalone.jar' package.

When trying to submit the pure Java topology (ExclamationTopology), I
got the following error:

$ storm jar storm-starter-0.0.1-SNAPSHOT-standalone.jar
storm.starter.ExclamationToplogy
Running: export STORM_JAR=storm-starter-0.0.1-SNAPSHOT-standalone.jar;
java -client -Djava.library.path=/home/archive/idas/lib -cp /home/
archive/idas/opt/storm-0.5.4/storm-0.5.4.jar:/home/archive/idas/
opt/storm-0.5.4/lib/clj-time-0.3.0.jar:/home/archive/idas/opt/

zookeeper-3.3.2.jar:storm-starter-0.0.1-SNAPSHOT-standalone.jar:/
gscuser/idas/.storm:/home/archive/idas/opt/storm-0.5.4/bin
storm.starter.ExclamationToplogy
Exception in thread "main" java.lang.NoClassDefFoundError: storm/
starter/ExclamationToplogy
Caused by: java.lang.ClassNotFoundException:
storm.starter.ExclamationToplogy
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:
301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
Could not find the main class: storm.starter.ExclamationToplogy.
Program will exit.

Did I mis-compile the project via lein?

When trying to run the python/multilang example (WordCountTopology), I
got this error as well:

$ storm jar storm-starter-0.0.1-SNAPSHOT-standalone.jar
storm.starter.WordCountTopology
Running: export STORM_JAR=storm-starter-0.0.1-SNAPSHOT-standalone.jar;

java -client -Djava.library.path=/home/archive/idas/lib -cp /home/
archive/idas/opt/storm-0.5.4/storm-0.5.4.jar:/home/archive/idas/opt/
storm-0.5.4/lib/clj-time-0.3.0.jar:/home/archive/idas/opt/storm-0.5.4/

zookeeper-3.3.2.jar:storm-starter-0.0.1-SNAPSHOT-standalone.jar:/
gscuser/idas/.storm:/home/archive/idas/opt/storm-0.5.4/bin
storm.starter.WordCountTopology

0 [main] ERROR org.apache.zookeeper.server.NIOServerCnxn - Thread
Thread[main,5,main] died
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:
119)
at
sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
at
sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52)
at org.apache.zookeeper.server.NIOServerCnxn
$Factory.<init>(NIOServerCnxn.java:144)
at org.apache.zookeeper.server.NIOServerCnxn
$Factory.<init>(NIOServerCnxn.java:125)
at backtype.storm.zookeeper
$mk_inprocess_zookeeper.invoke(zookeeper.clj:101)
at backtype.storm.testing
$mk_local_storm_cluster.doInvoke(testing.clj:101)
at clojure.lang.RestFn.invoke(RestFn.java:398)
at backtype.storm.LocalCluster$_init.invoke(LocalCluster.clj:
10)
at backtype.storm.LocalCluster.<init>(Unknown Source)
at storm.starter.WordCountTopology.main(WordCountTopology.java:
85)

On my single node cluster, I assume I have to have the zookeeper
daemon up and running so that nimbus and the supervisor(s) can
communicate with each other. Do I need to have another zookeeper
configuration parameter setup in storm.yaml so that I can submit a
storm job to nimbus?

Many Thanks again for the help and tips so far.

Sincerely,
Indraniel

On Nov 19, 1:53 pm, nathanmarz <nathan.m...@gmail.com> wrote:
> Hey Indraniel,
>
> Getting the multilang stuff working can be frustrating. The behavior
> seems to be different depending on what OS you're using.
>
> One question I have for you: are you at least able to run pure Java
> topologies on your remote cluster (e.g., ExclamationTopology)?
>
> One thing you can try to get multilang working is upgrading your OS to
> Ubuntu 10.10. This is what we use for our clusters and the multilang
> works without a hitch.
>
> As you're debugging this, it also might be helpful to look at the code

> for how Storm workers launch the multilang scripts:https://github.com/nathanmarz/storm/blob/master/src/jvm/backtype/stor...

indraniel

unread,
Nov 22, 2011, 4:15:26 PM11/22/11
to storm-user
Okay, I think I got the ExclamationTopology going by finagling the
zookeeper and storm invocations. I started up zookeeper to get nimbus
up
and running, but then stopped zookeeper to enable storm job
submissions
to nimbus.

The process is described in detail towards the end of this email.

After getting the ExclamationTopology (pure java) example going, I
similarly attempted the WordCountTopology (multilang example). But
again,
I got the 'Cannot run program "python"' output.

Is starting and stopping the zookeeper daemon as described below,
appropriate for a one-node cluster test environment? Was the
successful ExclamationTopology invocation really running in "local"
mode?

-Indraniel

# Invocation Details ########

1. Started up 4 xterms
2. xterm[1]: $ zkServer.sh start # started zookeeper
3. xterm[2]: $ storm nimbus # started nimbus daemon

saw the following in the nimbus.log:

...
2011-11-22 14:22:41 ClientCnxn [INFO] Opening socket connection to
server /127.0.0.1:2181
2011-11-22 14:22:41 ClientCnxn [INFO] Socket connection established
to localhost.localdomain/127.0.0.1:2181, initiating session
2011-11-22 14:22:41 ClientCnxn [INFO] Session establishment
complete on server localhost.localdomain/127.0.0.1:2181, sessionid =
0x133ccf10e960001, negotiated timeout = 20000
2011-11-22 14:22:41 zookeeper [INFO] Zookeeper state
update: :connected:none
2011-11-22 14:22:41 ClientCnxn [INFO] EventThread shut down
2011-11-22 14:22:41 nimbus [INFO] Starting Nimbus server...

4. xterm[3]: $ storm supervisor # started supervisor daemon


saw the following in the supervisor.log:

2011-11-22 14:24:20 ZooKeeper [INFO] Initiating client connection,
connectString=127.0.0.1:2181/storm sessionTimeout=20000
watcher=backtype.storm.zookeeper$mk_client$reify__1247@1cecd92c
2011-11-22 14:24:20 ClientCnxn [INFO] Opening socket connection to
server /127.0.0.1:2181
2011-11-22 14:24:20 ClientCnxn [INFO] Socket connection established
to localhost.localdomain/127.0.0.1:2181, initiating session
2011-11-22 14:24:20 ClientCnxn [INFO] Session establishment
complete on server localhost.localdomain/127.0.0.1:2181, sessionid =
0x133ccf10e960003, negotiated timeout = 20000

5. xterm[4]: $ storm jar storm-starter-0.0.1-SNAPSHOT-standalone.jar
storm.starter.ExclamationTopology
$ # submitted the storm exclamation topology (pure java
example)


saw the following output error:

Running: export STORM_JAR=storm-starter-0.0.1-SNAPSHOT-
standalone.jar; java -client -Djava.library.path=/home/archive/idas/
lib -cp /home/archive/idas/opt/storm-0.5.4/storm-0.5.4.jar:/home/
archive/idas/opt/storm-0.5.4/lib/clj-time-0.3.0.j
ar:/home/archive/idas/opt/storm-0.5.4/lib/clojure-1.2.0.jar:/home/
archive/idas/opt/storm-0.5.4/lib/clojure-contrib-1.2.0.jar:/home/
archive/idas/opt/storm-0.5.4/lib/clout-0.4.1.jar:/home/archive/idas/
opt/storm-0.5.4/lib/commons-codec-1.3.j
ar:/home/archive/idas/opt/storm-0.5.4/lib/commons-exec-1.1.jar:/
home/archive/idas/opt/storm-0.5.4/lib/commons-fileupload-1.2.1.jar:/
home/archive/idas/opt/storm-0.5.4/lib/commons-io-1.4.jar:/home/archive/
idas/opt/storm-0.5.4/lib/commons-lang-2.5.jar:/home/archive/idas/opt/
storm-0.5.4/lib/commons-logging-1.1.1.jar:/home/archive/idas/opt/


storm-0.5.4/lib/compojure-0.6.4.jar:/home/archive/idas/opt/storm-0.5.4/
lib/core.incubator-0.1.0.jar:/home/archive/idas/opt/storm-0.5.4/lib/
hiccup-0.3.6.jar:/home/archive/idas/opt/storm-0.5.4/lib/
httpclient-4.0.1.jar:/home/archive/idas/opt/storm-0.5.4/lib/
httpcore-4.0.1.jar:/home/archive/idas/opt/storm-0.5.4/lib/
jetty-6.1.26.jar:/home/archive/idas/opt/storm-0.5.4/lib/jetty-
util-6.1.26.jar:/home/archive/idas/opt/storm-0.5.4/lib/
jline-0.9.94.jar:/home/archive/idas/opt/storm-0.5.4/lib/joda-
time-1.6.jar:/home/archive/idas/opt/storm-0.5.4/lib/json-
simple-1.1.jar:/home/archive/idas/opt/storm-0.5.4/lib/junit-3.8.1.jar:/
home/archive/idas/opt/storm-0.5.4/lib/jvyaml-1.0.0.jar:/home/archive/
idas/opt/storm-0.5.4/lib/jzmq-2.1.0.jar:/home/archive/idas/opt/
storm-0.5.4/lib/libthrift7-0.7.0.jar:/home/archive/idas/opt/
storm-0.5.4/lib/log4j-1.2.16.jar:/home/archive/idas/opt/storm-0.5.4/
lib/ring-core-0.3.10.jar:/home/archive/idas/opt/storm-0.5.4/lib/ring-
jetty-adapter-0.3.11.jar:/home/archive/idas/opt/storm-0.5.4/lib/ring-
servlet-0.3.11.jar:/home/archive/idas/opt/storm-0.5.4/lib/servlet-
api-2.5-20081211.jar:/home/archive/idas/opt/storm-0.5.4/lib/servlet-
api-2.5.jar:/home/archive/idas/opt/storm-0.5.4/lib/slf4j-
api-1.5.8.jar:/home/archive/idas/opt/storm-0.5.4/lib/slf4j-
log4j12-1.5.8.jar:/home/archive/idas/opt/storm-0.5.4/lib/
tools.macro-0.1.0.jar:/home/archive/idas/opt/storm-0.5.4/lib/
zookeeper-3.3.2.jar:storm-starter-0.0.1-SNAPSHOT-standalone.jar:/
gscuser/idas/.storm:/home/archive/idas/opt/storm-0.5.4/bin

storm.starter.ExclamationTopology


0 [main] ERROR org.apache.zookeeper.server.NIOServerCnxn -
Thread Thread[main,5,main] died
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:
119)
at
sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
at
sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52)
at org.apache.zookeeper.server.NIOServerCnxn
$Factory.<init>(NIOServerCnxn.java:144)
at org.apache.zookeeper.server.NIOServerCnxn
$Factory.<init>(NIOServerCnxn.java:125)
at backtype.storm.zookeeper
$mk_inprocess_zookeeper.invoke(zookeeper.clj:101)
at backtype.storm.testing
$mk_local_storm_cluster.doInvoke(testing.clj:101)
at clojure.lang.RestFn.invoke(RestFn.java:398)
at backtype.storm.LocalCluster

$_init.invoke(LocalCluster.clj:10)


at backtype.storm.LocalCluster.<init>(Unknown Source)
at

storm.starter.ExclamationTopology.main(ExclamationTopology.java:67)

6. xterm[1]: $ zkServer.sh stop # stopped the zookeeper daemon
7. xterm[4]: $ storm jar storm-starter-0.0.1-SNAPSHOT-standalone.jar
storm.starter.ExclamationTopology
$ # 2nd time submitted the storm exclamation topology
(pure java example)

saw the following successful abridged output:

0 [main-EventThread] INFO backtype.storm.zookeeper -


Zookeeper state update: :connected:none

69 [main-EventThread] INFO backtype.storm.zookeeper -


Zookeeper state update: :connected:none

78 [main-EventThread] INFO backtype.storm.zookeeper -


Zookeeper state update: :connected:none

97 [main-EventThread] INFO backtype.storm.zookeeper -


Zookeeper state update: :connected:none

160 [main-EventThread] INFO backtype.storm.zookeeper -


Zookeeper state update: :connected:none

227 [main] INFO backtype.storm.daemon.nimbus - Received
topology submission for test with conf {"topology.serializations" {},
"storm.id" "test-1-1321994107", "topology.debug" true}
364 [main] INFO backtype.storm.daemon.nimbus - Task
test-1-1321994107:1 timed out
365 [main] INFO backtype.storm.daemon.nimbus - Task
test-1-1321994107:2 timed out

...

470 [main] INFO backtype.storm.daemon.nimbus - Available slots:
(["03586326-dcf8-45ed-81d6-a6b165c95424" 1] ["03586326-dcf8-45ed-81d6-
a6b165c95424" 2] ["03586326-dcf8-45ed-81d6-a6b165c95424" 3] ["83758bf1-
fde8-49e4-b739-1a80396a9901" 4] ["83758bf1-fde8-49e4-
b739-1a80396a9901" 5] ["83758bf1-fde8-49e4-b739-1a80396a9901" 6])
511 [main] INFO backtype.storm.daemon.nimbus - Setting new
assignment for storm id test-1-1321994107:
#:backtype.storm.daemon.common.Assignment{:master-code-dir "/tmp/
ec3780bc-5ee9-49d7-a3bd-8cd86e1e56a2/nimbus/stormdist/
test-1-1321994107", :node->host {"03586326-dcf8-45ed-81d6-
a6b165c95424" "indraniel.wustl.edu"}, :task->node+port {1 ["03586326-
dcf8-45ed-81d6-a6b165c95424" 1], 2 ["03586326-dcf8-45ed-81d6-
a6b165c95424" 1], 3 ["03586326-dcf8-45ed-81d6-a6b165c95424" 1], 4
["03586326-dcf8-45ed-81d6-a6b165c95424" 1], 5 ["03586326-
dcf8-45ed-81d6-a6b165c95424" 1], 6 ["03586326-dcf8-45ed-81d6-
a6b165c95424" 1], 7 ["03586326-dcf8-45ed-81d6-a6b165c95424" 1], 8
["03586326-dcf8-45ed-81d6-a6b165c95424" 1], 9 ["03586326-
dcf8-45ed-81d6-a6b165c95424" 1], 10 ["03586326-dcf8-45ed-81d6-
a6b165c95424" 1], 11 ["03586326-dcf8-45ed-81d6-a6b165c95424" 1], 12
["03586326-dcf8-45ed-81d6-a6b165c95424" 1], 13 ["03586326-
dcf8-45ed-81d6-a6b165c95424" 1], 14 ["03586326-dcf8-45ed-81d6-
a6b165c95424" 1], 15 ["03586326-dcf8-45ed-81d6-a6b165c95424" 1], 16
["03586326-dcf8-45ed-81d6-a6b165c95424" 1]}, :task->start-time-secs {1
1321994108, 2 1321994108, 3 1321994108, 4 1321994108, 5 1321994108, 6
1321994108, 7 1321994108, 8 1321994108, 9 1321994108, 10 1321994108,
11 1321994108, 12 1321994108, 13 1321994108, 14 1321994108, 15
1321994108, 16 1321994108}}
525 [main] INFO backtype.storm.daemon.nimbus - Activating test:
test-1-1321994107
537 [Thread-11] INFO backtype.storm.daemon.supervisor -
Downloading code for storm id test-1-1321994107 from /tmp/
ec3780bc-5ee9-49d7-a3bd-8cd86e1e56a2/nimbus/stormdist/
test-1-1321994107
539 [Thread-11] INFO backtype.storm.daemon.supervisor -
Finished downloading code for storm id test-1-1321994107 from /tmp/
ec3780bc-5ee9-49d7-a3bd-8cd86e1e56a2/nimbus/stormdist/
test-1-1321994107
540 [Thread-6] INFO backtype.storm.daemon.supervisor -
Downloading code for storm id test-1-1321994107 from /tmp/
ec3780bc-5ee9-49d7-a3bd-8cd86e1e56a2/nimbus/stormdist/
test-1-1321994107
541 [Thread-6] INFO backtype.storm.daemon.supervisor - Finished
downloading code for storm id test-1-1321994107 from /tmp/
ec3780bc-5ee9-49d7-a3bd-8cd86e1e56a2/nimbus/stormdist/
test-1-1321994107
583 [Thread-7] INFO backtype.storm.daemon.supervisor -


Launching worker with assignment
#:backtype.storm.daemon.supervisor.LocalAssignment{:storm-id

"test-1-1321994107", :task-ids (1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
16)} for this supervisor 03586326-dcf8-45ed-81d6-a6b165c95424 on port
1 with id addf9f43-d784-4fb3-ba17-fa5795c3774e
584 [Thread-7] INFO backtype.storm.daemon.worker - Launching
worker for test-1-1321994107 on 03586326-dcf8-45ed-81d6-a6b165c95424:1
with id addf9f43-d784-4fb3-ba17-fa5795c3774e
587 [Thread-7-EventThread] INFO backtype.storm.zookeeper -


Zookeeper state update: :connected:none

863 [Thread-30] INFO backtype.storm.daemon.task - Emitting:
class backtype.storm.testing.TestWordSpout source: 1:6, stream: 1, id:
{}, [nathan]
877 [Thread-34] INFO backtype.storm.daemon.task - Emitting:
class backtype.storm.testing.TestWordSpout source: 1:7, stream: 1, id:
{}, [bertels]
890 [Thread-38] INFO backtype.storm.daemon.task - Emitting:
class backtype.storm.testing.TestWordSpout source: 1:8, stream: 1, id:
{}, [jackson]
901 [Thread-42] INFO backtype.storm.daemon.task - Emitting:
class backtype.storm.testing.TestWordSpout source: 1:9, stream: 1, id:
{}, [mike]
914 [Thread-46] INFO backtype.storm.daemon.task - Emitting:
class backtype.storm.testing.TestWordSpout source: 1:10, stream: 1,
id: {}, [jackson]
924 [Thread-50] INFO backtype.storm.daemon.task - Emitting:
class backtype.storm.testing.TestWordSpout source: 1:11, stream: 1,
id: {}, [jackson]
926 [Thread-21] INFO backtype.storm.daemon.task - Emitting:
class storm.starter.ExclamationTopology$ExclamationBolt source: 2:2,
stream: 1, id: {}, [nathan!!!]
926 [Thread-21] INFO backtype.storm.daemon.task - Emitting:
class storm.starter.ExclamationTopology$ExclamationBolt source: 2:2,
stream: 1, id: {}, [jackson!!!]
926 [Thread-23] INFO backtype.storm.daemon.task - Emitting:
class storm.starter.ExclamationTopology$ExclamationBolt source: 2:3,
stream: 1, id: {}, [mike!!!]
927 [Thread-19] INFO backtype.storm.daemon.task - Emitting:
class storm.starter.ExclamationTopology$ExclamationBolt source: 2:1,
stream: 1, id: {}, [bertels!!!]

...

10563 [main] INFO backtype.storm.daemon.nimbus - Deactivated
test and scheduled to be killed
10563 [main] INFO backtype.storm.daemon.nimbus - Shutting down
master
10563 [Thread-5] INFO backtype.storm.util - Async loop
interrupted!
10564 [Thread-50] INFO backtype.storm.daemon.task - Emitting:
class backtype.storm.testing.TestWordSpout source: 1:11, stream: 1,
id: {}, [bertels]
10566 [Thread-21] INFO backtype.storm.daemon.task - Emitting:
class storm.starter.ExclamationTopology$ExclamationBolt source: 2:2,
stream: 1, id: {}, [bertels!!!]
10566 [Thread-1] INFO backtype.storm.event - Event manager
interrupted
10566 [Thread-25] INFO backtype.storm.daemon.task - Emitting:
class storm.starter.ExclamationTopology$ExclamationBolt source: 3:4,
stream: 1, id: {}, [bertels!!!!!!]
10567 [Thread-2] INFO backtype.storm.daemon.nimbus - Entering
kill loop for {"test-1-1321994107" 1321994148}
10567 [Thread-2] INFO backtype.storm.daemon.nimbus - Waiting for
30 seconds to kill topology test-1-1321994107
10567 [Thread-2] INFO backtype.storm.event - Event manager
interrupted
10571 [main] INFO backtype.storm.daemon.nimbus - Shut down
master
10574 [main] INFO backtype.storm.daemon.supervisor - Shutting
down 03586326-dcf8-45ed-81d6-a6b165c95424:addf9f43-d784-4fb3-ba17-
fa5795c3774e
10574 [main] INFO backtype.storm.process-simulator - Killing
process c5149a1a-8e56-4f88-bee8-d9c7b38520ef

...

10691 [Thread-12] INFO backtype.storm.event - Event manager
interrupted
10695 [main] INFO backtype.storm.testing - Shutting down in
process zookeeper
10697 [main] INFO backtype.storm.testing - Done shutting down in
process zookeeper
10697 [main] INFO backtype.storm.testing - Deleting temporary
path /tmp/ec3780bc-5ee9-49d7-a3bd-8cd86e1e56a2
10698 [main] INFO backtype.storm.testing - Deleting temporary
path /tmp/89bf8386-d90a-4b65-8f3e-08235f3f20c1
10698 [main] INFO backtype.storm.testing - Deleting temporary
path /tmp/4bd81876-b148-4829-8303-26580f9c8fcf
10699 [main] INFO backtype.storm.testing - Deleting temporary
path /tmp/c542b49e-165a-41f5-88b4-2ad2ddcdb5f7


7. xterm[4]: $ storm jar storm-starter-0.0.1-SNAPSHOT-standalone.jar
storm.starter.WordCountTopology
$ # 1st time submitting the storm exclamation topology
(python/multilang example)

saw the following abridged failure output:

0 [main-EventThread] INFO backtype.storm.zookeeper -


Zookeeper state update: :connected:none

63 [main-EventThread] INFO backtype.storm.zookeeper -


Zookeeper state update: :connected:none

76 [main-EventThread] INFO backtype.storm.zookeeper -


Zookeeper state update: :connected:none

98 [main-EventThread] INFO backtype.storm.zookeeper -


Zookeeper state update: :connected:none

168 [main-EventThread] INFO backtype.storm.zookeeper -


Zookeeper state update: :connected:none

244 [main] INFO backtype.storm.daemon.nimbus - Received


topology submission for word-count with conf

{"topology.serializations" {}, "storm.id" "word-count-1-1321994666",
"topology.max.task.parallelism" 3, "topology.debug" true}
355 [main] INFO backtype.storm.daemon.nimbus - Task word-
count-1-1321994666:1 timed out
357 [main] INFO backtype.storm.daemon.nimbus - Task word-
count-1-1321994666:2 timed out
358 [main] INFO backtype.storm.daemon.nimbus - Task word-
count-1-1321994666:3 timed out
359 [main] INFO backtype.storm.daemon.nimbus - Task word-
count-1-1321994666:4 timed out
361 [main] INFO backtype.storm.daemon.nimbus - Task word-
count-1-1321994666:5 timed out
362 [main] INFO backtype.storm.daemon.nimbus - Task word-
count-1-1321994666:6 timed out
364 [main] INFO backtype.storm.daemon.nimbus - Task word-
count-1-1321994666:7 timed out
365 [main] INFO backtype.storm.daemon.nimbus - Task word-
count-1-1321994666:8 timed out
366 [main] INFO backtype.storm.daemon.nimbus - Task word-
count-1-1321994666:9 timed out
368 [main] INFO backtype.storm.daemon.nimbus - Task word-
count-1-1321994666:10 timed out
371 [main] INFO backtype.storm.daemon.nimbus - Reassigning word-
count-1-1321994666 to 1 slots
453 [main] INFO backtype.storm.daemon.nimbus - Reassign ids: [1


2 3 4 5 6 7 8 9 10]

470 [main] INFO backtype.storm.daemon.nimbus - Available slots:
(["91d33af3-358c-472c-b9d0-5d79a63de995" 4] ["91d33af3-358c-472c-
b9d0-5d79a63de995" 5] ["91d33af3-358c-472c-b9d0-5d79a63de995" 6]
["85242e75-9541-490d-9a6b-eea17e1b5da2" 1] ["85242e75-9541-490d-9a6b-
eea17e1b5da2" 2] ["85242e75-9541-490d-9a6b-eea17e1b5da2" 3])
511 [main] INFO backtype.storm.daemon.nimbus - Setting new
assignment for storm id word-count-1-1321994666:
#:backtype.storm.daemon.common.Assignment{:master-code-dir "/tmp/
f4704980-9df6-4640-b6c2-7bf08be52511/nimbus/stormdist/word-
count-1-1321994666", :node->host {"91d33af3-358c-472c-
b9d0-5d79a63de995" "indraniel.gsc.wustl.edu"}, :task->node+port {1
["91d33af3-358c-472c-b9d0-5d79a63de995" 4], 2 ["91d33af3-358c-472c-
b9d0-5d79a63de995" 4], 3 ["91d33af3-358c-472c-b9d0-5d79a63de995" 4], 4
["91d33af3-358c-472c-b9d0-5d79a63de995" 4], 5 ["91d33af3-358c-472c-
b9d0-5d79a63de995" 4], 6 ["91d33af3-358c-472c-b9d0-5d79a63de995" 4], 7
["91d33af3-358c-472c-b9d0-5d79a63de995" 4], 8 ["91d33af3-358c-472c-
b9d0-5d79a63de995" 4], 9 ["91d33af3-358c-472c-b9d0-5d79a63de995" 4],
10 ["91d33af3-358c-472c-b9d0-5d79a63de995" 4]}, :task->start-time-secs
{1 1321994666, 2 1321994666, 3 1321994666, 4 1321994666, 5 1321994666,
6 1321994666, 7 1321994666, 8 1321994666, 9 1321994666, 10
1321994666}}
526 [main] INFO backtype.storm.daemon.nimbus - Activating word-
count: word-count-1-1321994666
540 [Thread-11] INFO backtype.storm.daemon.supervisor -
Downloading code for storm id word-count-1-1321994666 from /tmp/
f4704980-9df6-4640-b6c2-7bf08be52511/nimbus/stormdist/word-
count-1-1321994666
541 [Thread-6] INFO backtype.storm.daemon.supervisor -
Downloading code for storm id word-count-1-1321994666 from /tmp/
f4704980-9df6-4640-b6c2-7bf08be52511/nimbus/stormdist/word-
count-1-1321994666
542 [Thread-11] INFO backtype.storm.daemon.supervisor -
Finished downloading code for storm id word-count-1-1321994666 from /
tmp/f4704980-9df6-4640-b6c2-7bf08be52511/nimbus/stormdist/word-
count-1-1321994666
543 [Thread-6] INFO backtype.storm.daemon.supervisor - Finished
downloading code for storm id word-count-1-1321994666 from /tmp/
f4704980-9df6-4640-b6c2-7bf08be52511/nimbus/stormdist/word-
count-1-1321994666
582 [Thread-12] INFO backtype.storm.daemon.supervisor -


Launching worker with assignment
#:backtype.storm.daemon.supervisor.LocalAssignment{:storm-id "word-

count-1-1321994666", :task-ids (1 2 3 4 5 6 7 8 9 10)} for this
supervisor 91d33af3-358c-472c-b9d0-5d79a63de995 on port 4 with id
329fcacd-161a-48a8-a5d3-153cbf0b315c
582 [Thread-12] INFO backtype.storm.daemon.worker - Launching
worker for word-count-1-1321994666 on 91d33af3-358c-472c-
b9d0-5d79a63de995:4 with id 329fcacd-161a-48a8-a5d3-153cbf0b315c
586 [Thread-12-EventThread] INFO backtype.storm.zookeeper -


Zookeeper state update: :connected:none

717 [Thread-12] ERROR backtype.storm.daemon.worker - Error on
initialization of server mk-worker


java.lang.RuntimeException: java.lang.RuntimeException:
java.io.IOException: Cannot run program "python" (in directory "/tmp/

b694d5e2-984c-4be3-8505-5e1e4e9aec7d/supervisor/stormdist/word-
count-1-1321994666/resources"): java.io.IOException: error=2, No such


file or directory
at clojure.lang.LazySeq.sval(LazySeq.java:47)

at clojure.lang.LazySeq.seq(LazySeq.java:56)
at clojure.lang.RT.seq(RT.java:450)
at clojure.core$seq.invoke(core.clj:122)
at clojure.core$dorun.invoke(core.clj:2450)
at clojure.core$doall.invoke(core.clj:2465)
at backtype.storm.daemon.worker
$fn__3074$exec_fn__858__auto____3075.invoke(worker.clj:172)
at clojure.lang.AFn.applyToHelper(AFn.java:187)
at clojure.lang.AFn.applyTo(AFn.java:151)
at clojure.core$apply.invoke(core.clj:540)
at backtype.storm.daemon.worker
$fn__3074$mk_worker__3216.doInvoke(worker.clj:78)
at clojure.lang.RestFn.invoke(RestFn.java:513)
at backtype.storm.daemon.supervisor
$fn__3668.invoke(supervisor.clj:398)
at clojure.lang.MultiFn.invoke(MultiFn.java:188)
at backtype.storm.daemon.supervisor
$fn__3452$exec_fn__858__auto____3453$sync_processes__3455$iter__3512__3516$fn__3517.invoke(supervisor.clj:
233)
at clojure.lang.LazySeq.sval(LazySeq.java:42)
at clojure.lang.LazySeq.seq(LazySeq.java:56)
at clojure.lang.RT.seq(RT.java:450)

...

Caused by: java.lang.RuntimeException: java.io.IOException: Cannot


run program "python" (in directory "/tmp/

b694d5e2-984c-4be3-8505-5e1e4e9aec7d/supervisor/stormdist/word-
count-1-1321994666/resources"): java.io.IOException: error=2, No su
at
backtype.storm.task.ShellBolt.initializeSubprocess(ShellBolt.java:83)

...

Caused by: java.io.IOException: Cannot run program "python" (in
directory "/tmp/b694d5e2-984c-4be3-8505-5e1e4e9aec7d/supervisor/
stormdist/word-count-1-1321994666/resources"): java.io.IOException:


error=2, No such file or directory

at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
at
backtype.storm.task.ShellBolt.initializeSubprocess(ShellBolt.java:74)
... 32 more

Caused by: java.io.IOException: java.io.IOException: error=2, No
such file or directory


at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
... 33 more

746 [Thread-12] INFO backtype.storm.util - Halting process:

nathanmarz

unread,
Nov 22, 2011, 11:11:56 PM11/22/11
to storm-user
OK, I see what you're doing wrong here. You most definitely should not
touch zookeeper once you have it running.

If you look at the code for ExclamationTopology, you can see that it
submits in local mode if you provide no arguments to the main
function. Otherwise, it submits it in remote mode with the name of the
topology being the first argument to the main function. So you should
try submitting it in remote mode by providing an argument, e.g.:

storm jar storm-starter-0.0.1-SNAPSHOT-standalone.jar
storm.starter.ExclamationTopology test-topology

You'll be able to see that it's working by seeing messages being
logged to the worker logs.

It sounds like you've been running WordCountTopology in local mode as
well, and the python isn't working from there. Have you tried using
Ubuntu 10.10?

-Nathan

> 16)} for this supervisor 03586326-dcf8-45ed-81d6-a6b165c95424 ...
>
> read more »

indraniel

unread,
Nov 28, 2011, 11:43:40 PM11/28/11
to storm-user
Nathan,

Your observation/suggestion seemed to have worked. I am now able to
submit storm jobs to my currently one-node cluster. Both the
ExclamationTopology and the WordCountTopology now work for me.

I also had to supply an argument for both the storm-starter topology
examples to enable remote job submissions as well.

storm jar storm-starter-0.0.1-SNAPSHOT-standalone.jar
storm.starter.ExclamationTopology et-test
storm jar storm-starter-0.0.1-SNAPSHOT-standalone.jar
storm.starter.WordCountTopology wc-test

I have not tried using Ubuntu 10.10 yet. The Linux distributions that
are currently available to me at my place of work, The Genome
Institute at Washington University -- http://genome.wustl.edu, are
based upon the Ubuntu long-term-support (LTS) distributions. Ubuntu
10.10 was not a LTS distribution, but 10.04 was. As an initial foray,
I was experimenting with Storm based upon the latest Ubuntu LTS
distribution available.

I will now continue onwards exploring the possible things I can
attempt with Storm. It seems like an interesting platform to possibly
improve the performance of a few genomics-related pipelines that occur
here. Thanks again for the help and advice so far!

-Indraniel

> ...
>
> read more »

Nathan Marz

unread,
Nov 29, 2011, 2:16:08 AM11/29/11
to storm...@googlegroups.com
I'm glad that you can at least get pure Java topologies working. I opened up an issue to try to reproduce the multilang issues on 10.04: https://github.com/nathanmarz/storm/issues/55

-Nathan

--
Twitter: @nathanmarz
http://nathanmarz.com

sushma shankar

unread,
Apr 12, 2012, 9:28:34 AM4/12/12
to storm...@googlegroups.com
Hi Nathan,
Iam trying to run exclamationTopology on remote mode..but it is giving the following error..
am using storm 0.7.0
Local mode is working fine.
here is the error
0    [main] INFO  backtype.storm.StormSubmitter  - Jar not uploaded to master yet. Submitting jar...
Exception in thread "main" java.lang.IllegalArgumentException: Nimbus host is not set
        at backtype.storm.utils.NimbusClient.<init>(NimbusClient.java:30)
        at backtype.storm.utils.NimbusClient.getConfiguredClient(NimbusClient.java:17)
        at backtype.storm.StormSubmitter.submitJar(StormSubmitter.java:78)
        at backtype.storm.StormSubmitter.submitJar(StormSubmitter.java:71)
        at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:50)
        at sushma.starter.ExclamationTopology.main(ExclamationTopology.java:67)
 
 
Plz help..

James Xu

unread,
Apr 12, 2012, 12:01:40 PM4/12/12
to storm...@googlegroups.com
For submitting topology to real cluster you need a config file in ~/.storm/storm.yaml. And config nimbus.host. For more, take a look at the wiki.

sushma shankar

unread,
Apr 16, 2012, 1:14:00 AM4/16/12
to storm-user
Thanks it worked fine. Can i submit the topology remotely to storm
cluster through eclipse itself?
> >>https://github.com/**nathanmarz/storm/issues/55<https://github.com/nathanmarz/storm/issues/55>
>
> >> -Nathan
>
> >> On Mon, Nov 28, 2011 at 11:43 PM, indraniel <indran...@gmail.com<javascript:_e({}, 'cvml', 'indran...@gmail.com');>
> >> > wrote:
>
> >>> Nathan,
>
> >>> Your observation/suggestion seemed to have worked. I am now able to
> >>> submit storm jobs to my currently one-node cluster. Both the
> >>> ExclamationTopology and the WordCountTopology now work for me.
>
> >>> I also had to supply an argument for both the storm-starter topology
> >>> examples to enable remote job submissions as well.
>
> >>> storm jar storm-starter-0.0.1-SNAPSHOT-**standalone.jar
> >>> storm.starter.**ExclamationTopology et-test
> >>> storm jar storm-starter-0.0.1-SNAPSHOT-**standalone.jar
> >>> storm.starter.**WordCountTopology wc-test
>
> >>> I have not tried using Ubuntu 10.10 yet. The Linux distributions that
> >>> are currently available to me at my place of work, The Genome
> >>> Institute at Washington University --http://genome.wustl.edu, are
> >>> based upon the Ubuntu long-term-support (LTS) distributions. Ubuntu
> >>> 10.10 was not a LTS distribution, but 10.04 was. As an initial foray,
> >>> I was experimenting with Storm based upon the latest Ubuntu LTS
> >>> distribution available.
>
> >>> I will now continue onwards exploring the possible things I can
> >>> attempt with Storm. It seems like an interesting platform to possibly
> >>> improve the performance of a few genomics-related pipelines that occur
> >>> here. Thanks again for the help and advice so far!
>
> >>> -Indraniel
>
> >>> On Nov 22, 10:11 pm, nathanmarz <nathan.m...@gmail.com> wrote:
> >>> > OK, I see what you're doing wrong here. You most definitely should not
> >>> > touch zookeeper once you have it running.
>
> >>> > If you look at the code for ExclamationTopology, you can see that it
> >>> > submits in local mode if you provide no arguments to the main
> >>> > function. Otherwise, it submits it in remote mode with the name of the
> >>> > topology being the first argument to the main function. So you should
> >>> > try submitting it in remote mode by providing an argument, e.g.:
>
> >>> > storm jar storm-starter-0.0.1-SNAPSHOT-**standalone.jar
> >>> > storm.starter.**ExclamationTopology test-topology
> >>> > > to localhost.localdomain/127.0.0.**1:2181 <http://127.0.0.1:2181>,
> >>> initiating session
> >>> > > 2011-11-22 14:22:41 ClientCnxn [INFO] Session establishment
> >>> > > complete on server localhost.localdomain/127.0.0.**1:2181<http://127.0.0.1:2181>,
> >>> sessionid =
> >>> > > 0x133ccf10e960001, negotiated timeout = 20000
> >>> > > 2011-11-22 14:22:41 zookeeper [INFO] Zookeeper state
> >>> > > update: :connected:none
> >>> > > 2011-11-22 14:22:41 ClientCnxn [INFO] EventThread shut down
> >>> > > 2011-11-22 14:22:41 nimbus [INFO] Starting Nimbus server...
>
> >>> > > 4. xterm[3]: $ storm supervisor # started supervisor daemon
>
> >>> > > saw the following in the supervisor.log:
>
> >>> > > 2011-11-22 14:24:20 ZooKeeper [INFO] Initiating client connection,
> >>> > > connectString=127.0.0.1:2181/**storm <http://127.0.0.1:2181/storm>sessionTimeout=20000
> >>> > > watcher=backtype.storm.**zookeeper$mk_client$reify__**1247@1cecd92c
> >>> > > 2011-11-22 14:24:20 ClientCnxn [INFO] Opening socket connection to
> >>> > > server /127.0.0.1:2181
> >>> > > 2011-11-22 14:24:20 ClientCnxn [INFO] Socket connection
> >>> established
> >>> > > to localhost.localdomain/127.0.0.**1:2181 <http://127.0.0.1:2181>,
> >>> initiating session
> >>> > > 2011-11-22 14:24:20 ClientCnxn [INFO] Session establishment
> >>> > > complete on server localhost.localdomain/127.0.0.**1:2181<http://127.0.0.1:2181>,
> >>> sessionid =
> >>> > > 0x133ccf10e960003, negotiated timeout = 20000
>
> >>> > > 5. xterm[4]: $ storm jar storm-starter-0.0.1-SNAPSHOT-**
> >>> standalone.jar
> >>> > > storm.starter.**ExclamationTopology
> >>> > > $ # submitted the storm exclamation topology (pure java
> >>> > > example)
>
> >>> > > saw the following output error:
>
> >>> > > Running: export STORM_JAR=storm-starter-0.0.1-**SNAPSHOT-
> >>> > > standalone.jar; java -client -Djava.library.path=/home/**
> >>> archive/idas/
> >>> > > lib -cp /home/archive/idas/opt/storm-**0.5.4/storm-0.5.4.jar:/home/
> >>> > > archive/idas/opt/storm-0.5.4/**lib/clj-time-0.3.0.j
> >>> > > ar:/home/archive/idas/opt/**storm-0.5.4/lib/clojure-1.2.0.**
> >>> jar:/home/
> >>> > > archive/idas/opt/storm-0.5.4/**lib/clojure-contrib-1.2.0.jar:**
> >>> /home/
> >>> > > archive/idas/opt/storm-0.5.4/**lib/clout-0.4.1.jar:/home/**
> >>> archive/idas/
> >>> > > opt/storm-0.5.4/lib/commons-**codec-1.3.j
> >>> > > ar:/home/archive/idas/opt/**storm-0.5.4/lib/commons-exec-**
> >>> 1.1.jar:/
> >>> > > home/archive/idas/opt/storm-0.**5.4/lib/commons-fileupload-1.**
> >>> 2.1.jar:/
> >>> > > home/archive/idas/opt/storm-0.**5.4/lib/commons-io-1.4.jar:/**
> >>> home/archive/
> >>> > > idas/opt/storm-0.5.4/lib/**commons-lang-2.5.jar:/home/**
> >>> archive/idas/opt/
> >>> > > storm-0.5.4/lib/commons-**logging-1.1.1.jar:/home/**
> >>> archive/idas/opt/
> >>> > > storm-0.5.4/lib/compojure-0.6.**4.jar:/home/archive/idas/opt/**
> >>> storm-0.5.4/
> >>> > > lib/core.incubator-0.1.0.jar:/**home/archive/idas/opt/storm-0.**
> >>> 5.4/lib/
> >>> > > hiccup-0.3.6.jar:/home/**archive/idas/opt/storm-0.5.4/**lib/
> >>> > > httpclient-4.0.1.jar:/home/**archive/idas/opt/storm-0.5.4/**lib/
> >>> > > httpcore-4.0.1.jar:/home/**archive/idas/opt/storm-0.5.4/**lib/
> >>> > > jetty-6.1.26.jar:/home/**archive/idas/opt/storm-0.5.4/**lib/jetty-
> >>> > > util-6.1.26.jar:/home/archive/**idas/opt/storm-0.5.4/lib/
> >>> > > jline-0.9.94.jar:/home/**archive/idas/opt/storm-0.5.4/**lib/joda-
> >>> > > time-1.6.jar:/home/archive/**idas/opt/storm-0.5.4/lib/json-
> >>> > > simple-1.1.jar:/home/archive/**idas/opt/storm-0.5.4/lib/**
> >>> junit-3.8.1.jar:/
> >>> > > home/archive/idas/opt/storm-0.**5.4/lib/jvyaml-1.0.0.jar:/**
> >>> home/archive/
> >>> > > idas/opt/storm-0.5.4/lib/jzmq-**2.1.0.jar:/home/archive/idas/**opt/
> >>> > > storm-0.5.4/lib/libthrift7-0.**7.0.jar:/home/archive/idas/**opt/
> >>> > > storm-0.5.4/lib/log4j-1.2.16.**jar:/home/archive/idas/opt/**
> >>> storm-0.5.4/
> >>> > > lib/ring-core-0.3.10.jar:/**home/archive/idas/opt/storm-0.**
> >>> 5.4/lib/ring-
> >>> > > jetty-adapter-0.3.11.jar:/**home/archive/idas/opt/storm-0.**
> >>> 5.4/lib/ring-
> >>> > > servlet-0.3.11.jar:/home/**archive/idas/opt/storm-0.5.4/**
> >>> lib/servlet-
> >>> > > api-2.5-20081211.jar:/home/**archive/idas/opt/storm-0.5.4/**
> >>> lib/servlet-
> >>> > > api-2.5.jar:/home/archive/**idas/opt/storm-0.5.4/lib/**slf4j-
> >>> > > api-1.5.8.jar:/home/archive/**idas/opt/storm-0.5.4/lib/**slf4j-
> >>> > > log4j12-1.5.8.jar:/home/**archive/idas/opt/storm-0.5.4/**lib/
> >>> > > tools.macro-0.1.0.jar:/home/**archive/idas/opt/storm-0.5.4/**lib/
> >>> > > zookeeper-3.3.2.jar:storm-**starter-0.0.1-SNAPSHOT-**
> >>> standalone.jar:/
> >>> > > gscuser/idas/.storm:/home/**archive/idas/opt/storm-0.5.4/**bin
>
> ...
>
> read more >>- Hide quoted text -
>
> - Show quoted text -

Nathan Marz

unread,
Apr 18, 2012, 3:10:09 AM4/18/12
to storm...@googlegroups.com
I haven't tried that, and I'm not sure how much work that would be to enable.

2012/4/15 sushma shankar <sushma...@gmail.com>

RP2

unread,
Jun 9, 2012, 3:15:13 AM6/9/12
to storm...@googlegroups.com
Indraneil and Nathan:

I have also been struggling to get the multilang topology to work in remote mode. I have posted a question in that regard https://groups.google.com/forum/?fromgroups#!topic/storm-user/HT6qM3BGm6w

I have also been getting this error. 
Caused by: java.io.IOException: Cannot run program "python" (in directory "/tmp/storm/supervisor/stormdist/zmet-1-1339167237/resources"): java.io.IOException: error=2, No such file or directory 

I have tried with all possible paths to python. Do you know how to fix this?

=Aman

RP2

unread,
Jun 9, 2012, 11:18:03 AM6/9/12
to storm...@googlegroups.com
My bad. I didnt have the resources directory in the jar I was uploading and storm tries to extract that. I added the <resources> in the pom and it worked. 

=Aman
Reply all
Reply to author
Forward
Message has been deleted
Message has been deleted
0 new messages