Submitting multilang to a production cluster
Ashley Brown
We were able to quickly get a Storm cluster running in our test
environment yesterday, and have successfully run Java-only test
topologies.
However, submitting the word-count example gave us the error below
(full error at bottom of this message):
===
java.lang.RuntimeException: java.lang.RuntimeException:
java.io.IOException: Cannot run program "/usr/bin/python" (in
directory "/ebs_data/storm/data/stormdist/word-count-4-1316610254/resources"):
java.io.IOException: error=2, No such file or directory
[snip]
===
/ebs_data/storm/data/ is our 'storm.local.dir'.
On investigation, it seems that 'resources' in
'stormdist/word-count-4-1316610254/' does not exist. I presume that
'resources' should have been extracted from the JAR file (confirmed
that it does exist in the JAR - I'm able to extract it manually) to
provide splitsentence.py and storm.py.
I'm using the File | Export | ... JAR File method in Eclipse
(remembering to include the resources).
Any thoughts on whether this is a problem with our JAR file creation
(in which case - what's the best way to do it?) or something wrong on
our production setup?
A
====
The full error:
2011-09-21 13:21:54 worker [ERROR] Error on initialization of server mk-worker
java.lang.RuntimeException: java.lang.RuntimeException:
java.io.IOException: Cannot run program "/usr/bin/python" (in
directory "/ebs_data/storm/data/stormdist/word-count-4-1316610254/resources"):
java.io.IOException: error=2, No such file or directory
at clojure.lang.LazySeq.sval(LazySeq.java:47)
at clojure.lang.LazySeq.seq(LazySeq.java:56)
at clojure.lang.RT.seq(RT.java:450)
at clojure.core$seq.invoke(core.clj:122)
at clojure.core$dorun.invoke(core.clj:2450)
at clojure.core$doall.invoke(core.clj:2465)
at backtype.storm.daemon.worker$fn__2916$exec_fn__837__auto____2917.invoke(worker.clj:169)
at clojure.lang.AFn.applyToHelper(AFn.java:180)
at clojure.lang.AFn.applyTo(AFn.java:151)
at clojure.core$apply.invoke(core.clj:540)
at backtype.storm.daemon.worker$fn__2916$mk_worker__3058.doInvoke(worker.clj:78)
at clojure.lang.RestFn.invoke(RestFn.java:483)
at backtype.storm.daemon.worker$_main.invoke(worker.clj:254)
at clojure.lang.AFn.applyToHelper(AFn.java:174)
at clojure.lang.AFn.applyTo(AFn.java:151)
at backtype.storm.daemon.worker.main(Unknown Source)
Caused by: java.lang.RuntimeException: java.io.IOException: Cannot run
program "/usr/bin/python" (in directory
"/ebs_data/storm/data/stormdist/word-count-4-1316610254/resources"):
java.io.IOException: error=2, No such file or directory
at backtype.storm.task.ShellBolt.initializeSubprocess(ShellBolt.java:83)
at backtype.storm.task.ShellBolt.prepare(ShellBolt.java:89)
at backtype.storm.daemon.task$fn__2867.invoke(task.clj:456)
at clojure.lang.MultiFn.invoke(MultiFn.java:194)
at backtype.storm.daemon.task$mk_task.invoke(task.clj:275)
at backtype.storm.daemon.worker$fn__2916$exec_fn__837__auto____2917$iter__2962__2966$fn__2967$fn__2968.invoke(worker.clj:169)
at backtype.storm.daemon.worker$fn__2916$exec_fn__837__auto____2917$iter__2962__2966$fn__2967.invoke(worker.clj:169)
at clojure.lang.LazySeq.sval(LazySeq.java:42)
... 15 more
Caused by: java.io.IOException: Cannot run program "/usr/bin/python"
(in directory "/ebs_data/storm/data/stormdist/word-count-4-1316610254/resources"):
java.io.IOException: error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
at backtype.storm.task.ShellBolt.initializeSubprocess(ShellBolt.java:74)
... 22 more
Caused by: java.io.IOException: java.io.IOException: error=2, No such
file or directory
at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
... 23 more
nathanmarz
Still working on the docs for multilang, but glad to see you're
plowing ahead. To get multilang to work, all your non-Java source
files must be in the resources subdirectory of the jar. To see an
example of what this should look like, run "lein uberjar" in the storm-
starter project. Here's what the internals of that jar look like:
[nathan@Tw-MBP13-NMarz ~/opensource/storm-starter (master)]$ jar -tf
storm-starter-0.0.1-SNAPSHOT-standalone.jar | grep resources
resources/splitsentence.py
resources/storm.py
resources/storm.pyc
resources/storm.rb
You'll see that it packs the "multilang/resources" directory in the
project into the resources subdir in the jar. Script locations in
Shell Bolt are resolved relative to the resources directory -- again,
look at storm-starter for an example of this.
Hope that helps,
Nathan
> at clojure.lang.AFn.applyTo(AFn.java:151)
> at clojure.core$apply.invoke(core.clj:540)
> at backtype.storm.daemon.worker$fn__2916$mk_worker__3058.doInvoke(worker.clj:7 8)
> at clojure.lang.RestFn.invoke(RestFn.java:483)
> at backtype.storm.daemon.worker$_main.invoke(worker.clj:254)
> at clojure.lang.AFn.applyToHelper(AFn.java:174)
> at clojure.lang.AFn.applyTo(AFn.java:151)
> at backtype.storm.daemon.worker.main(Unknown Source)
> Caused by: java.lang.RuntimeException: java.io.IOException: Cannot run
> program "/usr/bin/python" (in directory
> "/ebs_data/storm/data/stormdist/word-count-4-1316610254/resources"):
> java.io.IOException: error=2, No such file or directory
> at backtype.storm.task.ShellBolt.initializeSubprocess(ShellBolt.java:83)
> at backtype.storm.task.ShellBolt.prepare(ShellBolt.java:89)
> at backtype.storm.daemon.task$fn__2867.invoke(task.clj:456)
> at clojure.lang.MultiFn.invoke(MultiFn.java:194)
> at backtype.storm.daemon.task$mk_task.invoke(task.clj:275)
> at backtype.storm.daemon.worker$fn__2916$exec_fn__837__auto____2917$iter__2962 __2966$fn__2967$fn__2968.invoke(worker.clj:169)
Ashley Brown
[snip]
That's what it looked like... I'll run a couple of experiments
tomorrow now we have ant generating jars.
nathanmarz
shell out to the "unzip" command to extract the resources folder from
the jar. Can you check that "unzip" is installed on the worker nodes?
Also, can you check the supervisor logs for any messages of the form
"Error when trying to extract ..."?
-Nathan
nathanmarz
Ashley Brown
It wasn't, but it is now. I've restarted the supervisors and
redeployed the topology - same problem. Manually executing the unzip
command as the same user as the supervisor creates the resources
directory as expected and the topology is able to start, so the
problem must lie with executing unzip within the supervisor.
> Also, can you check the supervisor logs for any messages of the form
> "Error when trying to extract ..."?
Nothing.
I don't have a Clojure dev setup here at the moment (and this is the
first time I've had a look at code in Clojure) so figuring out exactly
what is wrong may be a bit tricky :) Any further hints appreciated!
nathanmarz
(defn exec-command! [command]
(let [[comm-str & args] (seq (.split command " "))
command (CommandLine. comm-str)]
(doseq [a args]
(.addArgument command a))
(.execute (DefaultExecutor.) command)
))
(defn extract-dir-from-jar [jarpath dir destdir]
(try
(exec-command! (str "unzip -qq " jarpath " " dir "/** -d "
destdir))
(catch ExecuteException e
(log-message "Error when trying to extract " dir " from "
jarpath))
))
Can you verify that that this exact unzip command works on the jar the
supervisors are downloading:
unzip -qq {JAR-PATH} resources/** -d {EXTRACT-LOCATION}
It's also possible that "unzip" isn't on the path of the supervisors
for some reason. That's worth looking into. Although I would think
that you would see messages logged about the error in that case.
Finally, what OS are you running Storm on?
-Nathan
Ashley Brown
> supervisors are downloading:
>
> unzip -qq {JAR-PATH} resources/** -d {EXTRACT-LOCATION}
Yes, that works (JAR-PATH is a jar in the 'inbox' directory and
EXTRACT-LOCATION is sourced from the error message when python fails
to execute).
> Finally, what OS are you running Storm on?
Ubuntu 10.10.
Will investigate supervisor paths etc.
nathanmarz
jarring process, can you try using the exact same process I use to
submit a topology to the cluster? You'll need to do the following:
1. Install Leiningen if you haven't already
2. Modify storm-starter's WordCountTopology to submit to a remote
cluster (replace LocalCluster with StormSubmitter)
3. Build the storm-starter jar with "lein deps && lein compile && lein
uberjar"
4. Run the topology with storm jar and verify the debug output
-Nathan
Ashley Brown
jar is uploaded:
2011-09-23 11:43:54 supervisor [INFO] Launching worker with assignment
#:backtype.storm.daemon.supervisor.LocalAssignment{:storm-id
"word-count-3-1316778233", :task-ids (1 2 3 4 5 6 7 8 9 10)} for this
supervisor 9b209e0f-67bb-4523-99cc-e04cba516ccf on port 6700 with id
1d66e4f7-7b0f-4a52-8fe2-3bcb05bf9fd8
2011-09-23 11:43:54 supervisor [INFO]
1d66e4f7-7b0f-4a52-8fe2-3bcb05bf9fd8 still hasn't started
2011-09-23 11:43:54 event [ERROR] Error when processing event
backtype.storm.daemon.supervisor$fn__3298$exec_fn__837__auto____3299$sync_processes__3301@5ffe40d5
java.lang.NullPointerException
at backtype.storm.utils.VersionedStore.listDir(VersionedStore.java:161)
at backtype.storm.utils.VersionedStore.getAllVersions(VersionedStore.java:119)
at backtype.storm.utils.VersionedStore.mostRecentVersion(VersionedStore.java:43)
at backtype.storm.utils.VersionedStore.mostRecentVersionPath(VersionedStore.java:31)
at backtype.storm.utils.LocalState.snapshot(LocalState.java:22)
at backtype.storm.utils.LocalState.get(LocalState.java:28)
at backtype.storm.daemon.supervisor$wait_for_worker_launch.invoke(supervisor.clj:116)
at backtype.storm.daemon.supervisor$wait_for_workers_launch.invoke(supervisor.clj:132)
at backtype.storm.daemon.supervisor$fn__3298$exec_fn__837__auto____3299$sync_processes__3301.invoke(supervisor.clj:218)
at backtype.storm.event$event_manager$fn__1750$fn__1751.invoke(event.clj:25)
At which point the supervisor stops and is picked back up by monit.
I also see a number of exceptions after killing a running topology
(although they look like they are not directly related):
2011-09-23 11:44:00 supervisor [ERROR] Failed to cleanup worker
72397b9d-3c85-4bb9-89c7-2678d86f4939. Will retry later
java.lang.RuntimeException: Failed to delete
/ebs_data/storm/data/workers/72397b9d-3c85-4bb9-89c7-2678d86f4939/pids
at backtype.storm.util$rmpath.invoke(util.clj:263)
at backtype.storm.daemon.supervisor$try_cleanup_worker.invoke(supervisor.clj:142)
at backtype.storm.daemon.supervisor$shutdown_worker.invoke(supervisor.clj:158)
at backtype.storm.daemon.supervisor$fn__3298$exec_fn__837__auto____3299$sync_processes__3301.invoke(supervisor.clj:206)
at backtype.storm.event$event_manager$fn__1750$fn__1751.invoke(event.clj:25)
at backtype.storm.event$event_manager$fn__1750.invoke(event.clj:22)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:662)
The directory it is trying to delete doesn't actually exist. If I
create it, it gets deleted again but still says it hasn't deleted it.
It continues to repeat that error/exception on a loop every few
seconds.
A
nathanmarz
which doesn't make sense given that you were able to run Java-only
topologies.
Are you on AWS by any chance (I ask because one of your paths says
"ebs")? If so, you can deploy a cluster using the storm deploy for AWS
( https://github.com/nathanmarz/storm-deploy ) which should solve all
your issues.
-Nathan
On Sep 23, 4:50 am, Ashley Brown <ash...@spider.io> wrote:
> No change, but looking through logs I do see the following after the
> jar is uploaded:
>
> 2011-09-23 11:43:54 supervisor [INFO] Launching worker with assignment
> #:backtype.storm.daemon.supervisor.LocalAssignment{:storm-id
> "word-count-3-1316778233", :task-ids (1 2 3 4 5 6 7 8 9 10)} for this
> supervisor 9b209e0f-67bb-4523-99cc-e04cba516ccf on port 6700 with id
> 1d66e4f7-7b0f-4a52-8fe2-3bcb05bf9fd8
> 2011-09-23 11:43:54 supervisor [INFO]
> 1d66e4f7-7b0f-4a52-8fe2-3bcb05bf9fd8 still hasn't started
> 2011-09-23 11:43:54 event [ERROR] Error when processing event
> backtype.storm.daemon.supervisor$fn__3298$exec_fn__837__auto____3299$sync_p rocesses__3301@5ffe40d5
> java.lang.NullPointerException
> at backtype.storm.utils.VersionedStore.listDir(VersionedStore.java:161)
> at backtype.storm.utils.VersionedStore.getAllVersions(VersionedStore.java:119)
> at backtype.storm.utils.VersionedStore.mostRecentVersion(VersionedStore.java:4 3)
> at backtype.storm.utils.LocalState.get(LocalState.java:28)
> at backtype.storm.daemon.supervisor$wait_for_worker_launch.invoke(supervisor.c lj:116)
> at backtype.storm.daemon.supervisor$wait_for_workers_launch.invoke(supervisor. clj:132)
>
> At which point the supervisor stops and is picked back up by monit.
>
> I also see a number of exceptions after killing a running topology
> (although they look like they are not directly related):
>
> 2011-09-23 11:44:00 supervisor [ERROR] Failed to cleanup worker
> 72397b9d-3c85-4bb9-89c7-2678d86f4939. Will retry later
> java.lang.RuntimeException: Failed to delete
> /ebs_data/storm/data/workers/72397b9d-3c85-4bb9-89c7-2678d86f4939/pids
> at backtype.storm.util$rmpath.invoke(util.clj:263)
> at backtype.storm.daemon.supervisor$try_cleanup_worker.invoke(supervisor.clj:1 42)
> at backtype.storm.daemon.supervisor$shutdown_worker.invoke(supervisor.clj:158)
> at backtype.storm.event$event_manager$fn__1750.invoke(event.clj:22)
> at clojure.lang.AFn.run(AFn.java:24)
> at java.lang.Thread.run(Thread.java:662)
>
> The directory it is trying to delete doesn't actually exist. If I
> create it, it gets deleted again but still says it hasn't deleted it.
> It continues to repeat that error/exception on a loop every few
> seconds.
>
> A
>
Ashley Brown
> which doesn't make sense given that you were able to run Java-only
> topologies.
I'll look into sandboxing and permissions issues when I come back to
it. Most of our code is Java only and it's working really well for
that so far.
> Are you on AWS by any chance (I ask because one of your paths says
> "ebs")? If so, you can deploy a cluster using the storm deploy for AWS
> ( https://github.com/nathanmarz/storm-deploy ) which should solve all
> your issues.
storm-deploy was next on my list of things to look at, but we have a
large investment in our existing AWS infrastructure, which includes
ZooKeeper and monitoring systems, so would like to get storm working
with those systems.
I'll report back on what we find - we're using a standard maverick AMI
with custom-rolled APT packages to deploy code.
Thanks for all your work, storm is just what we needed!
Ashley Brown
> which doesn't make sense given that you were able to run Java-only
> topologies.
I've come back to this today (our Java topologies are running fine,
but now we have some Python we need to add!)
In one of the other threads you mentioned that nimbus and supervisor
needed separate directories - we were using the same directory for
both with one of our supervisors co-located with nimbus. Stopping that
supervisor to force allocation on another machine has solved the
problem.
I was wondering why they need separate directories? Is it
straightforward to split the configuration so they can be co-located
without installing two copies of storm with different configs?
nathanmarz
Supervisor can't share a local dir -- I just need to do a little bit
of refactoring :) I opened up an issue for this and will try to have
this done for the next release (which will be soon). Issue is here:
https://github.com/nathanmarz/storm/issues/12
-Nathan
nathanmarz
-Nathan
Steve
Did working out the Nimbus/Supervisor directory issue solve the
initial python problem, or was that a separate issue? I'm not running
a Supervisor on the Nimbus host, but I do get the same python error
when using ShellBolt.
My topology works fine locally (through Eclipse) and the Storm Cluster
on AWS is running fine for Java-only topologies. Python is definitely
installed on the nodes and I created my jar using lein (lein deps &&
lein compile && lein uberjar). I've verified that the python file is
in the jar in the resources directory. Unzip is also installed on the
machines. Is there anything specific that needs to be configured to
use Python in deployed mode (ie in the project.clj, environment
variables, storm.yaml, etc)? FYI I've tested this with both my own
topologies and WordCountTopology and in all situations I get the
python 'No such file or directory' error.
And Nathan... I know... I'll try to get storm-deploy working (will
have some feedback shortly on the other thread) which will take care
of this all, but it would still be good to understand the magic going
on under the hood :)
Thanks,
Steve
On Sep 28, 4:41 pm, nathanmarz <nathan.m...@gmail.com> wrote:
> And... this is now fixed and released in 0.5.3.
>
> -Nathan
>
> On Sep 28, 11:49 am, nathanmarz <nathan.m...@gmail.com> wrote:
>
>
>
>
>
>
>
> > Ah ok, this makes more sense now. There's no reason why Nimbus and
> > Supervisor can't share a local dir -- I just need to do a little bit
> > of refactoring :) I opened up an issue for this and will try to have
> > this done for the next release (which will be soon). Issue is here:
>
> >https://github.com/nathanmarz/storm/issues/12
>
> > -Nathan
>
> > On Sep 28, 4:02 am, Ashley Brown <ash...@spider.io> wrote:
>
> > > > It sounds like it's having trouble doing filesystem operations --
> > > > which doesn't make sense given that you were able to run Java-only
> > > > topologies.
>
> > > I've come back to this today (our Java topologies are running fine,
Nathan Marz
Steve
Unfortunately I still get the same error after cloning the latest (as
of Wednesday morning) storm-deploy. I also updated to 0.5.4 when I
did the test using the default configuration in clusters.yaml. The
error I'm getting now is not the mkdir issue though. Instead I'm
getting the supervisor security group error that I posted earlier.
I'm plodding along with the manual setup, but I'm still hoping that we
can figure this out. Is there anything else you suggest trying at
this point?
nathanmarz
the many other people who have been using the storm-deploy. I did run
into that security group issue once, but it went away when I tried
again (for me, it was a rare random error). The only thing I can
suggest is to talk to Hugo Duncan (cc'd. Hugo who created pallet, the
library that storm-deploy uses for the deploy) and see if he has any
insight into this.
-Nathan
Steve
It turned out that my config.clj was setup incorrectly. I was not
using the main account number for the aws-user-id. I was using my
individual IAM account name. I did not realize I was using the wrong
ID from the error message, but Hugo took a look at the logs and
identified that error. After setting that correctly storm-deploy does
work! One thing to note though is that I've found that it actually
fails to cleanly launch from time to time and that I just have to kill
and re-run the cluster until it launches fine. Not sure what this is,
but I'm wondering if you've seen this before based on the fact that
you were asking how many times I tried launching with the previous
errors.
Another question I have is regarding access to the machines started by
storm-deploy. Is there somewhere to configure passwords for root/
ubuntu users in any of the configuration files? I can't take a look
at logs on the machines because they require su access.
Thanks again for the help,
Steve
On Oct 26, 10:59 am, nathanmarz <nathan.m...@gmail.com> wrote:
> I really wish I knew what was different about your setup than mine or
> the many other people who have been using the storm-deploy. I did run
> into that security group issue once, but it went away when I tried
> again (for me, it was a rare random error). The only thing I can
nathanmarz
I do see random errors from time to time when running storm-deploy,
but they're pretty rare. I'd say it happens once every 50 times I run
the deploy.
The machines are configured with public key authentication, so you can
ssh in without any passwords. Just use the key you configured in
":private-key-path" in the pallet config to ssh in. Then you can ssh
with "ssh -i {private key path} storm@{node ip}". The "storm" user
should be able to read the log files.
-Nathan