Running Storm with additional CLASSPATH

[ashu]

Hello,

I have built some codes under storm-starter. However, I need
additional CLASSPATH setup in my code, when running on a cluster use:

storm jar my.jar storm.starter.mycode

Is there anyway to add additional CLASSPATH info for Storm to pick up?

Thanks,
Ashu

[Nathan Marz]

Generally you should package all the dependencies of your code with the jar you submit to the cluster. For example, you can do this with storm-starter with Leiningen by running the command "lein uberjar". 

Alternatively, any jars you put in the lib/ directory of the storm distribution on your cluster will be included on the classpath.

--
Twitter: @nathanmarz
http://nathanmarz.com

[ashu]

Thanks. Nathan.

The problem is that besides .jar files, I need to include in my
CLASSPATH the path to .so files (linux) and xml configuration files.
Can I put those in storm's lib/ directory? Or should those be in
my .jar file submitted to Storm?

If I put those in Storm's lib/, then I need to put them in all worker
machines as well, right?

If I put those in the .jar file I submitted to Storm, then my
storm.starter.mycode knows where to pick those up? In a relative path?

Thanks,
Ashu

[Nathan Marz]

Your XML files should be in your jar... you can use the "Thread.currentThread().getContextClassLoader().getResources(...)" function to find resources on your classpath.

As for your .so files, I would install those on each machine and update your storm.yaml's "java.library.path" config to point to them.

[ashu]

I put the wrapper_xxx.so directoy in storm.yaml's "java.library.path",
like "/root/so/lib:/root/sth-else/". I got the following errors. If I
ran it with -Djava.library.path=/root/so/lib, it works. How to do it
with "Storm jar " command? --Thanks.

java.lang.UnsatisfiedLinkError: no wrapper_xxx in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1709)
at java.lang.Runtime.loadLibrary0(Runtime.java:823)
at java.lang.System.loadLibrary(System.java:1028)
at
com.alibaba.apsara.pangu.PanguFileSystem.<clinit>(PanguFileSystem.java:
581)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:
703)
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:748)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1433)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:
56)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:
1450)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:215)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:120)
at storm.starter.RecordInput.nextTuple(RecordInput.java:50)
at backtype.storm.daemon.task
$fn__3452$fn__3501.invoke(task.clj:322)
at backtype.storm.daemon.task$mk_task
$iter__3397__3401$fn__3402$fn__3403$fn__3404.invoke(task.clj:210)

> Twitter: @nathanmarzhttp://nathanmarz.com- Hide quoted text -
>
> - Show quoted text -

[ashu]

I fixed the errors.
28 [main] INFO backtype.storm.StormSubmitter - Uploading topology
jar cssd.jar to assigned location: ["/apang/disk1/stormtmp"]/nimbus/
inbox/stormjar-e8ec2bd7-997b-4708-b292-d3ac59fb8d1e.jar
39 [main] INFO backtype.storm.StormSubmitter - Successfully
uploaded topology jar to assigned location: ["/apang/disk1/stormtmp"]/
nimbus/inbox/stormjar-e8ec2bd7-997b-4708-b292-d3ac59fb8d1e.jar
40 [main] INFO backtype.storm.StormSubmitter - Submitting topology
test in distributed mode with conf {"topology.workers":
3,"topology.debug":true}
99 [main] INFO backtype.storm.StormSubmitter - Finished submitting
topology: test

Now I have two questions.

1. The topology is submitter to the cluster. How do I check it is
running? For nimbus machine, I can check storm/logs directory. In
worker node machines, I also installed storm. In storm/logs, I did not
see any new logs.
2. In my program I write to a local file some results. When running
with "java prog", it can see the file. When running in a cluster, I
can not file the file. Will it still able to write to a local file?

Thanks,
Ashu

> > Twitter: @nathanmarzhttp://nathanmarz.com-Hide quoted text -
>
> > - Show quoted text -- Hide quoted text -

[Nathan Marz]

On Wed, Apr 4, 2012 at 1:09 AM, ashu <ash...@gmail.com> wrote:

I fixed the errors.
28   [main] INFO  backtype.storm.StormSubmitter  - Uploading topology
jar cssd.jar to assigned location: ["/apang/disk1/stormtmp"]/nimbus/
inbox/stormjar-e8ec2bd7-997b-4708-b292-d3ac59fb8d1e.jar
39   [main] INFO  backtype.storm.StormSubmitter  - Successfully
uploaded topology jar to assigned location: ["/apang/disk1/stormtmp"]/
nimbus/inbox/stormjar-e8ec2bd7-997b-4708-b292-d3ac59fb8d1e.jar
40   [main] INFO  backtype.storm.StormSubmitter  - Submitting topology
test in distributed mode with conf {"topology.workers":
3,"topology.debug":true}
99   [main] INFO  backtype.storm.StormSubmitter  - Finished submitting
topology: test

Now I have two questions.

1. The topology is submitter to the cluster. How do I check it is
running? For nimbus machine, I can check storm/logs directory. In
worker node machines, I also installed storm. In storm/logs, I did not
see any new logs.

If you're not seeing any logs in that directory on the worker machines, then something is going wrong when launching your workers.

 

2. In my program I write to a local file some results. When running
with "java prog", it can see the file. When running in a cluster, I
can not file the file. Will it still able to write to a local file?

Storm isn't stopping you from writing to the local filesystem. Though from above it sounds like your workers aren't launching.

[ashu]

Thanks. Nathan.

1. To confirm one thing. I use storm.starter package and put my
implementation there. When packaging my own .jar file, should I
generate it from storm-starter top directory since it may need the
files in storm-starter/lib?

2. Can I use the command "jar cf jar-file directory" to generate
the .jar file? And this .jar file needs to be in nimbus machine only
(no need in worker machines)?

3. Is there a way to kill nimbus or supervisor? I use "ps aux" and
could not find the processes.

Thanks,
Ashu

> > > > Twitter: @nathanmarzhttp://nathanmarz.com-Hidequoted text -

>
> > > > - Show quoted text -- Hide quoted text -
>
> > > - Show quoted text -
>
> --

> Twitter: @nathanmarzhttp://nathanmarz.com- Hide quoted text -

[ashu]

4. One more question. When I submitted ExclamationTopology to the
cluster, all the print-outs were gone. Did it work differently in a
cluster?

Thanks.

> > > > > Twitter: @nathanmarzhttp://nathanmarz.com-Hidequotedtext -

>
> > > > > - Show quoted text -- Hide quoted text -
>
> > > > - Show quoted text -
>
> > --

> > Twitter: @nathanmarzhttp://nathanmarz.com-Hide quoted text -

>
> > - Show quoted text -- Hide quoted text -
>

[ashu]

If I ran my jar file on local mode, it worked.

If I ran it on a cluster, it failed with the following errors from
nimbus.log. Why it got timed out?

-----------------------------
2012-04-05 06:17:51 ClientCnxn [INFO] Opening socket connection to
server t01a08081.corp-01.xxx.com/10.1.152.81:2181
2012-04-05 06:17:51 ClientCnxn [INFO] Socket connection established to
t01a08081.corp-01.xxx.com/10.1.152.81:2181, initiating session
2012-04-05 06:17:51 ClientCnxn [INFO] Session establishment complete
on server t01a08081.corp-01.xxx.com/10.1.152.81:2181, sessionid =
0x1367f6f6c800000, negotiated timeout = 20000
2012-04-05 06:17:51 zookeeper [INFO] Zookeeper state
update: :connected:none
2012-04-05 06:17:51 ZooKeeper [INFO] Session: 0x1367f6f6c800000 closed
2012-04-05 06:17:51 ClientCnxn [INFO] EventThread shut down
2012-04-05 06:17:51 CuratorFrameworkImpl [INFO] Starting
2012-04-05 06:17:51 ZooKeeper [INFO] Initiating client connection,
connectString=10.1.152.81:2181/storm sessionTimeout=20000
watcher=com.netflix.curator.ConnectionState@192425a
2012-04-05 06:17:51 ClientCnxn [INFO] Opening socket connection to
server /10.1.152.81:2181
2012-04-05 06:17:51 ClientCnxn [INFO] Socket connection established to
t01a08081.corp-01.xxx.com/10.1.152.81:2181, initiating session
2012-04-05 06:17:51 ClientCnxn [INFO] Session establishment complete
on server t01a08081.corp-01.xxx.com/10.1.152.81:2181, sessionid =
0x1367f6f6c800001, negotiated timeout = 20000
2012-04-05 06:17:51 nimbus [INFO] Starting Nimbus server...
2012-04-05 06:18:32 nimbus [INFO] Uploading file from client to ["/
apang/disk1/stormtmp"]/nimbus/inbox/stormjar-c9a2edb5-43e5-4b48-
acdd-039a1b767e9c.jar
2012-04-05 06:18:32 nimbus [INFO] Finished uploading file from client:
["/apang/disk1/stormtmp"]/nimbus/inbox/stormjar-c9a2edb5-43e5-4b48-
acdd-039a1b767e9c.jar
2012-04-05 06:18:32 nimbus [INFO] Received topology submission for
test with conf {"topology.ackers" 1, "topology.kryo.register" nil,
"storm.id" "test-1-1333577912", "topology.workers" 3, "topology.debug"
true}
2012-04-05 06:18:32 nimbus [INFO] Task test-1-1333577912:1 timed out
2012-04-05 06:18:32 nimbus [INFO] Task test-1-1333577912:2 timed out
2012-04-05 06:18:32 nimbus [INFO] Task test-1-1333577912:3 timed out
2012-04-05 06:18:32 nimbus [INFO] Task test-1-1333577912:4 timed out
2012-04-05 06:18:32 nimbus [INFO] Task test-1-1333577912:5 timed out
2012-04-05 06:18:32 nimbus [INFO] Task test-1-1333577912:6 timed out
2012-04-05 06:18:32 nimbus [INFO] Setting new assignment for storm id
test-1-1333577912: #:backtype.storm.daemon.common.Assignment{:master-
code-dir "[\"/apang/disk1/stormtmp\"]/nimbus/stormdist/
test-1-1333577912", :node->host {}, :task->node+port {}, :task->start-
time-secs {}}
2012-04-05 06:18:32 nimbus [INFO] Activating test: test-1-1333577912
2012-04-05 06:18:42 nimbus [INFO] Task test-1-1333577912:1 timed out

> > > > > com.a.ap.pan.PanguFileSystem.<clinit>(PanguFileSystem.java:

> > > Twitter: @nathanmarzhttp://nathanmarz.com-Hidequoted text -

>
> > > - Show quoted text -- Hide quoted text -
>
> > > - Show quoted text -- Hide quoted text -
>

[Nathan Marz]

On Wed, Apr 4, 2012 at 12:59 PM, ashu <ash...@gmail.com> wrote:

Thanks. Nathan.

1. To confirm one thing. I use storm.starter package and put my
implementation there. When packaging my own .jar file, should I
generate it from storm-starter top directory since it may need the
files in storm-starter/lib?

Use Leiningen's "lein uberjar" command to see what a proper jar should look like.

 

2. Can I use the command "jar cf jar-file directory" to generate
the .jar file? And this .jar file needs to be in nimbus machine only
(no need in worker machines)?

You shouldn't put the jar on Nimbus or the workers yourself. Let the "storm jar" command do that for you.

 

3. Is there a way to kill nimbus or supervisor? I use "ps aux" and
could not find the processes.

I kill them by doing "ps ax" to find the pid and the I kill -9 them. I don't know why you wouldn't be able to find them... the Storm processes are just like any other process.

[Nathan Marz]

On Wed, Apr 4, 2012 at 1:19 PM, ashu <ash...@gmail.com> wrote:

4. One more question. When I submitted ExclamationTopology to the
cluster, all the print-outs were gone. Did it work differently in a
cluster?

Storm captures stdout and stderr and puts them into the log files. How were you doing your print outs? 

[Nathan Marz]

What version of Storm is this with? That last timeout is concerning. Did you override nimbus.task.timeout.secs or nimbus.task.launch.secs ? 

[ashu]

I did not. But it works now.

> ...
>
> read more »- Hide quoted text -