Error when hitting the storm-ui URL

[YogiP]

Hi,

I am working with storm-0.9.0-wip4 package on CentOS-5.8.

I am following this wiki post. I start the storm-ui by running the command bin/storm ui. When i hit the URL 'localhost:8080' i get this error:

org.apache.thrift7.transport.TTransportException: java.net.ConnectException: Connection refused

at org.apache.thrift7.transport.TSocket.open(TSocket.java:183)

at org.apache.thrift7.transport.TFramedTransport.open(TFramedTransport.java:81)

at backtype.storm.thrift$nimbus_client_and_conn.invoke(thrift.clj:56)

at backtype.storm.ui.core$main_page.invoke(core.clj:124)

at backtype.storm.ui.core$fn__7548.invoke(core.clj:732)

at compojure.core$if_route$fn__6058.invoke(core.clj:39)

at compojure.core$if_method$fn__6051.invoke(core.clj:24)

at compojure.core$routing$fn__6073.invoke(core.clj:98)

at clojure.core$some.invoke(core.clj:2390)

at compojure.core$routing.doInvoke(core.clj:98)

at clojure.lang.RestFn.applyTo(RestFn.java:139)

at clojure.core$apply.invoke(core.clj:603)

at compojure.core$routes$fn__6077.invoke(core.clj:103)

at backtype.storm.ui.core$catch_errors$fn__7586.invoke(core.clj:789)

at ring.middleware.keyword_params$wrap_keyword_params$fn__6629.invoke(keyword_params.clj:21)

at ring.middleware.nested_params$wrap_nested_params$fn__6666.invoke(nested_params.clj:64)

at ring.middleware.params$wrap_params$fn__6606.invoke(params.clj:76)

at ring.middleware.multipart_params$wrap_multipart_params$fn__6684.invoke(multipart_params.clj:80)

at ring.middleware.session$wrap_session$fn__6838.invoke(session.clj:40)

at ring.middleware.cookies$wrap_cookies$fn__6775.invoke(cookies.clj:132)

at ring.adapter.jetty$proxy_handler$fn__7029.invoke(jetty.clj:16)

at ring.adapter.jetty.proxy$org.mortbay.jetty.handler.AbstractHandler$0.handle(Unknown Source)

at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)

at org.mortbay.jetty.Server.handle(Server.java:326)

at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)

at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)

at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)

at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)

at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)

at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)

at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

Caused by: java.net.ConnectException: Connection refused

at java.net.PlainSocketImpl.socketConnect(Native Method)

at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)

at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)

at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)

at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)

at java.net.Socket.connect(Socket.java:529)

at org.apache.thrift7.transport.TSocket.open(TSocket.java:178)

... 30 more

Any ideas as to what this error could be and how can i resolve the issue?. Any suggestions would help a lot.

Thanks & Regards

Yogi P 

[Viral Bajaria]

That means your Nimbus thrift is not running.

-Viral

--
You received this message because you are subscribed to the Google Groups "storm-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[YogiP]

Thanks a lot Viral. Does that mean, by only running the command bin/storm nimbus, i would be able to run the UI? Because, i tested this, i ran all the 3 commands, but still storm-UI is not accessible through that browser. Any other points to keep in mind while starting the bin/storm nimbus command?

Thanks,

Yogi P

[Viral Bajaria]

UI is a separate daemon. If you run UI without Nimbus, you will get the exception that you saw. UI gathers all the information from Nimbus.

-Viral

[YogiP]

Oh sorry Viral..i was not clear in explaining that part. What i meant is, i started nimbus as well as UI, but still i got this error. Are there any specific configuration to be made when running the nimbus?

Thanks & Regards,

Yogi P

[YogiP]

Hi Viral,

I just noticed in the nimbus logs that it is not starting up properly. Following is the error in the nimbus logs:

2013-04-26 03:50:20 o.a.z.ClientCnxn [INFO] Opening socket connection to server localhost/127.0.0.1:2181

2013-04-26 03:50:20 o.a.z.ClientCnxn [WARN] Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect

java.net.ConnectException: Connection refused

at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.6.0_35]

at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599) ~[na:1.6.0_35]

at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119) ~[zookeeper-3.3.3.jar:3.3.3-1073969]

2013-04-26 03:50:21 o.a.z.ClientCnxn [INFO] Opening socket connection to server localhost/127.0.0.1:2181

2013-04-26 03:50:21 o.a.z.ClientCnxn [WARN] Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect

java.net.ConnectException: Connection refused

at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.6.0_35]

at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599) ~[na:1.6.0_35]

at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119) ~[zookeeper-3.3.3.jar:3.3.3-1073969]

2013-04-26 03:50:23 o.a.z.ClientCnxn [INFO] Opening socket connection to server localhost/127.0.0.1:2181

2013-04-26 03:50:23 o.a.z.ClientCnxn [WARN] Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect

java.net.ConnectException: Connection refused

at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.6.0_35]

at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599) ~[na:1.6.0_35]

at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119) ~[zookeeper-3.3.3.jar:3.3.3-1073969]

2013-04-26 03:50:24 o.a.z.ClientCnxn [INFO] Opening socket connection to server localhost/127.0.0.1:2181

2013-04-26 03:50:24 o.a.z.ClientCnxn [WARN] Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect

java.net.ConnectException: Connection refused

at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.6.0_35]

at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599) ~[na:1.6.0_35]

at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119) ~[zookeeper-3.3.3.jar:3.3.3-1073969]

I am sure i am missing some configuration. In the storm.yaml file, i have just uncommented the nimbu.host part as of now and set it to "127.0.0.1"

Thanks & Regards,

Yogi P

[Homer Strong]

You're missing ZooKeeper. Nimbus (and others) must be configured to talk to a running ZK server.

[YogiP]

Thanks a lot for the response Homer. How do i start a zookeeper? Is there a separate installation to be made? I just found a zookeeper jar inside the lib folder of storm. So , i thought storm would be starting it internally.  As i said, In the storm.yaml file, i have just uncommented the nimbu.host part as of now and set it to "127.0.0.1" and rest all is commented. Is there anything else to be done? How do i confirm zookeeper is running and is working properly?

Thanks & Regrads,

Yogesh

[Viral Bajaria]

multiple ways to check zookeeper is running in production: 

- echo ruok | nc {ip/host} 2181 (default zookeeper port) .... this command will return imok if everything is running fine.

- or the simplest is to just telnet to port 2181 and if it connects it's up or else it's not.

- or dirty way is to use jps (i think it only comes with sun jdk) or check by running ps auxwww | grep Quorum, you want to verify that there is a java process running with QuorumPeerMain class as the main class.

In LocalCluster mode, storm starts it's own zookeeper process, so you don't need to setup one for LocalCluster but if you are trying to setup a cluster then yes you need one. The zookeeper jar that ships with storm is for all the client interaction. You can download the whole zookeeper package from apache and start a single server quorum.

-Viral

[YogiP]

Thanks a lot Viral. Will try it out. So my steps would be to download and install zookeeper and confirm with the commands given by you that it starts properly.

So, it all started with starting of UI and it has reached to starting of Zookeeper. So, am i right in saying that since UI needs nimbus to start and since nimbus needs zookeeper, UI does not start on a local cluster?

Thanks & Regards,

Yogi P

[YogiP]

Thanks a lot Viral & Homer. I am able to start the storm nimbus, supervisor and ui without any errors. I am able to see the UI in the browser. Zookeeper was missing when trying to set up a cluster. Now, my next and final step in this cycle is to run a topology on the cluster. What are the steps?

Following is how i am trying to do it. Could you please correct me wherever I am wrong OR if i am missing something:

1. I have Storm-Master machine and Storm-Slave machine. I run Zoo-keeper on master machine, i run nimbus, supervisor and ui on master machine. I run all the same things on Storm-slave machine.

2. I run the WordCountTopology program on Master machine.

3. Now, this should run on the worker machine too right?? How do i check that the Master and Slave have identified eachother?? How do i check that the Topology is getting distributed on the cluster??

Following is storm.yaml of Master machine:

storm.zookeeper.servers:

     - "storm-master-IP"

     - "storm-slave-IP"

 

nimbus.host: "storm-master-IP"

Following is storm.yaml on Slave machine:

storm.zookeeper.servers:

     - "storm-master-IP"

     - "storm-slave-IP"

nimbus.host: "storm-slave-IP"

Thanks & Regards,

Yogi P

[Homer Strong]

No, you've almost set up 2 independent clusters. To have a simple cluster set up with 1 'master' and 1 'slave', run ZK+nimbus+ui on the 'master' and _only_ 'supervisor'. 

Not that you couldn't set it up in other ways (e.g. also running a supervisor on 'master') but this is how I would start if you want more than 1 node. NB you don't want an even number of ZK servers.

After submitting the topology it should appear in the UI, which will also show you the host(s) which are running the components.

The configuration I'm suggesting would look like this:

Master:

storm.zookeeper.servers:

     - "storm-master-IP"

 

nimbus.host: "storm-master-IP"

Slave:

storm.zookeeper.servers:

     - "storm-master-IP"

nimbus.host: "storm-master-IP"

[YogiP]

Hi Homer,

Thanks a lot for the response. That was very very clear and step-wise. I followed exactly what you said.

Everything starts fine, but i cant confirm the cluster formation. I don't see anything in the UI that indicates that the topology is running on 2 machines and is getting distributed. Is there any other way to check this?

Also, I am submitting the ExclamationTopology and WordCountTopoplogy examples that came with the storm-starter package. I hope even these topologies should run and get distributed on the cluster.

Thanks again!

-YogiP

[Homer Strong]

On the root UI page, there should be a table of supervisors. You should expect to see all of your supervisor processes there, so there should 1 row for you. If you only have 1 supervisor then your topology will only run on 1 machine.

After you submit a topology to nimbus the UI should report the # of used supervisor slots that the topology. If the slots are filled then your topology was distributed.

[Viral Bajaria]

On Wed, May 8, 2013 at 4:40 AM, YogiP <world...@gmail.com> wrote:

Also, I am submitting the ExclamationTopology and WordCountTopoplogy examples that came with the storm-starter package. I hope even these topologies should run and get distributed on the cluster.

You can submit these topologies to run in distributed mode by passing a command line argument. The argument is irrelevant to the topology so you can just pass "FOO"

[YogiP]

Hi Homer,

On the root UI page, i see the following when i run the WordCountTopology program:

Cluster summary shows 1 supervisor. This means that the 2 machines are identifying each other.

Topology Summary shows nothing.

Supervisor summary shows the details of 1 supervisor, # of slots is 4, but the # of used slots is 0.

Am i missing something? any particular known reasons for it not getting distributed?

Thanks!

-YogiP

[YogiP]

Hi Viral,

Thanks a lot for the response.

When i run the program with a command line argument from the command window, i get the following error:

[root@localhost bin]# java storm/starter/WordCountTopology "FOO"

1223 [main] INFO  backtype.storm.StormSubmitter - Jar not uploaded to master yet. Submitting jar...

Exception in thread "main" java.lang.RuntimeException: Must submit topologies using the 'storm' client script so that StormSubmitter knows which jar to upload.

at backtype.storm.StormSubmitter.submitJar(StormSubmitter.java:131)

at backtype.storm.StormSubmitter.submitJar(StormSubmitter.java:123)

at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:74)

at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:41)

at storm.starter.WordCountTopology.main(WordCountTopology.java:124)

 Do i need to change anything in the program?

Thanks & Regards,

YogiP

[Viral Bajaria]

use "storm jar <jar-name> <class-name> FOO" instead of running via java.

[YogiP]

Hi Viral,

Did exactly what you said. Getting this error now:

Exception in thread "main" java.lang.RuntimeException: Found multiple defaults.yaml resources. You're probably bundling the Storm jars with your topology jar.

at backtype.storm.utils.Utils.findAndReadConfigFile(Utils.java:115)

at backtype.storm.utils.Utils.readDefaultConfig(Utils.java:135)

at backtype.storm.utils.Utils.readStormConfig(Utils.java:155)

at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:62)

at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:41)

at storm.starter.WordCountTopology.main(WordCountTopology.java:124)

Something to do with multiple .yaml files.

Thanks,

YogiP

[Viral Bajaria]

On Thu, May 9, 2013 at 2:40 AM, YogiP <world...@gmail.com> wrote:

Exception in thread "main" java.lang.RuntimeException: Found multiple defaults.yaml resources. You're probably bundling the Storm jars with your topology jar.

The error is self-explanatory. When you build your jar-with-dependencies make sure the storm dependency is marked as provided. You don't need to bundle storm with your jar since those libs are already on your supervisor/worker nodes.

[YogiP]

Thanks a lot Viral & Homer. You and Homer have been of great help. The topology got distributed and I can see the topology summary, # of workers and # of slots in UI.

One more question for this thread, How can i stop a topology? or how can I deactivate a topology?

Thanks & Regards,

YogiP 

[YogiP]

Thanks a lot Viral & Homer for all the help. Got one full cycle understanding.

Got the kill command to kill the topology.

-YogiP

[YogiP]

Hi Viral/Homer,

I just observed that the topology is not running properly in a cluster mode.

I noticed the following error in worker logs:

2013-05-10 02:30:59 b.s.util [ERROR] Async loop died!

java.lang.RuntimeException: org.zeromq.ZMQException: Invalid argument(0x16)

at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:87) ~[storm-0.9.0-wip4.jar:na]

at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:58) ~[storm-0.9.0-wip4.jar:na]

at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:62) ~[storm-0.9.0-wip4.jar:na]

at backtype.storm.disruptor$consume_loop_STAR_$fn__3024.invoke(disruptor.clj:73) ~[storm-0.9.0-wip4.jar:na]

at backtype.storm.util$async_loop$fn__436.invoke(util.clj:377) ~[storm-0.9.0-wip4.jar:na]

at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na]

at java.lang.Thread.run(Thread.java:662) [na:1.6.0_35]

Caused by: org.zeromq.ZMQException: Invalid argument

at org.zeromq.ZMQ$Socket.send(Native Method) ~[jzmq-2.1.0.jar:na]

at zilch.mq$send.invoke(mq.clj:93) ~[storm-0.9.0-wip4.jar:na]

at backtype.storm.messaging.zmq.ZMQConnection.send(zmq.clj:40) ~[storm-0.9.0-wip4.jar:na]

at backtype.storm.daemon.worker$mk_transfer_tuples_handler$fn__4694$fn__4695.invoke(worker.clj:298) ~[storm-0.9.0-wip4.jar:na]

at backtype.storm.daemon.worker$mk_transfer_tuples_handler$fn__4694.invoke(worker.clj:287) ~[storm-0.9.0-wip4.jar:na]

at backtype.storm.disruptor$clojure_handler$reify__3011.onEvent(disruptor.clj:43) ~[storm-0.9.0-wip4.jar:na]

at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:84) ~[storm-0.9.0-wip4.jar:na]

... 6 common frames omitted

It is some ZeroMQ error. It is breaking the execution and the Topology is not running properly there after. 

Any ideas about the error? and as to how can i resolve it?

Thanks & Regards,

YogiP 

[Viral Bajaria]

Check your firewall rules. Are you sure all the worker ports are open for communication ?

-Viral

--

[YogiP]

Hi Viral,

The firewall is not running. And what ports do i need to check whether if they are open are not?

Thanks,

YogiP

[Homer Strong]

The worker ports are whatever you've configured in your storm.yaml, as in

https://github.com/nathanmarz/storm/blob/master/conf/defaults.yaml#L53

by convention they begin at 6700.

[YogiP]

Oops. Thanks a lot for the response. I never specified the ports in the storm.yaml file on the master and the worker machine. I did that and its working. 

supervisor.slots.ports:
    - 6700
    - 6701
    - 6702
    - 6703

I had the impression that those lines are optional and storm takes it by default.

Thanks & Regards,

YogiP

[YogiP]

Hi,

I am just facing a weird error when trying to run any example (WordCountTopology) in a distributed mode. It is also not creating the worker logs on the Slave machine. My setup is as follows:

Master-Machine: zookeeper+nimbus+ui

Slave-Machine: supervisor

Following is the error I am getting in the supervisor log on the slave machine:

2013-10-01 02:48:31 b.s.d.supervisor [INFO] Starting supervisor with id d500eb37-c075-43ec-aba4-cf43361b26f1 at host localhost

2013-10-01 02:48:32 b.s.event [ERROR] Error when processing event

java.lang.RuntimeException: java.io.EOFException

at backtype.storm.utils.Utils.deserialize(Utils.java:68) ~[storm-0.9.0-wip4.jar:na]

at backtype.storm.utils.LocalState.snapshot(LocalState.java:24) ~[storm-0.9.0-wip4.jar:na]

at backtype.storm.utils.LocalState.get(LocalState.java:28) ~[storm-0.9.0-wip4.jar:na]

at backtype.storm.daemon.supervisor$sync_processes.invoke(supervisor.clj:192) ~[storm-0.9.0-wip4.jar:na]

at clojure.lang.AFn.applyToHelper(AFn.java:161) [clojure-1.4.0.jar:na]

at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.4.0.jar:na]

at clojure.core$apply.invoke(core.clj:603) ~[clojure-1.4.0.jar:na]

at clojure.core$partial$fn__4070.doInvoke(core.clj:2343) ~[clojure-1.4.0.jar:na]

at clojure.lang.RestFn.invoke(RestFn.java:397) ~[clojure-1.4.0.jar:na]

at backtype.storm.event$event_manager$fn__3126.invoke(event.clj:24) ~[storm-0.9.0-wip4.jar:na]

at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na]

at java.lang.Thread.run(Thread.java:662) [na:1.6.0_35]

Caused by: java.io.EOFException: null

at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2280) ~[na:1.6.0_35]

at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2749) ~[na:1.6.0_35]

at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:779) ~[na:1.6.0_35]

at java.io.ObjectInputStream.<init>(ObjectInputStream.java:279) ~[na:1.6.0_35]

at backtype.storm.utils.Utils.deserialize(Utils.java:63) ~[storm-0.9.0-wip4.jar:na]

... 11 common frames omitted

Although, when run in local mode, the supervisor is working fine and also the worker logs get created.

Could you please help with the reason for it to not work in distributed mode. Supervisor seems to be the culprit, but not getting the exact error.

Thanks & Regards,

YogiP