Submitting Topology - Please help!

[Amedeo Merlo]

Hello everyone,

I'm not able to run the command storm jar allmycode-with-dependecies.jar org.me.MyTopology arg1 arg2 arg3
What I did until now is:
- Create a little topology
- Did what is explained there https://github.com/nathanmarz/storm/wiki/Running-topologies-on-a-production-cluster
- Installed the dependecies (0mq and jzmq)

Now I'd like to submit the topology using 4 virtual machines.
From the first I launch the command to submit. I have configurated the storm.yaml with the ip of the nimbus/zookeper machine
In the second machine I start the zookeper (just run zkServer.sh start from bin?) and start the nimbus running storm nimbus
In the third and the forth I run storm supervisor (Have I to configure the Ip of this machines? Where?)

What I get is Exception in thread "main" java.lang.RuntimeException: org.apache.thrift7.transport.TTransportException: java.net.ConnectException: Connection refused
Why?
Can you help me please? I don't know what to do or wich is the problem

(The topology works on a local cluster)

Cheers,
Amedeo

[Philip O'Toole]

You need to supply more context. What process is generating the log
message? Can you supply more of the log?

Philip

> --
> You received this message because you are subscribed to the Google Groups
> "storm-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to storm-user+...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Amedeo Merlo]

Thanks for answering :)
Now I'm trying to submit the topology from the same nimbus/zookeeper machine
I launch all from command line and I get:

Exception in thread "main" java.lang.RuntimeException: org.apache.thrift7.transport.TTransportException: java.net.ConnectException: Connection refused

at backtype.storm.utils.NimbusClient.<init>(NimbusClient.java:36)

at backtype.storm.utils.NimbusClient.getConfiguredClient(NimbusClient.java:17)

at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:69)

at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:40)

at Topology2.main(Topology2.java:40)

Caused by: org.apache.thrift7.transport.TTransportException: java.net.ConnectException: Connection refused

at org.apache.thrift7.transport.TSocket.open(TSocket.java:183)

at org.apache.thrift7.transport.TFramedTransport.open(TFramedTransport.java:81)

at backtype.storm.utils.NimbusClient.<init>(NimbusClient.java:34)

... 4 more

Caused by: java.net.ConnectException: Connection refused

at java.net.PlainSocketImpl.socketConnect(Native Method)

at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)

at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)

at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)

at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)

at java.net.Socket.connect(Socket.java:529)

at org.apache.thrift7.transport.TSocket.open(TSocket.java:178)

In the nimbus log there's:

2013-05-02 18:48:01 ZooKeeper [INFO] Client environment:zookeeper.version=3.3.3-1073969, built on 02/23/2011 22:27 GMT

2013-05-02 18:48:01 ZooKeeper [INFO] Client environment:host.name=VM-amedeo

2013-05-02 18:48:01 ZooKeeper [INFO] Client environment:java.version=1.6.0_45

2013-05-02 18:48:01 ZooKeeper [INFO] Client environment:java.vendor=Sun Microsystems Inc.

2013-05-02 18:48:01 ZooKeeper [INFO] Client environment:java.home=/usr/lib/jvm/java-6-oracle/jre

2013-05-02 18:48:01 ZooKeeper [INFO] Client environment:java.class.path=/home/amedeo/Scrivania/storm-0.8.2/storm-0.8.2.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/hiccup-0.3.6.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/guava-13.0.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/minlog-1.2.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/zookeeper-3.3.3.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/carbonite-1.5.0.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/math.numeric-tower-0.0.1.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/core.incubator-0.1.0.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/ring-servlet-0.3.11.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/disruptor-2.10.1.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/tools.macro-0.1.0.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/slf4j-log4j12-1.5.8.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/objenesis-1.2.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/httpclient-4.1.1.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/commons-codec-1.4.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/snakeyaml-1.9.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/jetty-util-6.1.26.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/tools.cli-0.2.2.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/kryo-2.17.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/curator-client-1.0.1.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/servlet-api-2.5-20081211.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/compojure-1.1.3.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/clojure-1.4.0.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/httpcore-4.1.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/commons-exec-1.1.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/tools.logging-0.2.3.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/reflectasm-1.07-shaded.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/jgrapht-0.8.3.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/slf4j-api-1.5.8.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/servlet-api-2.5.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/json-simple-1.1.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/libthrift7-0.7.0.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/joda-time-2.0.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/commons-io-1.4.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/commons-logging-1.1.1.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/curator-framework-1.0.1.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/clout-1.0.1.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/ring-jetty-adapter-0.3.11.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/jetty-6.1.26.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/commons-lang-2.5.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/ring-core-1.1.5.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/log4j-1.2.16.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/clj-time-0.4.1.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/jline-0.9.94.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/jzmq-2.1.0.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/junit-3.8.1.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/asm-4.0.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/commons-fileupload-1.2.1.jar:/home/amedeo/Scrivania/storm-0.8.2/log4j:/home/amedeo/Scrivania/storm-0.8.2/conf

2013-05-02 18:48:01 ZooKeeper [INFO] Client environment:java.library.path=/usr/local/lib:/opt/local/lib:/usr/lib

2013-05-02 18:48:01 ZooKeeper [INFO] Client environment:java.io.tmpdir=/tmp

2013-05-02 18:48:01 ZooKeeper [INFO] Client environment:java.compiler=<NA>

2013-05-02 18:48:01 ZooKeeper [INFO] Client environment:os.name=Linux

2013-05-02 18:48:01 ZooKeeper [INFO] Client environment:os.arch=i386

2013-05-02 18:48:01 ZooKeeper [INFO] Client environment:os.version=3.5.0-28-generic

2013-05-02 18:48:01 ZooKeeper [INFO] Client environment:user.name=amedeo

2013-05-02 18:48:01 ZooKeeper [INFO] Client environment:user.home=/home/amedeo

2013-05-02 18:48:01 ZooKeeper [INFO] Client environment:user.dir=/home/amedeo/workspace/Context-storm-new

2013-05-02 18:48:01 ZooKeeperServer [INFO] Server environment:zookeeper.version=3.3.3-1073969, built on 02/23/2011 22:27 GMT

2013-05-02 18:48:01 ZooKeeperServer [INFO] Server environment:host.name=VM-amedeo

2013-05-02 18:48:01 ZooKeeperServer [INFO] Server environment:java.version=1.6.0_45

2013-05-02 18:48:01 ZooKeeperServer [INFO] Server environment:java.vendor=Sun Microsystems Inc.

2013-05-02 18:48:01 ZooKeeperServer [INFO] Server environment:java.home=/usr/lib/jvm/java-6-oracle/jre

2013-05-02 18:48:01 ZooKeeperServer [INFO] Server environment:java.class.path=/home/amedeo/Scrivania/storm-0.8.2/storm-0.8.2.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/hiccup-0.3.6.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/guava-13.0.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/minlog-1.2.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/zookeeper-3.3.3.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/carbonite-1.5.0.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/math.numeric-tower-0.0.1.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/core.incubator-0.1.0.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/ring-servlet-0.3.11.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/disruptor-2.10.1.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/tools.macro-0.1.0.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/slf4j-log4j12-1.5.8.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/objenesis-1.2.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/httpclient-4.1.1.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/commons-codec-1.4.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/snakeyaml-1.9.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/jetty-util-6.1.26.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/tools.cli-0.2.2.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/kryo-2.17.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/curator-client-1.0.1.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/servlet-api-2.5-20081211.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/compojure-1.1.3.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/clojure-1.4.0.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/httpcore-4.1.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/commons-exec-1.1.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/tools.logging-0.2.3.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/reflectasm-1.07-shaded.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/jgrapht-0.8.3.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/slf4j-api-1.5.8.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/servlet-api-2.5.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/json-simple-1.1.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/libthrift7-0.7.0.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/joda-time-2.0.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/commons-io-1.4.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/commons-logging-1.1.1.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/curator-framework-1.0.1.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/clout-1.0.1.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/ring-jetty-adapter-0.3.11.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/jetty-6.1.26.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/commons-lang-2.5.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/ring-core-1.1.5.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/log4j-1.2.16.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/clj-time-0.4.1.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/jline-0.9.94.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/jzmq-2.1.0.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/junit-3.8.1.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/asm-4.0.jar:/home/amedeo/Scrivania/storm-0.8.2/lib/commons-fileupload-1.2.1.jar:/home/amedeo/Scrivania/storm-0.8.2/log4j:/home/amedeo/Scrivania/storm-0.8.2/conf

2013-05-02 18:48:01 ZooKeeperServer [INFO] Server environment:java.library.path=/usr/local/lib:/opt/local/lib:/usr/lib

2013-05-02 18:48:01 ZooKeeperServer [INFO] Server environment:java.io.tmpdir=/tmp

2013-05-02 18:48:01 ZooKeeperServer [INFO] Server environment:java.compiler=<NA>

2013-05-02 18:48:01 ZooKeeperServer [INFO] Server environment:os.name=Linux

2013-05-02 18:48:01 ZooKeeperServer [INFO] Server environment:os.arch=i386

2013-05-02 18:48:01 ZooKeeperServer [INFO] Server environment:os.version=3.5.0-28-generic

2013-05-02 18:48:01 ZooKeeperServer [INFO] Server environment:user.name=amedeo

2013-05-02 18:48:01 ZooKeeperServer [INFO] Server environment:user.home=/home/amedeo

2013-05-02 18:48:01 ZooKeeperServer [INFO] Server environment:user.dir=/home/amedeo/workspace/Context-storm-new

2013-05-02 18:48:01 nimbus [INFO] Starting Nimbus with conf {"dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 5000, "topology.skip.missing.kryo.registrations" false, "ui.childopts" "-Xmx768m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 500, "nimbus.monitor.freq.secs" 10, "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "storm-local", "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "localhost", "storm.zookeeper.port" 2181, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.zookeeper.servers" ["localhost"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" 1, "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" [6700 6701 6702 6703], "topology.debug" false, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 1, "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.acker.tasks" nil, "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.zmq", "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx1024m", "storm.cluster.mode" "distributed", "topology.optimize" true, "topology.max.task.parallelism" nil}

2013-05-02 18:48:01 nimbus [INFO] Using default scheduler

2013-05-02 18:48:02 CuratorFrameworkImpl [INFO] Starting

2013-05-02 18:48:02 ZooKeeper [INFO] Initiating client connection, connectString=localhost:2181 sessionTimeout=20000 watcher=com.netflix.curator.ConnectionState@161509b

2013-05-02 18:48:02 ClientCnxn [INFO] Opening socket connection to server localhost/127.0.0.1:2181

2013-05-02 18:48:02 ClientCnxn [WARN] Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect

java.net.ConnectException: Connection refused

at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)

at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119)

2013-05-02 18:48:03 ClientCnxn [INFO] Opening socket connection to server localhost/127.0.0.1:2181

2013-05-02 18:48:03 ClientCnxn [WARN] Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect

java.net.ConnectException: Connection refused

at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)

at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119)

2013-05-02 18:48:04 ClientCnxn [INFO] Opening socket connection to server localhost/127.0.0.1:2181

2013-05-02 18:48:04 ClientCnxn [WARN] Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect

java.net.ConnectException: Connection refused

at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)

at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119)

2013-05-02 18:48:06 ClientCnxn [INFO] Opening socket connection to server localhost/127.0.0.1:2181

2013-05-02 18:48:06 ClientCnxn [WARN] Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect

java.net.ConnectException: Connection refused

at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)

at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119)

Could you answer also to the question I asked before?
just run zkServer.sh start from bin for Zookepper?
Where to configure the Ip of the supervisor?

I'm a newble of storm, but I really want to understand better it.
Thanks again,

Amedeo

[Homer Strong]

The first problem is that nimbus can't talk to ZK. It's looking on localhost, and you say that nimbus and ZK are running on the same host, so my guess is that ZK is not running.

> just run zkServer.sh start from bin for Zookepper?

I assume that you mean from download of the Apache distribution. Yes, `sudo bin/zkServer.sh start` works for me.

> Where to configure the Ip of the supervisor?

You don't need to configure the supervisor IPs anywhere.

You might find this page useful:

https://github.com/nathanmarz/storm/wiki/Setting-up-a-Storm-cluster#fill-in-mandatory-configurations-into-stormyaml

[Amedeo Merlo]

Really thanks for your answer.

Yes I donloaded the apache distribution and I ran sudo bin/zkServer.sh start
It seems working. Then I run storm nimbus and finally submit the topology. I tried both from another virtual machine and from the same when I run zookeper and nimbus (changing the yaml configuration of course), with any success.

Any ideas?


Thank you again,
Amedeo

[Homer Strong]

You meant without any success? Are the same errors logged?

What happens when you run

$ echo ruok | nc 127.0.0.1 2181

on ZK machine?

It would be helpful if you supplied your storm.yaml.

[Amedeo Merlo]

Now I'm running Zookeper and Nimbus on the same machine. When I run $ echo ruok | nc 127.0.0.1 2181 It doesn't happen anything

My storm.yaml is that:

########### These all have default values as shown

########### Additional configuration goes into storm.yaml

java.library.path: "/usr/local/lib:/opt/local/lib:/usr/lib"

### storm.* configs are general configurations

# the local dir is where jars are kept

storm.local.dir: "storm-local"

storm.zookeeper.servers:

    - "localhost"

storm.zookeeper.port: 2181

storm.zookeeper.root: "/storm"

storm.zookeeper.session.timeout: 20000

storm.zookeeper.connection.timeout: 15000

storm.zookeeper.retry.times: 5

storm.zookeeper.retry.interval: 1000

storm.zookeeper.retry.intervalceiling.millis: 30000

storm.cluster.mode: "distributed" # can be distributed or local

storm.local.mode.zmq: false

storm.thrift.transport: "backtype.storm.security.auth.SimpleTransportPlugin"

storm.messaging.transport: "backtype.storm.messaging.zmq"

### nimbus.* configs are for the master

nimbus.host: "localhost"

nimbus.thrift.port: 6627

nimbus.childopts: "-Xmx1024m"

nimbus.task.timeout.secs: 30

nimbus.supervisor.timeout.secs: 60

nimbus.monitor.freq.secs: 10

nimbus.cleanup.inbox.freq.secs: 600

nimbus.inbox.jar.expiration.secs: 3600

nimbus.task.launch.secs: 120

nimbus.reassign: true

nimbus.file.copy.expiration.secs: 600

nimbus.topology.validator: "backtype.storm.nimbus.DefaultTopologyValidator"

### ui.* configs are for the master

ui.port: 8080

ui.childopts: "-Xmx768m"

drpc.port: 3772

drpc.worker.threads: 64

drpc.queue.size: 128

drpc.invocations.port: 3773

drpc.request.timeout.secs: 600

transactional.zookeeper.root: "/transactional"

transactional.zookeeper.servers: null

transactional.zookeeper.port: null

### supervisor.* configs are for node supervisors

# Define the amount of workers that can be run on this machine. Each worker is assigned a port to use for communication

supervisor.slots.ports:

    - 6700

    - 6701

    - 6702

    - 6703

supervisor.childopts: "-Xmx256m"

#how long supervisor will wait to ensure that a worker process is started

supervisor.worker.start.timeout.secs: 120

#how long between heartbeats until supervisor considers that worker dead and tries to restart it

supervisor.worker.timeout.secs: 30

#how frequently the supervisor checks on the status of the processes it's monitoring and restarts if necessary

supervisor.monitor.frequency.secs: 3

#how frequently the supervisor heartbeats to the cluster state (for nimbus)

supervisor.heartbeat.frequency.secs: 5

supervisor.enable: true

### worker.* configs are for task workers

worker.childopts: "-Xmx768m"

worker.heartbeat.frequency.secs: 1

task.heartbeat.frequency.secs: 3

task.refresh.poll.secs: 10

zmq.threads: 1

zmq.linger.millis: 5000

zmq.hwm: 0

### topology.* configs are for specific executing storms

topology.enable.message.timeouts: true

topology.debug: false

topology.optimize: true

topology.workers: 1

topology.acker.executors: 1

topology.acker.tasks: null

topology.tasks: null

# maximum amount of time a message has to complete before it's considered failed

topology.message.timeout.secs: 30

topology.skip.missing.kryo.registrations: false

topology.max.task.parallelism: null

topology.max.spout.pending: null

topology.state.synchronization.timeout.secs: 60

topology.stats.sample.rate: 0.05

topology.builtin.metrics.bucket.size.secs: 60

topology.fall.back.on.java.serialization: true

topology.worker.childopts: null

topology.executor.receive.buffer.size: 1024 #batched

topology.executor.send.buffer.size: 1024 #individual messages

topology.receiver.buffer.size: 8 # setting it too high causes a lot of problems (heartbeat thread gets starved, throughput plummets)

topology.transfer.buffer.size: 1024 # batched

topology.tick.tuple.freq.secs: null

topology.worker.shared.thread.pool.size: 4

topology.disruptor.wait.strategy: "com.lmax.disruptor.BlockingWaitStrategy"

topology.spout.wait.strategy: "backtype.storm.spout.SleepSpoutWaitStrategy"

topology.sleep.spout.wait.strategy.time.ms: 1

topology.error.throttle.interval.secs: 10

topology.max.error.report.per.interval: 5

topology.kryo.factory: "backtype.storm.serialization.DefaultKryoFactory"

topology.tuple.serializer: "backtype.storm.serialization.types.ListDelegateSerializer"

topology.trident.batch.emit.interval.millis: 500

dev.zookeeper.path: "/tmp/dev-storm-zookeeper"

I try to submit the topology and these are the errors:

Exception in thread "main" java.lang.RuntimeException: org.apache.thrift7.transport.TTransportException: java.net.ConnectException: Connection refused

at backtype.storm.utils.NimbusClient.<init>(NimbusClient.java:36)

at backtype.storm.utils.NimbusClient.getConfiguredClient(NimbusClient.java:17)

at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:69)

at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:40)

at Topology2.main(Topology2.java:40)

Caused by: org.apache.thrift7.transport.TTransportException: java.net.ConnectException: Connection refused

at org.apache.thrift7.transport.TSocket.open(TSocket.java:183)

at org.apache.thrift7.transport.TFramedTransport.open(TFramedTransport.java:81)

at backtype.storm.utils.NimbusClient.<init>(NimbusClient.java:34)

... 4 more

Caused by: java.net.ConnectException: Connection refused

at java.net.PlainSocketImpl.socketConnect(Native Method)

at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)

at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)

at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)

at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)

at java.net.Socket.connect(Socket.java:529)

at org.apache.thrift7.transport.TSocket.open(TSocket.java:178)

... 6 more

Thank you very much,
Amedeo

[Philip O'Toole]

Have you actually started Zookeeper?

--

[Amedeo Merlo]

Yes. I downloaded from http://mirrors.supportex.net/apache/zookeeper/zookeeper-3.4.5/ and run from bin with

sudo ./zkServer.sh start

Then it prints that:

JMX enabled by default

Using config: /home/amedeo/Scrivania/zookeeper-3.4.5/bin/../conf/zoo.cfg

Starting zookeeper ... STARTED

[Philip O'Toole]

What do you get if you type:

ps aux | grep zoo

and

telnet 127.0.0.1 2181

?

[Amedeo Merlo]

amedeo@VM-amedeo:~/Scrivania/zookeeper-3.4.5/bin$ ps aux | grep zoo

root      1906  5.0  3.4 251072 17308 pts/0    Sl   18:13   0:00 java -Dzookeeper.log.dir=. -Dzookeeper.root.logger=INFO,CONSOLE -cp /home/amedeo/Scrivania/zookeeper-3.4.5/bin/../build/classes:/home/amedeo/Scrivania/zookeeper-3.4.5/bin/../build/lib/*.jar:/home/amedeo/Scrivania/zookeeper-3.4.5/bin/../lib/slf4j-log4j12-1.6.1.jar:/home/amedeo/Scrivania/zookeeper-3.4.5/bin/../lib/slf4j-api-1.6.1.jar:/home/amedeo/Scrivania/zookeeper-3.4.5/bin/../lib/netty-3.2.2.Final.jar:/home/amedeo/Scrivania/zookeeper-3.4.5/bin/../lib/log4j-1.2.15.jar:/home/amedeo/Scrivania/zookeeper-3.4.5/bin/../lib/jline-0.9.94.jar:/home/amedeo/Scrivania/zookeeper-3.4.5/bin/../zookeeper-3.4.5.jar:/home/amedeo/Scrivania/zookeeper-3.4.5/bin/../src/java/lib/*.jar:/home/amedeo/Scrivania/zookeeper-3.4.5/bin/../conf: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.QuorumPeerMain /home/amedeo/Scrivania/zookeeper-3.4.5/bin/../conf/zoo.cfg

amedeo    1921  0.0  0.1   4664   836 pts/0    S+   18:13   0:00 grep --color=auto zoo

amedeo@VM-amedeo:~/Scrivania/zookeeper-3.4.5/bin$ telnet 127.0.0.1 2181

Trying 127.0.0.1...

telnet: Unable to connect to remote host: Connection refused

[Homer Strong]

Could you post your zoo.cfg and ZK log please?

You received this message because you are subscribed to a topic in the Google Groups "storm-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/storm-user/Fdb1Ds4lC7E/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to storm-user+...@googlegroups.com.

[Philip O'Toole]

>>Trying 127.0.0.1...

telnet: Unable to connect to remote host: Connection refused

I believe this is your problem. You need to fix that issue, if you really think Zookeeper is binding to the local interface. Show us your Zookeeper config file.

Philip

[Amedeo Merlo]

zoo.cfg:

tickTime=2000

dataDir=/var/zookeeper/

clientPort=2181

initLimit=5

syncLimit=2

server.1=zoo1:2888:3888

server.2=zoo2:2888:3888

server.3=zoo3:2888:3888

Where is the ZK log?


Amedeo

[Homer Strong]

That zoo.cfg looks wrong. Change

server.1=zoo1:2888:3888

server.2=zoo2:2888:3888

server.3=zoo3:2888:3888

to 

server.1=localhost:2888:3888

log may be in the zookeeper.out file in the ZK dir.

--

You received this message because you are subscribed to a topic in the Google Groups "storm-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/storm-user/Fdb1Ds4lC7E/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to storm-user+...@googlegroups.com.

[Amedeo Merlo]

THANKS!!
Now it seems working.
I didn't pay attention to that file... 
Anyway can you also tell me when can I see the results?
In my topology I have one spout and two bolts. The lastone prints something. How to see that?


Thanks again and sorry for some "stupid" question
Amedeo.

[Homer Strong]

Look in the worker logs.

[Amedeo Merlo]

Really thanks for the patience.
Unluckly I checked and there are storm exceptions... Can you help me to understand the problems?

Now I'm working in only one machine where I start ZK, nimbus and 4 supervisors

The storm yaml for the supervisor is:

supervisor.slots.ports:
    - 6700
    - 6701
    - 6702
    - 6703

In the logs folder i found supervisor.log and this is the exception:

java.io.FileNotFoundException: File 'storm-local/supervisor/stormdist/mytopology-1-1368240232/stormconf.ser' does not exist
	at org.apache.commons.io.FileUtils.openInputStream(FileUtils.java:137)
	at org.apache.commons.io.FileUtils.readFileToByteArray(FileUtils.java:1135)
	at backtype.storm.config$read_supervisor_storm_conf.invoke(config.clj:138)
	at backtype.storm.daemon.supervisor$fn__4793.invoke(supervisor.clj:414)
	at clojure.lang.MultiFn.invoke(MultiFn.java:177)
	at backtype.storm.daemon.supervisor$sync_processes$iter__4684__4688$fn__4689.invoke(supervisor.clj:249)
	at clojure.lang.LazySeq.sval(LazySeq.java:42)
	at clojure.lang.LazySeq.seq(LazySeq.java:60)
	at clojure.lang.RT.seq(RT.java:473)
	at clojure.core$seq.invoke(core.clj:133)
	at clojure.core$dorun.invoke(core.clj:2725)
	at clojure.core$doall.invoke(core.clj:2741)
	at backtype.storm.daemon.supervisor$sync_processes.invoke(supervisor.clj:237)
	at clojure.lang.AFn.applyToHelper(AFn.java:161)
	at clojure.lang.AFn.applyTo(AFn.java:151)
	at clojure.core$apply.invoke(core.clj:603)
	at clojure.core$partial$fn__4070.doInvoke(core.clj:2343)
	at clojure.lang.RestFn.invoke(RestFn.java:397)
	at backtype.storm.event$event_manager$fn__2507.invoke(event.clj:24)
	at clojure.lang.AFn.run(AFn.java:24)
	at java.lang.Thread.run(Thread.java:662)

The other files are worker-6701.log, worker 6702.log, worker-6703.log and I found:

2013-05-13 15:40:25 worker [ERROR] Error on initialization of server mk-worker
java.io.IOException: No such file or directory
	at java.io.UnixFileSystem.createFileExclusively(Native Method)
	at java.io.File.createNewFile(File.java:883)
	at backtype.storm.util$touch.invoke(util.clj:432)
	at backtype.storm.daemon.worker$fn__4348$exec_fn__1228__auto____4349.invoke(worker.clj:331)
	at clojure.lang.AFn.applyToHelper(AFn.java:185)
	at clojure.lang.AFn.applyTo(AFn.java:151)
	at clojure.core$apply.invoke(core.clj:601)
	at backtype.storm.daemon.worker$fn__4348$mk_worker__4404.doInvoke(worker.clj:323)
	at clojure.lang.RestFn.invoke(RestFn.java:512)
	at backtype.storm.daemon.worker$_main.invoke(worker.clj:433)
	at clojure.lang.AFn.applyToHelper(AFn.java:172)
	at clojure.lang.AFn.applyTo(AFn.java:151)
	at backtype.storm.daemon.worker.main(Unknown Source)

then

java.lang.IncompatibleClassChangeError: Found interface backtype.storm.tuple.Tuple, but class was expected
	at bolts.Bolt1.execute(Bolt1.java:23)
	at backtype.storm.topology.BasicBoltExecutor.execute(BasicBoltExecutor.java:32)
	at backtype.storm.daemon.executor$fn__4050$tuple_action_fn__4052.invoke(executor.clj:566)
	at backtype.storm.daemon.executor$mk_task_receiver$fn__3976.invoke(executor.clj:345)
	at backtype.storm.disruptor$clojure_handler$reify__1606.onEvent(disruptor.clj:43)
	at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:84)
	at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:58)
	at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:62)
	at backtype.storm.daemon.executor$fn__4050$fn__4059$fn__4106.invoke(executor.clj:658)
	at backtype.storm.util$async_loop$fn__465.invoke(util.clj:377)
	at clojure.lang.AFn.run(AFn.java:24)
	at java.lang.Thread.run(Thread.java:662)
2013-05-13 15:27:05 executor [ERROR] 
java.lang.IncompatibleClassChangeError: Found interface backtype.storm.tuple.Tuple, but class was expected
	at bolts.Bolt1.execute(Bolt1.java:23)
	at backtype.storm.topology.BasicBoltExecutor.execute(BasicBoltExecutor.java:32)
	at backtype.storm.daemon.executor$fn__4050$tuple_action_fn__4052.invoke(executor.clj:566)
	at backtype.storm.daemon.executor$mk_task_receiver$fn__3976.invoke(executor.clj:345)
	at backtype.storm.disruptor$clojure_handler$reify__1606.onEvent(disruptor.clj:43)
	at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:84)
	at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:58)
	at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:62)
	at backtype.storm.daemon.executor$fn__4050$fn__4059$fn__4106.invoke(executor.clj:658)
	at backtype.storm.util$async_loop$fn__465.invoke(util.clj:377)
	at clojure.lang.AFn.run(AFn.java:24)
	at java.lang.Thread.run(Thread.java:662)

I copied only the exceptions from one file. They are repeated in the file and the are both on worker-6701.log and worker-6702.log

Thank you,
Amedeo

[Amedeo Merlo]

I forgot to write what I have in Bolt1.java
These are the rows from 19 to 24:

public class Bolt1 extends BaseBasicBolt {

@Override

public void execute(Tuple input, BasicOutputCollector collector) {

String str = input.getString(0);

String start = "Raw Readings: ";
Amedeo