how to validate if the storm in yarn installed successfully?

[ch huang]

hi,maillist:

       i installed storm on yarn ,and use "storm-yarn launch" command start storm ,i see the following info in http://myresourcemanagerhost:8080/cluster/scheduler

 

application_1389249819948_0005

hdfs

Storm-on-Yarn

default

Thu, 09 Jan 2014 08:00:41 GMT

N/A

RUNNING

UNDEFINED

ApplicationMaster

 

but when i use the following process test,it failed ,i do not know why?

 

# wget https://github.com/nathanmarz/storm-starter/archive/master.zip

# mv master master.zip

# unzip master.zip

# cd storm-starter-master/

# mvn -f m2-pom.xml package

# ls
LICENSE     multilang    README.markdown  storm-starter.iml  test
m2-pom.xml  project.clj  src              target

# cd target/

 

[root@CH34 target]# storm jar storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar storm.starter.WordCountTopology WordCountTopology
Running: java -client -Dstorm.options= -Dstorm.home=/opt/storm-0.9.0-wip21 -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /opt/storm-0.9.0-wip21/storm-netty-0.9.0-wip21.jar:/opt/storm-0.9.0-wip21/storm-core-0.9.0-wip21.jar:/opt/storm-0.9.0-wip21/storm-console-logging-0.9.0-wip21.jar:/opt/storm-0.9.0-wip21/lib/hiccup-0.3.6.jar:/opt/storm-0.9.0-wip21/lib/curator-framework-1.0.1.jar:/opt/storm-0.9.0-wip21/lib/carbonite-1.5.0.jar:/opt/storm-0.9.0-wip21/lib/compojure-1.1.3.jar:/opt/storm-0.9.0-wip21/lib/jgrapht-0.8.3.jar:/opt/storm-0.9.0-wip21/lib/httpclient-4.1.1.jar:/opt/storm-0.9.0-wip21/lib/junit-3.8.1.jar:/opt/storm-0.9.0-wip21/lib/core.incubator-0.1.0.jar:/opt/storm-0.9.0-wip21/lib/netty-3.6.3.Final.jar:/opt/storm-0.9.0-wip21/lib/commons-codec-1.4.jar:/opt/storm-0.9.0-wip21/lib/slf4j-api-1.6.5.jar:/opt/storm-0.9.0-wip21/lib/tools.cli-0.2.2.jar:/opt/storm-0.9.0-wip21/lib/servlet-api-2.5-20081211.jar:/opt/storm-0.9.0-wip21/lib/jline-0.9.94.jar:/opt/storm-0.9.0-wip21/lib/logback-classic-1.0.6.jar:/opt/storm-0.9.0-wip21/lib/servlet-api-2.5.jar:/opt/storm-0.9.0-wip21/lib/ring-core-1.1.5.jar:/opt/storm-0.9.0-wip21/lib/jetty-6.1.26.jar:/opt/storm-0.9.0-wip21/lib/commons-exec-1.1.jar:/opt/storm-0.9.0-wip21/lib/ring-devel-0.3.11.jar:/opt/storm-0.9.0-wip21/lib/tools.macro-0.1.0.jar:/opt/storm-0.9.0-wip21/lib/ring-jetty-adapter-0.3.11.jar:/opt/storm-0.9.0-wip21/lib/jzmq-2.1.0.jar:/opt/storm-0.9.0-wip21/lib/mockito-all-1.9.5.jar:/opt/storm-0.9.0-wip21/lib/clj-time-0.4.1.jar:/opt/storm-0.9.0-wip21/lib/disruptor-2.10.1.jar:/opt/storm-0.9.0-wip21/lib/asm-4.0.jar:/opt/storm-0.9.0-wip21/lib/ring-servlet-0.3.11.jar:/opt/storm-0.9.0-wip21/lib/commons-logging-1.1.1.jar:/opt/storm-0.9.0-wip21/lib/jetty-util-6.1.26.jar:/opt/storm-0.9.0-wip21/lib/clout-1.0.1.jar:/opt/storm-0.9.0-wip21/lib/objenesis-1.2.jar:/opt/storm-0.9.0-wip21/lib/commons-fileupload-1.2.1.jar:/opt/storm-0.9.0-wip21/lib/commons-lang-2.5.jar:/opt/storm-0.9.0-wip21/lib/joda-time-2.0.jar:/opt/storm-0.9.0-wip21/lib/zookeeper-3.3.3.jar:/opt/storm-0.9.0-wip21/lib/clj-stacktrace-0.2.2.jar:/opt/storm-0.9.0-wip21/lib/tools.logging-0.2.3.jar:/opt/storm-0.9.0-wip21/lib/kryo-2.17.jar:/opt/storm-0.9.0-wip21/lib/minlog-1.2.jar:/opt/storm-0.9.0-wip21/lib/curator-client-1.0.1.jar:/opt/storm-0.9.0-wip21/lib/commons-io-1.4.jar:/opt/storm-0.9.0-wip21/lib/snakeyaml-1.11.jar:/opt/storm-0.9.0-wip21/lib/libthrift7-0.7.0-2.jar:/opt/storm-0.9.0-wip21/lib/log4j-over-slf4j-1.6.6.jar:/opt/storm-0.9.0-wip21/lib/guava-13.0.jar:/opt/storm-0.9.0-wip21/lib/json-simple-1.1.jar:/opt/storm-0.9.0-wip21/lib/clojure-1.4.0.jar:/opt/storm-0.9.0-wip21/lib/httpcore-4.1.jar:/opt/storm-0.9.0-wip21/lib/reflectasm-1.07-shaded.jar:/opt/storm-0.9.0-wip21/lib/logback-core-1.0.6.jar:/opt/storm-0.9.0-wip21/lib/math.numeric-tower-0.0.1.jar:storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar:/opt/storm-0.9.0-wip21/conf:/opt/storm-0.9.0-wip21/bin -Dstorm.jar=storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar storm.starter.WordCountTopology WordCountTopology
Exception in thread "main" java.lang.RuntimeException: org.apache.thrift7.transport.TTransportException: java.net.ConnectException: Connection refused
        at backtype.storm.utils.NimbusClient.getConfiguredClient(NimbusClient.java:21)
        at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:70)
        at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:41)
        at storm.starter.WordCountTopology.main(WordCountTopology.java:77)
Caused by: org.apache.thrift7.transport.TTransportException: java.net.ConnectException: Connection refused
        at org.apache.thrift7.transport.TSocket.open(TSocket.java:183)
        at org.apache.thrift7.transport.TFramedTransport.open(TFramedTransport.java:81)
        at backtype.storm.security.auth.SimpleTransportPlugin.connect(SimpleTransportPlugin.java:66)
        at backtype.storm.security.auth.ThriftClient.<init>(ThriftClient.java:46)
        at backtype.storm.utils.NimbusClient.<init>(NimbusClient.java:30)
        at backtype.storm.utils.NimbusClient.<init>(NimbusClient.java:26)
        at backtype.storm.utils.NimbusClient.getConfiguredClient(NimbusClient.java:19)
        ... 3 more
Caused by: java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:579)
        at org.apache.thrift7.transport.TSocket.open(TSocket.java:178)
        ... 9 more

 

one strange thing is i install storm on CH34, but when launch storm,the nimbus start on CH36,here is my storm.yaml content

 

 

########### These MUST be filled in for a storm configuration
storm.zookeeper.servers:
    - 192.168.10.220
    - 192.168.10.221
    - 192.168.10.222

storm.zookeeper.port: 2181
storm.zookeeper.root: "/storm"
storm.zookeeper.session.timeout: 20000
storm.zookeeper.connection.timeout: 15000
storm.zookeeper.retry.times: 5
storm.zookeeper.retry.interval: 1000
storm.zookeeper.retry.intervalceiling.millis: 30000
storm.cluster.mode: "distributed"

nimbus.host: localhost
nimbus.thrift.port: 8889
nimbus.thrift.max_buffer_size: 1048576
nimbus.childopts: "-Xmx1024m"
nimbus.task.timeout.secs: 30
nimbus.supervisor.timeout.secs: 60
nimbus.monitor.freq.secs: 10
nimbus.cleanup.inbox.freq.secs: 600
nimbus.inbox.jar.expiration.secs: 3600
nimbus.task.launch.secs: 120
nimbus.reassign: true
nimbus.file.copy.expiration.secs: 600
nimbus.topology.validator: "backtype.storm.nimbus.DefaultTopologyValidator"

ui.port: 8080
ui.childopts: "-Xmx768m"

logviewer.port: 8000
logviewer.childopts: "-Xmx128m"
logviewer.appender.name: "A1"

drpc.port: 3772
drpc.worker.threads: 64
drpc.queue.size: 128
drpc.invocations.port: 3773
drpc.request.timeout.secs: 600
drpc.childopts: "-Xmx768m"

transactional.zookeeper.root: "/transactional"
transactional.zookeeper.servers: null
transactional.zookeeper.port: null
### supervisor.* configs are for node supervisors
# Define the amount of workers that can be run on this machine. Each worker is assigned a port to use for communication
supervisor.slots.ports:
    - 6700
    - 6701
    - 6702
    - 6703
supervisor.childopts: "-Xmx256m"
#how long supervisor will wait to ensure that a worker process is started
supervisor.worker.start.timeout.secs: 120
#how long between heartbeats until supervisor considers that worker dead and tries to restart it
supervisor.worker.timeout.secs: 30
#how frequently the supervisor checks on the status of the processes it's monitoring and restarts if necessary
supervisor.monitor.frequency.secs: 3
#how frequently the supervisor heartbeats to the cluster state (for nimbus)
supervisor.heartbeat.frequency.secs: 5
supervisor.enable: true

### worker.* configs are for task workers
worker.childopts: "-Xmx768m"
worker.heartbeat.frequency.secs: 1

task.heartbeat.frequency.secs: 3
task.refresh.poll.secs: 10

zmq.threads: 1
zmq.linger.millis: 5000
zmq.hwm: 0

storm.messaging.netty.server_worker_threads: 1
storm.messaging.netty.client_worker_threads: 1
storm.messaging.netty.buffer_size: 5242880 #5MB buffer
storm.messaging.netty.max_retries: 30
storm.messaging.netty.max_wait_ms: 1000
storm.messaging.netty.min_wait_ms: 100
### topology.* configs are for specific executing storms
topology.enable.message.timeouts: true
topology.debug: false
topology.optimize: true
topology.workers: 1
topology.acker.executors: null
topology.tasks: null
# maximum amount of time a message has to complete before it's considered failed
topology.message.timeout.secs: 30
topology.skip.missing.kryo.registrations: false
topology.max.task.parallelism: null
topology.max.spout.pending: null
topology.state.synchronization.timeout.secs: 60
topology.stats.sample.rate: 0.05
topology.builtin.metrics.bucket.size.secs: 60
topology.fall.back.on.java.serialization: true
topology.worker.childopts: null
topology.executor.receive.buffer.size: 1024 #batched
topology.executor.send.buffer.size: 1024 #individual messages
topology.receiver.buffer.size: 8 # setting it too high causes a lot of problems (heartbeat thread gets starved, throughput plummets)
topology.transfer.buffer.size: 1024 # batched
topology.tick.tuple.freq.secs: null
topology.worker.shared.thread.pool.size: 4
topology.disruptor.wait.strategy: "com.lmax.disruptor.BlockingWaitStrategy"
topology.spout.wait.strategy: "backtype.storm.spout.SleepSpoutWaitStrategy"
topology.sleep.spout.wait.strategy.time.ms: 1
topology.error.throttle.interval.secs: 10
topology.max.error.report.per.interval: 5
topology.kryo.factory: "backtype.storm.serialization.DefaultKryoFactory"
topology.tuple.serializer: "backtype.storm.serialization.types.ListDelegateSerializer"
topology.trident.batch.emit.interval.millis: 500

dev.zookeeper.path: "/tmp/dev-storm-zookeeper"

[Andy Feng]

You need to get storm.yaml from storm-yarn. The storm.yaml mentioned in your email points nimbus to your local host, and it is not connecting to nimbus on YARN. 

--
You received this message because you are subscribed to the Google Groups "storm-yarn" group.
To unsubscribe from this group and stop receiving emails from it, send an email to storm-yarn+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.