after run mvn package ,how to install storm?

[ch huang]

hi,maillist:

             i run " mvn -f pom-cdh43.xml package" , it create two dir "storm-local,target" ,but when i run

 

"bin/storm-yarn launch" it says 

 

"bin/storm-yarn: line 16: read-link: command not found
storm is not installed"

 

           why ? how to install storm?

[ch huang]

i find some output from mvn -f pom-cdh43.xml package,something wrong?

13:28:40.950 [mainhread] com.yahoo.storm.yarn.StormOnYarn.getClient(StormOnYarn.java:132)evel com.yahoo.storm.yarn.StormOnYarn.getClient(StormOnYarn.java:132)ogger{36} - Attaching to CH34:9000 to talk to app master application_1388640446502_0001sg
13:28:41.065 [mainhread] org.apache.hadoop.yarn.service.AbstractService.stop(AbstractService.java:113)evel org.apache.hadoop.yarn.service.AbstractService.stop(AbstractService.java:113)ogger{36} - Service:org.apache.hadoop.yarn.client.YarnClientImpl is stopped.sg
Exception in thread "main" java.lang.RuntimeException: org.apache.thrift7.transport.TTransportException: java.net.ConnectException: Connection refused
        at com.yahoo.storm.yarn.MasterClient.getConfiguredClient(MasterClient.java:44)
        at com.yahoo.storm.yarn.StormOnYarn.getClient(StormOnYarn.java:133)
        at com.yahoo.storm.yarn.StormMasterCommand.process(StormMasterCommand.java:77)
        at com.yahoo.storm.yarn.Client.execute(Client.java:139)
        at com.yahoo.storm.yarn.Client.main(Client.java:145)
Caused by: org.apache.thrift7.transport.TTransportException: java.net.ConnectException: Connection refused
        at org.apache.thrift7.transport.TSocket.open(TSocket.java:183)
        at org.apache.thrift7.transport.TFramedTransport.open(TFramedTransport.java:81)
        at backtype.storm.security.auth.SimpleTransportPlugin.connect(SimpleTransportPlugin.java:66)
        at backtype.storm.security.auth.ThriftClient.<init>(ThriftClient.java:46)
        at com.yahoo.storm.yarn.MasterClient.<init>(MasterClient.java:50)
        at com.yahoo.storm.yarn.MasterClient.getConfiguredClient(MasterClient.java:38)
        ... 4 more
Caused by: java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:579)
        at org.apache.thrift7.transport.TSocket.open(TSocket.java:178)
        ... 9 more
13:28:42.075 [main] INFO  c.yahoo.storm.yarn.EmbeddedZKServer - shutdown embedded zookeeper server with port 60000

 

在 2014年1月2日星期四UTC+8上午11时58分08秒，ch huang写道：

[ch huang]

and find in nodemanager.log

 

2014-01-02 14:29:45,426 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 1487 for container-id containe
r_1388641982818_0004_01_000001: 427.4 MB of 5 GB physical memory used; 15.4 GB of 10.5 GB virtual memory used
2014-01-02 14:29:45,426 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Process tree for container: container_1388641982818_0004_01_000001 has processes older than 1 iteration running over the configured limit. Limit=11274289152, current usage = 13633417216
2014-01-02 14:29:45,428 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Container [pid=1487,containerID=container_1388641982818_0004_01_000001] is running beyond virtual memory limits. Current usage: 427.4 MB of 5 GB physical memory used; 15.4 GB of 10.5 GB virtual memory used. Killing container.
Dump of the process-tree for container_1388641982818_0004_01_000001 :
        |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
        |- 1523 1493 1487 1487 (java) 198 108 1599909888 28052 /usr/java/jdk1.7.0_45/bin/java -server -Dstorm.home=./storm/storm-0.9.0-wip21/ -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file=storm.yaml -cp conf:./storm/storm-0.9.0-wip21/:./storm/storm-0.9.0-wip21/lib/*:./storm/storm-0.9.0-wip21/* -Xmx1024m -Dlogfile.name=/var/log/hadoop-yarn/containers2/application_1388641982818_0004/container_1388641982818_0004_01_000001/nimbus.log backtype.storm.daemon.nimbus
        |- 1487 31281 1487 1487 (bash) 0 0 9457664 278 /bin/bash -c /usr/java/jdk1.7.0_45/bin/java -Dstorm.home=./storm/storm-0.9.0-wip21/ -Dlogfile.name=/var/log/hadoop-yarn/containers2/application_1388641982818_0004/container_1388641982818_0004_01_000001/master.log com.yahoo.storm.yarn.MasterServer 1>/var/log/hadoop-yarn/containers2/application_1388641982818_0004/container_1388641982818_0004_01_000001/stderr 2>/var/log/hadoop-yarn/containers2/application_1388641982818_0004/container_1388641982818_0004_01_000001/stdout
        |- 1524 1493 1487 1487 (java) 231 89 1323163648 32970 /usr/java/jdk1.7.0_45/bin/java -server -Dstorm.home=./storm/storm-0.9.0-wip21/ -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file=storm.yaml -cp conf:./storm/storm-0.9.0-wip21/:./storm/storm-0.9.0-wip21/lib/*:./storm/storm-0.9.0-wip21/* -Xmx768m -Dstorm.options=nimbus.host=localhost -Dlogfile.name=/var/log/hadoop-yarn/containers2/application_1388641982818_0004/container_1388641982818_0004_01_000001/ui.log backtype.storm.ui.core
        |- 1493 1487 1487 1487 (java) 377 75 13623959552 48124 /usr/java/jdk1.7.0_45/bin/java -Dstorm.home=./storm/storm-0.9.0-wip21/ -Dlogfile.name=/var/log/hadoop-yarn/containers2/application_1388641982818_0004/container_1388641982818_0004_01_000001/master.log com.yahoo.storm.yarn.MasterServer

 

 

在 2014年1月2日星期四UTC+8下午1时30分08秒，ch huang写道：

[Bobby Evans]

I am not totally sure what is going on here.  The first set of exceptions showed thrift trying to bind to a port that was not free. I think the tests have a set port, which stinks, and we should change it.

The second showed that the container was going way over the virtual memory limit (15.4 GB) but only using less than 430 MB.  I am not really sure what would be causing this without looking more deeply at the live App Master process.  It is using what looks like 12 or the 15GB or virtual memory and I really don’t know why.  It should be tiny.  If you can reproduce it please either turn off the virtual memory checks for the node manager

yarn.nodemanager.vmem-check-enabled=false

Or boost the vmdm/pmem ratio to be a lot higher, like 4 or 5.

yarn.nodemanager.vmem-pmem-ratio=5

Then take a look at the AppMaster, the one that has com.yahoo.storm.yarn.MasterServer on the command line.  Take a look at the stack trace (jstack) it could be that we are using a ton of threads (although that seems unlikely).  Try using pmap or /proc/<PID>/maps to give us an idea of where the memory table is allocated.   If it shows the majority of that 15GB as [anon] or blank, then it may be a lot more difficult to debug, if so please let me know and I’ll try to reproduce it locally.

—Bobby

--
You received this message because you are subscribed to the Google Groups "storm-yarn" group.
To unsubscribe from this group and stop receiving emails from it, send an email to storm-yarn+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.