building storm-yarn-master with mvn on cloud

615 views
Skip to first unread message

tsanghv

unread,
Nov 26, 2013, 5:29:39 PM11/26/13
to storm...@googlegroups.com
I am trying to follow the instructions on hortonworks - storm on yarn. I am very new to all the components that I am working with: Cloud/Hadoop/Storm.

I followed the instructions on that page to install the following on a 3-node cloud system on OpenStack
- JDK 1.7
- Apache Ambari
- Hadoop version 2.1.0.2.0.5.0-67
Everything seems to be working fine.


Now I am trying to install Storm on top of it. So
- installed Maven 3.1.1 on one of the machines in the cluster
- installed Storm on the same machine and tried to build the mvn package but I get many different errors (attaching the file which shows the entire logs):


21:21:32.103 [Public Localizer] INFO o.a.h.y.s.n.c.l.ResourceLocalizationService - Failed to download rsrc { { file:/home/hdfs/master/storm-yarn-master/lib/storm.zip, 1380896156000, ARCHIVE, null },pending,[(container_1385526077580_0001_01_000001)],968675556624033,DOWNLOADING} java.io.IOException: Resource file:/home/hdfs/master/storm-yarn-master/lib/storm.zip is not publicly accessable and as such cannot be part of the public cache. at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:182) ~[hadoop-yarn-common-2.1.0.2.0.5.0-67.jar:na] at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:276) ~[hadoop-yarn-common-2.1.0.2.0.5.0-67.jar:na]


21:21:32.124 [LocalizerRunner for container_1385526077580_0001_01_000001] WARN org.apache.hadoop.util.Shell - Interrupted while reading the error stream java.lang.InterruptedException: null at java.lang.Object.wait(Native Method) ~[na:1.7.0_45] at java.lang.Thread.join(Thread.java:1280) ~[na:1.7.0_45] at java.lang.Thread.join(Thread.java:1354) ~[na:1.7.0_45] at org.apache.hadoop.util.Shell.runCommand(Shell.java:450) [hadoop-common-2.1.0.2.0.5.0-67.jar:na]


21:21:33.729 [LocalizerRunner for container_1385526077580_0001_02_000001] INFO  o.a.h.y.s.n.c.l.ResourceLocalizationService - Localizer failed
java.io.IOException: java.lang.InterruptedException
at org.apache.hadoop.util.Shell.runCommand(Shell.java:461) ~[hadoop-common-2.1.0.2.0.5.0-67.jar:na]
at org.apache.hadoop.util.Shell.run(Shell.java:373) ~[hadoop-common-2.1.0.2.0.5.0-67.jar:na]


21:21:47.598 [mainhread] org.apache.hadoop.util.NativeCodeLoader.<clinit>(NativeCodeLoader.java:62)evel org.apache.hadoop.util.NativeCodeLoader.<clinit>(NativeCodeLoader.java:62)ogger{36} - Unable to load native-hadoop library for your platform... using builtin-java classes where applicablesg Exception in thread "main" java.lang.NullPointerException at com.yahoo.storm.yarn.StormMasterCommand.downloadStormYaml(StormMasterCommand.java:168)


Running: java -client -Dstorm.options= -Dstorm.home=/home/hdfs/master/storm-yarn-master/lib/storm-0.9.0-wip21 -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /home/hdfs/master/storm-yarn-master/lib/storm-0.9.0-wip21/storm-core-0.9.0-wip21.jar:/home/hdfs/master/storm-yarn-master/lib/storm-0.9.0-wip21/storm-console-logging-0.9.0-wip21.jar:/home/hdfs/master/storm-yarn-master/lib/storm-0.9.0-wip21/storm-netty-0.9.0-wip21.jar:/home/hdfs/master/storm-yarn-master/lib/storm-0.9.0-wip21/lib/math.numeric-tower-0.0.1.jar:/home/hdfs/master/storm-yarn-master/lib/storm-0.9.0-wip21/lib/tools.logging-0.2.3.jar:/home/hdfs/master/storm-yarn-master/lib/storm-0.9.0-wip21/lib/log4j-over-slf4j-1.6.6.jar:/home/hdfs/master/storm-yarn-master/lib/storm-0.9.0-wip21/lib/servlet-api-2.5-20081211.jar:/home/hdfs/master/storm-yarn-master/lib/storm-0.9.0-wip21/lib/tools.cli-0.2.2.jar:/home/hdfs/master/storm-yarn-master/lib/storm-0.9.0-wip21/lib/commons-codec-1.4.jar:/home/hdfs/master/storm-yarn-master/lib/storm-0.9.0-wip21/lib/clojure-1.4.0.jar:/home/hdfs/master/storm-yarn-master/lib/storm-0.9.0-wip21/lib/minlog-1.2.jar:/home/hdfs/master/storm-yarn-master/lib/storm-0.9.0-wip21/lib/reflectasm-1.07-shaded.jar:/home/hdfs/master/storm-yarn-master/lib/storm-0.9.0-wip21/lib/libthrift7-0.7.0-2.jar:/home/hdfs/master/storm-yarn-master/lib/storm-0.9.0-wip21/lib/tools.macro-0.1.0.jar:/home/hdfs/master/storm-yarn-master/lib/storm-0.9.0-wip21/lib/json-simple-1.1.jar:/home/hdfs/master/storm-yarn-master/lib/storm-0.9.0-wip21/lib/ring-servlet-0.3.11.jar:/home/hdfs/master/storm-yarn-master/lib/storm-0.9.0-wip21/lib/slf4j-api-1.6.5.jar:/home/hdfs/master/storm-yarn-master/lib/storm-0.9.0-wip21/lib/commons-lang-2.5.jar:/home/hdfs/master/storm-yarn-master/lib/storm-0.9.0-wip21/lib/ring-jetty-adapter-0.3.11.jar:/home/hdfs/master/storm-yarn-master/lib/storm-0.9.0-wip21/lib/clout-1.0.1.jar:/home/hdfs/master/storm-yarn-master/lib/storm-0.9.0-wip21/lib/jline-0.9.94.jar:/home/hdfs/master/storm-yarn-master/lib/storm-0.9.0-wip21/lib/logback-core-1.0.6.jar:/home/hdfs/master/storm-yarn-master/lib/storm-0.9.0-wip21/lib/objenesis-1.2.jar:/home/hdfs/master/storm-yarn-master/lib/storm-0.9.0-wip21/lib/logback-classic-1.0.6.jar:/home/hdfs/master/storm-yarn-master/lib/storm-0.9.0-wip21/lib/curator-client-1.0.1.jar:/home/hdfs/master/storm-yarn-master/lib/storm-0.9.0-wip21/lib/asm-4.0.jar:/home/hdfs/master/storm-yarn-master/lib/storm-0.9.0-wip21/lib/jzmq-2.1.0.jar:/home/hdfs/master/storm-yarn-master/lib/storm-0.9.0-wip21/lib/jgrapht-0.8.3.jar:/home/hdfs/master/storm-yarn-master/lib/storm-0.9.0-wip21/lib/snakeyaml-1.11.jar:/home/hdfs/master/storm-yarn-master/lib/storm-0.9.0-wip21/lib/carbonite-1.5.0.jar:/home/hdfs/master/storm-yarn-master/lib/storm-0.9.0-wip21/lib/mockito-all-1.9.5.jar:/home/hdfs/master/storm-yarn-master/lib/storm-0.9.0-wip21/lib/netty-3.6.3.Final.jar:/home/hdfs/master/storm-yarn-master/lib/storm-0.9.0-wip21/lib/ring-devel-0.3.11.jar:/home/hdfs/master/storm-yarn-master/lib/storm-0.9.0-wip21/lib/commons-io-1.4.jar:/home/hdfs/master/storm-yarn-master/lib/storm-0.9.0-wip21/lib/compojure-1.1.3.jar:/home/hdfs/master/storm-yarn-master/lib/storm-0.9.0-wip21/lib/zookeeper-3.3.3.jar:/home/hdfs/master/storm-yarn-master/lib/storm-0.9.0-wip21/lib/disruptor-2.10.1.jar:/home/hdfs/master/storm-yarn-master/lib/storm-0.9.0-wip21/lib/clj-time-0.4.1.jar:/home/hdfs/master/storm-yarn-master/lib/storm-0.9.0-wip21/lib/jetty-6.1.26.jar:/home/hdfs/master/storm-yarn-master/lib/storm-0.9.0-wip21/lib/jetty-util-6.1.26.jar:/home/hdfs/master/storm-yarn-master/lib/storm-0.9.0-wip21/lib/curator-framework-1.0.1.jar:/home/hdfs/master/storm-yarn-master/lib/storm-0.9.0-wip21/lib/commons-logging-1.1.1.jar:/home/hdfs/master/storm-yarn-master/lib/storm-0.9.0-wip21/lib/guava-13.0.jar:/home/hdfs/master/storm-yarn-master/lib/storm-0.9.0-wip21/lib/commons-exec-1.1.jar:/home/hdfs/master/storm-yarn-master/lib/storm-0.9.0-wip21/lib/commons-fileupload-1.2.1.jar:/home/hdfs/master/storm-yarn-master/lib/storm-0.9.0-wip21/lib/junit-3.8.1.jar:/home/hdfs/master/storm-yarn-master/lib/storm-0.9.0-Exception in thread "main" java.lang.RuntimeException: org.apache.thrift7.transport.TTransportException: java.net.ConnectException: Connection refused at backtype.storm.utils.NimbusClient.getConfiguredClient(NimbusClient.java:21) at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:70) at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:41) at storm.starter.ExclamationTopology.main(ExclamationTopology.java:60) Caused by: org.apache.thrift7.transport.TTransportException: java.net.ConnectException: Connection refused at org.apache.thrift7.transport.TSocket.open(TSocket.java:183) at org.apache.thrift7.transport.TFramedTransport.open(TFramedTransport.java:81) at backtype.storm.security.auth.SimpleTransportPlugin.connect(SimpleTransportPlugin.java:66) at backtype.storm.security.auth.ThriftClient.<init>(ThriftClient.java:46) at backtype.storm.utils.NimbusClient.<init>(NimbusClient.java:30) at backtype.storm.utils.NimbusClient.<init>(NimbusClient.java:26) at backtype.storm.utils.NimbusClient.getConfiguredClient(NimbusClient.java:19) ... 3 more Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391) at java.net.Socket.connect(Socket.java:579) at org.apache.thrift7.transport.TSocket.open(TSocket.java:178) ... 9 more


21:22:41.223 [mainhread] org.apache.hadoop.util.NativeCodeLoader.<clinit>(NativeCodeLoader.java:62)evel org.apache.hadoop.util.NativeCodeLoader.<clinit>(NativeCodeLoader.java:62)ogger{36} - Unable to load native-hadoop library for your platform... using builtin-java classes where applicablesg Exception in thread "main" java.lang.NullPointerException at com.yahoo.storm.yarn.StormMasterCommand.process(StormMasterCommand.java:110) at com.yahoo.storm.yarn.Client.execute(Client.java:139) at com.yahoo.storm.yarn.Client.main(Client.java:145)


I tried to look on this forum for some solutions to this problem but nothing was clear. There was some mention of using hadoop-0.23 branch for storm-yarn instead of master but that did not fix the problem either.
mvn_clean_package.logs

NiRmAl KuMaR

unread,
Nov 27, 2013, 12:08:17 AM11/27/13
to storm...@googlegroups.com
Hi,

Me too got the same error. The build though succeeds. Refer: https://groups.google.com/forum/#!topic/storm-yarn/KLYaQD9HoXA

But you can ignore them and move forward. 

Thanks,
-Nirmal

tsanghv

unread,
Nov 27, 2013, 2:55:24 AM11/27/13
to storm...@googlegroups.com

It seems like from the post below that the issue was resolved when you used 
* storm-yarn branch: hadoop-0.23 instead of master and 
* you used Apache Hadoop 0.23.9

I have hadoop distribution that came from Horton Works installed and the version is 2.1.0.2.0.5.0-67.

These are the steps I followed to use storm-yarn branch hadoop-0.23
* I downloaded storm-yarn branch hadoop-0.23 using  wget https://github.com/yahoo/storm-yarn/archive/hadoop-0.23.zip
* I did not make any changes to the pom-0.23.xml file which I did for master branch (I had made changes for master branch as mentioned for Storm on Yarn).
* I followed rest of the instructions which were similar for the master branch and then tried to build the storm-yarn package using mvn -f pom-0.23.xml package but it fails with the errors below.

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/client/YarnClientImpl
        at com.yahoo.storm.yarn.StormOnYarn.<init>(StormOnYarn.java:80)
        at com.yahoo.storm.yarn.StormOnYarn.<init>(StormOnYarn.java:75)
        at com.yahoo.storm.yarn.StormOnYarn.launchApplication(StormOnYarn.java:363)
        at com.yahoo.storm.yarn.LaunchCommand.process(LaunchCommand.java:55)
        at com.yahoo.storm.yarn.Client.execute(Client.java:139)
        at com.yahoo.storm.yarn.Client.main(Client.java:145)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.yarn.client.YarnClientImpl
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        ... 6 more
07:41:56.887 [main] ERROR com.yahoo.storm.yarn.TestIntegration - setup failure
java.io.FileNotFoundException: target/appId.txt (No such file or directory)
        at java.io.FileInputStream.open(Native Method) ~[na:1.7.0_45]
        at java.io.FileInputStream.<init>(FileInputStream.java:146) ~[na:1.7.0_45]
        at java.io.FileInputStream.<init>(FileInputStream.java:101) ~[na:1.7.0_45]
        at java.io.FileReader.<init>(FileReader.java:58) ~[na:1.7.0_45]
        at com.yahoo.storm.yarn.TestIntegration.setup(TestIntegration.java:102) ~[test-classes/:na]


Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.43 sec <<< FAILURE!
Results :
Failed tests:   com.yahoo.storm.yarn.TestIntegration: expected:<null> but was:<java.io.FileNotFoundException: target/appId.txt (No such file or directory)>
Tests in error:
  com.yahoo.storm.yarn.TestStormCluster: UI server already running.
Tests run: 3, Failures: 1, Errors: 1, Skipped: 0

Bobby Evans

unread,
Nov 27, 2013, 10:59:12 AM11/27/13
to tsanghv, storm...@googlegroups.com
Looking at the log you attached it shows that you had one of the tests fail because it tried to bring up the storm UI daemon and could not bind to the desired port.  The default ui port is 8080, which is rather common, and on some versions of hadoop it is the default port used for the shuffle service.  If you want to ignore the error you probably can, and just run

mvn package –DskipTests

Either way it sounds like you will want to change the ui port to be a different value when you run your cluster for real.

--Bobby

--
You received this message because you are subscribed to the Google Groups "storm-yarn" group.
To unsubscribe from this group and stop receiving emails from it, send an email to storm-yarn+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

tsanghv

unread,
Nov 27, 2013, 12:03:22 PM11/27/13
to storm...@googlegroups.com, tsanghv
Thanks for sharing the information about 0.23 and HDP 2.1 being incompatible. 

Now that I know master is supposed to work with HDP 2.1 - I think there are 2 suggestions that are made.

#Suggestion 1

Look at the POM file comments to see if I find anything. I did not see any comment around HDP or Hortonworks in POM.xml itself but HDP - Storm On Yarn suggest to do the following which I already did.

Edit the pom.xml repos and Hadoop version to point at Hortonworks.soya1

#Suggestion 2

Build storm-yarn without building the tests. I tried that as well and in this case when I try to get the stormConfig I see some errors. These are the commands I run:
1. mvn package -DskipTests
2. storm-yarn launch
3. yarn application -list (this does not show any running applications)
4. storm-yarn getStormConfig -appId application_1384576174968_0017 -output ~/.storm/storm.yaml
I get the application id from command 2 above so I still try to get the storm.yaml file but it fails with the following exception:

Exception in thread "main" java.lang.RuntimeException: org.apache.thrift7.transport.TTransportException: java.net.ConnectException: Connection refused
        at com.yahoo.storm.yarn.MasterClient.getConfiguredClient(MasterClient.java:42)
        at com.yahoo.storm.yarn.StormOnYarn.getClient(StormOnYarn.java:133)
        at com.yahoo.storm.yarn.StormMasterCommand.process(StormMasterCommand.java:76)
        at com.yahoo.storm.yarn.Client.execute(Client.java:139)
        at com.yahoo.storm.yarn.Client.main(Client.java:145)
Caused by: org.apache.thrift7.transport.TTransportException: java.net.ConnectException: Connection refused
        at org.apache.thrift7.transport.TSocket.open(TSocket.java:183)
        at org.apache.thrift7.transport.TFramedTransport.open(TFramedTransport.java:81)
        at backtype.storm.security.auth.SimpleTransportPlugin.connect(SimpleTransportPlugin.java:66)
        at backtype.storm.security.auth.ThriftClient.<init>(ThriftClient.java:46)
        at com.yahoo.storm.yarn.MasterClient.<init>(MasterClient.java:48)
        at com.yahoo.storm.yarn.MasterClient.getConfiguredClient(MasterClient.java:36)
        ... 4 more
Caused by: java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:579)
        at org.apache.thrift7.transport.TSocket.open(TSocket.java:178)
        ... 9 more

There was also the mention of changing the UI port but I am not sure where to change that.

Bobby Evans

unread,
Nov 27, 2013, 12:07:20 PM11/27/13
to tsanghv, storm...@googlegroups.com
The error logs you sent only indicate that there was an error while starting the storm cluster, but nothing about what that error could be.  What was the output from the storm-yarn launch command ?

tsanghv

unread,
Nov 27, 2013, 1:21:53 PM11/27/13
to storm...@googlegroups.com, tsanghv
I dont see any error when I run storm-yarn launch command. These are the outputs from running the commands.

[hdfs@mm-hdp-dn-95338 storm-yarn-master]$ mvn package -DskipTests
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building storm-yarn 1.0-alpha
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ storm-yarn ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 2 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.0:compile (default-compile) @ storm-yarn ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ storm-yarn ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 2 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.0:testCompile (default-testCompile) @ storm-yarn ---
[INFO] Changes detected - recompiling the module!
[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!
[INFO] Compiling 5 source files to /home/hdfs/master/storm-yarn-master/target/test-classes
[INFO]
[INFO] --- maven-jar-plugin:2.4:jar (default) @ storm-yarn ---
[INFO]
[INFO] --- maven-surefire-plugin:2.10:test (default-test) @ storm-yarn ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ storm-yarn ---
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 6.207s
[INFO] Finished at: Wed Nov 27 18:18:01 GMT-07:00 2013
[INFO] Final Memory: 14M/163M
[INFO] ------------------------------------------------------------------------
[hdfs@mm-hdp-dn-95338 storm-yarn-master]$

[hdfs@mm-hdp-dn-95338 storm-yarn-master]$ storm-yarn launch
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hdfs/master/storm/storm-0.9.0-wip21/lib/logback-classic-1.0.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
13/11/27 18:15:26 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
13/11/27 18:15:27 INFO client.RMProxy: Connecting to ResourceManager at mm-hdp-rm-95334.phx-os1.stratus.dev.ebay.com/10.9.209.246:8050
13/11/27 18:15:27 INFO yarn.StormOnYarn: Copy App Master jar from local filesystem and add to local environment
13/11/27 18:15:27 WARN hdfs.BlockReaderLocal: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
13/11/27 18:15:29 INFO yarn.StormOnYarn: Set the environment for the application master
13/11/27 18:15:29 INFO yarn.StormOnYarn: YARN CLASSPATH COMMAND = [[yarn, classpath]]
13/11/27 18:15:29 INFO yarn.StormOnYarn: YARN CLASSPATH = [/etc/hadoop/conf:/etc/hadoop/conf:/etc/hadoop/conf:/usr/lib/hadoop/lib/*:/usr/lib/hadoop/.//*:/usr/lib/hadoop-hdfs/./:/usr/lib/hadoop-hdfs/lib/*:/usr/lib/hadoop-hdfs/.//*:/usr/lib/hadoop-yarn/lib/*:/usr/lib/hadoop-yarn/.//*:/usr/lib/hadoop-mapreduce/lib/*:/usr/lib/hadoop-mapreduce/.//*::/usr/share/java/mysql-connector-java-5.1.17.jar:/usr/share/java/mysql-connector-java.jar:/usr/lib/hadoop-mapreduce/*:/usr/lib/hadoop-yarn/.//*:/usr/lib/hadoop-yarn/lib/*]
13/11/27 18:15:29 INFO yarn.StormOnYarn: Using JAVA_HOME = [/usr/java/jdk1.7.0_45]
13/11/27 18:15:29 INFO yarn.StormOnYarn: Setting up app master command:[/usr/java/jdk1.7.0_45/bin/java, -Dstorm.home=./storm/storm-0.9.0-wip21/, -Dlogfile.name=<LOG_DIR>/master.log, com.yahoo.storm.yarn.MasterServer, 1><LOG_DIR>/stderr, 2><LOG_DIR>/stdout]
13/11/27 18:15:29 INFO impl.YarnClientImpl: Submitted application application_1384576174968_0018 to ResourceManager at mm-hdp-rm-95334.phx-os1.stratus.dev.ebay.com/10.9.209.246:8050
application_1384576174968_0018

[hdfs@mm-hdp-dn-95338 storm-yarn-master]$ yarn application -list
13/11/27 18:16:32 INFO client.RMProxy: Connecting to ResourceManager at mm-hdp-rm-95334.phx-os1.stratus.dev.ebay.com/10.9.209.246:8050
Total number of applications (application-types: [] and states: [RUNNING]):0
                Application-Id      Application-Name        Application-Type          User           Queue                   State             Final-State             Progress                              Tracking-URL
[hdfs@mm-hdp-dn-95338 storm-yarn-master]$


[hdfs@mm-hdp-dn-95338 storm-yarn-master]$ storm-yarn getStormConfig -appId application_1384576174968_0018 -output ~/.storm/storm.yaml
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hdfs/master/storm/storm-0.9.0-wip21/lib/logback-classic-1.0.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
13/11/27 18:17:27 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
13/11/27 18:17:27 INFO client.RMProxy: Connecting to ResourceManager at mm-hdp-rm-95334.phx-os1.stratus.dev.ebay.com/10.9.209.246:8050
13/11/27 18:17:27 INFO yarn.StormOnYarn: application report for application_1384576174968_0018 :mm-hdp-dn-95338:9000
13/11/27 18:17:27 INFO yarn.StormOnYarn: Attaching to mm-hdp-dn-95338:9000 to talk to app master application_1384576174968_0018
Exception in thread "main" java.lang.RuntimeException: org.apache.thrift7.transport.TTransportException: java.net.ConnectException: Connection refused
        at com.yahoo.storm.yarn.MasterClient.getConfiguredClient(MasterClient.java:42)
        at com.yahoo.storm.yarn.StormOnYarn.getClient(StormOnYarn.java:133)
        at com.yahoo.storm.yarn.StormMasterCommand.process(StormMasterCommand.java:76)
        at com.yahoo.storm.yarn.Client.execute(Client.java:139)
        at com.yahoo.storm.yarn.Client.main(Client.java:145)
Caused by: org.apache.thrift7.transport.TTransportException: java.net.ConnectException: Connection refused
        at org.apache.thrift7.transport.TSocket.open(TSocket.java:183)
        at org.apache.thrift7.transport.TFramedTransport.open(TFramedTransport.java:81)
        at backtype.storm.security.auth.SimpleTransportPlugin.connect(SimpleTransportPlugin.java:66)
        at backtype.storm.security.auth.ThriftClient.<init>(ThriftClient.java:46)
        at com.yahoo.storm.yarn.MasterClient.<init>(MasterClient.java:48)
        at com.yahoo.storm.yarn.MasterClient.getConfiguredClient(MasterClient.java:36)
        ... 4 more
Caused by: java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:579)
        at org.apache.thrift7.transport.TSocket.open(TSocket.java:178)
        ... 9 more
[hdfs@mm-hdp-dn-95338 storm-yarn-master]$

Bobby Evans

unread,
Nov 27, 2013, 1:26:54 PM11/27/13
to tsanghv, storm...@googlegroups.com
That indicates that the storm app master has crashed.  You probably want to go to the RM UI and look for the application id associated with storm.  From there you should be able to find the logs for the AM.

tsanghv

unread,
Nov 27, 2013, 3:20:41 PM11/27/13
to storm...@googlegroups.com, tsanghv
Attaching YARN-Applications.png which shows all the applications submitted to YARN.

Storm-On-Yarn has always failed. This is the exception I see:

User:hdfs
Name:Storm-on-Yarn
Application Type:YARN
State:FAILED
FinalStatus:FAILED
Started:27-Nov-2013 18:37:13
Elapsed:32sec
Tracking URL:History
Diagnostics:
Application application_1384576174968_0019 failed 2 times due to AM Container for appattempt_1384576174968_0019_000002 exited with exitCode: 1 due to: Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:458)
at org.apache.hadoop.util.Shell.run(Shell.java:373)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:578)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:258)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:74)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
.Failing this attempt.. Failing the application.

Attaching Storm-On-Yarn.png which shows the error.
YARN-Applications.png
Storm-On-Yarn.png

Bobby Evans

unread,
Nov 27, 2013, 3:26:36 PM11/27/13
to tsanghv, storm...@googlegroups.com
In the lower right hand corner there is a logs link, next to the application attempt.  That should have something in it that would be useful.

tsanghv

unread,
Nov 27, 2013, 4:10:07 PM11/27/13
to storm...@googlegroups.com, tsanghv
When I click on that, this is what I see:

Error getting logs for container_1384576174968_0019_01_000001


I started looking at the code for storm-yarn to see where the exception might be coming from and it seems like the exception is coming when trying to reading the Master_Host or Master_Thrift_Port on line 32 or 33 here:

Exception in thread "main" java.lang.RuntimeException: org.apache.thrift7.transport.TTransportException: java.net.ConnectException: Connection refused
        at com.yahoo.storm.yarn.MasterClient.getConfiguredClient(MasterClient.java:42)
        at com.yahoo.storm.yarn.StormOnYarn.getClient(StormOnYarn.java:133)
        at com.yahoo.storm.yarn.StormMasterCommand.process(StormMasterCommand.java:76)
        at com.yahoo.storm.yarn.Client.execute(Client.java:139)
        at com.yahoo.storm.yarn.Client.main(Client.java:145)

I see following entries in one of the storm.yaml files:
master.host: localhost
master.thrift.port: 9000

Just wondering what is this port used for? Is there some service that should be up an running on that port ?

Bobby Evans

unread,
Nov 27, 2013, 5:05:39 PM11/27/13
to tsanghv, storm...@googlegroups.com
The app master has a thrift service just like nimbus does.  It is there so you can interact with it, launch new supervisors, get the current storm cluster config, etc.  It is a bit ugly but those configs are set dynamically after querying the resource manager.  They really should not be configs, but that is how they are right now.  That backtrace is simply saying that the client could not talk to the AM.  Why the AM was not listening is because it had crashed.  Why it crashed? I don't know the logs for the AM do not appear to have been written.  That probably means that it crashed very early on.

Sadly there are three different places in the code where it can exit with an error code of 1.  All of them correspond to some error happened and we cannot go on.  Without the logs I cannot really do more then that.  You might want to look at the logs for the node manager to see if they have anything in them for this, validate that you have log aggregation enabled, which might be causing the error getting the container logs.  I just don't know beyond that.

tsanghv

unread,
Nov 27, 2013, 7:08:28 PM11/27/13
to storm...@googlegroups.com, tsanghv
Where are the resource manager and node manager logs typically located ? If they are in some file - I can try to look at the logs there.

Bobby Evans

unread,
Nov 28, 2013, 7:50:16 AM11/28/13
to tapan....@gmail.com, storm...@googlegroups.com
That depends on your insulation and your settings. And I don't remember the setting off the top of my head. If you are using hdp or cdh you probably could check their documentation. 

--Bobby 
On my phone 

tsanghv

unread,
Dec 2, 2013, 4:56:27 PM12/2/13
to storm...@googlegroups.com, tapan....@gmail.com, Bobby Evans
These are the yarn logs. I see there are exceptions around yarn user not being able to write to hdfs file system - not sure how to fix that as storm-yarn seems to be executed as yarn user.
There is another error with bind exception for starting the Master Thrift server which seems to be happening because the port might already be in use. Is there a configuration I can change to use a different port ?

Container: container_1385621683413_0008_02_000001 on mm-hdp-dn-95338.phx-os1.stratus.dev.ebay.com_45454
=========================================================================================================
LogType: nimbus.log
LogLength: 8772
Log Contents:
2013-12-02 21:40:20 o.a.z.ZooKeeper [INFO] Client environment:zookeeper.version=3.3.3-1073969, built on 02/23/2011 22:27 GMT
2013-12-02 21:40:20 o.a.z.ZooKeeper [INFO] Client environment:host.name=mm-hdp-dn-95338.phx-os1.stratus.dev.ebay.com
2013-12-02 21:40:20 o.a.z.ZooKeeper [INFO] Client environment:java.version=1.7.0_45
2013-12-02 21:40:20 o.a.z.ZooKeeper [INFO] Client environment:java.vendor=Oracle Corporation
2013-12-02 21:40:20 o.a.z.ZooKeeper [INFO] Client environment:java.home=/usr/java/jdk1.7.0_45/jre
2013-12-02 21:40:20 o.a.z.ZooKeeper [INFO] Client environment:java.class.path=conf:./storm/storm-0.9.0-wip21/:./storm/storm-0.9.0-wip21/storm-core-0.9.0-wip21.jar:./storm/storm-0.9.0-wip21/storm-console-logging-0.9.0-wip21.jar:./storm/storm-0.9.0-wip21/storm-netty-0.9.0-wip21.jar:./storm/storm-0.9.0-wip21/lib/math.numeric-tower-0.0.1.jar:./storm/storm-0.9.0-wip21/lib/tools.logging-0.2.3.jar:./storm/storm-0.9.0-wip21/lib/log4j-over-slf4j-1.6.6.jar:./storm/storm-0.9.0-wip21/lib/servlet-api-2.5-20081211.jar:./storm/storm-0.9.0-wip21/lib/tools.cli-0.2.2.jar:./storm/storm-0.9.0-wip21/lib/commons-codec-1.4.jar:./storm/storm-0.9.0-wip21/lib/clojure-1.4.0.jar:./storm/storm-0.9.0-wip21/lib/minlog-1.2.jar:./storm/storm-0.9.0-wip21/lib/reflectasm-1.07-shaded.jar:./storm/storm-0.9.0-wip21/lib/libthrift7-0.7.0-2.jar:./storm/storm-0.9.0-wip21/lib/tools.macro-0.1.0.jar:./storm/storm-0.9.0-wip21/lib/json-simple-1.1.jar:./storm/storm-0.9.0-wip21/lib/ring-servlet-0.3.11.jar:./storm/storm-0.9.0-wip21/lib/slf4j-api-1.6.5.jar:./storm/storm-0.9.0-wip21/lib/commons-lang-2.5.jar:./storm/storm-0.9.0-wip21/lib/ring-jetty-adapter-0.3.11.jar:./storm/storm-0.9.0-wip21/lib/clout-1.0.1.jar:./storm/storm-0.9.0-wip21/lib/jline-0.9.94.jar:./storm/storm-0.9.0-wip21/lib/logback-core-1.0.6.jar:./storm/storm-0.9.0-wip21/lib/objenesis-1.2.jar:./storm/storm-0.9.0-wip21/lib/logback-classic-1.0.6.jar:./storm/storm-0.9.0-wip21/lib/curator-client-1.0.1.jar:./storm/storm-0.9.0-wip21/lib/asm-4.0.jar:./storm/storm-0.9.0-wip21/lib/jzmq-2.1.0.jar:./storm/storm-0.9.0-wip21/lib/jgrapht-0.8.3.jar:./storm/storm-0.9.0-wip21/lib/snakeyaml-1.11.jar:./storm/storm-0.9.0-wip21/lib/carbonite-1.5.0.jar:./storm/storm-0.9.0-wip21/lib/mockito-all-1.9.5.jar:./storm/storm-0.9.0-wip21/lib/netty-3.6.3.Final.jar:./storm/storm-0.9.0-wip21/lib/ring-devel-0.3.11.jar:./storm/storm-0.9.0-wip21/lib/commons-io-1.4.jar:./storm/storm-0.9.0-wip21/lib/compojure-1.1.3.jar:./storm/storm-0.9.0-wip21/lib/zookeeper-3.3.3.jar:./storm/storm-0.9.0-wip21/lib/disruptor-2.10.1.jar:./storm/storm-0.9.0-wip21/lib/clj-time-0.4.1.jar:./storm/storm-0.9.0-wip21/lib/jetty-6.1.26.jar:./storm/storm-0.9.0-wip21/lib/jetty-util-6.1.26.jar:./storm/storm-0.9.0-wip21/lib/curator-framework-1.0.1.jar:./storm/storm-0.9.0-wip21/lib/commons-logging-1.1.1.jar:./storm/storm-0.9.0-wip21/lib/guava-13.0.jar:./storm/storm-0.9.0-wip21/lib/commons-exec-1.1.jar:./storm/storm-0.9.0-wip21/lib/commons-fileupload-1.2.1.jar:./storm/storm-0.9.0-wip21/lib/junit-3.8.1.jar:./storm/storm-0.9.0-wip21/lib/ring-core-1.1.5.jar:./storm/storm-0.9.0-wip21/lib/core.incubator-0.1.0.jar:./storm/storm-0.9.0-wip21/lib/kryo-2.17.jar:./storm/storm-0.9.0-wip21/lib/hiccup-0.3.6.jar:./storm/storm-0.9.0-wip21/lib/joda-time-2.0.jar:./storm/storm-0.9.0-wip21/lib/clj-stacktrace-0.2.2.jar:./storm/storm-0.9.0-wip21/lib/httpclient-4.1.1.jar:./storm/storm-0.9.0-wip21/lib/httpcore-4.1.jar:./storm/storm-0.9.0-wip21/lib/servlet-api-2.5.jar
2013-12-02 21:40:20 o.a.z.ZooKeeper [INFO] Client environment:java.library.path=/usr/local/lib:/opt/local/lib:/usr/lib
2013-12-02 21:40:20 o.a.z.ZooKeeper [INFO] Client environment:java.io.tmpdir=/tmp
2013-12-02 21:40:20 o.a.z.ZooKeeper [INFO] Client environment:java.compiler=<NA>
2013-12-02 21:40:20 o.a.z.ZooKeeper [INFO] Client environment:os.name=Linux
2013-12-02 21:40:20 o.a.z.ZooKeeper [INFO] Client environment:os.arch=amd64
2013-12-02 21:40:20 o.a.z.ZooKeeper [INFO] Client environment:os.version=2.6.32-279.22.1.el6.x86_64
2013-12-02 21:40:20 o.a.z.ZooKeeper [INFO] Client environment:user.name=yarn
2013-12-02 21:40:20 o.a.z.ZooKeeper [INFO] Client environment:user.home=/home/yarn
2013-12-02 21:40:20 o.a.z.ZooKeeper [INFO] Client environment:user.dir=/hadoop/yarn/usercache/hdfs/appcache/application_1385621683413_0008/container_1385621683413_0008_02_000001
2013-12-02 21:40:20 o.a.z.s.ZooKeeperServer [INFO] Server environment:zookeeper.version=3.3.3-1073969, built on 02/23/2011 22:27 GMT
2013-12-02 21:40:20 o.a.z.s.ZooKeeperServer [INFO] Server environment:host.name=mm-hdp-dn-95338.phx-os1.stratus.dev.ebay.com
2013-12-02 21:40:20 o.a.z.s.ZooKeeperServer [INFO] Server environment:java.version=1.7.0_45
2013-12-02 21:40:20 o.a.z.s.ZooKeeperServer [INFO] Server environment:java.vendor=Oracle Corporation
2013-12-02 21:40:20 o.a.z.s.ZooKeeperServer [INFO] Server environment:java.home=/usr/java/jdk1.7.0_45/jre
2013-12-02 21:40:20 o.a.z.s.ZooKeeperServer [INFO] Server environment:java.class.path=conf:./storm/storm-0.9.0-wip21/:./storm/storm-0.9.0-wip21/storm-core-0.9.0-wip21.jar:./storm/storm-0.9.0-wip21/storm-console-logging-0.9.0-wip21.jar:./storm/storm-0.9.0-wip21/storm-netty-0.9.0-wip21.jar:./storm/storm-0.9.0-wip21/lib/math.numeric-tower-0.0.1.jar:./storm/storm-0.9.0-wip21/lib/tools.logging-0.2.3.jar:./storm/storm-0.9.0-wip21/lib/log4j-over-slf4j-1.6.6.jar:./storm/storm-0.9.0-wip21/lib/servlet-api-2.5-20081211.jar:./storm/storm-0.9.0-wip21/lib/tools.cli-0.2.2.jar:./storm/storm-0.9.0-wip21/lib/commons-codec-1.4.jar:./storm/storm-0.9.0-wip21/lib/clojure-1.4.0.jar:./storm/storm-0.9.0-wip21/lib/minlog-1.2.jar:./storm/storm-0.9.0-wip21/lib/reflectasm-1.07-shaded.jar:./storm/storm-0.9.0-wip21/lib/libthrift7-0.7.0-2.jar:./storm/storm-0.9.0-wip21/lib/tools.macro-0.1.0.jar:./storm/storm-0.9.0-wip21/lib/json-simple-1.1.jar:./storm/storm-0.9.0-wip21/lib/ring-servlet-0.3.11.jar:./storm/storm-0.9.0-wip21/lib/slf4j-api-1.6.5.jar:./storm/storm-0.9.0-wip21/lib/commons-lang-2.5.jar:./storm/storm-0.9.0-wip21/lib/ring-jetty-adapter-0.3.11.jar:./storm/storm-0.9.0-wip21/lib/clout-1.0.1.jar:./storm/storm-0.9.0-wip21/lib/jline-0.9.94.jar:./storm/storm-0.9.0-wip21/lib/logback-core-1.0.6.jar:./storm/storm-0.9.0-wip21/lib/objenesis-1.2.jar:./storm/storm-0.9.0-wip21/lib/logback-classic-1.0.6.jar:./storm/storm-0.9.0-wip21/lib/curator-client-1.0.1.jar:./storm/storm-0.9.0-wip21/lib/asm-4.0.jar:./storm/storm-0.9.0-wip21/lib/jzmq-2.1.0.jar:./storm/storm-0.9.0-wip21/lib/jgrapht-0.8.3.jar:./storm/storm-0.9.0-wip21/lib/snakeyaml-1.11.jar:./storm/storm-0.9.0-wip21/lib/carbonite-1.5.0.jar:./storm/storm-0.9.0-wip21/lib/mockito-all-1.9.5.jar:./storm/storm-0.9.0-wip21/lib/netty-3.6.3.Final.jar:./storm/storm-0.9.0-wip21/lib/ring-devel-0.3.11.jar:./storm/storm-0.9.0-wip21/lib/commons-io-1.4.jar:./storm/storm-0.9.0-wip21/lib/compojure-1.1.3.jar:./storm/storm-0.9.0-wip21/lib/zookeeper-3.3.3.jar:./storm/storm-0.9.0-wip21/lib/disruptor-2.10.1.jar:./storm/storm-0.9.0-wip21/lib/clj-time-0.4.1.jar:./storm/storm-0.9.0-wip21/lib/jetty-6.1.26.jar:./storm/storm-0.9.0-wip21/lib/jetty-util-6.1.26.jar:./storm/storm-0.9.0-wip21/lib/curator-framework-1.0.1.jar:./storm/storm-0.9.0-wip21/lib/commons-logging-1.1.1.jar:./storm/storm-0.9.0-wip21/lib/guava-13.0.jar:./storm/storm-0.9.0-wip21/lib/commons-exec-1.1.jar:./storm/storm-0.9.0-wip21/lib/commons-fileupload-1.2.1.jar:./storm/storm-0.9.0-wip21/lib/junit-3.8.1.jar:./storm/storm-0.9.0-wip21/lib/ring-core-1.1.5.jar:./storm/storm-0.9.0-wip21/lib/core.incubator-0.1.0.jar:./storm/storm-0.9.0-wip21/lib/kryo-2.17.jar:./storm/storm-0.9.0-wip21/lib/hiccup-0.3.6.jar:./storm/storm-0.9.0-wip21/lib/joda-time-2.0.jar:./storm/storm-0.9.0-wip21/lib/clj-stacktrace-0.2.2.jar:./storm/storm-0.9.0-wip21/lib/httpclient-4.1.1.jar:./storm/storm-0.9.0-wip21/lib/httpcore-4.1.jar:./storm/storm-0.9.0-wip21/lib/servlet-api-2.5.jar
2013-12-02 21:40:20 o.a.z.s.ZooKeeperServer [INFO] Server environment:java.library.path=/usr/local/lib:/opt/local/lib:/usr/lib
2013-12-02 21:40:20 o.a.z.s.ZooKeeperServer [INFO] Server environment:java.io.tmpdir=/tmp
2013-12-02 21:40:20 o.a.z.s.ZooKeeperServer [INFO] Server environment:java.compiler=<NA>
2013-12-02 21:40:20 o.a.z.s.ZooKeeperServer [INFO] Server environment:os.name=Linux
2013-12-02 21:40:20 o.a.z.s.ZooKeeperServer [INFO] Server environment:os.arch=amd64
2013-12-02 21:40:20 o.a.z.s.ZooKeeperServer [INFO] Server environment:os.version=2.6.32-279.22.1.el6.x86_64
2013-12-02 21:40:20 o.a.z.s.ZooKeeperServer [INFO] Server environment:user.name=yarn
2013-12-02 21:40:20 o.a.z.s.ZooKeeperServer [INFO] Server environment:user.home=/home/yarn
2013-12-02 21:40:20 o.a.z.s.ZooKeeperServer [INFO] Server environment:user.dir=/hadoop/yarn/usercache/hdfs/appcache/application_1385621683413_0008/container_1385621683413_0008_02_000001

LogType: stderr
LogLength: 8501
Log Contents:
21:40:16,335 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.groovy]
21:40:16,336 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
21:40:16,340 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [file:/hadoop/yarn/usercache/hdfs/appcache/application_1385621683413_0008/filecache/11/conf/logback.xml]
21:40:16,341 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs multiple times on the classpath.
21:40:16,341 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs at [file:/hadoop/yarn/usercache/hdfs/appcache/application_1385621683413_0008/filecache/11/conf/logback.xml]
21:40:16,341 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs at [jar:file:/hadoop/yarn/filecache/11/storm.zip/storm-0.9.0-wip21/storm-console-logging-0.9.0-wip21.jar!/logback.xml]
21:40:16,712 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - debug attribute not set
21:40:16,723 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - Setting ReconfigureOnChangeFilter scanning period to 1 minutes
21:40:16,723 |-INFO in ReconfigureOnChangeFilter{invocationCounter=0} - Will scan for changes in [[/hadoop/yarn/usercache/hdfs/appcache/application_1385621683413_0008/filecache/11/conf/logback.xml]] every 60 seconds. 
21:40:16,723 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - Adding ReconfigureOnChangeFilter as a turbo filter
21:40:16,740 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.rolling.RollingFileAppender]
21:40:16,755 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [A1]
21:40:17,012 |-INFO in ch.qos.logback.core.rolling.FixedWindowRollingPolicy@476d50ae - No compression will be used
21:40:17,042 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
21:40:17,186 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[A1] - Active log file name: /hadoop/yarn/application_1385621683413_0008/container_1385621683413_0008_02_000001/nimbus.log
21:40:17,186 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[A1] - File property is set to [/hadoop/yarn/application_1385621683413_0008/container_1385621683413_0008_02_000001/nimbus.log]
21:40:17,187 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.rolling.RollingFileAppender]
21:40:17,187 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [ACCESS]
21:40:17,189 |-INFO in ch.qos.logback.core.rolling.FixedWindowRollingPolicy@59cbf424 - No compression will be used
21:40:17,196 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
21:40:17,197 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[ACCESS] - Active log file name: access.log
21:40:17,197 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[ACCESS] - File property is set to [access.log]
21:40:17,197 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to INFO
21:40:17,198 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [A1] to Logger[ROOT]
21:40:17,198 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting additivity of logger [backtype.storm.security.auth.authorizer] to false
21:40:17,199 |-INFO in ch.qos.logback.classic.joran.action.LevelAction - backtype.storm.security.auth.authorizer level set to INFO
21:40:17,199 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [ACCESS] to Logger[backtype.storm.security.auth.authorizer]
21:40:17,199 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
21:40:17,213 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@5d09958a - Registering current configuration as safe fallback point

21:40:17,022 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.groovy]
21:40:17,023 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
21:40:17,023 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [file:/hadoop/yarn/usercache/hdfs/appcache/application_1385621683413_0008/filecache/11/conf/logback.xml]
21:40:17,025 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs multiple times on the classpath.
21:40:17,025 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs at [file:/hadoop/yarn/usercache/hdfs/appcache/application_1385621683413_0008/filecache/11/conf/logback.xml]
21:40:17,025 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs at [jar:file:/hadoop/yarn/filecache/11/storm.zip/storm-0.9.0-wip21/storm-console-logging-0.9.0-wip21.jar!/logback.xml]
21:40:17,152 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - debug attribute not set
21:40:17,157 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - Setting ReconfigureOnChangeFilter scanning period to 1 minutes
21:40:17,157 |-INFO in ReconfigureOnChangeFilter{invocationCounter=0} - Will scan for changes in [[/hadoop/yarn/usercache/hdfs/appcache/application_1385621683413_0008/filecache/11/conf/logback.xml]] every 60 seconds. 
21:40:17,157 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - Adding ReconfigureOnChangeFilter as a turbo filter
21:40:17,164 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.rolling.RollingFileAppender]
21:40:17,175 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [A1]
21:40:17,321 |-INFO in ch.qos.logback.core.rolling.FixedWindowRollingPolicy@e7f21d6 - No compression will be used
21:40:17,331 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
21:40:17,387 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[A1] - Active log file name: /hadoop/yarn/application_1385621683413_0008/container_1385621683413_0008_02_000001/ui.log
21:40:17,387 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[A1] - File property is set to [/hadoop/yarn/application_1385621683413_0008/container_1385621683413_0008_02_000001/ui.log]
21:40:17,388 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.rolling.RollingFileAppender]
21:40:17,388 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [ACCESS]
21:40:17,390 |-INFO in ch.qos.logback.core.rolling.FixedWindowRollingPolicy@3c867009 - No compression will be used
21:40:17,392 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
21:40:17,393 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[ACCESS] - Active log file name: access.log
21:40:17,393 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[ACCESS] - File property is set to [access.log]
21:40:17,393 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to INFO
21:40:17,393 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [A1] to Logger[ROOT]
21:40:17,395 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting additivity of logger [backtype.storm.security.auth.authorizer] to false
21:40:17,395 |-INFO in ch.qos.logback.classic.joran.action.LevelAction - backtype.storm.security.auth.authorizer level set to INFO
21:40:17,395 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [ACCESS] to Logger[backtype.storm.security.auth.authorizer]
21:40:17,395 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
21:40:17,397 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@1e257dea - Registering current configuration as safe fallback point


LogType: stdout
LogLength: 9764
Log Contents:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/hadoop/yarn/filecache/11/storm.zip/storm-0.9.0-wip21/lib/logback-classic-1.0.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
13/12/02 21:40:11 INFO yarn.MasterServer: Starting the AM!!!!
13/12/02 21:40:12 INFO yarn.MasterServer: appAttemptID from env:appattempt_1385621683413_0008_000002
13/12/02 21:40:12 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-nodemanagers-proxies : 500
13/12/02 21:40:13 INFO client.RMProxy: Connecting to ResourceManager at mm-hdp-rm-95334.phx-os1.stratus.dev.ebay.com/10.9.209.246:8030
13/12/02 21:40:13 INFO yarn.StormMasterServerHandler: Storm master host:10.9.210.218
13/12/02 21:40:13 INFO yarn.MasterServer: launch nimbus
13/12/02 21:40:13 INFO yarn.StormMasterServerHandler: starting nimbus...
13/12/02 21:40:13 INFO yarn.MasterServer: launch ui
13/12/02 21:40:13 INFO yarn.StormMasterServerHandler: starting UI...
13/12/02 21:40:13 INFO yarn.MasterServer: launch 1 supervisors
13/12/02 21:40:13 INFO yarn.StormMasterServerHandler: adding 1 supervisors...
13/12/02 21:40:13 INFO yarn.StormAMRMClient: Added 1 supervisors, but not requesting containers now.
13/12/02 21:40:13 INFO yarn.StormMasterServerHandler: Running: /usr/java/jdk1.7.0_45/bin/java -server -Dstorm.home=./storm/storm-0.9.0-wip21/ -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file=storm.yaml -cp conf:./storm/storm-0.9.0-wip21/:./storm/storm-0.9.0-wip21/*:./storm/storm-0.9.0-wip21/lib/* -Xmx1024m -Dlogfile.name=/hadoop/yarn/application_1385621683413_0008/container_1385621683413_0008_02_000001/nimbus.log backtype.storm.daemon.nimbus
13/12/02 21:40:13 INFO yarn.StormMasterServerHandler: Running: /usr/java/jdk1.7.0_45/bin/java -server -Dstorm.home=./storm/storm-0.9.0-wip21/ -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file=storm.yaml -cp conf:./storm/storm-0.9.0-wip21/:./storm/storm-0.9.0-wip21/*:./storm/storm-0.9.0-wip21/lib/* -Xmx768m -Dstorm.options=nimbus.host=localhost -Dlogfile.name=/hadoop/yarn/application_1385621683413_0008/container_1385621683413_0008_02_000001/ui.log backtype.storm.ui.core
13/12/02 21:40:14 INFO yarn.MasterServer: Got a registration response maximumCapability { memory: 6144 virtual_cores: 32 }
13/12/02 21:40:14 INFO yarn.MasterServer: Max Capability <memory:6144, vCores:32>
13/12/02 21:40:14 INFO yarn.StormAMRMClient: Max Capability is now <memory:6144, vCores:32>
13/12/02 21:40:14 INFO yarn.MasterServer: Starting HB thread
13/12/02 21:40:14 INFO yarn.MasterServer: Starting launcher
13/12/02 21:40:14 INFO yarn.MasterServer: Starting Master Thrift Server
13/12/02 21:40:16 INFO yarn.MasterServer: HB: Received allocated containers (1)
13/12/02 21:40:16 INFO yarn.MasterServer: HB: Supervisors are to run, so queueing (1) containers...
13/12/02 21:40:16 INFO yarn.MasterServer: LAUNCHER: Taking container with id (container_1385621683413_0008_02_000002) from the queue.
13/12/02 21:40:16 INFO yarn.MasterServer: LAUNCHER: Supervisors are to run, so launching container id (container_1385621683413_0008_02_000002)
13/12/02 21:40:18 ERROR yarn.MasterServer: Launcher thread I/O exception : 
org.apache.hadoop.security.AccessControlException: Permission denied: user=yarn, access=WRITE, inode="/user":hdfs:hdfs:drwxr-xr-x
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:234)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:214)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:158)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5152)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5134)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:5108)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:3368)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:3338)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3310)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:694)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:502)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:59598)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2352)
at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2321)
at org.apache.hadoop.hdfs.DistributedFileSystem$16.doCall(DistributedFileSystem.java:828)
at org.apache.hadoop.hdfs.DistributedFileSystem$16.doCall(DistributedFileSystem.java:824)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:78)
at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:824)
at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:817)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1929)
at com.yahoo.storm.yarn.Util.createConfigurationFileInFs(Util.java:139)
at com.yahoo.storm.yarn.StormAMRMClient.launchSupervisorOnContainer(StormAMRMClient.java:185)
at com.yahoo.storm.yarn.MasterServer$2.run(MasterServer.java:199)
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=yarn, access=WRITE, inode="/user":hdfs:hdfs:drwxr-xr-x
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:234)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:214)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:158)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5152)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5134)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:5108)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:3368)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:3338)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3310)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:694)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:502)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:59598)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)

at org.apache.hadoop.ipc.Client.call(Client.java:1347)
at org.apache.hadoop.ipc.Client.call(Client.java:1300)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy11.mkdirs(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:188)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy11.mkdirs(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:467)
at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2350)
... 10 more

LogType: ui.log
LogLength: 0
Log Contents:



Container: container_1385621683413_0008_01_000001 on mm-hdp-rm-95334.phx-os1.stratus.dev.ebay.com_45454
=========================================================================================================
LogType: nimbus.log
LogLength: 15793
Log Contents:
2013-12-02 21:41:24 o.a.z.ZooKeeper [INFO] Client environment:zookeeper.version=3.3.3-1073969, built on 02/23/2011 22:27 GMT
2013-12-02 21:41:24 o.a.z.ZooKeeper [INFO] Client environment:host.name=mm-hdp-rm-95334.phx-os1.stratus.dev.ebay.com
2013-12-02 21:41:24 o.a.z.ZooKeeper [INFO] Client environment:java.version=1.7.0_45
2013-12-02 21:41:24 o.a.z.ZooKeeper [INFO] Client environment:java.vendor=Oracle Corporation
2013-12-02 21:41:24 o.a.z.ZooKeeper [INFO] Client environment:java.home=/usr/java/jdk1.7.0_45/jre
2013-12-02 21:41:24 o.a.z.ZooKeeper [INFO] Client environment:java.class.path=conf:./storm/storm-0.9.0-wip21/:./storm/storm-0.9.0-wip21/storm-core-0.9.0-wip21.jar:./storm/storm-0.9.0-wip21/storm-console-logging-0.9.0-wip21.jar:./storm/storm-0.9.0-wip21/storm-netty-0.9.0-wip21.jar:./storm/storm-0.9.0-wip21/lib/math.numeric-tower-0.0.1.jar:./storm/storm-0.9.0-wip21/lib/tools.logging-0.2.3.jar:./storm/storm-0.9.0-wip21/lib/log4j-over-slf4j-1.6.6.jar:./storm/storm-0.9.0-wip21/lib/servlet-api-2.5-20081211.jar:./storm/storm-0.9.0-wip21/lib/tools.cli-0.2.2.jar:./storm/storm-0.9.0-wip21/lib/commons-codec-1.4.jar:./storm/storm-0.9.0-wip21/lib/clojure-1.4.0.jar:./storm/storm-0.9.0-wip21/lib/minlog-1.2.jar:./storm/storm-0.9.0-wip21/lib/reflectasm-1.07-shaded.jar:./storm/storm-0.9.0-wip21/lib/libthrift7-0.7.0-2.jar:./storm/storm-0.9.0-wip21/lib/tools.macro-0.1.0.jar:./storm/storm-0.9.0-wip21/lib/json-simple-1.1.jar:./storm/storm-0.9.0-wip21/lib/ring-servlet-0.3.11.jar:./storm/storm-0.9.0-wip21/lib/slf4j-api-1.6.5.jar:./storm/storm-0.9.0-wip21/lib/commons-lang-2.5.jar:./storm/storm-0.9.0-wip21/lib/ring-jetty-adapter-0.3.11.jar:./storm/storm-0.9.0-wip21/lib/clout-1.0.1.jar:./storm/storm-0.9.0-wip21/lib/jline-0.9.94.jar:./storm/storm-0.9.0-wip21/lib/logback-core-1.0.6.jar:./storm/storm-0.9.0-wip21/lib/objenesis-1.2.jar:./storm/storm-0.9.0-wip21/lib/logback-classic-1.0.6.jar:./storm/storm-0.9.0-wip21/lib/curator-client-1.0.1.jar:./storm/storm-0.9.0-wip21/lib/asm-4.0.jar:./storm/storm-0.9.0-wip21/lib/jzmq-2.1.0.jar:./storm/storm-0.9.0-wip21/lib/jgrapht-0.8.3.jar:./storm/storm-0.9.0-wip21/lib/snakeyaml-1.11.jar:./storm/storm-0.9.0-wip21/lib/carbonite-1.5.0.jar:./storm/storm-0.9.0-wip21/lib/mockito-all-1.9.5.jar:./storm/storm-0.9.0-wip21/lib/netty-3.6.3.Final.jar:./storm/storm-0.9.0-wip21/lib/ring-devel-0.3.11.jar:./storm/storm-0.9.0-wip21/lib/commons-io-1.4.jar:./storm/storm-0.9.0-wip21/lib/compojure-1.1.3.jar:./storm/storm-0.9.0-wip21/lib/zookeeper-3.3.3.jar:./storm/storm-0.9.0-wip21/lib/disruptor-2.10.1.jar:./storm/storm-0.9.0-wip21/lib/clj-time-0.4.1.jar:./storm/storm-0.9.0-wip21/lib/jetty-6.1.26.jar:./storm/storm-0.9.0-wip21/lib/jetty-util-6.1.26.jar:./storm/storm-0.9.0-wip21/lib/curator-framework-1.0.1.jar:./storm/storm-0.9.0-wip21/lib/commons-logging-1.1.1.jar:./storm/storm-0.9.0-wip21/lib/guava-13.0.jar:./storm/storm-0.9.0-wip21/lib/commons-exec-1.1.jar:./storm/storm-0.9.0-wip21/lib/commons-fileupload-1.2.1.jar:./storm/storm-0.9.0-wip21/lib/junit-3.8.1.jar:./storm/storm-0.9.0-wip21/lib/ring-core-1.1.5.jar:./storm/storm-0.9.0-wip21/lib/core.incubator-0.1.0.jar:./storm/storm-0.9.0-wip21/lib/kryo-2.17.jar:./storm/storm-0.9.0-wip21/lib/hiccup-0.3.6.jar:./storm/storm-0.9.0-wip21/lib/joda-time-2.0.jar:./storm/storm-0.9.0-wip21/lib/clj-stacktrace-0.2.2.jar:./storm/storm-0.9.0-wip21/lib/httpclient-4.1.1.jar:./storm/storm-0.9.0-wip21/lib/httpcore-4.1.jar:./storm/storm-0.9.0-wip21/lib/servlet-api-2.5.jar
2013-12-02 21:41:24 o.a.z.ZooKeeper [INFO] Client environment:java.library.path=/usr/local/lib:/opt/local/lib:/usr/lib
2013-12-02 21:41:24 o.a.z.ZooKeeper [INFO] Client environment:java.io.tmpdir=/tmp
2013-12-02 21:41:24 o.a.z.ZooKeeper [INFO] Client environment:java.compiler=<NA>
2013-12-02 21:41:24 o.a.z.ZooKeeper [INFO] Client environment:os.name=Linux
2013-12-02 21:41:24 o.a.z.ZooKeeper [INFO] Client environment:os.arch=amd64
2013-12-02 21:41:24 o.a.z.ZooKeeper [INFO] Client environment:os.version=2.6.32-279.22.1.el6.x86_64
2013-12-02 21:41:24 o.a.z.ZooKeeper [INFO] Client environment:user.name=yarn
2013-12-02 21:41:24 o.a.z.ZooKeeper [INFO] Client environment:user.home=/home/yarn
2013-12-02 21:41:24 o.a.z.ZooKeeper [INFO] Client environment:user.dir=/hadoop/yarn/usercache/hdfs/appcache/application_1385621683413_0008/container_1385621683413_0008_01_000001
2013-12-02 21:41:24 o.a.z.s.ZooKeeperServer [INFO] Server environment:zookeeper.version=3.3.3-1073969, built on 02/23/2011 22:27 GMT
2013-12-02 21:41:24 o.a.z.s.ZooKeeperServer [INFO] Server environment:host.name=mm-hdp-rm-95334.phx-os1.stratus.dev.ebay.com
2013-12-02 21:41:24 o.a.z.s.ZooKeeperServer [INFO] Server environment:java.version=1.7.0_45
2013-12-02 21:41:24 o.a.z.s.ZooKeeperServer [INFO] Server environment:java.vendor=Oracle Corporation
2013-12-02 21:41:24 o.a.z.s.ZooKeeperServer [INFO] Server environment:java.home=/usr/java/jdk1.7.0_45/jre
2013-12-02 21:41:24 o.a.z.s.ZooKeeperServer [INFO] Server environment:java.class.path=conf:./storm/storm-0.9.0-wip21/:./storm/storm-0.9.0-wip21/storm-core-0.9.0-wip21.jar:./storm/storm-0.9.0-wip21/storm-console-logging-0.9.0-wip21.jar:./storm/storm-0.9.0-wip21/storm-netty-0.9.0-wip21.jar:./storm/storm-0.9.0-wip21/lib/math.numeric-tower-0.0.1.jar:./storm/storm-0.9.0-wip21/lib/tools.logging-0.2.3.jar:./storm/storm-0.9.0-wip21/lib/log4j-over-slf4j-1.6.6.jar:./storm/storm-0.9.0-wip21/lib/servlet-api-2.5-20081211.jar:./storm/storm-0.9.0-wip21/lib/tools.cli-0.2.2.jar:./storm/storm-0.9.0-wip21/lib/commons-codec-1.4.jar:./storm/storm-0.9.0-wip21/lib/clojure-1.4.0.jar:./storm/storm-0.9.0-wip21/lib/minlog-1.2.jar:./storm/storm-0.9.0-wip21/lib/reflectasm-1.07-shaded.jar:./storm/storm-0.9.0-wip21/lib/libthrift7-0.7.0-2.jar:./storm/storm-0.9.0-wip21/lib/tools.macro-0.1.0.jar:./storm/storm-0.9.0-wip21/lib/json-simple-1.1.jar:./storm/storm-0.9.0-wip21/lib/ring-servlet-0.3.11.jar:./storm/storm-0.9.0-wip21/lib/slf4j-api-1.6.5.jar:./storm/storm-0.9.0-wip21/lib/commons-lang-2.5.jar:./storm/storm-0.9.0-wip21/lib/ring-jetty-adapter-0.3.11.jar:./storm/storm-0.9.0-wip21/lib/clout-1.0.1.jar:./storm/storm-0.9.0-wip21/lib/jline-0.9.94.jar:./storm/storm-0.9.0-wip21/lib/logback-core-1.0.6.jar:./storm/storm-0.9.0-wip21/lib/objenesis-1.2.jar:./storm/storm-0.9.0-wip21/lib/logback-classic-1.0.6.jar:./storm/storm-0.9.0-wip21/lib/curator-client-1.0.1.jar:./storm/storm-0.9.0-wip21/lib/asm-4.0.jar:./storm/storm-0.9.0-wip21/lib/jzmq-2.1.0.jar:./storm/storm-0.9.0-wip21/lib/jgrapht-0.8.3.jar:./storm/storm-0.9.0-wip21/lib/snakeyaml-1.11.jar:./storm/storm-0.9.0-wip21/lib/carbonite-1.5.0.jar:./storm/storm-0.9.0-wip21/lib/mockito-all-1.9.5.jar:./storm/storm-0.9.0-wip21/lib/netty-3.6.3.Final.jar:./storm/storm-0.9.0-wip21/lib/ring-devel-0.3.11.jar:./storm/storm-0.9.0-wip21/lib/commons-io-1.4.jar:./storm/storm-0.9.0-wip21/lib/compojure-1.1.3.jar:./storm/storm-0.9.0-wip21/lib/zookeeper-3.3.3.jar:./storm/storm-0.9.0-wip21/lib/disruptor-2.10.1.jar:./storm/storm-0.9.0-wip21/lib/clj-time-0.4.1.jar:./storm/storm-0.9.0-wip21/lib/jetty-6.1.26.jar:./storm/storm-0.9.0-wip21/lib/jetty-util-6.1.26.jar:./storm/storm-0.9.0-wip21/lib/curator-framework-1.0.1.jar:./storm/storm-0.9.0-wip21/lib/commons-logging-1.1.1.jar:./storm/storm-0.9.0-wip21/lib/guava-13.0.jar:./storm/storm-0.9.0-wip21/lib/commons-exec-1.1.jar:./storm/storm-0.9.0-wip21/lib/commons-fileupload-1.2.1.jar:./storm/storm-0.9.0-wip21/lib/junit-3.8.1.jar:./storm/storm-0.9.0-wip21/lib/ring-core-1.1.5.jar:./storm/storm-0.9.0-wip21/lib/core.incubator-0.1.0.jar:./storm/storm-0.9.0-wip21/lib/kryo-2.17.jar:./storm/storm-0.9.0-wip21/lib/hiccup-0.3.6.jar:./storm/storm-0.9.0-wip21/lib/joda-time-2.0.jar:./storm/storm-0.9.0-wip21/lib/clj-stacktrace-0.2.2.jar:./storm/storm-0.9.0-wip21/lib/httpclient-4.1.1.jar:./storm/storm-0.9.0-wip21/lib/httpcore-4.1.jar:./storm/storm-0.9.0-wip21/lib/servlet-api-2.5.jar
2013-12-02 21:41:24 o.a.z.s.ZooKeeperServer [INFO] Server environment:java.library.path=/usr/local/lib:/opt/local/lib:/usr/lib
2013-12-02 21:41:24 o.a.z.s.ZooKeeperServer [INFO] Server environment:java.io.tmpdir=/tmp
2013-12-02 21:41:24 o.a.z.s.ZooKeeperServer [INFO] Server environment:java.compiler=<NA>
2013-12-02 21:41:24 o.a.z.s.ZooKeeperServer [INFO] Server environment:os.name=Linux
2013-12-02 21:41:24 o.a.z.s.ZooKeeperServer [INFO] Server environment:os.arch=amd64
2013-12-02 21:41:24 o.a.z.s.ZooKeeperServer [INFO] Server environment:os.version=2.6.32-279.22.1.el6.x86_64
2013-12-02 21:41:24 o.a.z.s.ZooKeeperServer [INFO] Server environment:user.name=yarn
2013-12-02 21:41:24 o.a.z.s.ZooKeeperServer [INFO] Server environment:user.home=/home/yarn
2013-12-02 21:41:24 o.a.z.s.ZooKeeperServer [INFO] Server environment:user.dir=/hadoop/yarn/usercache/hdfs/appcache/application_1385621683413_0008/container_1385621683413_0008_01_000001
2013-12-02 21:41:25 b.s.d.nimbus [INFO] Starting Nimbus with conf {"dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 5000, "topology.skip.missing.kryo.registrations" false, "ui.childopts" "-Xmx768m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "nimbusui.startup.ms" 10000, "topology.trident.batch.emit.interval.millis" 500, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib", "storm.zip.visibility" "PUBLIC", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "storm-local", "storm.messaging.netty.buffer_size" 1048576, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "localhost", "storm.messaging.netty.min_wait_ms" 1000, "storm.zookeeper.port" 2181, "master.container.size-mb" 5120, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.zookeeper.servers" ["localhost"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "master.thrift.port" 9000, "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "yarn.report.wait.millis" 10000, "master.initial-num-supervisors" 1, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 100, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "master.heartbeat.interval.millis" 1000, "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" [6700 6701 6702 6703], "master.host" "localhost", "topology.debug" false, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 1, "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "storm.zip.path" "/lib/storm/0.9.0-wip21/storm.zip", "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "storm.messaging.netty.max_wait_ms" 5000, "drpc.request.timeout.secs" 600, "master.timeout.secs" 1000, "storm.local.mode.zmq" false, "ui.port" 7070, "nimbus.childopts" "-Xmx1024m", "storm.cluster.mode" "distributed", "topology.optimize" true, "topology.max.task.parallelism" nil, "master.container.priority" 0}
2013-12-02 21:41:25 b.s.d.nimbus [INFO] Using default scheduler
2013-12-02 21:41:25 c.n.c.f.i.CuratorFrameworkImpl [INFO] Starting
2013-12-02 21:41:25 o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=localhost:2181 sessionTimeout=20000 watcher=com.netflix.curator.ConnectionState@389ea6dd
2013-12-02 21:41:25 o.a.z.ClientCnxn [INFO] Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181
2013-12-02 21:41:25 o.a.z.ClientCnxn [INFO] Socket connection established to localhost/0:0:0:0:0:0:0:1:2181, initiating session
2013-12-02 21:41:25 o.a.z.ClientCnxn [INFO] Session establishment complete on server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x3426e17ea96002a, negotiated timeout = 20000
2013-12-02 21:41:25 b.s.zookeeper [INFO] Zookeeper state update: :connected:none
2013-12-02 21:41:25 o.a.z.ZooKeeper [INFO] Session: 0x3426e17ea96002a closed
2013-12-02 21:41:25 o.a.z.ClientCnxn [INFO] EventThread shut down
2013-12-02 21:41:25 c.n.c.f.i.CuratorFrameworkImpl [INFO] Starting
2013-12-02 21:41:25 o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=localhost:2181/storm sessionTimeout=20000 watcher=com.netflix.curator.ConnectionState@213b0eef
2013-12-02 21:41:25 o.a.z.ClientCnxn [INFO] Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181
2013-12-02 21:41:25 o.a.z.ClientCnxn [INFO] Socket connection established to localhost/0:0:0:0:0:0:0:1:2181, initiating session
2013-12-02 21:41:25 o.a.z.ClientCnxn [INFO] Session establishment complete on server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x3426e17ea96002b, negotiated timeout = 20000
2013-12-02 21:41:25 b.s.d.nimbus [INFO] Starting Nimbus server...
2013-12-02 21:41:46 b.s.d.nimbus [ERROR] Error when processing event
java.io.IOException: Unable to create directory storm-local/nimbus
at org.apache.commons.io.FileUtils.forceMkdir(FileUtils.java:1488) ~[commons-io-1.4.jar:1.4]
at backtype.storm.config$master_local_dir.invoke(config.clj:112) ~[storm-core-0.9.0-wip21.jar:na]
at backtype.storm.config$master_stormdist_root.invoke(config.clj:118) ~[storm-core-0.9.0-wip21.jar:na]
at backtype.storm.daemon.nimbus$code_ids.invoke(nimbus.clj:737) ~[storm-core-0.9.0-wip21.jar:na]
at backtype.storm.daemon.nimbus$cleanup_storm_ids.invoke(nimbus.clj:746) ~[storm-core-0.9.0-wip21.jar:na]
at backtype.storm.daemon.nimbus$do_cleanup$fn__5456.invoke(nimbus.clj:811) ~[storm-core-0.9.0-wip21.jar:na]
at backtype.storm.daemon.nimbus$do_cleanup.invoke(nimbus.clj:810) ~[storm-core-0.9.0-wip21.jar:na]
at backtype.storm.daemon.nimbus$fn__5508$exec_fn__1224__auto____5509$fn__5514.invoke(nimbus.clj:890) ~[storm-core-0.9.0-wip21.jar:na]
at backtype.storm.timer$schedule_recurring$this__3005.invoke(timer.clj:77) ~[storm-core-0.9.0-wip21.jar:na]
at backtype.storm.timer$mk_timer$fn__2988$fn__2989.invoke(timer.clj:33) ~[storm-core-0.9.0-wip21.jar:na]
at backtype.storm.timer$mk_timer$fn__2988.invoke(timer.clj:26) [storm-core-0.9.0-wip21.jar:na]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
2013-12-02 21:41:46 b.s.util [INFO] Halting process: ("Error when processing an event")

LogType: stderr
LogLength: 8501
Log Contents:
21:41:21,526 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.groovy]
21:41:21,526 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
21:41:21,528 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [file:/hadoop/yarn/usercache/hdfs/appcache/application_1385621683413_0008/filecache/11/conf/logback.xml]
21:41:21,532 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs multiple times on the classpath.
21:41:21,532 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs at [file:/hadoop/yarn/usercache/hdfs/appcache/application_1385621683413_0008/filecache/11/conf/logback.xml]
21:41:21,532 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs at [jar:file:/hadoop/yarn/filecache/11/storm.zip/storm-0.9.0-wip21/storm-console-logging-0.9.0-wip21.jar!/logback.xml]
21:41:21,662 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - debug attribute not set
21:41:21,667 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - Setting ReconfigureOnChangeFilter scanning period to 1 minutes
21:41:21,667 |-INFO in ReconfigureOnChangeFilter{invocationCounter=0} - Will scan for changes in [[/hadoop/yarn/usercache/hdfs/appcache/application_1385621683413_0008/filecache/11/conf/logback.xml]] every 60 seconds. 
21:41:21,667 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - Adding ReconfigureOnChangeFilter as a turbo filter
21:41:21,676 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.rolling.RollingFileAppender]
21:41:21,686 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [A1]
21:41:21,853 |-INFO in ch.qos.logback.core.rolling.FixedWindowRollingPolicy@476d50ae - No compression will be used
21:41:21,863 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
21:41:21,919 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[A1] - Active log file name: /hadoop/yarn/application_1385621683413_0008/container_1385621683413_0008_01_000001/nimbus.log
21:41:21,919 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[A1] - File property is set to [/hadoop/yarn/application_1385621683413_0008/container_1385621683413_0008_01_000001/nimbus.log]
21:41:21,922 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.rolling.RollingFileAppender]
21:41:21,922 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [ACCESS]
21:41:21,924 |-INFO in ch.qos.logback.core.rolling.FixedWindowRollingPolicy@59cbf424 - No compression will be used
21:41:21,927 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
21:41:21,928 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[ACCESS] - Active log file name: access.log
21:41:21,928 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[ACCESS] - File property is set to [access.log]
21:41:21,928 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to INFO
21:41:21,928 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [A1] to Logger[ROOT]
21:41:21,929 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting additivity of logger [backtype.storm.security.auth.authorizer] to false
21:41:21,929 |-INFO in ch.qos.logback.classic.joran.action.LevelAction - backtype.storm.security.auth.authorizer level set to INFO
21:41:21,929 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [ACCESS] to Logger[backtype.storm.security.auth.authorizer]
21:41:21,929 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
21:41:21,931 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@5d09958a - Registering current configuration as safe fallback point

21:41:22,245 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.groovy]
21:41:22,255 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
21:41:22,255 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [file:/hadoop/yarn/usercache/hdfs/appcache/application_1385621683413_0008/filecache/11/conf/logback.xml]
21:41:22,256 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs multiple times on the classpath.
21:41:22,256 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs at [file:/hadoop/yarn/usercache/hdfs/appcache/application_1385621683413_0008/filecache/11/conf/logback.xml]
21:41:22,256 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs at [jar:file:/hadoop/yarn/filecache/11/storm.zip/storm-0.9.0-wip21/storm-console-logging-0.9.0-wip21.jar!/logback.xml]
21:41:22,362 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - debug attribute not set
21:41:22,365 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - Setting ReconfigureOnChangeFilter scanning period to 1 minutes
21:41:22,365 |-INFO in ReconfigureOnChangeFilter{invocationCounter=0} - Will scan for changes in [[/hadoop/yarn/usercache/hdfs/appcache/application_1385621683413_0008/filecache/11/conf/logback.xml]] every 60 seconds. 
21:41:22,365 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - Adding ReconfigureOnChangeFilter as a turbo filter
21:41:22,368 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.rolling.RollingFileAppender]
21:41:22,380 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [A1]
21:41:22,516 |-INFO in ch.qos.logback.core.rolling.FixedWindowRollingPolicy@e7f21d6 - No compression will be used
21:41:22,526 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
21:41:22,576 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[A1] - Active log file name: /hadoop/yarn/application_1385621683413_0008/container_1385621683413_0008_01_000001/ui.log
21:41:22,576 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[A1] - File property is set to [/hadoop/yarn/application_1385621683413_0008/container_1385621683413_0008_01_000001/ui.log]
21:41:22,578 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.rolling.RollingFileAppender]
21:41:22,578 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [ACCESS]
21:41:22,580 |-INFO in ch.qos.logback.core.rolling.FixedWindowRollingPolicy@3c867009 - No compression will be used
21:41:22,581 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
21:41:22,582 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[ACCESS] - Active log file name: access.log
21:41:22,582 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[ACCESS] - File property is set to [access.log]
21:41:22,582 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to INFO
21:41:22,582 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [A1] to Logger[ROOT]
21:41:22,583 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting additivity of logger [backtype.storm.security.auth.authorizer] to false
21:41:22,583 |-INFO in ch.qos.logback.classic.joran.action.LevelAction - backtype.storm.security.auth.authorizer level set to INFO
21:41:22,583 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [ACCESS] to Logger[backtype.storm.security.auth.authorizer]
21:41:22,583 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
21:41:22,584 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@1e257dea - Registering current configuration as safe fallback point


LogType: stdout
LogLength: 10965
Log Contents:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/hadoop/yarn/filecache/11/storm.zip/storm-0.9.0-wip21/lib/logback-classic-1.0.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
13/12/02 21:41:16 INFO yarn.MasterServer: Starting the AM!!!!
13/12/02 21:41:17 INFO yarn.MasterServer: appAttemptID from env:appattempt_1385621683413_0008_000001
13/12/02 21:41:17 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-nodemanagers-proxies : 500
13/12/02 21:41:18 INFO client.RMProxy: Connecting to ResourceManager at mm-hdp-rm-95334.phx-os1.stratus.dev.ebay.com/10.9.209.246:8030
13/12/02 21:41:18 INFO yarn.StormMasterServerHandler: Storm master host:10.9.209.246
13/12/02 21:41:18 INFO yarn.MasterServer: launch nimbus
13/12/02 21:41:18 INFO yarn.StormMasterServerHandler: starting nimbus...
13/12/02 21:41:18 INFO yarn.MasterServer: launch ui
13/12/02 21:41:18 INFO yarn.StormMasterServerHandler: starting UI...
13/12/02 21:41:18 INFO yarn.MasterServer: launch 1 supervisors
13/12/02 21:41:18 INFO yarn.StormMasterServerHandler: adding 1 supervisors...
13/12/02 21:41:18 INFO yarn.StormAMRMClient: Added 1 supervisors, but not requesting containers now.
13/12/02 21:41:18 INFO yarn.StormMasterServerHandler: Running: /usr/java/jdk1.7.0_45/bin/java -server -Dstorm.home=./storm/storm-0.9.0-wip21/ -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file=storm.yaml -cp conf:./storm/storm-0.9.0-wip21/:./storm/storm-0.9.0-wip21/*:./storm/storm-0.9.0-wip21/lib/* -Xmx768m -Dstorm.options=nimbus.host=localhost -Dlogfile.name=/hadoop/yarn/application_1385621683413_0008/container_1385621683413_0008_01_000001/ui.log backtype.storm.ui.core
13/12/02 21:41:18 INFO yarn.StormMasterServerHandler: Running: /usr/java/jdk1.7.0_45/bin/java -server -Dstorm.home=./storm/storm-0.9.0-wip21/ -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file=storm.yaml -cp conf:./storm/storm-0.9.0-wip21/:./storm/storm-0.9.0-wip21/*:./storm/storm-0.9.0-wip21/lib/* -Xmx1024m -Dlogfile.name=/hadoop/yarn/application_1385621683413_0008/container_1385621683413_0008_01_000001/nimbus.log backtype.storm.daemon.nimbus
13/12/02 21:41:19 INFO yarn.MasterServer: Got a registration response maximumCapability { memory: 6144 virtual_cores: 32 }
13/12/02 21:41:19 INFO yarn.MasterServer: Max Capability <memory:6144, vCores:32>
13/12/02 21:41:19 INFO yarn.StormAMRMClient: Max Capability is now <memory:6144, vCores:32>
13/12/02 21:41:19 INFO yarn.MasterServer: Starting HB thread
13/12/02 21:41:19 INFO yarn.MasterServer: Starting launcher
13/12/02 21:41:19 INFO yarn.MasterServer: Starting Master Thrift Server
Exception in thread "main" java.net.BindException: Address already in use
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:376)
at java.net.ServerSocket.bind(ServerSocket.java:376)
at java.net.ServerSocket.<init>(ServerSocket.java:237)
at java.net.ServerSocket.<init>(ServerSocket.java:181)
at org.mortbay.jetty.bio.SocketConnector.newServerSocket(SocketConnector.java:80)
at org.mortbay.jetty.bio.SocketConnector.open(SocketConnector.java:73)
at org.mortbay.jetty.AbstractConnector.doStart(AbstractConnector.java:283)
at org.mortbay.jetty.bio.SocketConnector.doStart(SocketConnector.java:147)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.jetty.Server.doStart(Server.java:235)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at ring.adapter.jetty$run_jetty.invoke(jetty.clj:65)
at backtype.storm.ui.core$start_server_BANG_.invoke(core.clj:810)
at backtype.storm.ui.core$_main.invoke(core.clj:813)
at clojure.lang.AFn.applyToHelper(AFn.java:159)
at clojure.lang.AFn.applyTo(AFn.java:151)
at backtype.storm.ui.core.main(Unknown Source)
13/12/02 21:41:34 INFO yarn.MasterServer: HB: Received allocated containers (1)
13/12/02 21:41:34 INFO yarn.MasterServer: HB: Supervisors are to run, so queueing (1) containers...
13/12/02 21:41:34 INFO yarn.MasterServer: LAUNCHER: Taking container with id (container_1385621683413_0008_01_000003) from the queue.
13/12/02 21:41:34 INFO yarn.MasterServer: LAUNCHER: Supervisors are to run, so launching container id (container_1385621683413_0008_01_000003)
13/12/02 21:41:35 ERROR yarn.MasterServer: Launcher thread I/O exception : 
org.apache.hadoop.security.AccessControlException: Permission denied: user=yarn, access=WRITE, inode="/user":hdfs:hdfs:drwxr-xr-x
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:234)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:214)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:158)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5152)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5134)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:5108)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:3368)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:3338)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3310)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:694)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:502)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:59598)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2352)
at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2321)
at org.apache.hadoop.hdfs.DistributedFileSystem$16.doCall(DistributedFileSystem.java:828)
at org.apache.hadoop.hdfs.DistributedFileSystem$16.doCall(DistributedFileSystem.java:824)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:78)
at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:824)
at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:817)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1929)
at com.yahoo.storm.yarn.Util.createConfigurationFileInFs(Util.java:139)
at com.yahoo.storm.yarn.StormAMRMClient.launchSupervisorOnContainer(StormAMRMClient.java:185)
at com.yahoo.storm.yarn.MasterServer$2.run(MasterServer.java:199)
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=yarn, access=WRITE, inode="/user":hdfs:hdfs:drwxr-xr-x
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:234)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:214)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:158)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5152)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5134)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:5108)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:3368)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:3338)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3310)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:694)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:502)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:59598)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)

at org.apache.hadoop.ipc.Client.call(Client.java:1347)
at org.apache.hadoop.ipc.Client.call(Client.java:1300)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy11.mkdirs(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:188)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy11.mkdirs(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:467)
at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2350)
... 10 more

LogType: ui.log
LogLength: 372
Log Contents:
2013-12-02 21:41:25 o.m.log [INFO] Logging to Logger[org.mortbay.log] via org.mortbay.log.Slf4jLog
2013-12-02 21:41:25 o.m.log [INFO] jetty-6.1.26
2013-12-02 21:41:25 o.m.log [WARN] failed SocketC...@0.0.0.0:7070: java.net.BindException: Address already in use
2013-12-02 21:41:25 o.m.log [WARN] failed Server@74a21183: java.net.BindException: Address already in use

Bobby Evans

unread,
Dec 2, 2013, 5:46:09 PM12/2/13
to tsanghv, storm...@googlegroups.com
Do you have a home directory on your cluster?

The permission denied error is happening when the AM is trying to write a config file out to HDFS in yarn's home directory under <home directory>/.storm/<app attempt id>.  The error is saying that the user yarn does not have permission to change the /user directory, probably because it is trying to create the directory /user/yarn and gets an error.

In general you do not want to run as a special system user like yarn.  This opens up a number of security holes and it would really be a lot better to run as someone else.

tsanghv

unread,
Dec 2, 2013, 7:07:29 PM12/2/13
to storm...@googlegroups.com, tsanghv
By default when I deployed the Hadoop Cluster using Apache Ambari - I dont think it gave me any options to add any users to HDFS. The only user which has access to HDFS is hdfs
Now when I am trying to install Storm on YARN - I am logged in as hdfs on the VM and trying to run storm-yarn launch but it seems like storm-yarn is running as yarn. If there is a way to run it as hdfs I can try that but I am not sure how to do that.
...

Bobby Evans

unread,
Dec 3, 2013, 10:51:34 AM12/3/13
to tsanghv, storm...@googlegroups.com
I am really not that familiar with ambari, you might want to try asking on the ambari mailing lists how to add in new users.  Are you running with security enabled?

--Bobby

--

tsanghv

unread,
Dec 3, 2013, 1:14:52 PM12/3/13
to storm...@googlegroups.com, tsanghv
I was able to fix the problem by creating the folder /user/yarn under HDFS and everything seems to work fine. Thanks for your help on that! I am running with security enabled and that is the reason the access permissions were failing the application master to register.

Being new to HDFS (Hadoop) - I think there must be a way to add new users to allow access to the file system otherwise all the applications trying to use HDFS would have to run as the user hdfs which is not ideal. We should be able to add users to HDFS outside of Ambari as well as Ambari is just a means to install/deploy on the cluster. 
...
Reply all
Reply to author
Forward
0 new messages