mvn package fails with storm-yarn 1.0-alpha

[James Sirota]

I am running HortonWorks CDH 2.0 CP and I am trying to install Yarn-Storm.  I have jdk1.7.0_25 using Maven 2.6.32-358.el6.x86_64 to build it.  The CDH cluster is up and running.  Zookeeper is running as well.  

When I do mvm package I get the following error.  When I look inside the jar after the maven build finishes I don't have the YarnClient class there.  What am I doing wrong?  Thank you in advance for your help.  Much appreciated

[root@master storm-yarn-master]# mvn package

[INFO] Scanning for projects...

[INFO]                                                                         

[INFO] ------------------------------------------------------------------------

[INFO] Building storm-yarn 1.0-alpha

[INFO] ------------------------------------------------------------------------

[INFO] 

[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ storm-yarn ---

[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!

[INFO] Copying 1 resource

[INFO] 

[INFO] --- maven-compiler-plugin:3.0:compile (default-compile) @ storm-yarn ---

[INFO] Nothing to compile - all classes are up to date

[INFO] 

[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ storm-yarn ---

[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!

[INFO] Copying 2 resources

[INFO] 

[INFO] --- maven-compiler-plugin:3.0:testCompile (default-testCompile) @ storm-yarn ---

[INFO] Nothing to compile - all classes are up to date

[INFO] 

[INFO] --- maven-jar-plugin:2.4:jar (default) @ storm-yarn ---

[INFO] 

[INFO] --- maven-surefire-plugin:2.10:test (default-test) @ storm-yarn ---

[INFO] Surefire report directory: /root/storm-yarn-master/target/surefire-reports

-------------------------------------------------------

 T E S T S

-------------------------------------------------------

Running com.yahoo.storm.yarn.TestStormCluster

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/root/.m2/repository/ch/qos/logback/logback-classic/1.0.6/logback-classic-1.0.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/root/.m2/repository/org/slf4j/slf4j-log4j12/1.6.1/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

18:47:21,127 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.groovy]

18:47:21,127 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]

18:47:21,127 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [file:/root/storm-yarn-master/target/test-classes/logback.xml]

18:47:21,231 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]

18:47:21,235 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [STDOUT]

18:47:21,283 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property

18:47:21,359 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to WARN

18:47:21,359 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to Logger[ROOT]

18:47:21,360 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.

18:47:21,362 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@19f9e414 - Registering current configuration as safe fallback point

SLF4J: Actual binding is of type [ch.qos.logback.classic.selector.DefaultContextSelector]

Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.455 sec <<< FAILURE!

Running com.yahoo.storm.yarn.TestIntegration

18:47:22.715 [main] WARN  o.a.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

18:47:24.673 [Thread-5] WARN  o.a.h.metrics2.impl.MetricsConfig - Cannot locate configuration: tried hadoop-metrics2-resourcemanager.properties,hadoop-metrics2.properties

18:47:25.548 [main] WARN  o.a.h.y.s.n.c.m.ContainersMonitorImpl - NodeManager configured with 4 G physical memory allocated to containers, which is more than 80% of the total physical memory available (988.8 M). Thrashing might happen.

Jul 22, 2013 6:47:26 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register

INFO: Registering org.apache.hadoop.yarn.server.resourcemanager.webapp.JAXBContextResolver as a provider class

Jul 22, 2013 6:47:26 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register

INFO: Registering org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices as a root resource class

Jul 22, 2013 6:47:26 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register

INFO: Registering org.apache.hadoop.yarn.webapp.GenericExceptionHandler as a provider class

Jul 22, 2013 6:47:26 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate

INFO: Initiating Jersey application, version 'Jersey: 1.8 06/24/2011 12:17 PM'

Jul 22, 2013 6:47:26 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider

INFO: Binding org.apache.hadoop.yarn.server.resourcemanager.webapp.JAXBContextResolver to GuiceManagedComponentProvider with the scope "Singleton"

Jul 22, 2013 6:47:26 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider

INFO: Binding org.apache.hadoop.yarn.webapp.GenericExceptionHandler to GuiceManagedComponentProvider with the scope "Singleton"

Jul 22, 2013 6:47:27 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider

INFO: Binding org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices to GuiceManagedComponentProvider with the scope "Singleton"

Jul 22, 2013 6:47:27 PM com.google.inject.servlet.GuiceFilter setPipeline

WARNING: Multiple Servlet injectors detected. This is a warning indicating that you have more than one GuiceFilter running in your web application. If this is deliberate, you may safely ignore this message. If this is NOT deliberate however, your application may not work as expected.

18:47:27.490 [main] WARN  o.a.h.y.s.n.c.m.ContainersMonitorImpl - NodeManager configured with 4 G physical memory allocated to containers, which is more than 80% of the total physical memory available (988.8 M). Thrashing might happen.

Jul 22, 2013 6:47:27 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register

INFO: Registering org.apache.hadoop.yarn.server.nodemanager.webapp.NMWebServices as a root resource class

Jul 22, 2013 6:47:27 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register

INFO: Registering org.apache.hadoop.yarn.webapp.GenericExceptionHandler as a provider class

Jul 22, 2013 6:47:27 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register

INFO: Registering org.apache.hadoop.yarn.server.nodemanager.webapp.JAXBContextResolver as a provider class

Jul 22, 2013 6:47:27 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate

INFO: Initiating Jersey application, version 'Jersey: 1.8 06/24/2011 12:17 PM'

Jul 22, 2013 6:47:27 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider

INFO: Binding org.apache.hadoop.yarn.server.nodemanager.webapp.JAXBContextResolver to GuiceManagedComponentProvider with the scope "Singleton"

Jul 22, 2013 6:47:27 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider

INFO: Binding org.apache.hadoop.yarn.webapp.GenericExceptionHandler to GuiceManagedComponentProvider with the scope "Singleton"

Jul 22, 2013 6:47:27 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider

INFO: Binding org.apache.hadoop.yarn.server.nodemanager.webapp.NMWebServices to GuiceManagedComponentProvider with the scope "Singleton"

Jul 22, 2013 6:47:27 PM com.google.inject.servlet.GuiceFilter setPipeline

WARNING: Multiple Servlet injectors detected. This is a warning indicating that you have more than one GuiceFilter running in your web application. If this is deliberate, you may safely ignore this message. If this is NOT deliberate however, your application may not work as expected.

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/root/storm-yarn-master/lib/storm-0.9.0-wip19/lib/logback-classic-1.0.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/client/YarnClient

at com.yahoo.storm.yarn.LaunchCommand.process(LaunchCommand.java:58)

at com.yahoo.storm.yarn.Client.execute(Client.java:139)

at com.yahoo.storm.yarn.Client.main(Client.java:145)

Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.yarn.client.YarnClient

at java.net.URLClassLoader$1.run(URLClassLoader.java:366)

at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

... 3 more

18:47:47.077 [main] ERROR com.yahoo.storm.yarn.TestIntegration - setup failure

java.io.FileNotFoundException: target/appId.txt (No such file or directory)

at java.io.FileInputStream.open(Native Method) ~[na:1.7.0_25]

at java.io.FileInputStream.<init>(FileInputStream.java:138) ~[na:1.7.0_25]

at java.io.FileInputStream.<init>(FileInputStream.java:97) ~[na:1.7.0_25]

at java.io.FileReader.<init>(FileReader.java:58) ~[na:1.7.0_25]

at com.yahoo.storm.yarn.TestIntegration.setup(TestIntegration.java:102) ~[test-classes/:na]

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_25]

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_25]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_25]

at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_25]

at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45) ~[junit-4.10.jar:na]

at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) ~[junit-4.10.jar:na]

at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42) ~[junit-4.10.jar:na]

at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27) ~[junit-4.10.jar:na]

at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30) ~[junit-4.10.jar:na]

at org.junit.runners.ParentRunner.run(ParentRunner.java:300) ~[junit-4.10.jar:na]

at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53) ~[na:na]

at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123) ~[na:na]

at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:104) ~[na:na]

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_25]

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_25]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_25]

at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_25]

at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:164) ~[surefire-api-2.10.jar:2.10]

at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:110) ~[surefire-booter-2.10.jar:2.10]

at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:175) ~[surefire-booter-2.10.jar:2.10]

at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcessWhenForked(SurefireStarter.java:107) ~[surefire-booter-2.10.jar:2.10]

at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:68) ~[surefire-booter-2.10.jar:2.10]

18:47:47.092 [Container Monitor] WARN  o.a.h.y.s.n.c.m.ContainersMonitorImpl - org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl is interrupted. Exiting.

18:47:47.222 [Container Monitor] WARN  o.a.h.y.s.n.c.m.ContainersMonitorImpl - org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl is interrupted. Exiting.

18:47:47.229 [Thread[Thread-11,5,main]] ERROR o.a.h.s.t.d.AbstractDelegationTokenSecretManager - InterruptedExcpetion recieved for ExpiredTokenRemover thread java.lang.InterruptedException: sleep interrupted

18:47:47.232 [ApplicationMaster Launcher] WARN  o.a.h.y.s.r.a.ApplicationMasterLauncher - org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread interrupted. Returning.

18:47:47.252 [ResourceManager Event Processor] ERROR o.a.h.y.s.r.ResourceManager - Returning, interrupted : java.lang.InterruptedException

Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 25.941 sec <<< FAILURE!

Running com.yahoo.storm.yarn.TestStormMaster

18:47:48.776 [Thread-266] ERROR b.storm.security.auth.ThriftServer - ThriftServer is being stopped due to: org.apache.thrift7.transport.TTransportException: Could not create ServerSocket on address 0.0.0.0/0.0.0.0:9000.

org.apache.thrift7.transport.TTransportException: Could not create ServerSocket on address 0.0.0.0/0.0.0.0:9000.

at org.apache.thrift7.transport.TNonblockingServerSocket.<init>(TNonblockingServerSocket.java:89) ~[libthrift7-0.7.0-2.jar:0.7.0-2]

at org.apache.thrift7.transport.TNonblockingServerSocket.<init>(TNonblockingServerSocket.java:68) ~[libthrift7-0.7.0-2.jar:0.7.0-2]

at org.apache.thrift7.transport.TNonblockingServerSocket.<init>(TNonblockingServerSocket.java:61) ~[libthrift7-0.7.0-2.jar:0.7.0-2]

at backtype.storm.security.auth.SimpleTransportPlugin.getServer(SimpleTransportPlugin.java:47) ~[storm-core-0.9.0-wip21.jar:na]

at backtype.storm.security.auth.ThriftServer.serve(ThriftServer.java:52) ~[storm-core-0.9.0-wip21.jar:na]

at com.yahoo.storm.yarn.TestStormMaster$1.run(TestStormMaster.java:72) ~[test-classes/:na]

at java.lang.Thread.run(Thread.java:724) ~[na:1.7.0_25]

Results :

Failed tests:   com.yahoo.storm.yarn.TestIntegration: expected:<null> but was:<java.io.FileNotFoundException: target/appId.txt (No such file or directory)>

Tests in error: 

  com.yahoo.storm.yarn.TestStormCluster: UI server already running.

Tests run: 2, Failures: 1, Errors: 1, Skipped: 0

[INFO] ------------------------------------------------------------------------

[INFO] BUILD FAILURE

[INFO] ------------------------------------------------------------------------

[INFO] Total time: 33.626s

[INFO] Finished at: Mon Jul 22 18:47:49 EDT 2013

[INFO] Final Memory: 10M/25M

[INFO] ------------------------------------------------------------------------

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.10:test (default-test) on project storm-yarn: There are test failures.

[ERROR] 

[ERROR] Please refer to /root/storm-yarn-master/target/surefire-reports for the individual test results.

[ERROR] -> [Help 1]

[ERROR] 

[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.

[ERROR] Re-run Maven using the -X switch to enable full debug logging.

[ERROR] 

[ERROR] For more information about the errors and possible solutions, please read the following articles:

[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

[Andy Feng]

Can your run that following commands and send us the result?

• yarn version
• yarn classpath

Which branch do you use? Master or hadoop-0.23?

From: James Sirota <jmy...@gmail.com>
Date: Monday, July 22, 2013 3:55 PM
To: <storm...@googlegroups.com>
Subject: mvn package fails with storm-yarn 1.0-alpha

--
You received this message because you are subscribed to the Google Groups "storm-yarn" group.
To unsubscribe from this group and stop receiving emails from it, send an email to storm-yarn+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Grant Liu]

Same error here. I think he meant HDP 2.0 CP. I'm using the hadoop-2.1.0 branch of storm-yarn. FWIW here's my output

[root@cos6-04 storm-yarn]# yarn version

Hadoop 2.1.0.2.0.4.0-38

Subversion g...@github.com:hortonworks/hadoop.git -r 1c6feea9d537846789eb3337dc5b1a8911cfd60a

Compiled by jenkins on 2013-07-08T10:29Z

From source with checksum d1403d7842ef98c85d5f3d1332fa4

This command was run using /usr/lib/hadoop/hadoop-common-2.1.0.2.0.4.0-38.jar

[root@cos6-04 storm-yarn]# yarn classpath

/etc/hadoop/conf:/etc/hadoop/conf:/etc/hadoop/conf:/usr/lib/hadoop/lib/*:/usr/lib/hadoop/.//*:/usr/lib/hadoop-hdfs/./:/usr/lib/hadoop-hdfs/lib/*:/usr/lib/hadoop-hdfs/.//*:/usr/lib/hadoop-yarn/lib/*:/usr/lib/hadoop-yarn/.//*:/usr/lib/hadoop-mapreduce/lib/*:/usr/lib/hadoop-mapreduce/.//*:/usr/lib/hadoop-yarn/.//*:/usr/lib/hadoop-yarn/lib/*

[Grant Liu]

Ok, found it off another thread and confirmed: org/apache/hadoop/yarn/client/api/YarnClient

The code base still looks for the old org/apache/hadoop/yarn/client/YarnClient path. Is a port/change underway? I see the 2.1 branch was already merged into trunk a while ago. I'll try it off of your repo, afeng, I'm assuming there's changes there not merged back in.

[Grant Liu]

For the benefit of the group, per offline discussions, use the 2.1 branch in afeng's repo for now. I got it past the error above (still getting errors around storm.zip that I'm tackling). It is not merged back yet. Per another thread in this group, HDP beta is when the api should settle and merging back can happen:

git clone https://github.com/anfeng/storm-yarn storm-yarn-afeng

cd storm-yarn-afeng

git fetch

git checkout -b hadoop-2.1.0-local origin/hadoop-2.1.0