Build error on mvn package

[Lining Sun]

Hello,

I got a build error as below. My java is 1.6.0_25, maven is 3.0.5.

Any clue and troubleshooting tip is truly appropriated.

Thanks in advance.

Lining

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.0:compile (default-compile) on project storm-yarn: Compilation failure -> [Help 1]

org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.0:compile (default-compile) on project storm-yarn: Compilation failure

        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:213)

        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)

        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)

        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)

        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)

        at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)

        at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)

        at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320)

        at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)

        at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)

        at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)

        at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

        at java.lang.reflect.Method.invoke(Method.java:597)

        at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)

        at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)

        at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)

        at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)

Caused by: org.apache.maven.plugin.compiler.CompilationFailureException: Compilation failure

        at org.apache.maven.plugin.compiler.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:749)

        at org.apache.maven.plugin.compiler.CompilerMojo.execute(CompilerMojo.java:118)

        at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)

        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)

        ... 19 more

[Andy Feng]

Lining:

Storm-yarn currently requires JDK 7. Can you use JDK 7 and let us know the result?

Andy

--
You received this message because you are subscribed to the Google Groups "storm-yarn" group.
To unsubscribe from this group and stop receiving emails from it, send an email to storm-yarn+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[lin...@gmail.com]

With JDK 7, build was succeeded. test failed as below,

 

Thanks

 

Lining

 

-------------------------------------------------------------------------------

Test set: com.yahoo.storm.yarn.TestIntegration

-------------------------------------------------------------------------------

Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 24.398 sec <<< FAILURE!

com.yahoo.storm.yarn.TestIntegration  Time elapsed: 0 sec  <<< FAILURE!

java.lang.AssertionError: expected:<null> but was:<java.io.FileNotFoundException: target/appId.txt (No such file or directory)>

        at org.junit.Assert.fail(Assert.java:93)

        at org.junit.Assert.failNotEquals(Assert.java:647)

        at org.junit.Assert.assertEquals(Assert.java:128)

        at org.junit.Assert.assertEquals(Assert.java:147)

        at com.yahoo.storm.yarn.TestIntegration.setup(TestIntegration.java:106)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)

        at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)

        at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)

        at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)

        at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)

        at org.junit.runners.ParentRunner.run(ParentRunner.java:300)

        at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)

        at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123)

        at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:104)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:164)

        at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:110)

        at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:175)

        at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcessWhenForked(SurefireStarter.java:107)

        at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:68)

~

[Andy Feng]

Please find the following files in your /target directory.

• surefire-reports/com.yahoo.storm.yarn.TestIntegration.txt
• com.yahoo.storm.yarn.TestIntegration/com.yahoo.storm.yarn.TestIntegration-logDir-nm-0_0/application_NN/container_MM/stderr 
• com.yahoo.storm.yarn.TestIntegration/com.yahoo.storm.yarn.TestIntegration-logDir-nm-0_0/application_NN/container_MM/stdout

We need to examine these files to see what went wrong.

Andy

[lin...@gmail.com]

Hi Andy,

 

I attached the 1st file you asked, but, there are no 2nd and 3rd files. I also attached ‘mvn test’ log that has an error,

 

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.yarn.client.YarnClientImpl.<init>(Ljava/net/InetSocketAddress;)V

 

It might be the problem with yarn setup. I followed Cloudera instruction at

 

http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/4.2.0/CDH4-Quick-Start/cdh4qs_topic_3_3.html

 

 

Thanks

 

Lining

[Andy Feng]

Lining:

Likely, this is caused by incompatible YARN installation. 

11:46:17.340 [main] INFO com.yahoo.storm.yarn.TestIntegration - bin/storm-yarn launch ./conf/storm.yaml --stormZip lib/storm.zip --appname storm-on-yarn-test --output target/appId.txt 

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.yarn.client.YarnClientImpl.<init>(Ljava/net/InetSocketAddress;)V 

 at com.yahoo.storm.yarn.StormOnYarn.<init>(StormOnYarn.java:74) 

 at com.yahoo.storm.yarn.StormOnYarn.<init>(StormOnYarn.java:68) 

 at com.yahoo.storm.yarn.StormOnYarn.launchApplication(StormOnYarn.java:340) 

[main] ERROR com.yahoo.storm.yarn.TestIntegration - setup failure java.io.FileNotFoundException: target/appId.txt (No such file or directory) 

 at java.io.FileInputStream.open(Native Method) ~[na:1.7.0_25] 

 at java.io.FileInputStream.<init>(FileInputStream.java:138) ~[na:1.7.0_25] 

 [main] INFO com.yahoo.storm.yarn.TestIntegration - java.lang.NullPointerException

Can you run the following command? I will then compare it with  hadoop 2.0.4-alpha, which we use for storm-yarn dev.

• yarn classpath

Andy

--

[lin...@gmail.com]

Hi Andy,

 

 

yarn classpath output is below,

/etc/hadoop/conf:/etc/hadoop/conf:/etc/hadoop/conf:/usr/lib/hadoop/lib/*:/usr/lib/hadoop/.//*:/usr/lib/hadoop-hdfs/./:/usr/lib/hadoop-hdfs/lib/*:/usr/lib/hadoop-hdfs/.//*:/usr/lib/hadoop-yarn/lib/*:/usr/lib/hadoop-yarn/.//*:/usr/lib/hadoop-mapreduce/lib/*:/usr/lib/hadoop-mapreduce/.//*:/usr/lib/hadoop-yarn/.//*:/usr/lib/hadoop-yarn/lib/*

 

Hadoop-Yarn version are below,

 

hadoop-mapreduce-2.0.0+1357-1.cdh4.3.0.p0.21.el5

hadoop-hdfs-secondarynamenode-2.0.0+1357-1.cdh4.3.0.p0.21.el5

hadoop-2.0.0+1357-1.cdh4.3.0.p0.21.el5

hadoop-hdfs-datanode-2.0.0+1357-1.cdh4.3.0.p0.21.el5

hadoop-yarn-2.0.0+1357-1.cdh4.3.0.p0.21.el5

hadoop-yarn-nodemanager-2.0.0+1357-1.cdh4.3.0.p0.21.el5

hadoop-hdfs-2.0.0+1357-1.cdh4.3.0.p0.21.el5

hadoop-mapreduce-historyserver-2.0.0+1357-1.cdh4.3.0.p0.21.el5

hadoop-conf-pseudo-2.0.0+1357-1.cdh4.3.0.p0.21.el5

hadoop-hdfs-namenode-2.0.0+1357-1.cdh4.3.0.p0.21.el5

hadoop-yarn-resourcemanager-2.0.0+1357-1.cdh4.3.0.p0.21.el5

 

Lining

[Andy Feng]

Look like that CDH 2.0 is essentially Apache Hadoop 0.23. We will need to modify storm-yarn code for that. 

A new issue has been created for us to tracking the progress.

https://github.com/yahoo/storm-yarn/issues/23

From: <lin...@gmail.com>
Date: Wednesday, July 10, 2013 3:28 PM
To: Andy Feng <andy...@gmail.com>, <storm...@googlegroups.com>
Subject: Re: Build error on mvn package

--

[Andy Feng]

Lining:

Would you try out the code at https://github.com/anfeng/storm-yarn/tree/hadoop-0.23?

• git clone g...@github.com:anfeng/storm-yarn.git –b hadoop-0.23 storm-yarn-0.23
• mvn test

It works with Apache Hadoop 0.23, but I have not tested it against CDH 2.0.0

Andy

From: <lin...@gmail.com>
Date: Wednesday, July 10, 2013 3:28 PM
To: Andy Feng <andy...@gmail.com>, <storm...@googlegroups.com>
Subject: Re: Build error on mvn package

--

[Bin Wang]

Andy,

Can you update the readme to add prerequire installation for build?

Thanks,

-Bin

[lin...@gmail.com]

Hi Andy,

 

This time, I got

 

12:08:34.878 [Socket Reader #1 for port 46283] WARN  org.apache.hadoop.ipc.Server - Incorrect header or version mismatch from 172.17.103.243:52573 got version 8 expected version 5

Exception in thread "main" java.lang.reflect.UndeclaredThrowableException

 

mvn test log attached.

 

Thanks

[Andy Feng]

We will add them soon. Thanks for being patient.

[tsanghv]

I am using hortonworks hadoop distribution with version 2.1.0.2.0.5.0-67 and I am seeing the following error even when I am using hadoop-0.23 branch for storm-yarn.

Is there a fix that needs to be made in storm-yarn to support Hortonworks hadoop distribution as well ?

This is the error I see:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/client/YarnClientImpl

        at com.yahoo.storm.yarn.StormOnYarn.<init>(StormOnYarn.java:80)

        at com.yahoo.storm.yarn.StormOnYarn.<init>(StormOnYarn.java:75)

        at com.yahoo.storm.yarn.StormOnYarn.launchApplication(StormOnYarn.java:363)

        at com.yahoo.storm.yarn.LaunchCommand.process(LaunchCommand.java:55)

        at com.yahoo.storm.yarn.Client.execute(Client.java:139)

        at com.yahoo.storm.yarn.Client.main(Client.java:145)

Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.yarn.client.YarnClientImpl

        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)

        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

        at java.security.AccessController.doPrivileged(Native Method)

        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)

        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

        ... 6 more

07:22:48.357 [main] ERROR com.yahoo.storm.yarn.TestIntegration - setup failure

java.io.FileNotFoundException: target/appId.txt (No such file or directory)

        at java.io.FileInputStream.open(Native Method) ~[na:1.7.0_45]

        at java.io.FileInputStream.<init>(FileInputStream.java:146) ~[na:1.7.0_45]

        at java.io.FileInputStream.<init>(FileInputStream.java:101) ~[na:1.7.0_45]

        at java.io.FileReader.<init>(FileReader.java:58) ~[na:1.7.0_45]

        at com.yahoo.storm.yarn.TestIntegration.setup(TestIntegration.java:102) ~[test-classes/:na]

[Bobby Evans]

0.23 and HDP-2.1 are not compatible with each other.  Andy just recently put in some changes to master that let it work with HDP-2.1.  Open the pom and look for the commented out sections.

--Bobby

[Tapan Sanghvi]

Thanks for sharing the information about 0.23 and HDP 2.1 being incompatible. 

Now that I master is supposed to work with HDP 2.1 - I think there are suggestions that you made.

#Suggestion 1

Look at the POM file comments to see if I find anything. I did not see any comment around HDP or Hortonworks in POM.xml itself but HDP - Storm On Yarn suggest to do the following which I already did.

Edit the pom.xml repos and Hadoop version to point at Hortonworks.

#Suggestion 2

Build storm-yarn without building the tests. I tried that as well and in this case when I try to get the stormConfig I see some errors. These are the commands I run:

1. mvn package -DskipTests

2. storm-yarn launch

3. yarn application -list (this does not show any running applications)

4. storm-yarn getStormConfig -appId application_1384576174968_0017 -output ~/.storm/storm.yaml

I get the application id from command 2 above so I still try to get the storm.yaml file but it fails with the following exception:

Exception in thread "main" java.lang.RuntimeException: org.apache.thrift7.transport.TTransportException: java.net.ConnectException: Connection refused

        at com.yahoo.storm.yarn.MasterClient.getConfiguredClient(MasterClient.java:42)

        at com.yahoo.storm.yarn.StormOnYarn.getClient(StormOnYarn.java:133)

        at com.yahoo.storm.yarn.StormMasterCommand.process(StormMasterCommand.java:76)

        at com.yahoo.storm.yarn.Client.execute(Client.java:139)

        at com.yahoo.storm.yarn.Client.main(Client.java:145)

Caused by: org.apache.thrift7.transport.TTransportException: java.net.ConnectException: Connection refused

        at org.apache.thrift7.transport.TSocket.open(TSocket.java:183)

        at org.apache.thrift7.transport.TFramedTransport.open(TFramedTransport.java:81)

        at backtype.storm.security.auth.SimpleTransportPlugin.connect(SimpleTransportPlugin.java:66)

        at backtype.storm.security.auth.ThriftClient.<init>(ThriftClient.java:46)

        at com.yahoo.storm.yarn.MasterClient.<init>(MasterClient.java:48)

        at com.yahoo.storm.yarn.MasterClient.getConfiguredClient(MasterClient.java:36)

        ... 4 more

Caused by: java.net.ConnectException: Connection refused

        at java.net.PlainSocketImpl.socketConnect(Native Method)

        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)

        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)

        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)

        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)

        at java.net.Socket.connect(Socket.java:579)

        at org.apache.thrift7.transport.TSocket.open(TSocket.java:178)

        ... 9 more

There was also the mention of changing the UI port but I am not sure where to change that.