About JDK7 and Hadoop2.1

[Sean Zhong]

Why we choose JDK7, and make storm-yarn incompatible with JDK6? What specific features do we use in JDK7?

And question about hadoop2.1, can the storm-yarn works with hadoop2.04? We plan to contribute some code to bigtop, seems it use hadoop2.04.

Thank you

Sean

[Andy Feng]

Master branch works with 2.0.4.

Sean, we will welcome you to make storm-yarn works with JDK 6. To find out what needs to be changed, you could change pom.xml to use 1.6 compiler.

Thanks,

Andy

Andy

--
You received this message because you are subscribed to the Google Groups "storm-yarn" group.
To unsubscribe from this group and stop receiving emails from it, send an email to storm-yarn+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Sean Zhong(clockfly)]

OK, Thanks for your suggestion.

after compiled with 1.6, I identified several compilation error, including java.nio and ProcessBuilder.

[ERROR] /myData/storm/storm_yarn/storm-yarn/src/main/java/com/yahoo/storm/yarn/Util.java:[25,30] package java.nio.file.attribute does not exist

[ERROR] /myData/storm/storm_yarn/storm-yarn/src/main/java/com/yahoo/storm/yarn/Util.java:[26,20] package java.nio.file does not exist

[ERROR] /myData/storm/storm_yarn/storm-yarn/src/main/java/com/yahoo/storm/yarn/Util.java:[27,20] package java.nio.file does not exist

[ERROR] /myData/storm/storm_yarn/storm-yarn/src/main/java/com/yahoo/storm/yarn/Util.java:[28,20] package java.nio.file does not exist

[ERROR] /myData/storm/storm_yarn/storm-yarn/src/main/java/com/yahoo/storm/yarn/Util.java:[29,20] package java.nio.file does not exist

[ERROR] /myData/storm/storm_yarn/storm-yarn/src/main/java/com/yahoo/storm/yarn/Util.java:[30,20] package java.nio.file does not exist

[ERROR] /myData/storm/storm_yarn/storm-yarn/src/main/java/com/yahoo/storm/yarn/StormOnYarn.java:[23,31] cannot find symbol

[ERROR] symbol  : class Redirect

[ERROR] location: class java.lang.ProcessBuilder

[ERROR] /myData/storm/storm_yarn/storm-yarn/src/main/java/com/yahoo/storm/yarn/StormMasterServerHandler.java:[22,31] cannot find symbol

[ERROR] symbol  : class Redirect

[ERROR] location: class java.lang.ProcessBuilder

[ERROR] /myData/storm/storm_yarn/storm-yarn/src/main/java/com/yahoo/storm/yarn/Util.java:[179,49] cannot find symbol

[ERROR] symbol  : variable FileSystems

[ERROR] location: class com.yahoo.storm.yarn.Util

[ERROR] /myData/storm/storm_yarn/storm-yarn/src/main/java/com/yahoo/storm/yarn/Util.java:[232,19] package java.nio.file does not exist

[ERROR] /myData/storm/storm_yarn/storm-yarn/src/main/java/com/yahoo/storm/yarn/Util.java:[232,36] cannot find symbol

[ERROR] symbol  : variable FileSystems

[ERROR] location: class com.yahoo.storm.yarn.Util

[ERROR] /myData/storm/storm_yarn/storm-yarn/src/main/java/com/yahoo/storm/yarn/Util.java:[233,12] cannot find symbol

[ERROR] symbol  : class PathMatcher

[ERROR] location: class com.yahoo.storm.yarn.Util

[ERROR] /myData/storm/storm_yarn/storm-yarn/src/main/java/com/yahoo/storm/yarn/Util.java:[236,23] package java.nio.file does not exist

[ERROR] /myData/storm/storm_yarn/storm-yarn/src/main/java/com/yahoo/storm/yarn/Util.java:[237,40] cannot find symbol

[ERROR] symbol  : class SimpleFileVisitor

[ERROR] location: class com.yahoo.storm.yarn.Util

[ERROR] /myData/storm/storm_yarn/storm-yarn/src/main/java/com/yahoo/storm/yarn/Util.java:[237,71] package java.nio.file does not exist

[ERROR] /myData/storm/storm_yarn/storm-yarn/src/main/java/com/yahoo/storm/yarn/Util.java:[237,10] cannot find symbol

[ERROR] symbol  : variable Files

[ERROR] location: class com.yahoo.storm.yarn.Util

[ERROR] /myData/storm/storm_yarn/storm-yarn/src/main/java/com/yahoo/storm/yarn/StormOnYarn.java:[192,81] cannot find symbol

[ERROR] symbol  : variable Redirect

[ERROR] location: class com.yahoo.storm.yarn.StormOnYarn

[ERROR] /myData/storm/storm_yarn/storm-yarn/src/main/java/com/yahoo/storm/yarn/StormOnYarn.java:[194,40] cannot find symbol

[ERROR] symbol  : variable Redirect

[ERROR] location: class java.lang.ProcessBuilder

[ERROR] /myData/storm/storm_yarn/storm-yarn/src/main/java/com/yahoo/storm/yarn/StormMasterServerHandler.java:[136,38] cannot find symbol

[ERROR] symbol  : variable Redirect

[ERROR] location: class com.yahoo.storm.yarn.StormMasterServerHandler.StormProcess

[ERROR] /myData/storm/storm_yarn/storm-yarn/src/main/java/com/yahoo/storm/yarn/StormMasterServerHandler.java:[137,39] cannot find symbol

[ERROR] symbol  : variable Redirect

[ERROR] location: class com.yahoo.storm.yarn.StormMasterServerHandler.StormProcess

[ERROR] -> [Help 1]

org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.0:compile (default-compile) on project storm-yarn: Compilation failure

        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:213)

        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)

        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)

        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)

        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)

        at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)

        at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)

        at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320)

        at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)

        at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)

        at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)

        at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

        at java.lang.reflect.Method.invoke(Method.java:597)

        at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)

        at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)

        at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)

        at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)

Caused by: org.apache.maven.plugin.compiler.CompilationFailureException: Compilation failure

        at org.apache.maven.plugin.compiler.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:745)

        at org.apache.maven.plugin.compiler.CompilerMojo.execute(CompilerMojo.java:118)

        at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)

        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)

        ... 19 more

[Robert Evans]

Sean,

Yes the one piece of functionality that had us switch to JDK7 was the ability to redirect stdout and stderr of a child process to the stdout and stderr of the JVM.  Otherwise we would have to spawn a new thread to read from them and write/log the results, and one line of code is a lot nicer to deal with then is 20+.  If you want to put the thread in that will read the processes output and log it or write it to stderr so we can continue to support JDK1.6 or above you can.  Although be aware that Hadoop itself is discussing moving all of their development to JDK7 now that JDK6 it has reached End Of Life according to Oracle.  In that case we may need to move to JDK7 simply because YARN will be there.

--Bobby

[Sean Zhong]

Thank you for your comments.

Yes, totally agree the JDK7 should be future. But we may want to support both at least for now. Consider 

1. In a cluster, there are lots of bigdata software, we need to make sure all supports JDK7 before switching the full cluster JDK7, otherwise we need to stich to JDK6. 

2. There are legacy software in the enterprise env which is tested and assured only on JDK6, 

3. Migration cost. 

[Robert Evans]

That is fine with me.  If you want to submit a pull request with a a signed contributor license agreement I would be happy to pull it in.  If you cannot do that, because of whatever reason, and I know that there are often some.  Please file an issue, and we will look at doing it.  It is mostly a question of how long it will take to happen.  There are only so many hours in the day.

--Bobby

[Sean Zhong]

I ported it to JDK6. https://github.com/clockfly/storm-yarn

Still testing it, if it is stable, then I will submit a pull request.

[Robert Evans]

Awesome.

I look forward to the patch.

[michae...@hotmail.com]