Re: Build failures for Azkaban on Java 1.6

[David Chen]

+correct email address for azkaban-dev mailing list.

On Tue, Oct 6, 2015 at 2:43 PM, David Chen <d...@google.com> wrote:

Thanks for catching this! I'll see if I can modify the build script to use the correct version of error-prone when building with Java 1.6.

I have opened the following GitHub issue to track: https://github.com/azkaban/azkaban/issues/515

On Tue, Oct 6, 2015 at 2:23 PM, Gaurav Aggarwal <gagg...@linkedin.com> wrote:

Hi David,

There seems to be some build failures due to error prone plugin in Azkaban master branch when we use Java 1.6.

[gaggarwa@gaggarwa-ld1 azkaban]$ ./gradlew distTar

FAILURE: Build failed with an exception.

* Where:

Build file '/home/gaggarwa/bug/azkaban/build.gradle' line: 39

* What went wrong:

A problem occurred evaluating root project 'azkaban'.

> java.lang.UnsupportedClassVersionError: net/ltgt/gradle/errorprone/ErrorPronePlugin : Unsupported major.minor version 51.0

* Try:

Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

Total time: 1.771 secs

Can you please have a look ?

--

Thanks,

Gaurav

[David Chen]

Hi Gaurav,

Can you send me your full Java version (output of javac -version)?

Thanks,

David

[Gaurav Aggarwal]

javac 1.6.0_27

--

Thanks,

Gaurav

[Gaurav Aggarwal]

Hi David,

We are planning to release Azkaban 3.0 sometime around late next week for which we waiting on following two things:-

1. Execute-as-user feature: https://github.com/azkaban/azkaban/pull/532 
   
2. Fixing build failures for master.

We have been pulling code in multipleexecutors_canary as release candidate branch for Azkaban 3.0. We are thinking of a command line merge to master as we have already reviewed all the changes and it will be hard to re-review a ~65 files change. What's your thought on that ?

Will you be able to fix master branch build failures as per Azkaban 3.0 timeline ?

--

Thanks,

Gaurav

[David Chen]

+Hien

Hi Gaurav,

Doing a command line merge from multipleexecutors_canary to master sounds good to me given that all the changes have been reviewed.

For the future, I am wondering whether it would be better to commit these changes to master rather than keeping long-running feature branches. I think it is fine for master to be unstable (this is the case for many other projects generally) and have release branches if people want to build a stable release from source. As discussed in #517, feature branches should be short-lived and frequently rebased on master. This would prevent branches from diverging far from master and would make resolving merge conflicts much more manageable. We can have a separate discussion on this offline.

For the Java 6 build failure, I discussed this with Hien in a separate thread, and we think that given that both Java 6 and 7 are now deprecated and that LinkedIn has migrated to Java 8, Azkaban 3 should no longer support Java 6 and should move to Java 8. There are already some Apache projects, such as Apache Tajo, that are requiring Java 8 to build. This way, people would be encouraged to move to an officially supported version of Java, and we would be able to use the new features in Java 8.

Thanks,

David

[Gaurav Aggarwal]

Hi David,

Thanks for the inputs. I agree with working on master branch and short-lived feature branches. Going forward we will work of the master branch. I think we will need some time to test and validate Azkaban java 1.8 build, so we can avoid any surprises, and this will further push Azkaban 3.0 release. I think it will be great if we can have Azkaban 3.0 on java 1.6 and upgrade in next releases.

Moreover, I see some unit test failures with JDK-1_7_0_51 and JDK-1_8_0_40 in master branch. Can you please have a look ?

Failed tests:-

azkaban.jobExecutor.JavaProcessJobTest > testJavaJobHashmap FAILED

    java.lang.RuntimeException at JavaProcessJobTest.java:147

        Caused by: azkaban.jobExecutor.utils.process.ProcessFailureException at JavaProcessJobTest.java:147

azkaban.jobExecutor.JavaProcessJobTest > testJavaJob FAILED

    java.lang.RuntimeException at JavaProcessJobTest.java:135

        Caused by: azkaban.jobExecutor.utils.process.ProcessFailureException at JavaProcessJobTest.java:135

azkaban.jobExecutor.AllJobExecutorTests > azkaban.jobExecutor.JavaProcessJobTest.testJavaJobHashmap FAILED

    java.lang.RuntimeException at JavaProcessJobTest.java:147

        Caused by: azkaban.jobExecutor.utils.process.ProcessFailureException at JavaProcessJobTest.java:147

azkaban.jobExecutor.AllJobExecutorTests > azkaban.jobExecutor.JavaProcessJobTest.testJavaJob FAILED

    java.lang.RuntimeException at JavaProcessJobTest.java:135

        Caused by: azkaban.jobExecutor.utils.process.ProcessFailureException at JavaProcessJobTest.java:135

--

Thanks,

Gaurav

[David Chen]

Hi Gaurav,

Thanks! Sounds good.

For the Java version support, I agree that at it would be good to hold off actually using the Java 8 features until after the 3.0 release. I don't think is necessary for us to bump the sourceCompatibility requirement to 1.8 right now, but I am thinking perhaps we should drop support for 1.6 and require at least 1.7. My motivations for this are two-fold:

1. 1.6 has been deprecated for a long time and even 1.7 is now deprecated.
   
2. error-prone requires at least 1.7

Hien - what do you advise?

Thanks for catching the test failures. However, I am unable to reproduce those test failures on 1.8.0_20. Can you attach the test report generated by Gradle so I can take a look at the stack traces?

Thanks,

David

[Hien Luu]

I think it is fine to drop support for 1.6. 

It is reasonable to hold off the Java 8 requirement until after 3.0 release, this will give folks running on Java 1.7 to leverage the multi-executor feature.

Thanks,

Hien

[David Chen]

Thanks, Hien. That sounds good.

It would be great to run our CI against the different JDK versions that we support. It seems that Travis CI supports testing against multiple JDK versions. I'll go ahead and send a PR for running our CI against both Java 7 and 8.

Once I have repros for the test failures Gaurav mentioned, I'll open bugs and add them to the rollup ticket: http://blog.travis-ci.com/support_for_multiple_jdks/

Thanks,

David

[Gaurav Aggarwal]

Hi David,

PFA test failure details from Java 1.8.0.40.

--

Thanks,

Gaurav

[David Chen]

Hi Gaurav,

This file only contains Gradle's stack trace caused by the fact that there was a test failure. Can you send the HTML Gradle test report? It's this HTML file mentioned in the log you sent: /home/gaggarwa/bug/azkaban/azkaban-common/build/reports/tests/index.html

Thanks,

David

[Gaurav Aggarwal]

PFA html files.

--

Thanks,

Gaurav

[David Chen]

Hi Gaurav,

I ran the tests on several machines but have not been able to reproduce the failures.

I am sending you a patch that adds some additional debug prints to print the error codes and log snippets to stderr. Since you are able to reproduce the failures, can you apply the patch, run the tests again, and send me the stderr logs?

Thanks,

David

[Gaurav Aggarwal]

Hi David,

I tried your patch, but I am still getting test failures. PFA test reports.

--

Thanks,

Gaurav

[David Chen]

Hi Gaurav,

Thanks for sending me the test reports. However, I am still not seeing the error codes or log snippets in the test output that the debug prints in my patch should have dumped.

Can you try to get the error code and log snippet in the ProcessFailureExceptions to print and then send me the output? Without that output, it would not be possible for me to debug this test failure because it has not reproduced on any of the 4 machines I have ran the tests on.

Thanks,

David

[Gaurav Aggarwal]

Hi David,

Somehow I see that stacktrace is truncated. I found that there are other failures as well (PFA details).

How about we have a hangout to resolve this issue ?

--

Thanks,

Gaurav

[David Chen]

Hi Gaurav,

Sorry for the delay. Your email somehow got filtered to a different mailbox, and I didn't see it until today.

Many of the other failures are due to the other tests that are currently broken. As you can see from the issue I opened (https://github.com/azkaban/azkaban/issues/504), there are quite a number of them.

Thanks for sending me the logs. I will take a look. How about let's have a meeting on Hangouts either Friday or early next week?

Thanks,

David

[David Chen]

I took a look at the logs. While I am seeing the stack trace in the logs and that a ProcessFailureException was thrown when the test calls ProcessJob.run(), I am not seeing the actual contents of the ProcessFailureException (.getLogSnippet() and .getExitCode()), which I need to find the root cause of this test failure. Is this test failing on the master branch or multipleexecutors_canary? I have not been able to reproduce on either branch.

I am also available tomorrow before 3 PM if you would like to chat over Hangouts. You can reach me at dzc...@gmail.com

Thanks,

David 

[Gaurav Aggarwal]

Apologies, I was not able to ping today. I will ping you around noon ?

All these test failures are for maser branch.

--

Thanks,

Gaurav

[David Chen]

Around noon tomorrow works for me.

In the meantime, can you take a look at my PR for running our CI against several JDK versions (https://github.com/azkaban/azkaban/pull/548)?

Talk with you tomorrow!

Thanks,

David
 

[Gaurav Aggarwal]

I and David synced up today to debug build issues. We tried a few things by catching exceptions but couldn't make it work. Then I tried few things independently and found that error was not in stderror but stdout of test process.

Issue was that my Azkaban build was running on java 1.8.0.40, but tests were running on older version of java as I had changed java version by setting environment variable. Setting java at system level (using update-alternative) solved the issue. Thanks David !!

Catch process exception Exception in thread "main" java.lang.UnsupportedClassVersionError: azkaban/jobExecutor/WordCountLocal : Unsupported major.minor version 52.0
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:791)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
	at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:482), 1

--

Thanks,

Gaurav

[Hien Luu]

Glad to hear issue is resolved.  Here comes Azkaban 3.0 :)

Hien

[Gaurav Aggarwal]

Update:-

I have resolved conflicts and merged multiexecutor_canary on my local git repo. I have also deployed Azkaban 3.0 package to our test instance for further testing. I will keep a close watch for sometime before pushing it to Azkaban master. 

FYI, changes made in https://github.com/azkaban/azkaban/pull/558 are breaking changes for LinkedIn's Azkaban ldap-manager-plugin. I have made changed to propagate fixes for ldap-manager-plugin.

--

Thanks,

Gaurav

[Hien Luu]

Nice.  Thanks.

Hien

[David Chen]

Awesome! Great to hear. Looking forward to the 3.0 release!