Azkaban 3.0.0 Released

[Gaurav Aggarwal]

Hi Folks,

I have just released Azkaban released Azkaban and Azkaban-plugins 3.0.0 on Github.

Azkaban 3.0.0 release: https://github.com/azkaban/azkaban/releases/tag/3.0.0
Azkaban plugin 3.0.0 release: https://github.com/azkaban/azkaban-plugins/releases/tag/3.0.0

We have also created Azkaban 3.0.0 documentation (under review): https://github.com/azkaban/azkaban/pull/574

On Mon, Nov 16, 2015 at 3:49 PM, David Chen <d...@google.com> wrote:

Awesome! Great to hear. Looking forward to the 3.0 release!

On Fri, Nov 13, 2015 at 9:19 AM, Hien Luu <hl...@linkedin.com> wrote:

Nice.  Thanks.

Hien

On Fri, Nov 13, 2015 at 9:11 AM, Gaurav Aggarwal <gagg...@linkedin.com> wrote:

Update:-

I have resolved conflicts and merged multiexecutor_canary on my local git repo. I have also deployed Azkaban 3.0 package to our test instance for further testing. I will keep a close watch for sometime before pushing it to Azkaban master. 

FYI, changes made in https://github.com/azkaban/azkaban/pull/558 are breaking changes for LinkedIn's Azkaban ldap-manager-plugin. I have made changed to propagate fixes for ldap-manager-plugin.

On Wed, Nov 11, 2015 at 3:22 PM, Hien Luu <hl...@linkedin.com> wrote:

Glad to hear issue is resolved.  Here comes Azkaban 3.0 :)

Hien

On Wed, Nov 11, 2015 at 3:13 PM, Gaurav Aggarwal <gagg...@linkedin.com> wrote:

I and David synced up today to debug build issues. We tried a few things by catching exceptions but couldn't make it work. Then I tried few things independently and found that error was not in stderror but stdout of test process.

Issue was that my Azkaban build was running on java 1.8.0.40, but tests were running on older version of java as I had changed java version by setting environment variable. Setting java at system level (using update-alternative) solved the issue. Thanks David !!

Catch process exception Exception in thread "main" java.lang.UnsupportedClassVersionError: azkaban/jobExecutor/WordCountLocal : Unsupported major.minor version 52.0
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:791)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
	at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:482), 1

On Thu, Oct 22, 2015 at 4:16 PM, David Chen <d...@google.com> wrote:

Around noon tomorrow works for me.

In the meantime, can you take a look at my PR for running our CI against several JDK versions (https://github.com/azkaban/azkaban/pull/548)?

Talk with you tomorrow!

Thanks,

David
 

On Thu, Oct 22, 2015 at 4:04 PM, Gaurav Aggarwal <gagg...@linkedin.com> wrote:

Apologies, I was not able to ping today. I will ping you around noon ?

All these test failures are for maser branch.

On Thu, Oct 22, 2015 at 2:12 AM, David Chen <d...@google.com> wrote:

I took a look at the logs. While I am seeing the stack trace in the logs and that a ProcessFailureException was thrown when the test calls ProcessJob.run(), I am not seeing the actual contents of the ProcessFailureException (.getLogSnippet() and .getExitCode()), which I need to find the root cause of this test failure. Is this test failing on the master branch or multipleexecutors_canary? I have not been able to reproduce on either branch.

I am also available tomorrow before 3 PM if you would like to chat over Hangouts. You can reach me at dzc...@gmail.com

Thanks,

David 

On Thu, Oct 22, 2015 at 1:46 AM, David Chen <d...@google.com> wrote:

Hi Gaurav,

Sorry for the delay. Your email somehow got filtered to a different mailbox, and I didn't see it until today.

Many of the other failures are due to the other tests that are currently broken. As you can see from the issue I opened (https://github.com/azkaban/azkaban/issues/504), there are quite a number of them.

Thanks for sending me the logs. I will take a look. How about let's have a meeting on Hangouts either Friday or early next week?

Thanks,

David

On Fri, Oct 16, 2015 at 6:09 PM, Gaurav Aggarwal <gagg...@linkedin.com> wrote:

Hi David,

Somehow I see that stacktrace is truncated. I found that there are other failures as well (PFA details).

How about we have a hangout to resolve this issue ?

On Thu, Oct 15, 2015 at 3:13 PM, David Chen <d...@google.com> wrote:

Hi Gaurav,

Thanks for sending me the test reports. However, I am still not seeing the error codes or log snippets in the test output that the debug prints in my patch should have dumped.

Can you try to get the error code and log snippet in the ProcessFailureExceptions to print and then send me the output? Without that output, it would not be possible for me to debug this test failure because it has not reproduced on any of the 4 machines I have ran the tests on.

Thanks,

David

On Thu, Oct 15, 2015 at 1:36 PM, Gaurav Aggarwal <gagg...@linkedin.com> wrote:

Hi David,

I tried your patch, but I am still getting test failures. PFA test reports.

On Wed, Oct 14, 2015 at 5:33 PM, David Chen <d...@google.com> wrote:

Hi Gaurav,

I ran the tests on several machines but have not been able to reproduce the failures.

I am sending you a patch that adds some additional debug prints to print the error codes and log snippets to stderr. Since you are able to reproduce the failures, can you apply the patch, run the tests again, and send me the stderr logs?

Thanks,

David

On Tue, Oct 13, 2015 at 3:45 PM, Gaurav Aggarwal <gagg...@linkedin.com> wrote:

PFA html files.

On Tue, Oct 13, 2015 at 3:43 PM, David Chen <d...@google.com> wrote:

Hi Gaurav,

This file only contains Gradle's stack trace caused by the fact that there was a test failure. Can you send the HTML Gradle test report? It's this HTML file mentioned in the log you sent: /home/gaggarwa/bug/azkaban/azkaban-common/build/reports/tests/index.html

Thanks,

David

On Tue, Oct 13, 2015 at 3:40 PM, Gaurav Aggarwal <gagg...@linkedin.com> wrote:

Hi David,

PFA test failure details from Java 1.8.0.40.

On Tue, Oct 13, 2015 at 3:34 PM, David Chen <d...@google.com> wrote:

Thanks, Hien. That sounds good.

It would be great to run our CI against the different JDK versions that we support. It seems that Travis CI supports testing against multiple JDK versions. I'll go ahead and send a PR for running our CI against both Java 7 and 8.

Once I have repros for the test failures Gaurav mentioned, I'll open bugs and add them to the rollup ticket: http://blog.travis-ci.com/support_for_multiple_jdks/

Thanks,

David

On Tue, Oct 13, 2015 at 3:16 PM, Hien Luu <hl...@linkedin.com> wrote:

I think it is fine to drop support for 1.6. 

It is reasonable to hold off the Java 8 requirement until after 3.0 release, this will give folks running on Java 1.7 to leverage the multi-executor feature.

Thanks,

Hien

On Tue, Oct 13, 2015 at 3:06 PM, David Chen <d...@google.com> wrote:

Hi Gaurav,

Thanks! Sounds good.

For the Java version support, I agree that at it would be good to hold off actually using the Java 8 features until after the 3.0 release. I don't think is necessary for us to bump the sourceCompatibility requirement to 1.8 right now, but I am thinking perhaps we should drop support for 1.6 and require at least 1.7. My motivations for this are two-fold:

1. 1.6 has been deprecated for a long time and even 1.7 is now deprecated.
   
2. error-prone requires at least 1.7

Hien - what do you advise?

Thanks for catching the test failures. However, I am unable to reproduce those test failures on 1.8.0_20. Can you attach the test report generated by Gradle so I can take a look at the stack traces?

Thanks,

David

On Tue, Oct 13, 2015 at 1:37 PM, Gaurav Aggarwal <gagg...@linkedin.com> wrote:

Hi David,

Thanks for the inputs. I agree with working on master branch and short-lived feature branches. Going forward we will work of the master branch. I think we will need some time to test and validate Azkaban java 1.8 build, so we can avoid any surprises, and this will further push Azkaban 3.0 release. I think it will be great if we can have Azkaban 3.0 on java 1.6 and upgrade in next releases.

Moreover, I see some unit test failures with JDK-1_7_0_51 and JDK-1_8_0_40 in master branch. Can you please have a look ?

Failed tests:-

azkaban.jobExecutor.JavaProcessJobTest > testJavaJobHashmap FAILED

    java.lang.RuntimeException at JavaProcessJobTest.java:147

        Caused by: azkaban.jobExecutor.utils.process.ProcessFailureException at JavaProcessJobTest.java:147

azkaban.jobExecutor.JavaProcessJobTest > testJavaJob FAILED

    java.lang.RuntimeException at JavaProcessJobTest.java:135

        Caused by: azkaban.jobExecutor.utils.process.ProcessFailureException at JavaProcessJobTest.java:135

azkaban.jobExecutor.AllJobExecutorTests > azkaban.jobExecutor.JavaProcessJobTest.testJavaJobHashmap FAILED

    java.lang.RuntimeException at JavaProcessJobTest.java:147

        Caused by: azkaban.jobExecutor.utils.process.ProcessFailureException at JavaProcessJobTest.java:147

azkaban.jobExecutor.AllJobExecutorTests > azkaban.jobExecutor.JavaProcessJobTest.testJavaJob FAILED

    java.lang.RuntimeException at JavaProcessJobTest.java:135

        Caused by: azkaban.jobExecutor.utils.process.ProcessFailureException at JavaProcessJobTest.java:135

On Mon, Oct 12, 2015 at 3:27 PM, David Chen <d...@google.com> wrote:

+Hien

Hi Gaurav,

Doing a command line merge from multipleexecutors_canary to master sounds good to me given that all the changes have been reviewed.

For the future, I am wondering whether it would be better to commit these changes to master rather than keeping long-running feature branches. I think it is fine for master to be unstable (this is the case for many other projects generally) and have release branches if people want to build a stable release from source. As discussed in #517, feature branches should be short-lived and frequently rebased on master. This would prevent branches from diverging far from master and would make resolving merge conflicts much more manageable. We can have a separate discussion on this offline.

For the Java 6 build failure, I discussed this with Hien in a separate thread, and we think that given that both Java 6 and 7 are now deprecated and that LinkedIn has migrated to Java 8, Azkaban 3 should no longer support Java 6 and should move to Java 8. There are already some Apache projects, such as Apache Tajo, that are requiring Java 8 to build. This way, people would be encouraged to move to an officially supported version of Java, and we would be able to use the new features in Java 8.

Thanks,

David

On Mon, Oct 12, 2015 at 2:44 PM, Gaurav Aggarwal <gagg...@linkedin.com> wrote:

Hi David,

We are planning to release Azkaban 3.0 sometime around late next week for which we waiting on following two things:-

1. Execute-as-user feature: https://github.com/azkaban/azkaban/pull/532 
   
2. Fixing build failures for master.

We have been pulling code in multipleexecutors_canary as release candidate branch for Azkaban 3.0. We are thinking of a command line merge to master as we have already reviewed all the changes and it will be hard to re-review a ~65 files change. What's your thought on that ?

Will you be able to fix master branch build failures as per Azkaban 3.0 timeline ?

On Wed, Oct 7, 2015 at 2:30 PM, Gaurav Aggarwal <gagg...@linkedin.com> wrote:

javac 1.6.0_27

On Wed, Oct 7, 2015 at 2:29 PM, David Chen <d...@google.com> wrote:

Hi Gaurav,

Can you send me your full Java version (output of javac -version)?

Thanks,

David

On Tue, Oct 6, 2015 at 2:47 PM, David Chen <d...@google.com> wrote:

+correct email address for azkaban-dev mailing list.

On Tue, Oct 6, 2015 at 2:43 PM, David Chen <d...@google.com> wrote:

Thanks for catching this! I'll see if I can modify the build script to use the correct version of error-prone when building with Java 1.6.

I have opened the following GitHub issue to track: https://github.com/azkaban/azkaban/issues/515

On Tue, Oct 6, 2015 at 2:23 PM, Gaurav Aggarwal <gagg...@linkedin.com> wrote:

Hi David,

There seems to be some build failures due to error prone plugin in Azkaban master branch when we use Java 1.6.

[gaggarwa@gaggarwa-ld1 azkaban]$ ./gradlew distTar

FAILURE: Build failed with an exception.

* Where:

Build file '/home/gaggarwa/bug/azkaban/build.gradle' line: 39

* What went wrong:

A problem occurred evaluating root project 'azkaban'.

> java.lang.UnsupportedClassVersionError: net/ltgt/gradle/errorprone/ErrorPronePlugin : Unsupported major.minor version 51.0

* Try:

Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

Total time: 1.771 secs

Can you please have a look ?

--

Thanks,

Gaurav

--

Thanks,

Gaurav

--

Thanks,

Gaurav

--

Thanks,

Gaurav

--

Thanks,

Gaurav

--

Thanks,

Gaurav

--

Thanks,

Gaurav

--

Thanks,

Gaurav

--

Thanks,

Gaurav

--

Thanks,

Gaurav

--

Thanks,

Gaurav

--

Thanks,

Gaurav

[David Chen]

Woohoo! \o/ Congrats! Really excited for the 3.0.0 release!

[Hien Luu]

Hi David,

Do you happen to know how to upload Azkaban jar files to maven central?  I spent sometime on this a few months back, but didn't make much progress.

Thanks,

Hien

[Gaurav Aggarwal]

Hi David/Hien,

We also need to update sql package uploaded on amazon aws and linked at http://azkaban.github.io/downloads.html. Can you share details about how should we update packages on amazon aws ?

--

Thanks,

Gaurav

[Hien Luu]

I wonder who is the owner of that s3 folder https://s3.amazonaws.com/azkaban2/

[David Chen]

Hi Hien and Gaurav,

Sorry, I don't think I ever had the instructions or credentials for the AWS account, nor have I pushed any of the Azkaban jars to Maven.

I can reach out to Richard Park to see if he remembers.

Thanks,

David

[Gaurav Aggarwal]

Hi David,

I think it will be good to figure out who owns AWS account for Azkaban and update azkaban on AWS.

--

Thanks,

Gaurav

[Hien Luu]

That would be awesome.    Thanks David.

[tiger...@ahamobile.com]

Trying to build a new Azkaban server with the latest version, I noticed the download page (http://azkaban.github.io/downloads.html) mentioned in the document (http://azkaban.github.io/azkaban/docs/latest/#overview) only have versions up to 2.5

Thank you for your work.

[David Chen]

I have reached out to Richard. He said that the AWS account is the Data Infra AWS account set up by Vincent Kwong.

As for pushing Azkaban artifacts to Maven, he said that he pushed to Sonatype Nexus, but he doesn't quite remember how exactly he did it since it was over 2 years ago. I think it should be fine to follow the documentation for Sonatype Nexus and open a ticket on their JIRA if we run into any issues.

[jeroe...@gmail.com]

Hi guys,

We get an exception right of the bat after starting the webserver:

Exception in thread "main" azkaban.executor.ExecutorManagerException: Error fetching active flows
       at azkaban.executor.JdbcExecutorLoader.fetchActiveFlows(JdbcExecutorLoader.java:216)
       at azkaban.executor.ExecutorManager.loadRunningFlows(ExecutorManager.java:431)
       at azkaban.executor.ExecutorManager.<init>(ExecutorManager.java:126)
       at azkaban.webapp.AzkabanWebServer.loadExecutorManager(AzkabanWebServer.java:261)
       at azkaban.webapp.AzkabanWebServer.<init>(AzkabanWebServer.java:197)
       at azkaban.webapp.AzkabanWebServer.main(AzkabanWebServer.java:739)
Caused by: java.sql.SQLException: Unknown column 'ex.executor_id' in 'on clause' Query: SELECT ex.exec_id exec_id, ex.enc_type enc_type, ex.flow_data flow_data, et.host host, et.port port, ax.update_time axUpdateTime, et.id executorId,
et.active executorStatus FROM execution_flows ex INNER JOIN  active_executing_flows ax ON ex.exec_id = ax.exec_id INNER JOIN  executors et ON ex.executor_id = et.id Parameters: []
       at org.apache.commons.dbutils.AbstractQueryRunner.rethrow(AbstractQueryRunner.java:363)
       at org.apache.commons.dbutils.QueryRunner.query(QueryRunner.java:350)
       at org.apache.commons.dbutils.QueryRunner.query(QueryRunner.java:306)
       at azkaban.executor.JdbcExecutorLoader.fetchActiveFlows(JdbcExecutorLoader.java:212)

Shouldn't that column name be 'exec_id' instead of 'executor_id'? The conflicting line would be JdbcExecutorLoader line 1333.

[jeroe...@gmail.com]

Sorry, I guess I should have just opened an issue at GitHub: https://github.com/azkaban/azkaban/issues/582

On Saturday, November 21, 2015 at 8:09:52 AM UTC+1, Gaurav Aggarwal wrote:

[tiger...@ahamobile.com]

 Hi,

Could you please update the document on how to build/install the servers with details, step by step? I am an operation engineer and have no experience of building/installing Java package.

The online Document still mentioned 2.5, building with ant, etc; and the README.md in the download of Azkaban 3 still mentioned Azkaban2 and bulid with gradlew

Thanks,
Tiger

[engelbe...@gmail.com]

As mentioned above the documentation for 3.0 points you to the download page but the download page only shows tars 2.1 and 2.5. Any chance we could get some updated files or clarification around the install? We have cloned the git repo and will start down the path of using that code but would like to feel that we are doing things correctly and using the correct code.

Thanks,

Simon

[jeroe...@gmail.com]

Agreed! I've opened a GitHub issue for this as well: https://github.com/azkaban/azkaban/issues/583

[Gaurav Aggarwal]

Hi  folks,

Actually, we don't own that download page anymore. Let try and find a better place to host downloads.

--

Thanks,

Gaurav

[gune...@gmail.com]

Hello All,

0 down vote favorite

I have set up the Azkaban Multiple Executor set by following Azkaban 3.0 Documentation. I have configured the Websers and Executor servers. Still I'm getting the following errors while running an uploaded flow: [root@sandbox azkaban-exec-server-3.0.0]# azkaban.executor.ExecutorManagerException: Error loading flow with exec 45 at azkaban.execapp.FlowRunnerManager.submitFlow(FlowRunnerManager.java:459) at azkaban.execapp.ExecutorServlet.handleAjaxExecute(ExecutorServlet.java:264) at azkaban.execapp.ExecutorServlet.doGet(ExecutorServlet.java:111) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) ^C Can anyone suggest how to overcome this?