Azkaban 2.1 Release

[Richard]

Announcing Azkaban 2.1

After releasing Azkaban 2.0, we've been working non-stop on the next version of Azkaban. We're now proud to announce the release of Azkaban 2.1. For this release, we took several suggestions from our internal users as well as the open source community when deciding what features we'd implement.

New features include:

• SLA features to notify users or kill flows if expected runtimes are exceeded.
• Job edits
• New ui for executing jobs
• Pipeline execution for concurrent running jobs
• Auto retries of failed jobs
• Manually invoke retries of failed jobs in a flow
• Proxy user checks to prevent users from running as incorrect headless accounts
• Refactored execution code for simplicity and extensibility
• Ability to prevent Project creation except for those with the proper permissions.
• JMX support
• Log size caps
• Log retention

Additionally, we've created a new Azkaban website to download packages and view documentation. We've worked hard to document the features of Azkaban, and although there's still work to be done, we believe we've made a good start. You can go to our new website at http://azkaban.github.io/azkaban2/. We're looking at getting a domain name for this site, so stay tuned.

Note that upgrading Azkaban 2.0 to 2.1 requires a few changes.

• There is an update_2.0_to_2.1.sql script provided that needs to be run to update the db. You can view the instructions here.
• If you are using azkaban-plugins, please update them as well. You will get the benefit of getting a few new job types as well as several bug fixes. We've also added secure Yarn support.
• If you've implemented a custom UserManager for your own authentication, you will need to build a new version of it against the Azkaban 2.1 jars. There were a few additional method interfaces implemented that prevent it from being backwards compatible.

Issues and feature requests can be submitted through github issues at https://github.com/azkaban/azkaban2/issues. We feel that there is plenty of room for Azkaban to grow, and our goal is to make it a very compelling option when it comes to Hadoop and workflow scheduling. Achieving this goal will require help from the community. Suggestions and/or code patches would be much appreciated.

Thanks,

-Richard Park

[Dan Streit]

Richard thanks for the update, I appreciate the work you guys are doing. The new SLA feature and Job Execution UI changes look good. The new documentation is definitely an improvement as well.

I ran into a few problems that might have just been from me rushing through updating to 2.1 on a test server. I'm going to try it again tomorrow on another server and reply with the specific issues if I see them again.

Dan