Announcing Azkaban 2 (update)

[Richard]

Edit: Fixed links. Duplicate message

Sorry for being quiet on this group for so long. An Azkaban update is long overdue.

Azkaban has not been abandoned. In the second half of last year, I was tasked with updating Azkaban to solve some of the pains we were facing. After evaluating the problem further, instead of patching around various issues, we decided to rebuild Azkaban from the ground up.

There were several reasons for this decision, but key motivators included LinkedIn's move to a secure Hadoop 1.0 and a new interface that was designed by one of LinkedIn's UI designers to fix our usability issues. Additionally, the code for various features in Azkaban was needlessly intertwined which made code changes and maintenance difficult.

What resulted was a new version of Azkaban with the following features:

• Completely new UI
• Uses MySQL DB as a datastore
• Project isolation
• User Authentication/Authorization
• Auditing of user actions
• Separation of the execution engine from web server
• Plugin-able executors and viewers
• Backwards compatible with previous azkaban jobs and zips

The changes that have been made will make it a lot easier to extend and maintain Azkaban for the future. We have committed resources to it and have a road map for various new exciting features.

You can find the code here:

https://github.com/azkaban/azkaban2

and the executor and hdfs browser plugin here:

https://github.com/azkaban/azkaban-plugins

An early version of the install instructions can be found in the docs folder in the repository. We will be adding more documents and instructions in the future.

Until we get a jira set up, any discovered bugs and feature request can be submitted as issues on github.

This version of Azkaban has been running in LinkedIn for the last couple months. It still has some sharp corners and jagged edges, but they are quickly being smoothed out. Expect more changes and updates in the future.

I again have to apologize for my absence on this group, and I endeavor to be more responsive in the future.

Thanks,

-Richard Park

[we...@dancingtrout.net]

Hi,
I couldn't open the setup manual .rtf under docs on my mac. textedit normally works fine for rtf, if not that, then word.

Wes

[Richard Park]

I'll have a look. What formats would be best for instructions?

I was considering just having html, but google docs doesn't format the html nicely, so diffs in github would be near impossible. I'll hunt around for better html editors.

[Felix GV]

Hello,

I've tried opening the rtf file on ubuntu and couldn't either. Is it possible the file is corrupted?

It'd be nice to have a valid version of that file in rtf format. I see no need to settle on a format before fixing the rtf version of the file...

Definitely looking forward to trying out Azkaban 2 !

Thanks for your help and contributions :) ...!

--

Félix

--
 
 

[Richard Park]

I've uploaded an html version. I'm going to try out other html editors which spit out nicely formatted text.

Please try it out. If there's confusion on what to add, please inform me and we'll fix/add it to the documents.

--
You received this message because you are subscribed to the Google Groups "azkaban" group.
To unsubscribe from this group and stop receiving emails from it, send an email to azkaban-dev...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Felix GV]

Hello,

Thanks a lot for the HTML version of the doc :) !

I was side-tracked for a little while but I finally got around to trying Azkaban. Here's what I came up with so far:

I followed the instructions for setting up the DB, the web server and the executor server. I also installed the jobtypes plugin and the executor server logs say that it got loaded so I guess that's fine...

I want to get Azkaban to run the Camus shaded jar that was recently open-sourced by the generous folks at LinkedIn ;) ... I'm able to run that job correctly (without Azkaban) by executing the CamusJob class inside of the jar.

Now, I can log into azkaban and create a project, but I can't get it to run a job. I go in my project and I click Upload and try to upload various things, but I get a blank page. After the blank page, I can hit back and I then see the following error messages towards the top of the page, depending on whether I try to upload the camus jar, the camus.properties file or a tar containing these two things, respectively:

File type application/x-java-archive unrecognized.
File type application/octet-stream unrecognized.

File type application/x-tar unrecognized.

I then tried creating a zip containing the same two files and uploading that caused the page to reload without showing a blank page nor any errors at the top.... I see logs in the azkaban web server as well as rows in the MySQL DB that seem to indicate the upload completed properly, but I see no change in the project, no workflow, etc...

Am I doing the right thing? What is it that I'm supposed to upload? How do I configure the project or workflow...?

I'm using the azkaban user that comes pre-configured in conf/azkaban-users.xml and which has ADMIN permissions, so I guess I must be seeing everything I can possibly see in the web UI, but I don't get how to make it work...

Any help would be greatly appreciated :) !

Thanks again :) !

--

Felix

--
Felix

[Richard Park]

Hey Felix,
Can you send me a version of a zip that doesn't work for you?
I'll have a look at it.

Thanks,
-Richard

---

From: azkab...@googlegroups.com [azkab...@googlegroups.com] on behalf of Felix GV [fe...@mate1inc.com]
Sent: Friday, February 08, 2013 5:44 PM
To: azkab...@googlegroups.com
Subject: Re: [azkaban] Re: Announcing Azkaban 2 (update)

[Felix GV]

Thanks for the quick reply Richard :) !

I can't really send that shaded camus jar because it pulls in a couple of our own proprietary jars in it...

I'm not at work anymore, but what I'll do on Monday is pull out a simple word count type of MR job, and try to package that in a zip, run it through Azkaban2, and then go from there with the troubleshooting.

That being said, now that I think of it, I may have figured out what I was doing wrong... I remember reading that Azkaban2 is compatible with Azkaban1 jobs, which, I guess, means that I need to include a .job file in the package to specify the type of job and possibly some other parameters. I didn't include this file in the zips I tried uploading today. If that's what I was missing, then it would probably be a good idea to give a proper error message explaining what's missing.

Anyway, I'll try all of this out and report back.

In any case, I think it'd be useful to have a dummy example job of each type, or something like that, just so users can easily validate if their setup is working correctly or not :)

Thanks again for the quick reply and for inducing my brain to kick into gear ;)

--

Felix

--
Felix

[Felix GV]

That was indeed the problem :) ... I had not included a .job file in my zip!

I got camus running through Azkaban2, but I have a couple of comments:

• Non-zip uploads should fail in a more graceful manner. As I said, when I upload a non-zip file, I see a blank page, and if I hit back there's an error message at the top of the page, but I think it would make sense to (fix the blank page, obviously, and) simplify the error message to something along the lines of "This type of upload is not supported. You can only upload a zip archive containing a .job file".
• When uploading a zip archive that doesn't contain a .job file, there should be a similarly helpful error message rather than just failing silently...
• I've been able to run the camus shaded jar as a "command" type of job, where the command property is something along the lines of "java -cp the-name-of-that-shaded-jar.jar:some-other-stuff-hadoop-needs-on-the-classpath com.linkedin.camus.etl.kafka.CamusJob -P camus.properties" but I have not been able to run a "javaprocess" type of job. I expect that I should be able to do so, because the camus shaded jar contains a main in the CamusJob class. However, the error messages I get in the executor's (and job's) logs are quite opaque and not very informative. Would there be a way to bubble up more info as to what actually happened that prevented the javaprocess job from running? This is the sort of log I'm getting with a failed javaprocess kind of job:
  

2013/02/11 16:48:31.210 -0500 INFO [camus] [Azkaban] Running execid:3 flow:camus project:1 version:6

2013/02/11 16:48:31.210 -0500 INFO [camus] [Azkaban] Creating active reference

2013/02/11 16:48:31.277 -0500 INFO [camus] [Azkaban] Updating initial flow directory.

2013/02/11 16:48:31.353 -0500 INFO [camus] [Azkaban] Fetching job and shared properties.

2013/02/11 16:48:31.353 -0500 INFO [camus] [Azkaban] Queuing initial jobs.

2013/02/11 16:48:31.353 -0500 INFO [camus] [Azkaban] Starting flows

2013/02/11 16:48:31.428 -0500 INFO [camus] [Azkaban] Job Started camus

2013/02/11 16:48:31.495 -0500 INFO [camus] [Azkaban] Starting job camus at 1360619311428

2013/02/11 16:48:31.495 -0500 INFO [camus] [Azkaban] Building javaprocess job executor. 

2013/02/11 16:48:31.496 -0500 INFO [camus] [Azkaban] job built.

2013/02/11 16:48:31.621 -0500 INFO [camus] [Azkaban] No classpath specified. Trying to load classes from /home/felixg/Downloads/azkaban-2.0/executions/3/azkaban-pack

2013/02/11 16:48:31.622 -0500 ERROR [camus] [Azkaban] Job set up failed null

2013/02/11 16:48:31.622 -0500 ERROR [camus] [Azkaban] Job run failed!

2013/02/11 16:48:31.622 -0500 ERROR [camus] [Azkaban] 

2013/02/11 16:48:31.623 -0500 INFO [camus] [Azkaban] Finishing job camus at 1360619311623

2013/02/11 16:48:31.738 -0500 INFO [camus] [Azkaban] Job Finished camus

2013/02/11 16:48:31.739 -0500 INFO [camus] [Azkaban] Flow appears finished. Cleaning up.

2013/02/11 16:48:31.739 -0500 INFO [camus] [Azkaban] FlowRunner thread has been interrupted.

2013/02/11 16:48:31.740 -0500 INFO [camus] [Azkaban] Finishing up flow. Awaiting Termination

2013/02/11 16:48:31.814 -0500 INFO [camus] [Azkaban] Setting flow status to Failed.

• Also, one last comment: is there any point in releasing the web server, the execution server and the jobtypes plugin as separate tarballs? I understand that making these things modular is great for updating them independently of one another, but it seems like there's little point in installing all of these things together manually, since most people probably need to have the three to achieve anything useful anyway, no? The documentaton was pretty good and I could get it working, but it seems like there is little point in having everyone put these things together manually, if everyone ends up with the same set up anyway... Perhaps I'm missing something and not seeing the big picture properly...?
  

Anyway, all in all, Azkaban2 seems to work well once we understand it a bit better. I definitely think there are some rough edges that make the barrier to entry a little higher than it needs to be, but I'm sure this can be improved easily, and I'm still planning to start using it for our production needs as soon as I finish setting up some other related things.

Definitely looking forward to using this more and seeing it evolve as we go!

Thanks a lot for your help and contributions :) !

--
Felix

[Richard Park]

Hey Dan,

1. Absolutely we should be releasing the roadmap. I'll put them up in the wiki on Tues when I'm back in the office. Off the top of my head, features include SLA actions (notification and killing), pipelining and queueing of flows, editing of jobs and conditional branching.

2.  We do this now for our Hadoop jobs. In azkaban-plugins, there are Hadoop executors that can be dropped in. We use these executors with the 'user.to.proxy' parameter set with the user we want to run as. It will grab a security token from Hadoop and run the Hadoop job under the user. I'll have someone write these settings in the azkaban-plugins wiki.

3. Sure. Your help is welcome. After I get the roadmap out, I'll be creating open Github issues on both the roadmap and any community requests. You could pick and issue that you feel like working on and add to them. One thing the community could do is to add their common plugins in azkaban-plugins. We're striving to make Azkaban more modular, and we could take advantage of this.

Oh, and I'll be fixing the broken urls.

Thanks,

-Richard

On Wed, Feb 13, 2013 at 12:14 PM, Dan Streit <dan...@gmail.com> wrote:

Richard,

First, thanks for the update!

We've been evaluating different work flow schedulers and Azkaban is my favorite so far. I have a couple of questions:

1) Can you share the roadmap so we can see what features are going to be added?

2) How difficult would it be to add support for running jobs as the user who submitted them? The reason I ask is that we plan to use Azkaban for running a variety of jobs like ETL, Map/Reduce and possibly publishing to data marts. It would make things easier for us if jobs were able to be run as other users.

3) I am interested in contributing to the project, is there anything I can do to help?

Also, the download links in the readme.md (https://github.com/azkaban/azkaban2) aren't correct url's. However, they are correct on the wiki (https://github.com/azkaban/azkaban2/wiki/Download-Packages).

Thanks,
Dan

--

[Jonathan Hodges]

Hi Felix,

Did you ever figure out how to execute Camus as a javaprocess?  We are also trying to do the same, but not exactly sure what the .job file should like.  We are able to run Camus fine with the hadoop jar command line.  To run in Azakaban we just took the camus.properties and added the following two lines and renamed as a .job file.

type=java

job.class=com.linkedin.camus.etl.kafka.CamusJob

We packaged this in a zip with the dependent jars and it uploaded fine in the UI.  However when we execute it gives a NullPointerException getting the Hadoop job configuration getConf().

Could you supply a sample .job by chance?  Even if you never got the javaprocess one working, the command example would be helpful.

Thanks in advance,

Jonathan

[Felix GV]

Hi Jonathan,

Sorry for the super late reply. I was away from work last week. I guess you may have already figured this out on your own by now, but I'll reply anyway...

This is the content of my camus.job file:

type=command

command=java -cp camus-mate1-job-0.1.0-SNAPSHOT-shaded.jar:/etc/hadoop/conf:/usr/lib/hadoop-hdfs/hadoop-hdfs.jar com.linkedin.camus.etl.kafka.CamusJob -P camus.properties

Notes:

• This camus.job file is packaged in a zip that also contains camus-mate1-job-0.1.0-SNAPSHOT-shaded.jar and the camus.properties file.
• There are no other extra jars packaged in the zip azkaban job (everything it needs is either included in the shaded jar or gotten from the extra entries in the -cp argument).
• camus-mate1-job-0.1.0-SNAPSHOT-shaded.jar is generated from a copy of the camus-example subproject that has minor differences in its maven pom files so that it pulls in some extra internal dependencies for our custom decoder implementation and other such camus-api-related classes that need to be re-implemented on a case-by-case basis.
  
• The hadoop-related paths I'm adding in the classpath are based on what gets created by the standard Hadoop 2.0.0-cdh4.1.2 apt-get debian packages.

Hopefully, that can be helpful for some people out there...!

--
Felix