RE: [azkaban] Azkaban 2 release management and automation

[Richard Park]

There is actually a way of doing it, that I have yet to document.

For each http command, you're able to also include a username and password with the request so you don't need a session.

Alternatively, you can curl/http request a login action, and then use the return session id in all your calls. The session is somewhat temporary though, but you can run multiple commands with the same session id.

I'll copy and paste the instructions I wrote up for someone else for auto upload. Scheduling and executing would be similar. Would this fit in with your use case?

You have to use POST for this to work.  CURL or HTTP request doesn't really matter.

There are two ways of doing this:
1. The session method
POST to any azkaban url a with the following parameters:
action=login
username=<username>
password=<password>

If the login was successful, you will get a json response:
{"status":"success", "session.id": <some string value>}

When doing an upload:
POST a multiform request to <azkaban url>/manager
session.id=<some session id>

action=upload
project=<project you want to upload to>
file=<file>

With that given session id, you should be able to do several operations without logging in again. Make sure that the user has write permissions for the project.

2. Temp auth method
Do an upload:
POST a multiform request to <azkaban url>/manager

action=upload
username=<username>
password=<password>

project=<project you want to upload to>
file=<file>

---

From: azkab...@googlegroups.com [azkab...@googlegroups.com] on behalf of Maxime Brugidou [maxime....@gmail.com]
Sent: Thursday, January 31, 2013 9:12 AM
To: azkab...@googlegroups.com
Subject: [azkaban] Azkaban 2 release management and automation

Hi all,

I am currently testing Azkaban 2 and it has some really nice features and keeps things simple. We are currently using a bot to deploy our workflows to production, and switching to Azkaban would require us to get some sort of automated way to upload, configure and schedule flows (API or something else).

The simplified setup would be:

* have a "test" and "prod" azkaban

* test would be open for read/write/execute/schedule permissions to the project owners so they can tweak things at will

* prod would be read only for users

* the bot would deploy the zip file along with the new/updated schedule setup (from source control) to test then prod when the release is approved

* (optional) we would like to disable project creations for users in prod and this does not seem possible yet (but easy to add a role feature)

I think Azkaban 2 is really close to what we need. But through reading the code I didn't find some steps to upload, configure automatically and schedule flows.

Any idea? Do you guys at LinkedIn follow some sort of automated deployment or is it manual? Do you thing we could design these features in a simple manner?

Any help appreciated, and i'd be glad to provide code/pull requests if you think this is feasible.

Thanks!

Maxime

--
You received this message because you are subscribed to the Google Groups "azkaban" group.
To unsubscribe from this group and stop receiving emails from it, send an email to azkaban-dev...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Maxime Brugidou]

Thanks. I will try this and it will probably be enough for our use case. Setting the schedule using http requests and making sure the bot has password access.

What about deactivating project creation to have read only users?

As a side note if we migrate we will probably later implement kerberos authentication.

[Richard Park]

Deactivating project creation is something we should put into Azkaban.

It is fairly easy, and we can have roles that have this feature disabled by default.