Re: Edit Jobs, Smart Scheduling

[yu.ch...@gmail.com]

You can view job file content and property file content.

There are multiple ways to get to the job page. One of them is to go to the execution page where you see the DAG graph, and right click on the job you want to see, and click "open". That should open the job page. On the job page, you have links to properties files. We are working on docs right now which should address this type of questions in future.

Regarding your equestion 2, we don't have "smart" scheduling as you describe. But 1) you can achieve the same scheduling with existing simple scheduling ones. 2) we have plan to allow flexible triggers, that is on our work list.

On Tuesday, May 7, 2013 11:40:06 AM UTC-7, Aaron Zimmerman wrote:

I'm taking a look at azkaban 2, and I have a few questions.

Is there a way to view/edit the properties of a job from the web console?  In Azkaban 1 this was very easy, but I can't find a way to even view the .job files.  

Is there any plan to incorporate smarter job scheduling than just a recurring interval?  Specifically I'm looking for a way to change the execution interval at points in the day.  So run a job every 30 minutes during business hours, and then run once or twice overnight so the cluster is free to do longer running, once a day jobs.  Is there a way to do this with Azkaban 2?  

thanks,

Aaron Zimmerman

[Richard]

Hi Aaron,

Yes, there is job edits, but not in the version of Azkaban you're probably using. 

We're releasing 2.1 soon, I'll make an announcement shortly (before the end of the week).

The smart scheduling is interesting. We could extend the scheduler to change recurrence dynamically. When the scheduler submits a flow to execute, removes the scheduled flow from the scheduler and re-adds it with the next runtime if recurrence is set. It should be easy to have it change the next runtime recurrence at this time, although I'm not sure how the UI would work yet.

-Richard

[Aaron Zimmerman]

Cool, I'll stay posted for that update.  

I'd be willing to pitch in on the smart scheduling.  I've also been thinking about hadoop smart scheduling in general.  I want the cluster to be running at peak efficiency as much as possible.    So it would be even cooler if we could give it some fuzzy logic, like an overall job priority, and approximately how often it needs to run, and let azkaban decide when to run which jobs, while monitoring the hadoop cluster for available task slots and starting the next job when the cluster has capacity.  That is probably outside of the capacity for the base project - but maybe the scheduling logic could be pluginable?   

AZ