Availabilty

[Jonathan Hodges]

Hi,

I was curious what the recommended approach is for higher availability with Azkaban2.  Will having multiple executor or web client processes across machines corrupt the underlying MySQL store?

Thanks in advance for the help.

-Jonathan

[Richard Park]

Multiple executors are allowable. Azkaban was designed to support that scenario, although with the limitation that there is one main executor that workflows are submitted to.

In fact, at LinkedIn, we use this feature to have an uninterrupted upgrade process. When upgrading, we start the new executor on a new port without shutting down the old executor, and restart the web client to repoint to the new executor port. The bounced web client picks up where it left off and starts submitting new jobs to the new executor. The old executor keeps running and we shut it off when it finishes its current executions.

It would be fairly trivial to allow for more than one active executor. It was always planned that we'd take this next step. However since the vast majority of our executing jobs are submissions to Hadoop, we found that most of the time, those job processes were idling. Priority was lowered for this feature. However, it hasn't been forgotten so if this feature is necessary, we'll add it to the issues on github.

Multiple web clients on the other hand may cause problems. Some of the issues have to do with inconsistent state in the clients. The scheduler on the web client also has a potential of colliding with each other. We have had discussions on how we'd proceed with doing this. Is this a highly desired feature?

-Richard

-Jonathan

--
You received this message because you are subscribed to the Google Groups "azkaban" group.
To unsubscribe from this group and stop receiving emails from it, send an email to azkaban-dev...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Jonathan Hodges]

Hi Richard,

Thanks for your detailed response.  We will also be using Azkaban mainly for Hadoop based flows so this isn’t a very high priority for us either.  I just wanted to confirm what was currently available.  The information you shared regarding uninterrupted upgrades will definitely come in handy for us in the future.

I have one last question regarding the scheduler.  If we wanted to swap in a different job scheduler technology would we just extend the Scheduler and Executor classes?  I noticed someone implemented a Quartz based patch in the old version of Azkaban and we might want to pursue something similar but with UC4.

Thanks again,

Jonathan

--
You received this message because you are subscribed to a topic in the Google Groups "azkaban" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/azkaban-dev/GG9B8dXeNMY/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to azkaban-dev...@googlegroups.com.

[Richard Park]

The scheduler should be modular enough that it can be swapped out by another scheduler. Ideally we'd abstract or interface the SchedulerManager and link to another implementation, much like what we've done for the UserManager.

We don't have any plans on swapping out the scheduler, but there's no reason why you can't. For now, if you want to implement your own version of the SchedulerManager, we'll accept the patch and then do the work to make it modular and swappable.

-Richard

[Jonathan Hodges]

Cool.  We are just in the investigative phase at this point, but if we decide to implement our own version of SchedulerManager we will gladly contribute back as a patch.  I should know more of our direction in the next few weeks.

-Jonathan

[Richard Park]

For the most part, with a few changes, the Scheduler code was ported from previous Azkaban instead of re-written. I'd expect that the port would be easy to do given the old quartz patch.