Job definition:
===========
Batch jobs are defined in apache config format. For example:
// python function to check if it is tuesday
checkIfTuesdayAndNotHoliday()
<job>
calendar: indian
<dependency: arbitrary_python_code: checkIfTuesdayAndNotHoliday()>
<command>
check availability of flight
</command>
<success: notify: email: agrawall/>
<failure: notify: email: ops>
<command>
some command to book my flight
</command>
</job>
<crontab> 00 17 * * 1-5 </crontab>
Batch installer script:
================
Users use this script to install (create schedule) the batches by giving the job definition file (mentioned in example above) to this script.
By using our load balancing infrastructure these schedules are distributed to different machines in the cluster. Finally, installer script creates a crontab entry like below:
Batch executor script:
================
This script takes care of executing the batch job, each of the command in the batch job are again distributed to run on different hosts in the cluster (this internally uses our cluster load balancing infra, that internally uses something similar to mesos). This is the script that takes of checking the dependencies, rescheduling the jobs, retrying, notifying etc.
Thanks