How to limit concurrent executions of a task during a backfill?

476 views
Skip to first unread message

Lance N.

unread,
Mar 27, 2016, 8:03:12 PM3/27/16
to Airflow

This is a DAG during a backfill run. Scheduling is random, which is almost find. I would like to limit each task (row) to only run one day at a time. Otherwise random scheduling is great. I'm already using the Pool feature for groups of tasks (each task reads from a table on one of several MySQL instances, so each instance gets one Pool).


These are the concurrency settings:

    max_active_runs=1,
    concurrency=3


Is there a specific limit to the number of concurrent executions of a particular task? Here you can see that two days of the same task are running simultaneously:




Thanks,

Lance Norskog




Reply all
Reply to author
Forward
0 new messages