Is it safe to run multiple airflow scheduler instance simultaneously ?

[Wu XIANG]

Hi,

  If I run multiple airflow schedulers simultaneously, will one particular task be scheduled more than once ?

thanks

wu

[Maxime Beauchemin]

It may work, but that would be by accident. We're planning on fully supporting this soon, but haven't had the need for it just yet as the scheduling lag is still under a minute or so.

The way it will work is that the scheduler will take a lock against the DAG it is scheduling to avoid other schedulers from stepping on its toes.

Feel free to open a Github issue to track this item.

Max