Multiple Dag directories

[Saurabh Saxena]

Hi,

Is there a way to have multiple dag directories ?
We have a requirement to segregate dags based on project names, we don't want to put all dags in one directory.
It would be nice to have different folders based on project names and having all dags related to the project in their respective directory.
If that is not possible, what can be other ways to segregate dags.

Regards,
Saurabh

[Arthur Wiedmer]

Hi,

You can have one dags folder with subfolders for DAGs based on projects (For instance ./dags/search/search_dag.py, ./dags/growth/growth.py, with DAGS_FOLDER = ./dags ). The scheduler does an os.walk which will recurse through the subfolders.

Let me know if that does not answer your question.

Best,

Arthur

[Saurabh Saxena]

Thanks Arthur. That answers my questions, I will give it a try.

Regards,
Saurabh

[Maxime Beauchemin]

Note that it will also follow symlinks, so that can be a way to integrate pipelines coming from different code repos for instance. Here's the related line of code:

https://github.com/airbnb/airflow/blob/master/airflow/models.py#L283

[Saurabh Saxena]

Thats great !! Thanks Maxime.

-Saurabh