DAG code dependencies

[William Cox]

I have a subfolder in my dags folder:

~/airflow/dags/subfolder/ 

Inside that folder I have several files:

dag_demo.py

sql_demo_1.py

sql_demo_2.py

Inside my dag definition I do an import:

import sql_demo_1

import sql_demo_2

I can test the dag via: python ~/airflow/dags/subfolder/dag_demo.py and it works fine.

However, if I do a `airflow list_dags` command I get the error:

[2016-02-15 18:05:09,389] {models.py:203} ERROR - Failed to import: /home/ubuntu/airflow/dags/subfolder/dag_demo.py

Traceback (most recent call last):

  File "/home/ubuntu/anaconda2/envs/airflow/lib/python2.7/site-packages/airflow/models.py", line 201, in process_file

    m = imp.load_source(mod_name, filepath)

  File "/home/ubuntu/airflow/dags/subfolder/dag_demo.py", line 10, in <module>

    import sql_demo_1

ImportError: No module named sql_demo_1

It's as if the dependencies, in the DAG subfolder, are not being seen by Airflow. Why is this?

I've also tried:

$ airflow list_dags -sd ~/airflow/dags/subfolder/

with the same result.

[William Cox]

As an update, if I move all the files out of the subfolder and into the base ~/airflow/dags folder it seems to behave. Though putting everything in the base folder seems like a maintenance nightmare. 

[cha...@chartboost.com]

Can you try:

import subfolder.sql_demo1 ?

or

from subfolder.sql_demo1 import myfunction

all the imports should be relative to the base dag folder.

[William Cox]

Thanks Charlie. I guess that should have been evident.

I added an empty __init__.py file to the subfolder and then, as you suggested, `from subfolder import sql_demo1` worked!

Thanks.