Issue with Importing Local Modules with inner-dependencies

[Reza Qorbani]

I have an Airflow DAG with local python modules with the following structure:

• dags/
  • __init__.py
  • company/
    • __init__.py
    • common/
      • __init__.py
      • security/
        • __init__.py
        • connection.py
      • storage/
        • __init__.py
        • s3.py
    • providers/
      • __init__.py
      • all.py
      • provider-a/
        • __init__.py
        • process.py
  • my-dag.py

Inside 'dags/my-dag.py' load 'dags/company/providers/all.py' which that module load 'dags/company/providers/provider-a/process.py' which depends on two modules 'dags/company/common/storage/s3.py' and 'dags/company/common/security/connection.py'. 

All setup works perfectly on a local Airflow and compiles nicely, but when I copy all the above to Google Cloud Composer, it failed with this error:

Broken DAG: [/home/airflow/gcs/dags/my-dag.py] module 'company' has no attribute 'common'.

I searched Stackoverflow and found others also have the exact same problem. The interesting part is when I drop all structure into only 1-level ('company/*.py') and refactor my module naming and references, then it works locally as well as in Composer. 

It seems like the Composer has some different behavior or limitation than normal Airflow when it gets to a multi-hierarchy folder/module structure with inner-dependency. 

I would appreciate it if anyone can help me with this. 

Cheers,

Reza

[Bob Muscovite]

Hey Reza,

What you may want to try is using zipped dags. My understanding is that the zipped dag gets added to the python path, so this may solve this issue for you, although it poses its own problems if you try to reference non-python files.

Regards,

Boris

[Bob Muscovite]

I think in particular with https://groups.google.com/g/cloud-composer-discuss/c/TYVX4UpM7iE/m/3Qa63fGHCAAJ we saw similar behaviour.

[Reza Qorbani]

Hi Boris, 

Thank you for the response. I've never used zipped dags but I'll look into it. I only use python files so the limitation you referring to should not affect me. 

In the past 2 days, I end up building a build process that takes my multi-hierarchy python files and flattening them into a 1-level folder and refactors code with the new flatten module structure so it can run without error. It turned out to be a very nice hack that does the job for me for now but I truly wonder how come others not facing such an issue already! Also wondering why there is a difference between Composer's Airflow and Local Airflow!

Cheers,

Reza

[Reza Qorbani]

Hi Boris, 

I just did Zipped DAGs and still got the same error message :( Any other idea?

Cheers,

Reza

On Monday, November 9, 2020 at 4:28:00 AM UTC-8 muscov...@gmail.com wrote: