Custom plugin dependencies are not loaded

400 views
Skip to first unread message

ar...@myndyou.com

unread,
Apr 30, 2019, 8:33:06 AM4/30/19
to cloud-composer-discuss
Hi,

I'm trying to create a custom Airflow sensor.

I've created a plugin in the /plugins GCS folder and tested the plugin and it worked well.. up until the point I needed to add some external dependencies.
The problem is that I was unable to add external dependencies as I did with DAGs.
I tried adding a /plugins/dependencies directory and importing a dependency it like so:

from dependencies.test import hello


This pattern works well for DAGs but I get a "module not found" error when I do that in the plugin.

What is the correct way to import local python dependencies?

Thanks

ar...@myndyou.com

unread,
May 2, 2019, 10:59:19 AM5/2/19
to cloud-composer-discuss
A quick update on the solution:

The Cloud Composer plugins document point to the official Airflow plugins page which in turn instructs to create a Airflow plugin class like so:

class AirflowTestPlugin(AirflowPlugin):
    ...

Unfortunately, this approach didn't work for me.
Apparently, there's no need for the Airflow plugins class boilerplate - just create the plugin or sensor like so:

class MySensor(BaseSensorOperator):

    @apply_defaults
    def __init__(self, *args, **kwargs):
        super(CallsSensor, self).__init__(*args, **kwargs)

    def poke(self, context):
        return False


You can now import anything from the plugins folder.

Import this sensor from a DAG just by calling
from sensor_file import MySensor

Arik
Reply all
Reply to author
Forward
0 new messages