Custom secret backend

[Artur Gukasian]

I'm trying to create a custom Secrets Backend in Airflow. But have an issue with Airflow Scheduler that doesn't load it.

Secret Backend is a package in pypi package that I install in my Composer environment.
But Airflow Scheduler can't find my package for some reason, and errors during startup with a module not found error.

All other Airflow components, such as a DAG processor or workers, load my backend successfully. 

I'm using composer-3-airflow-2.11.1-build.1 version (was trying on 2.10 as well)
What could be a reason that only the Scheduler can't load my module?

Also, there is composer-dev cli that works with a custom backend, but only on old versions: 0.10.2. All new versions 0.11.x can't find the package with the custom backend.

[Michał Modras]

Hi,

In Composer 3, custom packages are not synced to the scheduler. This is by design, to ensure that scheduler functionality runs on a verified Composer image and custom packages do not affect its core functionality.

The custom backends (e.g. secrets backend) that rely on custom packages can be used in airflow-workers which are designed to run user code.

Best,

Michal

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/cloud-composer-discuss/d08dc8a5-d15d-4a4f-95f4-a15f7e5f81a5n%40googlegroups.com.