Secret Manager Backend fails on composer-1.12.4 / airflow-1.10.10

[Lucas Esteves Cardozo]

Hi, everyone!

I'm trying to set up a Composer cluster with GCP Secret Manager backend for connections/variables, but it's failing on the GKE airflow-database-init-job with the following error:  ImportError: No module named providers.google.cloud.secrets.secret_manager. I've pulled the image that is being used by this job on GKE and, indeed, this module is not there. According to Composer docs for Secret Manager integration:

"To use Secret Manager, your Cloud Composer environment must use Airflow 1.10.10 or later and Python 3.6 or later (Python 2 is not supported)."

however, the airflow-database-init-job image seems to be using Python 2.7. Shouldn't it be >=3.6? This error can also be reproduced with the composer-1.12.3/airflow-1.10.10. 

Is anyone experiencing this?

Thanks!

[Mateusz Henc]

Hi,

Yes, this is a known issue. We are working on fixing it.

The simplest workaround is to create an environment without "secrets-backend" configuration and add it later, updating the existing environment.

Best regards,

Mateusz Henc

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/72bdb4fe-de27-498f-bb6b-5ae36ea1bda0n%40googlegroups.com.

[Lucas Esteves Cardozo]

Indeed this workaround seems to be the way to go. Thanks, Mateusz!