airflow monitoring liveness probe crashes when service account only has composer.worker

[Pedro Jacinto]

Hi,

We're experiencing crashes in the airflow-monitoring deployment (in the composer GKE cluster) due to a failed liveness probe.

The liveness probe (/home/airflow/metric_prober.py) fails due to:

Traceback (most recent call last):
  File "/home/airflow/metric_prober.py", line 87, in <module>
    main()
  File "/home/airflow/metric_prober.py", line 76, in main
    view=monitoring_v3.enums.ListTimeSeriesRequest.TimeSeriesView.FULL))
(...)
google.api_core.exceptions.PermissionDenied: 403 Permission monitoring.timeSeries.list denied (or the resource may not exist).

The service account that we use to run composer only has the composer.worker role, which does not contain that permission

❯ gcloud iam roles describe roles/composer.worker | grep timeSeries

- monitoring.timeSeries.create

We'll be granting extra permissions to that svc account, but is this an expected behaviour?

Shouldn't composer.worker have this extra permission as it's required by airflow-monitoring?

Thanks!

[Rafal Biegacz]

Hi,

This is not expected behaviour. We already fixed this issue and the fix is in the process of being rolled out to production.

Yes - the composer.work role definition will be expanded to include monitoring.timeSeries.list permissions.

As a workaround, you can navigate to the Cloud IAM section of GCP console (or use gcloud commands) to modify the definition of the permissions for the service account used for running GKE cluster in Composer and include the missing permission.

I'm sorry for inconvenience.

Regards, Rafal.

This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/a3c57883-e150-4720-82e5-f0262e41c42d%40googlegroups.com.