Cannot create Cloud Composer 2 - GKE pods fail and missing IAM roles
Jomyuth Kiengee
Problem: I'm trying to create a Cloud Composer environment, but the CREATE operation is failing with the following error message:
Some of the GKE pods failed to become healthy. Please check the GKE logs for details, and retry the operation.
Additional Information:
The error occurred two days ago and hasn't resolved itself. I suspect the issue might be related to missing IAM roles in the following Service Accounts: service-...@cloudcomposer-accounts.iam.gserviceaccount.com in project 79xxxxxxx4 is missing the roles/composer.ServiceAgentV2Ext role. However, I have some concerns:
The error message mentions checking GKE logs for details, but I'm not sure how to interpret them. While the message suggests missing roles, I'm using custom IAM roles for some Service Accounts. It's possible the custom roles already grant the necessary permissions, but the warning is ignoring them. What I've tried so far:
I've reviewed the Cloud Composer documentation and troubleshooting guides, but haven't found a solution specific to this scenario. I've tried adding the roles/composer.ServiceAgentV2Ext role to the mentioned Service Account, but the CREATE operation still fails.
- How can I further investigate the GKE logs to pinpoint the cause of the pod health issues?
- Is there a way to verify if my custom IAM roles actually grant the necessary permissions for Cloud Composer?
- Are there any alternative solutions to address this CREATE operation failure?
Malek Dbouba
After migrating to use version 3 of Composer, the issue was resolved. It's important to note that this version is not mentioned in the console, so you need to use the command line to specify it. Here is the gcloud config that worked for me:
--image-version composer-3-airflow-2.7.3-build.5
--location us-central1
I hope this helps you if you face a similar issue.