Incident - Cloud Composer - Global

[Steve]

Hi everyone, 

I saw in the google service health cloud status

we are having an incident

it is said that there is a Workaround: Customers can retry creation or create newer Composer version (>=2.1)

but, I have tried to create airflow version >=2.1 using these commands :

gcloud composer environments create demo-environment \
  --location us-central1 \
  --image-version composer-2.2.0-airflow-2.5.1
in the Cloud Composer console, it is spinning all the time (40 minutes to 1 hour)

I have changed the image version into :

composer-2.1.15-airflow-2.5.1

composer-2.1.14-airflow-2.3.4

composer-2.1.13-airflow-2.4.3
composer-2.1.12-airflow-2.3.4
composer-2.1.9-airflow-2.4.3
composer-2.1.1-airflow-2.3.4

but nothing works, I have waited 40 minutes - 1 hour hoping it will be created, 

but it got an error timeout, is there anyone who succeeded in creating airflow using CLI?

I have tried to create Cloud Composer using Cloud Console but it can't be created because of a service account issue.

Best regards

Steve

[Rafal Biegacz]

Hi,

I'm from the Composer team and I can confirm that Composer creations for in us-central1 (and other regions) work fine for versions >= 2.1.

BTW - creations of Composer 2 environments for older Composer versions are also possible.

- if you select "Private Service Connect" as the method (as shown here: https://cloud.google.com/composer/docs/composer-2/configure-private-service-connect)

- even with "VPC Peering" it's possible to create Composer environments; though, you might need to re-try creation operation in case the first attempt is not successful.

I would not change composer versions in your creation attempts. What error do you get after the creation attempt when you use composer-2.2.0-airflow-2.5.1 ?

 

Regards, Rafal.

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/f813e4ef-05eb-47fa-9afb-31a5c884ed2bn%40googlegroups.com.

[Steve]

Hi Rafal,

Thank you for your reply,

There is no error in the log because it is still creating an environment (the loading icon keeps spinning indefinitely), 

there is an error in the cloud shell terminal(CLI) through :

ERROR: (gcloud.composer.environments.create) Operation projects/[project_id]/locations/us-central1/operations/3d498e83-5017-40a0-a511-8c7d8063d0a3 has not finished in 3600 seconds. The operations may still be underway remotely and may still succeed; use gcloud list and describe commands or https://console.developers.google.com/ to check resource state.

I have tried to run this command and waited for about 1 hour and 50 minutes just now:

• gcloud composer environments create demo-environment \
    --location us-central1 \
    --image-version composer-2.2.0-airflow-2.5.1

The user that I use to execute this command in Cloud Shell has these roles:

• AI Platform Admin
  
• App Engine Admin
  
• BigQuery Admin
  
• Composer Administrator
  
• Editor
  
• Owner
  
• Viewer
  

I attached the enabled services

When I tried to use UI (Console) to create composer version 2, 

it asks me to click the checkbox to grant, and when I tried to click create button it gives me this error :

• Composer API Service Agent service account (service-41...@cloudcomposer-accounts.iam.gserviceaccount.com) does not have required permissions set. Cloud Composer API Service Agent role is missing. Please bind roles/composer.serviceAgent back on service-41...@cloudcomposer-accounts.iam.gserviceaccount.com.

after this, I can't execute this command at all :

• gcloud composer environments create demo-environment \
    --location us-central1 \
    --image-version composer-2.2.0-airflow-2.5.1

it gives me 

ERROR: (gcloud.composer.environments.create) FAILED_PRECONDITION: Composer API Service Agent service account (service-41...@cloudcomposer-accounts.iam.gserviceaccount.com) does not have required permissions set. Cloud Composer API Service Agent role is missing. Please bind roles/composer.serviceAgent back on service-41...@cloudcomposer-accounts.iam.gserviceaccount.com.

am I missing something? 

Note: 

Sorry for the long response, I need to run the Cloud Composer again,

Best regards

Steve

[Rafal Biegacz]

Please, go to Cloud IAM and check the permissions of the Composer Agent Service Account. It should look something like this:

Regards, Rafal.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/9cd9d187-8df4-4910-bf56-e71b3e990d54n%40googlegroups.com.

[Steve]

Hi Rafal,

When I did this in UI (cloud console) :
it asks me to click the checkbox to grant, and when I tried to click "Create" button it gives me this error :

Composer API Service Agent service account (service-41...@cloudcomposer-accounts.iam.gserviceaccount.com) does not have required permissions set. Cloud Composer API Service Agent role is missing. Please bind roles/composer.serviceAgent back on service-41...@cloudcomposer-accounts.iam.gserviceaccount.com.

I didn't see any composer service agent inside the IAM > Service Accounts, so I choose to add it by CLI/cloud shell
so I did add the policy-binding using CLI to the service account requested above

and I can confirm that the Environment is created correctly,

thank you so much Rafal, I really appreciate it,

Best regards

Steve

[Malek Dbouba]

  Hello,   

  After migrating to use version 3 of Composer, the issue was resolved.  It's important to note that this version is not mentioned in the console, so you need to use the command line to specify it. Here is the gcloud config that worked for me:  

  --image-version composer-3-airflow-2.7.3-build.5  
  --location us-central1

I hope this helps you if you face a similar issue.