Unable to create/delete Cloud Composer Environments

[we...@betabrand.com]

I am currently running into trouble spinning up a new Cloud Composer environment. Every attempt to do so eventually times out, and the environment ends up in state: ERROR.

When I try to delete the failed Cloud Composer environment, I get: "Composer Backend timed out. Currently running tasks are [stage: CP_DEPLOYMENT_DELETING response_timestamp { seconds: 1530913576 nanos: 548000000 } ]."

How do I resolve this?

[Cheng Liu]

Hi Wells,

I often find creating the environment using cmd works when the web UI is failing. You may want to give it a shot.

https://cloud.google.com/sdk/gcloud/reference/beta/composer/ 

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsub...@googlegroups.com.
To post to this group, send email to cloud-composer-discuss@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/351be5e8-09b1-4fd8-bced-cdd2eda89081%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Feng Lu]

Could you paste the deletion operation id for me? Thanks. 

On Fri, Jul 6, 2018 at 4:35 PM Cheng Liu <orang...@gmail.com> wrote:

Hi Wells,

I often find creating the environment using cmd works when the web UI is failing. You may want to give it a shot.

https://cloud.google.com/sdk/gcloud/reference/beta/composer/ 

On Fri, Jul 6, 2018 at 3:54 PM, <we...@betabrand.com> wrote:

I am currently running into trouble spinning up a new Cloud Composer environment. Every attempt to do so eventually times out, and the environment ends up in state: ERROR.

When I try to delete the failed Cloud Composer environment, I get: "Composer Backend timed out. Currently running tasks are [stage: CP_DEPLOYMENT_DELETING response_timestamp { seconds: 1530913576 nanos: 548000000 } ]."

How do I resolve this?

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To post to this group, send email to cloud-compo...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/351be5e8-09b1-4fd8-bced-cdd2eda89081%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/CAENJcPNydoAG_btrb70nOwCaPNHW%2BgKNTwLvtpFQd7NdtddEyw%40mail.gmail.com.

[Flávio Ferreira]

Did you find any solution? It is also happening to me both in the UI and Console. 

$ gcloud --project k8s-staging-176212 composer environments delete test  --location europe-west1

Deleting the following environments:

 - [test] in [europe-west1]

Do you want to continue (Y/n)?  Y

Delete in progress for environment [projects/k8s-staging-176212/locations/europe-west1/environments/test] with operation [projects/k8s-staging-176212/locations/europe-west1/operations/a58432fe-5e36-470d-9f43-0177b40f3c12].

Waiting for [projects/k8s-staging-176212/locations/europe-west1/environments/test] to be deleted...failed.

ERROR: Failed to delete environment [projects/k8s-staging-176212/locations/europe-west1/environments/test]: Operation [projects/k8s-staging-176212/locations/europe-west1/operations/a58432fe-5e36-470d-9f43-0177b40f3c12] failed: Composer Backend timed out. Currently running tasks are [stage: CP_DEPLOYMENT_DELETING

response_timestamp {

  seconds: 1534242877

  nanos: 845000000

}

].

ERROR: (gcloud.composer.environments.delete) Some requested deletions did not succeed.

I cannot also create new environment with success, it always fail with errors like this:

Resource not found (resource=europe-west1-staging-ccompo-7753c350-composer-backend-sub-7753c350-fd39-470b-bfa6-ce46afef7603). 

My account is Owner of project and I cannot find any relevant information reading logs. 

Could you provide any help?

[Flávio Ferreira]

That's weird but after a lot of tries I just decide to disable the Google Composer API and then re-enable it and it start working. :|

Maybe it can help anyone else in the future. 

[Maria Janczak]

Hi Flavio,

Sorry for the late reply.

When the Composer API is enabled, a Composer API service account is created in your Google Cloud Project. This gives the Composer API permissions to create and delete resources that are part of Composer Environments. If someone on your team modified the permissions for the service account, it could result in such errors, and disabling/reenabling the API would fix the permissions on the service account, and resolve the errors. The message displayed on failure should have been improved in the most recent Composer release (on 8/17).

-Maria

--

You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To post to this group, send email to cloud-compo...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/6fb76ad4-4dd7-4364-b475-060b4e9f7fe9%40googlegroups.com.

[Miro Karpis]

I have now spend almost 4 hours by creating a new environment :( ,...problem still not solved. Why does delete an environment take 1 hour?

Cheers,

Miro

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsub...@googlegroups.com.

[Feng Lu]

I suspect that the IAM binding of Composer service robot account in your project (e.g., servi...@cloudcomposer-accounts.iam.gserviceaccount.com) is probably broken (e.g., someone may change the IAM policy and break the Composer service), could you disable/re-enable the Composer API first and try again? 

Miro

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.

To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/6fb76ad4-4dd7-4364-b475-060b4e9f7fe9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.

To post to this group, send email to cloud-compo...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/1cf0e5d7-19ae-4905-b541-aa1a3cff4f46%40googlegroups.com.

[Miro Karpis]

Thanks,..but still no luck :(

The error I'm getting after approx of 1 hour waiting is:

Composer Backend timed out. Currently running tasks are [stage: CP_COMPOSER_AGENT_RUNNING description: "RPC Successful: Pull" response_timestamp { seconds: 1535703239 nanos: 57000000 } ].

Not sure what the problem is here yet,.but It would be great if some vital checks could be done during the process start,..

Cheers,

m.

Miro

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsub...@googlegroups.com.

To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/6fb76ad4-4dd7-4364-b475-060b4e9f7fe9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsub...@googlegroups.com.

[Feng Lu]

Did you specify a service account when creating the composer environment? If yes, please make sure that service account has composer.worker role. 

Miro

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.

To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/6fb76ad4-4dd7-4364-b475-060b4e9f7fe9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.

To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/1cf0e5d7-19ae-4905-b541-aa1a3cff4f46%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.

To post to this group, send email to cloud-compo...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/3e4ac709-5bec-4ee6-8d37-dfcf96fc76a4%40googlegroups.com.

[Miro Karpis]

Hi Feng Lu, I have solved the problem by cleaning all the related instances IAM, storage, compute engine..
Thanks for helping.

[Miro Karpis]

sorry for bothering again with this,..but keep facing the same issue (not able to create composer environment) - now for all projects :(

Composer Backend timed out. Currently running tasks are [stage: CP_COMPOSER_AGENT_RUNNING description: "RPC Successful: Pull" response_timestamp { seconds: 1536901379 nanos: 199000000 } ].

I have tried several time to completely remove all related iam's, enable/disable composer API, from web console, via gcloud - but no luck.

Please does anybody know if it is possible to get more logs or how can I investigate this?

Regards,

Miro

[Trevor Edwards]

You can check if the "composer agent" job is succeeding. Go to http://console.cloud.google.com/kubernetes/workload for your project and look for a workload for the agent. It likely is in error state, and you can click into it to see the logs.

[Ismail Zmerli]

Hello I know this has been a while but I am facing the same issue for 3 days cannot seem to work this out ! 

On Saturday, 7 July 2018 00:54:55 UTC+2, we...@betabrand.com wrote:

[Rafal Biegacz]

Hi Ismail,

I can help you. I will write an email directly to you to agree on the next steps. Once we discover the root cause of the issues we can upgrade the discussion group what it was.

Regards, Rafal.

--

You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/eaf29611-9399-4e75-bf3e-197287f03640o%40googlegroups.com.

[Arthur Lopes]

Hey everybody, 

Hope this conversation is still going on, 'cause I ran into that error as well getting my first way around GCP beyond BigQuery, so I probably messed something up...

Anyway, the error was:

"Got error "" during CP_DEPLOYMENT_DELETING [Rerunning Task. ]."

And what I did was:

- Created a CP environment

- Did things successfully

- Deleted the nodes via Kubernetes (studying on my own and want to keep costs low, so I wanted to narrow the uptime. This lead me to someone on StackOverflow talking about doing this instead of creating and deleting the env everyday)
- Error started

- Can't delete

- To add insult to injury, I deleted the service account linked to it (had other inactive ones created for other projects and decided to do some cleaning)

I'm just going to start working on a new one, but I would appreciate greatly if someone could help me with this one

Cheers,

Tuca