Error while operations/rollouts

[Rafi Shemesh]

App engine, nodejs. 

OpenApi endpoints.

swagger: "2.0"

So no changes were made to enpoints swagger, but the deploy now stuck and after ~4 min fail while trying to perform operation "rollouts". 

The command: 

 gcloud endpoints services deploy openapi-appengine.yaml --project PROJNAME

The error:

Waiting for async operation operations/rollouts.PROJNAME.appspot.com:ID to complete...

ERROR: (gcloud.endpoints.services.deploy) The operation with ID rollouts.PROJNAME.appspot.com:ID resulted in a failure.

In logs the error is:

 

 

}

It's strange, because no changes were made to enpoints.

[Juan Uys]

Same here. Minor change to the code, but can't deploy:

gcloud endpoints --verbosity=debug --project PROJECT_ID services deploy swagger.yaml

DEBUG: Running [gcloud.endpoints.services.deploy] with arguments: [--project: "PROJECT_ID", --verbosity: "debug", SERVICE_CONFIG_FILE:1: "[u'swagger.yaml']"]

INFO: No JSON detected in service config. Trying YAML...

Waiting for async operation operations/serviceConfigs.PROJECT_ID.appspot.com:577357ff-c3de-4fed-97cb-11e41349d5fe to complete...

Operation finished successfully. The following command can describe the Operation details:

 gcloud endpoints operations describe operations/serviceConfigs.PROJECT_ID.appspot.com:577357ff-c3de-4fed-97cb-11e41349d5fe

Waiting for async operation operations/rollouts.PROJECT_ID.appspot.com:256ae9ff-0498-4783-8e23-f2bb0b52e006 to complete...

DEBUG: (gcloud.endpoints.services.deploy) The operation with ID rollouts.PROJECT_ID.appspot.com:256ae9ff-0498-4783-8e23-f2bb0b52e006 resulted in a failure.

Traceback (most recent call last):

  File "/Users/juanuys/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 985, in Execute

    resources = calliope_command.Run(cli=self, args=args)

  File "/Users/juanuys/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py", line 795, in Run

    resources = command_instance.Run(args)

  File "/Users/juanuys/google-cloud-sdk/lib/surface/endpoints/services/deploy.py", line 393, in Run

    services_util.ProcessOperationResult(rollout_operation, args.async)

  File "/Users/juanuys/google-cloud-sdk/lib/googlecloudsdk/api_lib/endpoints/services_util.py", line 481, in ProcessOperationResult

    op = GetProcessedOperationResult(result, is_async)

  File "/Users/juanuys/google-cloud-sdk/lib/googlecloudsdk/api_lib/endpoints/services_util.py", line 526, in GetProcessedOperationResult

    op_ref, GetClientInstance()))

  File "/Users/juanuys/google-cloud-sdk/lib/googlecloudsdk/api_lib/endpoints/services_util.py", line 583, in WaitForOperation

    'The operation with ID {0} resulted in a failure.'.format(operation_id))

OperationErrorException: The operation with ID rollouts.PROJECT_ID.appspot.com:256ae9ff-0498-4783-8e23-f2bb0b52e006 resulted in a failure.

ERROR: (gcloud.endpoints.services.deploy) The operation with ID rollouts.PROJECT_ID.appspot.com:256ae9ff-0498-4783-8e23-f2bb0b52e006 resulted in a failure.

juanuys@Juans-MacBook-Pro [14:10:00] [~/Documents/muse/apps/factor] [web-auth *]

gcloud endpoints operations describe rollouts.PROJECT_ID.appspot.com:256ae9ff-0498-4783-8e23-f2bb0b52e006

done: true

error:

  code: 13

  message: Internal Error

metadata:

  '@type': type.googleapis.com/google.api.servicemanagement.v1.OperationMetadata

  progressPercentage: 100

  resourceNames:

  - services/PROJECT_ID.appspot.com/rollouts/2019-06-03r10

  startTime: 2019-06-03 14:06:25 BST

name: operations/rollouts.PROJECT_ID.appspot.com:256ae9ff-0498-4783-8e23-f2bb0b52e006

[toby_a...@threestrands.in]

I'm pretty sure the problem is at Google's end. Does anyone know if they are aware of it, or how we can make them aware? Do someone need to file a bug report somewhere?

[efrain...@tecnomotum.com]

Hello.

We are seeing the same problem here. However when we acces the Endpoints Portal we see the API deployed there

[nste...@gmail.com]

I have the same issue, I can see deployment in the cloud console but it seems failed

בתאריך יום שני, 3 ביוני 2019 בשעה 17:43:33 UTC+3, מאת efrain...@tecnomotum.com:

[on...@cloudwise.nl]

Hello, we are having the same problem. The gcloud command:

gcloud endpoints servives deploy openapi.json

fails, and we can see in Cloud Console > Endpoints > Services > Deployment History that our deployment shows up as Failed.

The error messages are exactly the same (except the project and operation names and IDs).

[Tim Cowley]

Hello Everyone,

There is currently problem with rolling out Endpoints services. The relevant parties at Google know and are actively working on the fix.

If you have support you can log into GCSC to follow and get updates.

[toby_a...@threestrands.in]

I'm glad the relevant people at Google are working on it. It's been about 20 hours since they found out about it (strange, since it was working for me about 10 hours ago). I know how hard it is having all this pressure to get something working that refuses to work. So here's some fruit for the guys at Google as encouragement 🍇 🍓 🍒 🍍 🍌

Anyway, even though it's still giving me an error, I can't see any way that's affecting me. My developer portal docs still get uploaded. The API is still usable, (even the parts I edited since it went down) the usage tracking is still working... If nothing is broken except that it gives a failure message, then that's not too bad!

[toby_a...@threestrands.in]

Hey it's fixed! (at least for me)

Well done Google people! Have some desert: 🥞 🥫 🍨 🍦 🍰 🍫

And beer is on the house 🍺 🍻

On Monday, June 3, 2019 at 9:33:59 PM UTC+5:30, Tim Cowley wrote:

[on...@cloudwise.nl]

Hey, unfortunately, the story is different for us.

We were deploying some breaking changes, and our frontend module -relying on the new API- was deployed before. 

The deployment for our new API, on the other hand, is still giving us the same problem. Not yet fixed for us, but we are hopeful and started trying again after seeing your message.

[sy...@slb.com]

Still failing for me

[Le Huynh]

I still have error. I was able to deploy once and when I tried again it failed. 

[nste...@gmail.com]

Still doesn't work and get same errors

בתאריך יום שלישי, 4 ביוני 2019 בשעה 05:08:12 UTC+3, מאת Le Huynh:

[notread...@gmail.com]

This has been raised as an issue, vote it up:

https://issuetracker.google.com/issues/134361013

[TRAPTI KHANDELWAL]

Facing issue .

Waiting for async operation operations/rollouts.gcpendpoints.appspot.com:ddaa2e19-26cd-4312-b38f-49069cb329dc to complete...

ERROR: (gcloud.endpoints.services.deploy) The operation with ID rollouts.gcpendpoints.appspot.com:ddaa2e19-26cd-4312-b38f-49069cb329dc resulted in a failure.

[TRAPTI KHANDELWAL]

Facing error :

Waiting for async operation operations/rollouts.gcpendpoints.appspot.com:ddaa2e19-26cd-4312-b38f-49069cb329dc to complete...

ERROR: (gcloud.endpoints.services.deploy) The operation with ID rollouts.gcpendpoints.appspot.com:ddaa2e19-26cd-4312-b38f-49069cb329dc resulted in a failure.

On Monday, June 3, 2019 at 2:01:00 PM UTC+5:30, Rafi Shemesh wrote:

[thibault...@carrefour.com]

Same error for me. 

It affects me as it seems that Cloud Endpoints ESP does not restart after deploying a new service, even if rollout_strategy is set to managed.  

[on...@cloudwise.nl]

Hello, could you please refer to https://issuetracker.google.com/issues/134361013 so we can raise awareness of this issue?

[on...@cloudwise.nl]

Hello, could you please refer to the issue that has been created for this problem? https://issuetracker.google.com/issues/134361013

[kabil...@gmail.com]

We are also facing a 20-30 minute latency on deploying a new endpoint service ended with a failure. 

[qiwz...@google.com]

Google is working this issue:

Issues with deploying and updating Cloud Endpoint services.

Status: Mitigation work is currently underway by our Engineering Team with one rollout complete. We are now waiting for our systems to get stabilized and will be rolling out configuration changes post that. We will provide another status update by Tuesday, 2019-06-04 22:00 US/Pacific with current details.

[qiwz...@google.com]

As for 14:50PST, the service config update issue has been mitigated.