Operation not complete. Waiting to retry. -> ERROR: (gcloud.app.deploy) Error Response: [13] An internal error occurred.

[Dario DCG]

Hi, I don't know if it's a coincidence, but since gcloud sdk 265 or 266 I have big difficulty trying to deploy my app to Google App Engine standard Python environment.

What happens is that the 'gcloud app deploy' command takes a very long time to run after successfully uploading the files, and issues many 'Updating service [default]...⠧DEBUG: Operation [...] not complete. Waiting to retry.' lines for several minutes until it A) succeeds, or B) (more often) fails or probably times out with 'Error Response: [13] An internal error occurred.'





The deployment can fail or succeed even on subsequent attempts with exactly the same version of the app (same files, same configuration etc.).





Before version 265, deployments were quick and always successful.





Below is a sample of the verbose output:


Do you want to continue (Y/n)?

Beginning deployment of service [default]...
╔════════════════════════════════════════════════════════════╗
╠═ Uploading 1 file to Google Cloud Storage                 ═╣
╚════════════════════════════════════════════════════════════╝
File upload done.
...
...
...
...
Updating service [default]...⠧DEBUG: Operation [apps/****/operations/e2aa6c6e-227c-4b59-beca-83fceaa55138] not complete. Waiting to retry.
Updating service [default]...⠏DEBUG: Operation [apps/****/operations/e2aa6c6e-227c-4b59-beca-83fceaa55138] not complete. Waiting to retry.
Updating service [default]...⠶DEBUG: Operation [apps/****/operations/e2aa6c6e-227c-4b59-beca-83fceaa55138] not complete. Waiting to retry.
Updating service [default]...⠹DEBUG: Operation [apps/****/operations/e2aa6c6e-227c-4b59-beca-83fceaa55138] not complete. Waiting to retry.
Updating service [default]...⠹DEBUG: Operation [apps/****/operations/e2aa6c6e-227c-4b59-beca-83fceaa55138] not complete. Waiting to retry.
INFO: Refreshing due to a 401 (attempt 1/2)
Updating service [default]...⠹INFO: Refreshing access_token
Updating service [default]...⠧DEBUG: Operation [apps/****/operations/e2aa6c6e-227c-4b59-beca-83fceaa55138] not complete. Waiting to retry.
Updating service [default]...⠏DEBUG: Operation [apps/****/operations/e2aa6c6e-227c-4b59-beca-83fceaa55138] not complete. Waiting to retry.
Updating service [default]...⠏DEBUG: Operation [apps/****/operations/e2aa6c6e-227c-4b59-beca-83fceaa55138] complete. Result: {
    "metadata": {
        "target": "apps/****/services/default/versions/v123alpha",
        "method": "google.appengine.v1.Versions.CreateVersion",
        "user": "******@gmail.com",
        "insertTime": "2019-10-21T07:20:10.069Z",
        "endTime": "2019-10-21T07:30:20.046Z",
        "@type": "type.googleapis.com/google.appengine.v1.OperationMetadataV1"
    },
    "done": true,
    "name": "apps/****/operations/e2aa6c6e-227c-4b59-beca-83fceaa55138",
    "error": {
        "message": "An internal error occurred.",
        "code": 13
    }
}
Updating service [default]...failed.
DEBUG: (gcloud.app.deploy) Error Response: [13] An internal error occurred.
Traceback (most recent call last):
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 983, in Execute
    resources = calliope_command.Run(cli=self, args=args)
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py", line 784, in Run
    resources = command_instance.Run(args)
  File "/usr/lib/google-cloud-sdk/lib/surface/app/deploy.py", line 99, in Run
    parallel_build=False)
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/command_lib/app/deploy_util.py", line 641, in RunDeploy
    ignore_file=args.ignore_file)
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/command_lib/app/deploy_util.py", line 431, in Deploy
    extra_config_settings)
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/api_lib/app/appengine_api_client.py", line 208, in DeployService
    poller=done_poller)
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/api_lib/app/operations_util.py", line 314, in WaitForOperation
    sleep_ms=retry_interval)
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/api_lib/util/waiter.py", line 264, in WaitFor
    sleep_ms, _StatusUpdate)
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/api_lib/util/waiter.py", line 326, in PollUntilDone
    sleep_ms=sleep_ms)
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/core/util/retry.py", line 229, in RetryOnResult
    if not should_retry(result, state):
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/api_lib/util/waiter.py", line 320, in _IsNotDone
    return not poller.IsDone(operation)
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/api_lib/app/operations_util.py", line 183, in IsDone
    encoding.MessageToPyValue(operation.error)))
OperationError: Error Response: [13] An internal error occurred.
ERROR: (gcloud.app.deploy) Error Response: [13] An internal error occurred.

I'll be happy to provide more information as needed, thank you for helping,

Dario

[George (Cloud Platform Support)]

Hello Dario, 

How does your app.yaml look like? 

To enable a closer examination of your installation, you need to run the following commands: 

gcloud components update

gcloud info

gcloud app deploy --verbosity=debug

What is the output from these commands? You should edit out all private data from the gcloud info command, app.yaml and generally from your reply. As you see, we need private information to examine your situation in depth. Groups threads do not offer proper tools to safeguard your information from public eyes. You are encouraged to open a private issue in Public Issue Tracker with similar contents, where visibility to postings can be restrained. 

[Dario DCG]

Thank you George,

As you suggested, I've opened an issue in the tracker, including all the information that you mentioned.

[Dario DCG]

If anyone experienced the same issue, downgrading the SDK to version 264 and progressively upgrading it back to 265, 266, 267 and 268 seems to have fixed the problem for me. It's possible that even a simple reinstalllation of the software would have been enough.

[George (Cloud Platform Support)]

Hi Dario, 

Glad to read you have found a solution. You are right, this looks like a corrupted Cloud SDK install, so a simple reinstall of the SDK would have very likely lead to normal behavior again.