Error Response: [13] The system encountered a fatal error

[Jose Montoya]

Hi App Engine Team,

We have been using App Engine normally and just this weekend start to fail the update. 

We update from a Docker container, so there is no cloud build and the app.yaml file has not changed.

After reviewing some suggestions online, we have tried:

• Updating the gcloud SDK
• Removing old versions in the App Engine console
• Validating the docker container into a VM in GCloud
• Run this quickstart in the same project

The problem is that the error is very generic. Do you have any suggestions?

Updating service [default] (this may take several minutes)...-DEBUG: Operation [apps/-----/operations/7162ec80-93f3-4f4e-b555-2ad82e76e90a] not complete. Waiting to retry.
Updating service [default] (this may take several minutes)...|DEBUG: Operation [apps/-----/operations/7162ec80-93f3-4f4e-b555-2ad82e76e90a] complete. Result:
{
    "done": true,
    "error": {
        "code": 13,
        "message": "The system encountered a fatal error"
    },
    "metadata": {
        "@type": "type.googleapis.com/google.appengine.v1.OperationMetadataV1",
        "endTime": "2022-01-25T13:24:24.274Z",
        "insertTime": "2022-01-25T12:58:17.876Z",
        "method": "google.appengine.v1.Versions.CreateVersion",
        "target": "apps/-----/services/default/versions/20220125t135813",
        "user": "--------@gmail.com"
    },
    "name": "apps/-----/operations/7162ec80-93f3-4f4e-b555-2ad82e76e90a"
}
Updating service [default] (this may take several minutes)...failed.
DEBUG: (gcloud.app.deploy) Error Response: [13] The system encountered a fatal error
Traceback (most recent call last):
  File "D:\Program Files (x86)\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\calliope\cli.py", line 987, in Execute
    resources = calliope_command.Run(cli=self, args=args)
  File "D:\Program Files (x86)\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\calliope\backend.py", line 809, in Run
    resources = command_instance.Run(args)
  File "D:\Program Files (x86)\Google\Cloud SDK\google-cloud-sdk\lib\surface\app\deploy.py", line 123, in Run
    return deploy_util.RunDeploy(
  File "D:\Program Files (x86)\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\command_lib\app\deploy_util.py", line 680, in RunDeploy
    deployer.Deploy(
  File "D:\Program Files (x86)\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\command_lib\app\deploy_util.py", line 461, in Deploy
    self.api_client.DeployService(new_version.service, new_version.id,
  File "D:\Program Files (x86)\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\api_lib\app\appengine_api_client.py", line 216, in DeployService
    return operations_util.WaitForOperation(
  File "D:\Program Files (x86)\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\api_lib\app\operations_util.py", line 307, in WaitForOperation
    completed_operation = waiter.WaitFor(
  File "D:\Program Files (x86)\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\api_lib\util\waiter.py", line 261, in WaitFor
    operation = PollUntilDone(
  File "D:\Program Files (x86)\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\api_lib\util\waiter.py", line 322, in PollUntilDone
    operation = retryer.RetryOnResult(
  File "D:\Program Files (x86)\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\core\util\retry.py", line 249, in RetryOnResult
    if not should_retry(result, state):
  File "D:\Program Files (x86)\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\api_lib\util\waiter.py", line 320, in _IsNotDone
    return not poller.IsDone(operation)
  File "D:\Program Files (x86)\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\api_lib\app\operations_util.py", line 182, in IsDone
    raise OperationError(requests.ExtractErrorMessage(
googlecloudsdk.api_lib.app.operations_util.OperationError: Error Response: [13] The system encountered a fatal error
ERROR: (gcloud.app.deploy) Error Response: [13] The system encountered a fatal error

[babdelrahman]

Hello, 

Is this is a Flex deployment? 

Were there any Org policy changes ?

Document [1] : 

You cannot use the App Engine flexible environment if the organization policy on your project restricts access to external IPs, for example:

• The effective policy for constraints/compute.vmExternalIpAccess is set to DENY_ALL.
• The effective policy for constraints/compute.vmExternalIpAccess is set to allow only specific VM instances.

Please check the Organization policy for your project, following steps in [1].

[1] https://cloud.google.com/appengine/docs/flexible/known-issues

[2] https://cloud.google.com/resource-manager/docs/organization-policy/using-constraints#override_boolean_policy

[Adriana Capua]

Hi, we are currently experiencing this same issue. May I know how this got fixed? Would really appreciate a response. Thank you!

btw- we have already submitted a ticket to Google for this but they haven't responded back yet.