GAE deployment throwing Error Response: [13] The system encountered a fatal error

[Adriana Capua]

Wondering if anybody here encountered this error when deploying to app engine? There are no changes with the app.yml file config and this was working before.

command used: gcloud app deploy --quiet --no-promote --no-stop-previous-version --version={VERSION_NUMBER}

I have submitted an issue here: https://issuetracker.google.com/issues/233865148

[Ernesto Contreras Pinon]

This error is rather generic and can be caused by multiple factors. Have you read past issues like this that have possible solutions? In this sample, the issue was solved through re-associating the App Engine service account with the app, for example. On a different occasion, it appears to be caused by transient network errors, are you still unable to deploy the version and are seeing the same error?

Otherwise, can you give more details about the changes that you added in the last deployment which failed? Confirming whether all your App Engine Flex quotas are under the limit would also be useful, which you can confirm by searching on the Cloud Audit logs for any quota exceeded errors.

[Adriana Capua]

Here's the part of the logs. Also, the weird thing is we are getting different error messages. Last time we were getting the error "[13] The system encountered a fatal error" and sometimes "[14] Cloud Container build either failed or interrupted.". For this instance when I tried to do a deployment again, it's the error 14 again.

[Adriana Capua]

Hi Ernesto,

Thanks a lot for the response. 

We have already tried the solutions we found online ie. validated billing account is okay, and adding permissions. etc. Specifics are below:

1. Verified almost all other threads
2. App.yaml has not defined min or max instance...just instances count is defined as 1
3. Not a transient network errors as we have tried it almost 15-20 times  with no network issues
4. Reg the code changes....even after reverting the code back to what we had in the successful build it's failing
5. Quotas are still under the limit

app.yaml:

runtime: custom

env: flex

resources:

  cpu: 1

  memory_gb: 2

  disk_size_gb: 15

manual_scaling:

  instances: 1

I will send the full logs as well.

One thing we haven't tried is re-associating the App Engine account to the app. However, we can't access this link in your comment. https://issuetracker.google.com/issues/201556420 Would you be able to give me access to view it or screenshots of the resolution if the access is not possible?

Many thanks!

On Friday, May 27, 2022 at 5:07:30 AM UTC+8 Ernesto Contreras Pinon wrote:

[Ernesto Contreras Pinon]

The solution from the IssueTracker URL you were not able to access is also published in this Stackoverflow thread. You can try to check if it helps, but from the new error that you have received, it seems this is more of an issue related to the process of building your custom runtime on App Engine Flex.

How long is the build process taking? In case it’s more than the default of 10 minutes, you’d have to set a larger timeout, which can be done alternatively with this command: “gcloud config set app/cloud_build_timeout <TIMEOUT_SECS>”. Otherwise, doing a fresh build without using cached layers, could also help. You can use the “--no-cache” flag when running gcloud app deploy to test this out. Moreover, have you also seen any difference when allocating additional memory and CPU to your App Engine instance?