Deadline exceeded error when deploying GAE flex app

[Jeff O'Neill]

I've been using GAE flex for about six months, and starting yesterday, I get this error when deploying with gcloud app deploy:

> ERROR: (gcloud.app.deploy) Error Response: [4] Flex operation

> projects/MY-PROJECT/regions/us-central1/operations/xxx

> error [DEADLINE_EXCEEDED]: An internal error occurred while processing

> task

> /appengine-flex-v1/insert_flex_deployment/flex_create_resources>2019-09-04T21:29:03.412Z8424.ow.0:

> Gave up polling Deployment Manager operation

> MY-PROJECT/operation-xxx.

I don't see any helpful info in my logs.

The app runs fine locally and I haven't done any major code changes since the a successful deploy a few days ago.

I wonder if I have somehow inadvertently changed a setting.  I did turn on Cloud Web Security Scanner, but I wouldn't expect that to cause the problem.

I set cloud_build_timeout to 600 (what I think is the default anyway) but that didn't help.

Any ideas how to fix this?  Should I set custom values for initial_delay_sec or app_start_timeout_sec?

[Jeff O'Neill]

I've been trying everything I can think of to fix this with no luck.  Here are some things I tried:

• deploying the exact code that previously deployed successfully
• deployed into a new app engine project
• increased cloud_build_timeout to 1200 (though I've read elsewhere that >600 doesn't work)
• maxed out all deadlines I could find and added the stuff below to my app.yaml

My app doesn't do anything out of the ordinary.  The build requires compiling C code (e.g., Spacy, Numpy) but nothing too complicated.

Any advice would be greatly appreciated.

=====

readiness_check:

  check_interval_sec: 300

  timeout_sec: 299

  failure_threshold: 2

  success_threshold: 2

  app_start_timeout_sec: 1800

liveness_check:

  check_interval_sec: 300

  timeout_sec: 299

  failure_threshold: 2

  success_threshold: 2

[Jeff O'Neill]

Now I tried building my own docker image, pushed the image to us.gcr.io and deployed from there.

I get the same error from my original email.

Hopefully, this is enough info for someone to give me some advice.

On Friday, September 6, 2019 at 7:57:05 AM UTC-4, Jeff O'Neill wrote:

[Jeff O'Neill]

I figured it out finally, and it is an ugly bug...

About 6 months ago, I set a location policy of US only.  The policy included US locations that existed at that time, but I think the policy did not update with new US locations as they became available.

My GAE app was being deployed to "us-central1-f" (I did not specifically pick this location; I had previously picked us-central).  This location was not in my location policy because I presume it did not exist then.  I removed my location policy, and I can now deploy my app.

I never got a meaningful error message in deploying my own app.  For a sanity check, I tried deploying Google's "hello world" GAE flex app, and that app gave an error message that included this:

> Location ZONE:us-central1-f violates constraint constraints/gcp.resourceLocations

So I was able to figure it out.

Could a Google Engineer please make an official bug report of this?

On Friday, September 6, 2019 at 7:57:05 AM UTC-4, Jeff O'Neill wrote:

[Aref Amiri (Cloud Platform Support)]

Hi Jeff,

I have reported this issue to the App Engine engineering team for further investigation. I have also created a Public Issue Tracker to follow the progress of the issue. You can star the page to be notified of the latest updates on the issue, as future communication will be shared on that thread.