Google App deploy fails on timeout: ERROR: CONTEXT DEADLINE EXCEEDED. How can i fix this?

[K. Ripping]

Hi All,

Since today I am unable to deploy my Google App Engine. The strange thing is, I've been using these same settings for months and had no issues.

I've added the log and app.yaml.

In the log, you will find warnings like:

Step #1: /usr/local/lib/python2.7/dist-packages/virtualenv.py:1041: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative usesStep #1: import imp

Step #1: google-auth 1.13.1 has requirement setuptools>=40.3.0, but you'll have setuptools 39.1.0 which is incompatible.

The actual error:

ERROR: (gcloud.app.deploy) Cloud build failed. Check logs at2 Failure status: UNKNOWN: Error Response: [4] DEADLINE_EXCEEDED

I'm not sure if it would solve the issue, but I've been trying to find a way to update setuptools, but on my own environment everything is up-to-date. 

2 questions

- How can i solve the error and deploy my app?

- What setuptools is google shell using when deploying?

Thanks,

K. Ripping

[Dustin Ingram]

Hi there, can you share your `requirements.txt` file with us as well?

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/ca106ff4-5dba-4f59-82f9-efbdb4de49c4%40googlegroups.com.

[K. Ripping]

Hi Dustin,

Yes, attached also the requirements.txt.

I've changed the timeout settings for deploying the app, which seems to solve the timeout error. 

Though i still get these lines:

Step #1: ?[91mYou are using pip version 10.0.1, however version 20.0.2 is available.

Step #1: You should consider upgrading via the 'pip install --upgrade pip' command.?[0m?[91m

Step #1: ?[91mgoogle-auth 1.13.1 has requirement setuptools>=40.3.0, but you'll have setuptools 39.1.0 which is incompatible.

When i look at the installed requirements on the cloud shell, using pip freeze i get the following lines:

Python 2 is deprecated. Upgrade to pip3 as soon as possible. 

See https://cloud.google.com/python/docs/python2-sunsetTo suppress this warning, create an empty ~/.cloudshell/no-pip-warning file.The command will automatically proceed in 5 seconds or on any key.********************************************************************************DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-supportabsl-py==0.9.0appdirs==1.4.3applicationinsights==0.11.9astor==0.8.1attrs==19.3.0autopep8==1.5backports.csv==1.0.7backports.shutil-get-terminal-size==1.0.0backports.weakref==1.0.post1cachetools==3.1.1certifi==2019.11.28chardet==3.0.4

cli-helpers==0.2.3Click==7.0configobj==5.0.6configparser==4.0.2contextlib2==0.6.0.post1crcmod==1.7decorator==4.4.2distlib==0.3.0enum34==1.1.10filelock==3.0.12funcsigs==1.0.2functools32==3.2.3.post2future==0.18.2futures==3.3.0gast==0.2.2google-api-core==1.16.0google-api-python-client==1.8.0google-auth==1.12.0google-auth-httplib2==0.0.3google-auth-oauthlib==0.4.1google-cloud-bigquery==1.24.0google-cloud-core==1.3.0google-cloud-datastore==1.11.0google-cloud-language==1.3.0google-cloud-logging==1.15.0google-cloud-spanner==1.15.0google-cloud-storage==1.26.0

google-cloud-translate==2.0.1google-cloud-videointelligence==1.14.0google-cloud-vision==1.0.0google-pasta==0.2.0google-resumable-media==0.5.0googleapis-common-protos==1.51.0grpc-google-iam-v1==0.12.3grpcio==1.27.2h5py==2.10.0httplib2==0.17.0humanize==1.0.0idna==2.9importlib-metadata==1.6.0importlib-resources==1.4.0invoke==1.4.1ipaddr==2.2.0ipython==5.9.0ipython-genutils==0.2.0jedi==0.15.2jsonschema==3.2.0Keras-Applications==1.0.8Keras-Preprocessing==1.1.0Markdown==3.1.1mccabe==0.6.1meld3==1.0.2mercurial==4.0

mock==3.0.5mssql-cli==0.18.2numpy==1.16.6oauth2==1.9.0.post1oauth2client==4.1.3oauthlib==3.1.0opt-einsum==2.3.2parso==0.6.2pathlib2==2.3.5pexpect==4.8.0pickleshare==0.7.5pluggy==0.13.1prompt-toolkit==1.0.18protobuf==3.11.3ptyprocess==0.6.0pyasn1==0.4.8pyasn1-modules==0.2.8pycodestyle==2.5.0pydocstyle==3.0.0pyflakes==2.1.1Pygments==2.5.2pyrsistent==0.16.0python-jsonrpc-server==0.3.4python-language-server==0.31.9pytz==2019.3requests==2.23.0requests-oauthlib==1.3.0

rope==0.16.0rsa==4.0scandir==1.10.0scipy==1.2.2simplegeneric==0.8.1singledispatch==3.4.0.3six==1.14.0snowballstemmer==2.0.0sqlparse==0.2.4supervisor==3.3.1tensorboard==2.1.0tensorflow==2.1.0tensorflow-estimator==2.1.0termcolor==1.1.0terminaltables==3.1.0traitlets==4.3.3typing==3.7.4.1ujson==1.35uritemplate==3.0.1urllib3==1.25.8virtualenv==20.0.15wcwidth==0.1.9Werkzeug==1.0.1wrapt==1.12.1yapf==0.29.0zipp==1.2.0

[David (Cloud Platform Support)]

“google-auth 1.13.1 has requirement setuptools>=40.3.0, but you'll have setuptools 39.1.0 which is incompatible” and “You are using pip version 10.0.1, however version 20.0.2 is available” are warning messages that show up as the version for these libraries are updated gradually and might not always be on the latest version. However, it should not produce any issues at the moment of deploying or any issues with the application itself.

[Shai Ben Tovim]

Having the exact same issue starting today:

code that deployed several times to GAE python-flex (same code same python dependencies) fails deploying due to same errors mentioned above by K. Ripping:

Failure status: UNKNOWN: Error Response: [4] DEADLINE_EXCEEDED

I can also see this error on the log (same as above):

Step #1: google-auth 1.13.1 has requirement setuptools>=40.3.0, but you'll have setuptools 39.1.0 which is incompatible.

I did notice that this process took much longer than usual:

Step #1:   Running setup.py bdist_wheel for grpcio: started
Step #1:   Running setup.py bdist_wheel for grpcio: still running...
Step #1:   Running setup.py bdist_wheel for grpcio: still running...
Step #1:   Running setup.py bdist_wheel for grpcio: still running...
Step #1:   Running setup.py bdist_wheel for grpcio: still running...
Step #1:   Running setup.py bdist_wheel for grpcio: still running...
Step #1:   Running setup.py bdist_wheel for grpcio: still running...
Step #1:   Running setup.py bdist_wheel for grpcio: still running...
Step #1:   Running setup.py bdist_wheel for grpcio: finished with status 'done'

[Shai Ben-Tovim]

Hi David,

It seems the problem is that the GAE Cloud Build process timesout and hence the Error Response: [4] DEADLINE_EXCEEDED.

One of the python google-cloud packages (probably pubsub) is trying to pull this package: grpcio.

The problem is that the grpcio package was updated a few days ago (April 3rd) to ver 1.28.1 that doesn't have a pre-compiled wheel version for the base GAE flex python image.

This results in the grpcio package being compiled from source during the GAE service deploy which takes well over 8 minutes causing the whole process to time out (seems the default timeout is 10 min)

I understood this is only happening now because in the past (before April 3rd) grpcio ver 1.27.1 was pulled and it does have a compiled wheel.

Shai

[David (Cloud Platform Support)]

Hello,

I’m seeing this excessive time now with Pub/Sub and Cloud Task libraries. You can follow this IssueTracker that I have created to raise awareness with the App Engine Flex engineering team. You can also report this yourself from an user perspective on Github. As for now, the workaround is to increase the timeout limit.

[K. Ripping]

Hi Shai,

As a workaround for the timeout issue, you can use following fix in your terminal:

gcloud config set app/cloud_build_timeout [ENTER SECONDS]

eg. i've set the timeout to 1200s, which should be enough.

This doesn't however solve the fact that the server that is building the app is working with an old version of Python and thus PIP and Setuptools.

Anyone who can help us fix that?

Koen

[Shai Ben-Tovim]

Thanks David.

I opened a GitHub issue: https://github.com/googleapis/google-cloud-python/issues/10407

[Shai Ben-Tovim]

Hi Keon,

This is exactly what I did (extend the deploy timeout) to work around the problem for now.

Shai

[Jason Brancazio]

Is there a way to set this configuration in Cloud Build? I can work around the problem by triggering builds locally, but builds triggered by our CI process, e.g. tagging a branch, are timing out.

[Probir Sil]

Hi Koen,

I am having the same issue. Even after increasing timeout to 1200s, it seems it is timing out around 10 min. I just restarted the build by going to Cloud build -> History -> Clicking on the build -> Restart. Do I need to do a fresh gcloud app deploy?

Thanks,

Probir

[David (Cloud Platform Support)]

Another workaround other than increasing the timeout limit would be adding “grpcio==1.27.2” to your requirements.txt and then try to redeploy

[Jason Brancazio]

For users specifically struggling to get an App Engine build triggered by Cloud Build to respect the timeouts specified in the build step yaml, using bash in the Cloud Build step worked for me:

- name: 'gcr.io/cloud-builders/gcloud'
  entrypoint: 'bash'
  args: ['-c', 'gcloud config set app/cloud_build_timeout 1800 && gcloud app deploy my-service/my-config.yaml']

Setting "timeout:" in the build step didn't work for me, perhaps because it doesn't actually call gcloud config set app/cloud_build_timeout.   

I found the tip here:

https://stackoverflow.com/questions/60070274/why-cant-i-override-the-timeout-on-my-google-cloud-build

[Matthew Svensson]

Thanks David!  That worked perfect for me!  I'll have to watch grpcio to see what caused this change.

[K. Ripping]

Great tip David, thanks! Also starred your raised issue.

[Ted song]

Thanks, David

works for me 

[Rossco]

I am attempting to deploy from my local machine to app engine standard, and nothing I do seems to increase the timeout from 10 mins, every deploy fails with the error 

ERROR: (gcloud.app.deploy) Error Response: [4] Cloud build did not succeed within 10m.

Any suggestions appreciated!

My cloudbuild.yaml is

steps:

- name: gcr.io/cloud-builders/npm

  args: [ install ]

- name: 'gcr.io/$PROJECT_ID/ng'

  args: ['build', '--prod']

- name: gcr.io/cloud-builders/gcloud

  args: [ app, deploy ]

  timeout: 2000s

timeout: 3600s

[Olu]

Hi, Rossco

I think the same concern was raised on this issue link[1] and the issue still seems to be evaluated. A workaround was provided on the link, however. I suggest you attempt the steps provided on the issue link using the [--timeout=DURATION] flag with gcloud builds submit [2] to change the timeout threshold.

[1]https://issuetracker.google.com/131889237
[2]https://cloud.google.com/sdk/gcloud/reference/builds/submit