GAE Python 3.7 cloud builder issues failing with pip

[Allan Wirth]

I am encountering a strange new issue with building my python 3.7 standard app when I modify the requirements.txt (if I don't modify it, I suppose it's using the cached deps). The issue started perhaps a few weeks ago.

In the build log (on the cloud build page, for the build triggered by 'gcloud app deploy app.yaml cron.yaml`, in the 'builder' step) I get the following failure:

INFO     pip_install_from_wheels took 0 seconds
INFO     starting: pip_install_from_wheels
INFO     pip_install_from_wheels /env/bin/python3.7 -m pip install --no-deps --prefix /tmp/tmp88q0Sy/env /tmp/tmpytZthV/wheel/pip-20.1.1-py2.py3-none-any.whl --disable-pip-version-check
INFO     `pip_install_from_wheels` stdout:
Processing /tmp/tmpytZthV/wheel/pip-20.1.1-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 19.2.3
    Uninstalling pip-19.2.3:
      Successfully uninstalled pip-19.2.3
Successfully installed pip-20.1.1

INFO     pip_install_from_wheels took 1 seconds
INFO     starting: pip_install_from_wheels
INFO     pip_install_from_wheels /env/bin/python3.7 -m pip install --no-deps --prefix /tmp/tmpghxqdf/env /tmp/tmpytZthV/wheel/cffi-1.14.0-cp37-cp37m-manylinux1_x86_64.whl --disable-pip-version-check
INFO     `pip_install_from_wheels` stdout:

INFO     `pip_install_from_wheels` had stderr output:
/env/bin/python3.7: No module named pip

ERROR    error: `pip_install_from_wheels` returned code: 1
INFO     pip_install_from_wheels took 0 seconds
INFO     build process for FTL image took 114 seconds
INFO     full build took 114 seconds
ERROR    `pip_install_from_wheels` had stderr output:
/env/bin/python3.7: No module named pip

error: `pip_install_from_wheels` returned code: 1
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/local/bin/ftl.par/__main__.py", line 65, in <module>
  File "/usr/local/bin/ftl.par/__main__.py", line 60, in main
  File "/usr/local/bin/ftl.par/__main__/ftl/common/ftl_error.py", line 77, in InternalErrorHandler
IOError: [Errno 2] No such file or directory: '""/output'

Note, I DO NOT have pip specified in my requirements.txt. If I freeze with --all (which includes my version of pip, 20.1.1), it also does not work.

However, If I downgrade pip in my virtualenv to 19.2.3 (the version from the error) AND freeze with --all, the build succeeds.

Furthermore, at the top of the build log, I get the following deprecation notice, presumably from https://github.com/GoogleCloudPlatform/cloud-builders

Pulling image: gcr.io/gae-runtimes/python37_app_builder:python37_20200601_3_7_7_RC00

                   ***** NOTICE *****

Alternative official `docker` images, including multiple versions across

multiple platforms, are maintained by the Docker Team. For details, please

visit https://hub.docker.com/_/docker.

However, because I am using the standard python 3.7 environment, as far as I know there is no configuration setting for me to change to docker image used for build/deployment.

Searching, I found the following links with others having similar errors, but I found no official statement from the app engine team:

- https://safecontrols.blog/2019/01/ (says google fixed june 1st)

- https://github.com/GoogleCloudPlatform/cloud-builders/issues/485 (closed WONTFIX and locked May 16)

- https://issuetracker.google.com/issues/131663964 (this says the issue is fixed June 17th)

- https://issuetracker.google.com/issues/159736137 (marked dupe of previous)

Am I doing something wrong? What is the long term solution to selecting pip versions? Is there an official statement I've missed?

Also, how can I fix the deprecation warning on the builder? As far as I know I am not manually selecting anything deprecated.

Thanks,

Allan Wirth

[nikolaosc]

Hello,
In the error I can see "...File "/usr/lib/python2.7/runpy.py",..." but you are using python 3.7. My assuption here is that something happening because of the python versions.

[Sergio Mujica]

add pip==19.2.3 to your rquirements.txt file

[Katayoon (Cloud Platform Support)]

Hi Allan,

If the issue still persists, I recommend to continiue with this Issue Tracker. I have forwarded this issue to the App Engine product team for further investigation. 

Note that Google groups are reserved for general questions on Google Cloud Platform-end products and for reporting an issue, we encourage you to report it to the Public Issue tracker in the future.