using 3rd party libraries on GAE standard with Python

[Dewey Gaedcke]

The amount of conflicting info about Python/GAE best practices is very frustrating.

Can someone please clarify:

Should we use the vendored approach:

from google.appengine.ext import vendor

vendor.add('lib')

Or should we use VENV?

And is there a hybrid approach using both to accommodate most Python IDEs?

If we use VENV (because our IDE requires it), how do we reconcile the two?

In other words, VENV is going to install modules inside of the "env/" directory where-as most tutorials instruct us to use:

pip install --upgrade --requirement requirements.txt --target lib/

Does the above advise change based on Python2.7 vs 3.7

Does it change when moving to Flex or Compute?

Google PLEASE!!! Remove all this contradictory info and clarify the CURRENT recommended approach.

[Dewey Gaedcke]

running on GAE-Std (Production), I'm getting this error:

mport_app __import__(module) File "/srv/main.py", line 3, in <module> from appengine_config import gateway # braintree File "/srv/appengine_config.py", line 2, in <module> from google.appengine.ext import vendor ModuleNotFoundError: No module named 'google.appengine'

[Stewart Reichling]

The `google.appengine` libraries are not supported in Python 3 on the App Engine standard environment. Those libraries are only supported in Python 2. Note that Python 2 will become unsupported by the Python language community on January 1, 2020. I would recommend that you use Python 3, not Python 2.

[Stewart Reichling]

Hi Dewey,

This page explains how to manage dependencies using Python 3. In general, the process looks like this:

• declare dependencies in a `requirements.txt` file (see "Declaring and managing dependencies")
• if you want to run/test your application locally, follow the steps to set up a virtual environment (see "Installing dependencies locally")
• when you deploy your app, only the dependencies listed in `requirements.txt` are installed. For all practical purposes, the contents of your env/ directory are ignored.

Note that use of `requirements.txt` and, if desired, `virtualenv` conforms to standard Python tooling. You shouldn't need to learn anything specific for App Engine. This is the current recommended approach. It is the same recommendation for the App Engine flexible environment and for Google Compute Engine.

Also note that the story is a bit different if you are using the Python 2.7 runtime on App Engine. This is a legacy runtime and does not conform to modern standards in Python development. My advice would be to use the Python 3.7 runtime on App Engine, not the Python 2.7 runtime.

Stewart

[Phillip Pearson]

Sorry for the confusion!  

The vendored approach (installing into lib/ and adding an appengine_config.py file) is correct for Python 2.7 on the standard environment:

  https://cloud.google.com/appengine/docs/standard/python/tools/using-libraries-python-27

Python 3.7 on the standard environment and all Python versions on the flexible environment use virtualenv and will install libraries specified in your requirements.txt, so you don't need to create the lib folder, and appengine_config.py is ignored:

  https://cloud.google.com/appengine/docs/standard/python3/specifying-dependencies

  https://cloud.google.com/appengine/docs/flexible/python/runtime#dependencies

Compute Engine is a whole different system that works in terms of VMs rather than apps, so none of this really applies there, I'm afraid.

Hope this helps!

Phil

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/e81c0386-af58-4b4d-b065-b655a9dad6af%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dewey Gaedcke]

Thanks for the response and clarification!!

I remember being told way back NOT to use venv with GAE & so all these posts where it is now being shown is very confusing.

Using Py3.7, I'm now getting:

ERROR: Python 3 and later is not compatible with the Google Cloud SDK. Please use Python version 2.7.x.

[Rahul Ravindran]

Could you paste the entire command surface?

Additionally, which version of Google Cloud SDK are you using?

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/d46e86e4-7494-46f0-97d4-d6c9e6bed2a7%40googlegroups.com.

[Dewey Gaedcke]

$ make serve

export CLOUDSDK_PYTHON=/Users/dgaedcke/dev/client/nmg/nmg_payments_api/env/bin

dev_appserver.py --clear_datastore 0 --logs_path=/tmp/gaelogs --log_level=warning \

--host 0.0.0.0 app.yaml

ERROR: Python 3 and later is not compatible with the Google Cloud SDK. Please use Python version 2.7.x.

If you have a compatible Python interpreter installed, you can use it by setting

the CLOUDSDK_PYTHON environment variable to point to it.

gcloud --version

Google Cloud SDK 217.0.0

app-engine-go 

app-engine-python 1.9.75

app-engine-python-extras 1.9.74

bq 2.0.34

cloud-datastore-emulator 2.0.2

core 2018.09.17

gsutil 4.34

Updates are available for some Cloud SDK components.

[Rahul Ravindran]

Unfortunately, dev_appserver does not yet work with Python 3.x(See https://cloud.google.com/appengine/docs/standard/python3/testing-and-deploying-your-app#local-dev-server). You need to run it from a virtualenv which is running python 2.7.

Alternatively, as specified in https://cloud.google.com/appengine/docs/standard/python3/testing-and-deploying-your-app, you can choose to not use dev_appserver at all.

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/11c13a74-2620-492c-919e-9481cb814894%40googlegroups.com.

[Vitaly Bogomolov]

My five cents.

I think, that real power of Google Cloud is a Standard Environment. But, in other hand, it is a sandbox with strong restrictions. You must use rules of this sanbox (vendor.add and so on). And, you must forgot about Python3 on Standard Environment.

If you care about Python3, you should use Flexible Environment, that is "different system that works in terms of VMs rather than apps", as it was said above.

Of course, this is just my opinion.

WBR, Vitaly.

[Stewart Reichling]

Both Python 3 environments (standard and flexible) are "vanilla" Python environments that use the open source runtime. Python 3 on the standard environment does not have the same restrictions as Python 2 on the standard environment.

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/c3124c8d-49e4-4ea7-8cfa-e5884d692b2b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Stewart Reichling | Product Manager, Serverless | stew...@google.com | 650.906.3381

[Dewey Gaedcke]

does anyone recognize these errors upon deploying a Golang project on Std.

com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:459)\\n\\tat com.google.gse.internal.DispatchQueueImpl$WorkerThread.run(DispatchQueueImpl.java:403)\\n\"",
            "  } ],",
            "  \"message\" : null",
            "}",
            "\t\tat com.google.apps.framework.producers.PresentImpl.get(PresentImpl.java:29)",
            "\t\tat com.google.apphosting.admin.flex.lib.dm.InsertDmDeploymentProducerModule.produceInsert(InsertDmDeploymentProducerModule.java:75)",
            "\tSuppressed: CriticalInputFailure: com.google.apphosting.admin.flex.lib.dm.InsertDmDeploymentProducerModule.produceDoneOrFailedOperation failed while trying to inject @com.google.apps.framework.producers.PrivateVisibility(annotation=\"@com.google.apphosting.admin.flex.lib.dm.InsertDmDeploymentProducerModule$InsertDmOperation\", module=com.google.apphosting.admin.flex.lib.dm.InsertDmDeploymentProducerModule.class) com.google.apphosting.admin.flex.lib.dm.internal.proto.Operation",
            "\tSuppressed: com.google.common.util.concurrent.UncheckedExecutionException: ",
            "\t\tat com.google.apps.framework.producers.Present.checkSucceeded(Present.java:54)",
            "\t\tat com.google.apphosting.admin.flex.resources.deployment.CreateResourcesProducerModule.produceBackendService(CreateResourcesProducerModule.java:128)",
            "\tSuppressed: CriticalInputFailure: com.google.apphosting.admin.flex.resources.deployment.CreateResourcesProducerModule.produceDeployedRecord failed while trying to inject @com.google.apps.framework.producers.PrivateVisibility(annotation=\"@com.google.apphosting.admin.flex.resources.deployment.CreateResourcesProducerModule$NewBackendService\", module=com.google.apphosting.admin.flex.resources.deployment.CreateResourcesProducerModule.class) java.util.Optional<com.google.api.services.compute.model.BackendService>"
          ]
        }
      }

[Dan S (Cloud Platform Support)]

Hi,

It seems that you’re experiencing an issue regarding access permissions(Java:403) and maybe you have used a not supported library (459 issue).

Kindly note that the Google Groups channel is designed to provide a free expression space to Google community rise discussions on different matters.

I believe that your questions should get a better assistance in the Stack Overflow[1] since it is related to a technical issue. There, you should receive help from the community fellow enthusiasts, and from Google engineers when you add a tag in your post according to the product where you require assistance, as you can check in our documentation[2]. Additionally, you can post some details regarding your scaling setup, project config, and some code samples.

[1] http://stackoverflow.com/

[2] https://cloud.google.com/support/docs/stackexchange