What's the "right" way to import from `google.xxx` packages?

[Jeffrey Eliasen]

I have a project that works in production just fine. I recently encountered a need to publish messages to PubSub and cannot for the life of me figure out how to install and reference packages in the `google.xxx` hierarchy. When I `pip install` such a package, my site-packages directory in my virtual environment receives a directory called `google` and this thwarts existing imports that rely on the packages inside the SDK (because that namespace is lower in priority than my venv).

I figure one of the following must be true:

1. I should not be installing the google packages using pip... in this case, how *should* I install them?
2. I need to install *everything* explicitly into my venv so that there is no fallback into the SDK packages... in this case, what's the full list of imports I should use for djangae to work?
3. I am missing something obvious... in this case, please thump me across the side of the head and then tell me what I'm doing wrong :)

Thanks in advance for any suggestions!

[Jacob G]

You can install google packages using pip. Scaffold your project using djangae-scaffold, and then add this code to the fix_path function in boot.py:

# google-auth is installed in sitepackages, but its package name `google`
# conflicts with the app engine sdk `google` package, so we need to add
# it to the sdk `google` package path.
import google
google_auth_path = join(PROD_SITEPACKAGES_DIR, 'google')
if google_auth_path not in google.__path__:
    google.__path__ = [google_auth_path] + google.__path__

[Jacob G]

Also I'm not sure you need venv for local app engine development, because the djangae-scaffold already sandboxes your sitepackages under the project folder.

[David Buxton]

Alternative approach is to use the JSON API with google-api-python-client. Example.

Last time I tried to use the google.cloud.pubsub package on App Engine standard 2.7, I concluded that not using google.cloud.pubsub was easier (I think there were further dependency problems, even after sorting out the package namespace problem).

Dave

[Jacob G]

Yeah I am also using google-api-python-client for pubsub. But there is also some google.xxx packages in there from what I remember.

[David Buxton]

I think it has the option to use google.auth.* - but I _think_ the default is to pull in oauth2client as a dependency. Not sure.

--
You received this message because you are subscribed to a topic in the Google Groups "djangae-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/djangae-users/LbjWPlmwqbs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to djangae-user...@googlegroups.com.
To post to this group, send email to djanga...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/djangae-users/687e07e3-9b81-4fc3-a00f-77648242e63a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

David Buxton
US mobile +1 628-202-9753

UK mobile  +44 (0)7905-645025
Email    da...@gasmark6.com

[David Buxton]

Looking at the commit history, google-api-python-client has purged the dependency on oauth2client as of version 1.7.0 (published July 2018). If you want to avoid the google.auth package, use 'google-api-python-client==1.6.7'.

(Most recent project I used this for we had pinned google-api-python-client before the 1.7.x releases.)

Dave

[Jeffrey Eliasen]

I ended up using this approach to solve the problem simply right away, and will investigate the more complete solution in the future. Thanks for all the suggestions, David and Jacob!

[Jacob G]

I didn't realize I was unnecessarily bringing in both oauth2client and google-auth. It looks like google-api-python-client is supporting both now, but oauth2client is deprecated. I ended up removing oauth2client, and now am just using google-auth. The fix_path code I posted earlier is necessary to use google-auth.

The change to my code to publish to PubSub using google-auth instead of oauth2client is easy enough:

import base64
import logging
import threading

from django.utils.encoding import force_bytes
from google.auth import app_engine
from googleapiclient import discovery

PUBSUB_SCOPES = ["https://www.googleapis.com/auth/pubsub"]

client_store = threading.local()

def publish_message(topic, data, attributes=None):
    body = {
        'messages': [
            {
                'attributes': attributes,
                'data': base64.b64encode(force_bytes(data))
            }
        ]
    }
    _get_client().projects().topics().publish(topic=topic, body=body).execute()
    logging.info('message published to pubsub topic {}'.format(topic))

def _get_client():
    if not hasattr(client_store, 'client'):
        credentials = app_engine.Credentials()
        if credentials.requires_scopes:
            credentials.with_scopes(scopes=[PUBSUB_SCOPES])
        client_store.client = discovery.build('pubsub', 'v1', credentials=credentials)
    return client_store.client

[Jacob G]

I spoke too soon. Remote shell requires oauth2client, at least in the GAE SDK version 57 that we have to use with djangae. Also, the latest version of urllib3 1.24 does not work with the old GAE SDK we are using, but version 1.23 does work. It looks like this commit broke it:

https://github.com/urllib3/urllib3/pull/1450/commits/01407533cbee3293043e13ce1eed8c00c063fe9f

[Jeffrey Eliasen]

So after digging around some, I've hit an interesting block. First, I don't use djangae-scaffold (mostly because I have a lot of tooling that works really well for everything from boilerplate to production and it was all built before I'd discovered the scaffold project)... so my setup is a little different. I'm happy to share it if that helps.

What works: Everything imports fine for ./manage.py shell, including all imports, all actual calls into google's libraries, etc. If I don't import any google libraries, my existing setup is nearly perfect (except for pesky softlinks, trying to figure a way out of that challenge as well).

What doesn't work: When I have installed any google libraries, things break as soon as wsgi.py is called, for reasons I can't quite make out. This is only true when using extra google libraries; if I haven't pip installed anything to the google namespace then all is well and has worked reliably like this for as long as I've been using djangae.

Basic layout:

project_root

site

manage.py

wsgi.py

config

settings.py

urls.py

apps

app1

app2

...

appN

site-packages

django (soft link to django in virtual environment site-packages)

djangae (soft link to djangae in virtual environment site-packages)

google (soft link to google in virtual environment site-packages)

...

env

(virtual env created any of the usual ways)

google_appengine

(downloaded using the same logic as install_deps uses)

My manage.py:

if __name__ == "__main__":

os.environ['DJANGAE_APP_YAML_LOCATION'] = os.path.abspath(os.path.dirname(__file__))

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'config.settings')

fix_path(include_dev=True)

from djangae.core.management import execute_from_command_line, test_execute_from_command_line

if 'test' in sys.argv:

test_execute_from_command_line(sys.argv)

else:

execute_from_command_line(sys.argv)

My fix_path:

def fix_path(include_dev=False):

PROJECT_PATH = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))

APPENGINE_PATH = os.path.join(PROJECT_PATH, 'google_appengine')

if include_dev:

sys.path.insert(1, APPENGINE_PATH)

VENV_PATH = os.path.join(PROJECT_PATH, 'env', 'lib', 'python2.7', 'site-packages')

if VENV_PATH not in sys.path:

sys.path.append(VENV_PATH)

sys.path.insert(1, os.path.join(PROJECT_PATH, 'site', 'site-packages'))

Without google/* imports and if running in the virtual env, I only needed to add the single line to install project_root/site/site-packages to the 1th element of sys.path. Historically I got google_appengine from my google-cloud-sdk path and didn't explicitly add that to the path. So my old setup had the last line of fix_path directly in manage.py where the call to fix_path is now placed.

The rest of the logic (other than the last line) is new and is putting google_appengine/* into sys.path and also bringing in the rest of site-packages if in 'dev' mode and running from outside the virtualenv so that all unit tests and such work.

My legacy wsgi.py always looked like this:

from google.appengine.ext import vendor

vendor.add('site-packages')

import os # noqa

from django.core.wsgi import get_wsgi_application # noqa

from djangae.wsgi import DjangaeApplication # noqa

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'config.settings')

application = DjangaeApplication(get_wsgi_application())

... and this worked great.

With google/* everything breaks when running the server locally. Most notably, if I break in manage.py and again in wsgi.py:

(env) $ python site/manage.py runserver

> /path/to/project_root/site/manage.py(16)<module>()

-> execute_from_command_line(sys.argv)

(Pdb) import google

(Pdb) google.__path__

['/path/to/project_root/google_appengine/google', '/path/to/project_root/env/lib/python2.7/site-packages/google']

(Pdb) c

INFO 2019-04-17 14:12:54,632 blobstore_service.py:74] Starting blobstore service on localhost:8012

> /path/to/project_root/site/apps/utils/transcription.py(5)<module>()

-> from google.cloud.speech import enums

(Pdb) c

Validating models...

Here' I've broken into the debugger in manage.py and can see that google.__path__ is what I expect... then when transcribe.py is loaded, I can successfully `from google.cloud.speech import enums` (and the actual functions in this module work just fine when running as shell).

... continuing:

System check identified no issues (0 silenced).

April 18, 2019 - 00:12:58

Djangae version 0.9.12

Django version 1.11.20, using settings 'config.settings'

Starting development server at http://127.0.0.1:8000/

Quit the server with CONTROL-C.

INFO 2019-04-18 00:12:58,332 devappserver2.py:116] Skipping SDK update check.

... truncated

> /path/to/project_root/site/wsgi.py(12)<module>()

-> application = DjangaeApplication(get_wsgi_application())

(Pdb) import google

(Pdb) google.__path__

['/path/to/project_root/env/lib/python2.7/site-packages/google', '/path/to/project_root/google_appengine/google']

(Pdb) c

Here I've broken in wsgi.py. At this point, if I inspect google.__path__, I see that (for reasons that aren't clear) the copy in my site-packages is first, and the copy in google_appengine is second.

... continuing by loading http://localhost:8000/ in the browser:

> /path/to/project_root/site/apps/utils/transcription.py(5)<module>()

-> from google.cloud.speech import enums

< now that we're under wsgi.py's control, the same line breaks... >

(Pdb) c

ERROR 2019-04-18 00:13:23,980 wsgi.py:263]

Traceback (most recent call last):

File "/path/to/project_root/google_appengine/google/appengine/runtime/wsgi.py", line 240, in Handle

handler = _config_handle.add_wsgi_middleware(self._LoadHandler())

File "/path/to/project_root/google_appengine/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler

handler, path, err = LoadObject(self._handler)

File "/path/to/project_root/google_appengine/google/appengine/runtime/wsgi.py", line 85, in LoadObject

obj = __import__(path[0])

File "/path/to/project_root/site/wsgi.py", line 12, in <module>

application = DjangaeApplication(get_wsgi_application())

File "/path/to/project_root/site/site-packages/django/core/wsgi.py", line 13, in get_wsgi_application

django.setup(set_prefix=False)

File "/path/to/project_root/site/site-packages/django/__init__.py", line 27, in setup

apps.populate(settings.INSTALLED_APPS)

File "/path/to/project_root/site/site-packages/django/apps/registry.py", line 108, in populate

app_config.import_models()

File "/path/to/project_root/site/site-packages/django/apps/config.py", line 202, in import_models

self.models_module = import_module(models_module_name)

File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/importlib/__init__.py", line 37, in import_module

__import__(name)

File "/path/to/project_root/site/apps/recordings/models.py", line 7, in <module>

from apps.utils import transcription

File "/path/to/project_root/site/apps/utils/transcription.py", line 5, in <module>

from google.cloud.speech import enums

File "/path/to/project_root/env/lib/python2.7/site-packages/google/cloud/speech.py", line 19, in <module>

from google.cloud.speech_v1 import SpeechClient

File "/path/to/project_root/env/lib/python2.7/site-packages/google/cloud/speech_v1/__init__.py", line 17, in <module>

from google.cloud.speech_v1.gapic import speech_client

INFO 2019-04-18 00:13:24,017 runserver.py:376] default: "GET / HTTP/1.1" 500 -

File "/path/to/project_root/env/lib/python2.7/site-packages/google/cloud/speech_v1/gapic/speech_client.py", line 18, in <module>

import pkg_resources

File "/path/to/project_root/google_appengine/google/appengine/tools/devappserver2/python/sandbox.py", line 1076, in load_module

raise ImportError('No module named %s' % fullname)

ImportError: No module named pkg_resources

Here I've broken again in the module that tries to import types from google.cloud.speech, and you can see that the import chokes.

My goal is to actually understand what's breaking here and remedy it, because, like I mentioned earlier, this project layout is strongly preferred if at all possible. This structure includes the ability to use pipenv and other tooling that's been evolved to maturity. If I *have* to switch to djangae-scaffold, I can, but part of my motivation is understanding more in-depth GAE/django/djangae functionality.

I'm happy to share anything else that would be helpful, just ask.

jeff

On Mon, Dec 31, 2018 at 7:43 AM Jacob G <ja...@fareclock.com> wrote:

I spoke too soon. Remote shell requires oauth2client, at least in the GAE SDK version 57 that we have to use with djangae. Also, the latest version of urllib3 1.24 does not work with the old GAE SDK we are using, but version 1.23 does work. It looks like this commit broke it:

https://github.com/urllib3/urllib3/pull/1450/commits/01407533cbee3293043e13ce1eed8c00c063fe9f

--
You received this message because you are subscribed to a topic in the Google Groups "djangae-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/djangae-users/LbjWPlmwqbs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to djangae-user...@googlegroups.com.
To post to this group, send email to djanga...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/djangae-users/5ec027b7-180b-41ac-bd28-e5eeeddc345a%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

jeffrey k eliasen

Find and follow me on:
Blog: http://jeff.jke.net
Twitter: http://twitter.com/jeffreyeliasen
Facebook: http://facebook.com/jeffrey.eliasen

[Grzes Furga]

The right way is to use google.appengine.ext.vendor.add to add your site-packages directory to the path instead of sys.path.insert directly. 

Google is using "namespaced packages", and they require a call to site.addsitedir to work I believe (otherwise only a single "google" package dir endsup on the sys.path)

You received this message because you are subscribed to the Google Groups "djangae-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to djangae-user...@googlegroups.com.

To post to this group, send email to djanga...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/djangae-users/CAJLzY0ikd6As5s25U8xruEXXVzVARgkPOa7%2BPD021UjHJ%3DuKdg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

--

Grześ Furga p.ota.to

[Grzes Furga]

Sorry Jeffrey, I only now noticed that you are doing that (or were) in terms of path manipulation that should be enough.

Now googling your error (pkg_resources + appengine) I found answers suggesting that this happens if you're using the library from the system install, or virtualenv and not a local directory.

Try installing the cloud speech with the "-t <your_libs_directory>" option, and then calling vendor.add(<your_libs_directory>)

In the traceback the cloud speech code comes from "project_root/env/lib/python2.7/site-packages/", you want it on project_root/<your_libs_directory/

I hope this helps.

[Jeffrey Eliasen]

Thanks! I ultimately found that using vendor.add() in both wsgi.py and manage.py solved all of the problems *really* cleanly, simplifying both to just a handful of lines of code each (with no callouts to boot.py even).  That function is fantastic.

You're right about that error. Once I got that resolved, I ultimately discovered that the package I wanted to use wasn't compatible with Python2 anyway, so the whole approach got nixed. But I learned a lot about python imports, sys.path, splitting packages across namespaces... fun journey!

----------

jeffrey k eliasen - technologist, philosopher, agent of change
blog | linkedin | google+ | facebook |