BigQuery Import not working on local dev_appserver. (Import error)

[Sumanau Sareen]

I am trying to use google BigQuery Client library : pip install --upgrade google-cloud-bigquery - t lib/ 

where lib is the folder where i have all python libraries. This successfully install the library lib/google/cloud/bigquery.

But when i try to import it in dev console by using

from google.cloud import bigquery

it gives an import error:

Traceback (most recent call last):
  File "/home/development/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/python/request_handler.py", line 226, in handle_interactive_request
    exec(compiled_code, self._command_globals)
  File "<string>", line 1, in <module>
  File "/home/development/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/python/sandbox.py", line 1001, in load_module
    raise ImportError('No module named %s' % fullname)
ImportError: No module named google.cloud.bigquery

I have also tried the approach mentioned on stackoverflow thread: http://stackoverflow.com/questions/40418201/google-cloud-bigquery-import-not-working-in-app-engine-project

It suggests to use:

from __future__ import absolute_import


But it still results in import error. Has anyone else faced similar issues in local dev app server before. In production appengine environment, everything works fine.




Thanks,

Sumanau

[Evan Jones]

It sounds like you are following the "Using third-party libraries" document below. If you follow these directions, everything will work if you use dev_appserver.py or run in production. However, when you run things locally, you will need to add the lib directory to your PYTHONPATH in some way. I've found the easiest way is to set the PYTHONPATH environment variable manually. For example:

PYTHONPATH=lib python (myscript.py)

I hope that helps,

Evan

[Attila-Mihaly Balazs]

An alternative to the PYTHONPATH solution is the following:

- create an "appengine_config.py" in the same directory you have your app.yaml
- put the following into it:

from google.appengine.ext import vendor
vendor.add('lib')

[Sumanau Sareen]

I am using the same approach.

The issue is only visible for bigquery library in appengine sandbox. In production everything is working fine.

where as if i try to use from gcloud import bigquery

it fails with error:

  File "/home/user/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/python/sandbox.py", line 1001, in load_module
    raise ImportError('No module named %s' % fullname)
ImportError: No module named google.protobuf

where protobuf is also installed. 

Other then these libraries, all other libraries are working fine. 

[Evan Jones]

This is probably happening due to Python package conflicts. By default, when you do "import google" it finds the first "google" package in your PYTHONPATH. It does *not* combine all the separate "google" packages together. There is a way to "opt in" to combining packages, but the App Engine packages and some of Google's other old Python packages don't do it, so you can run into this. We've hacked around this by adding a file named:

${VIRTUALENV}/lib/python2.7/site-packages/gae.pth

With the following contents:

${YOUR PATH TO google-cloud-sdk/platform/google_appengine}

import dev_appserver; dev_appserver.fix_sys_path()

import google

import pkgutil; pkgutil.extend_path(google.__path__, google.__name__)

This gets loaded by Python before anything else, and ensures that the multiple "google" packages you might have get combined together. There are probably other ways to hack around this, but the key line is inserting pkgutil.extend_path(google.__path__, google.__name__) somewhere in your code. Good luck!

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/C5LjgIiJwJY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/cd0067ba-e94b-4642-9133-d96ae2ab1483%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.