dev_appserver: Connect to remote datastore

642 views
Skip to first unread message

Filipe Caldas

unread,
Nov 22, 2017, 8:19:06 AM11/22/17
to Google App Engine
Hi,

  I currently have a python application and I am using the dev_appserver to run some tests locally, on some scenarios I need run tests with data on the production server, would it be possible to connect the dev_appserver to my production datastore?
If so how should this be implemented?

  I have tried to modify my appengine_config.py and add a connection there using remote_api_stub.ConfigureRemoteApiForOAuth() but that did not work.

Best regards

George (Cloud Platform Support)

unread,
Nov 22, 2017, 12:49:21 PM11/22/17
to Google App Engine
Hello Filipe, 

It is indeed possible to connect the dev_appserver to your production datastore. This has to be done in a way similar to what you already tried, by modifying appengine_config.py to call remote_api_stub.ConfigureRemoteApi(app_id=None, path='/_ah/remote_api',                                    auth_func=lambda: ('email', 'password'), servername='appid.apps  fpot.com') . More related detail may be gathered from the replies to the "Is it possible to run dev_appserver.py with the remote datastore?" question on stackoverflow. 

When you say that your attempt did not work, what were the errors? How did that fail, exactly? 

Filipe Caldas

unread,
Nov 23, 2017, 4:27:28 AM11/23/17
to Google App Engine
Hi George,

  So I did see that question on Stack overflow, but I could not reproduce the behaviour:

  I added:


if(BaseHandler.isDevAppServer()):
    from google.appengine.ext.remote_api import remote_api_stub
    from google.appengine.datastore.entity_pb import Reference

    remote_api_stub.ConfigureRemoteApi(app_id=None, path='/_ah/remote_api',
                                       auth_func=lambda: ('mye...@mycompany.com', 'mypassword'),
                                       servername='appid.appspot.com')

    if os.environ['SERVER_SOFTWARE'].startswith('Development'):
        Reference.app = lambda *args: os.environ['APPLICATION_ID'].replace('dev~', 's~')

To my appengine_config.py (I also changed the appid.appspot.com to my application id.appspot.com) but in both cases I got the following error:

Traceback (most recent call last):
  File "/home/fcaldas/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 240, in Handle
    handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
  File "/home/fcaldas/google-cloud-sdk/platform/google_appengine/google/appengine/api/lib_config.py", line 351, in __getattr__
    self._update_configs()
  File "/home/fcaldas/google-cloud-sdk/platform/google_appengine/google/appengine/api/lib_config.py", line 287, in _update_configs
    self._registry.initialize()
  File "/home/fcaldas/google-cloud-sdk/platform/google_appengine/google/appengine/api/lib_config.py", line 160, in initialize
    import_func(self._modname)
  File "/home/fcaldas/lipex/webapp_lipex/appengine_config.py", line 31, in <module>
    servername='appid.appspot.com')
  File "/home/fcaldas/google-cloud-sdk/platform/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 881, in ConfigureRemoteApi
    app_id = GetRemoteAppIdFromServer(server, path, rtok)
  File "/home/fcaldas/google-cloud-sdk/platform/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 586, in GetRemoteAppIdFromServer
    response = server.Send(path, payload=None, **urlargs)
  File "/home/fcaldas/google-cloud-sdk/platform/google_appengine/google/appengine/tools/appengine_rpc.py", line 393, in Send
    f = self.opener.open(req)
  File "/usr/lib64/python2.7/urllib2.py", line 435, in open
    response = meth(req, response)
  File "/usr/lib64/python2.7/urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib64/python2.7/urllib2.py", line 473, in error
    return self._call_chain(*args)
  File "/usr/lib64/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.7/urllib2.py", line 556, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 404: Not Found
INFO     2017-11-23 09:17:14,153 module.py:821] default: "GET / HTTP/1.1" 500 -

I do have the remote_api on and I use multiple CRON scripts to load data into the datastore running on Virtual machines, on those scripts I normally simply use:

from google.appengine.ext.remote_api import remote_api_stub

PROJECT_ID = "myprojectID"
remote_api_stub.ConfigureRemoteApiForOAuth(
    '{}.appspot.com'.format(PROJECT_ID),
    '/_ah/remote_api/')

But adding that to my appengine_config gives the following errors:

fcaldas@trinity:~/lipex/webapp_lipex$ dev_appserver.py app.yaml 
INFO     2017-11-23 09:23:14,125 devappserver2.py:105] Skipping SDK update check.
INFO     2017-11-23 09:23:19,250 api_server.py:300] Starting API server at: http://localhost:38963
INFO     2017-11-23 09:23:19,286 dispatcher.py:251] Starting module "default" running at: http://localhost:8080
INFO     2017-11-23 09:23:19,287 admin_server.py:116] Starting admin server at: http://localhost:8000
INFO     2017-11-23 09:23:28,484 transport.py:157] Attempting refresh to obtain initial access_token
INFO     2017-11-23 09:23:28,487 client.py:755] Refreshing access_token
ERROR    2017-11-23 09:23:38,957 Applicant.py:25] Could not import google.cloud.bigquery
ERROR    2017-11-23 09:23:38,963 Trademark.py:33] Could not import google.cloud.bigquery
ERROR    2017-11-23 09:23:39,031 Applicant.py:25] Could not import google.cloud.bigquery
ERROR    2017-11-23 09:23:39,034 Trademark.py:33] Could not import google.cloud.bigquery
ERROR    2017-11-23 09:23:39,322 PubSub.py:15] Could not import PubSub: watch reports will not work. (PubSub.py)
LOG 0 1511429019436934 _Authenticate configuring auth; needs_auth=False
LOG 0 1511429019437051 Sending request to https://lipexdb-test.appspot.com/_ah/remote_api/ headers={'X-appcfg-api-version': '1', 'content-length': '602', 'Content-Type': 'application/octet-stream'} body=ELIDED
LOG 0 1511429019437414 _Authenticate configuring auth; needs_auth=False
LOG 0 1511429019437465 Sending request to https://lipexdb-test.appspot.com/_ah/remote_api/ headers={'X-appcfg-api-version': '1', 'content-length': '1031', 'Content-Type': 'application/octet-stream'} body=ELIDED
LOG 0 1511429019437716 _Authenticate configuring auth; needs_auth=False
LOG 0 1511429019437761 Sending request to https://lipexdb-test.appspot.com/_ah/remote_api/ headers={'X-appcfg-api-version': '1', 'content-length': '1461', 'Content-Type': 'application/octet-stream'} body=ELIDED
LOG 0 1511429019437980 _Authenticate configuring auth; needs_auth=False
LOG 0 1511429019438023 Sending request to https://lipexdb-test.appspot.com/_ah/remote_api/ headers={'X-appcfg-api-version': '1', 'content-length': '1891', 'Content-Type': 'application/octet-stream'} body=ELIDED
LOG 0 1511429019438242 _Authenticate configuring auth; needs_auth=False
LOG 0 1511429019438284 Sending request to https://lipexdb-test.appspot.com/_ah/remote_api/ headers={'X-appcfg-api-version': '1', 'content-length': '2321', 'Content-Type': 'application/octet-stream'} body=ELIDED
LOG 0 1511429019438521 _Authenticate configuring auth; needs_auth=False
LOG 0 1511429019438568 Sending request to https://lipexdb-test.appspot.com/_ah/remote_api/ headers={'X-appcfg-api-version': '1', 'content-length': '2751', 'Content-Type': 'application/octet-stream'} body=ELIDED
LOG 0 1511429019438798 _Authenticate configuring auth; needs_auth=False
LOG 0 1511429019438844 Sending request to https://lipexdb-test.appspot.com/_ah/remote_api/ headers={'X-appcfg-api-version': '1', 'content-length': '3181', 'Content-Type': 'application/octet-stream'} body=ELIDED
LOG 0 1511429019439237 _Authenticate configuring auth; needs_auth=False
LOG 0 1511429019439308 Sending request to https://lipexdb-test.appspot.com/_ah/remote_api/ headers={'X-appcfg-api-version': '1', 'content-length': '3611', 'Content-Type': 'application/octet-stream'} body=ELIDED
LOG 0 1511429019439642 _Authenticate configuring auth; needs_auth=False
LOG 0 1511429019439716 Sending request to https://lipexdb-test.appspot.com/_ah/remote_api/ headers={'X-appcfg-api-version': '1', 'content-length': '4041', 'Content-Type': 'application/octet-stream'} body=ELIDED
LOG 0 1511429019440161 _Authenticate configuring auth; needs_auth=False
LOG 0 1511429019440234 Sending request to https://lipexdb-test.appspot.com/_ah/remote_api/ headers={'X-appcfg-api-version': '1', 'content-length': '4471', 'Content-Type': 'application/octet-stream'} body=ELIDED
LOG 0 1511429019440775 _Authenticate configuring auth; needs_auth=False
LOG 0 1511429019440892 Sending request to https://lipexdb-test.appspot.com/_ah/remote_api/ headers={'X-appcfg-api-version': '1', 'content-length': '4901', 'Content-Type': 'application/octet-stream'} body=ELIDED
LOG 0 1511429019441860 _Authenticate configuring auth; needs_auth=False
LOG 0 1511429019441960 Sending request to https://lipexdb-test.appspot.com/_ah/remote_api/ headers={'X-appcfg-api-version': '1', 'content-length': '5331', 'Content-Type': 'application/octet-stream'} body=ELIDED
LOG 0 1511429019442334 _Authenticate configuring auth; needs_auth=False
LOG 0 1511429019442386 Sending request to https://lipexdb-test.appspot.com/_ah/remote_api/ headers={'X-appcfg-api-version': '1', 'content-length': '5761', 'Content-Type': 'application/octet-stream'} body=ELIDED
LOG 0 1511429019442668 _Authenticate configuring auth; needs_auth=False
LOG 0 1511429019442718 Sending request to https://lipexdb-test.appspot.com/_ah/remote_api/ headers={'X-appcfg-api-version': '1', 'content-length': '6191', 'Content-Type': 'application/octet-stream'} body=ELIDED
LOG 0 1511429019442979 _Authenticate configuring auth; needs_auth=False
LOG 0 1511429019443025 Sending request to https://lipexdb-test.appspot.com/_ah/remote_api/ headers={'X-appcfg-api-version': '1', 'content-length': '6621', 'Content-Type': 'application/octet-stream'} body=ELIDED
LOG 0 1511429019443358 _Authenticate configuring auth; needs_auth=False
LOG 0 1511429019443407 Sending request to https://lipexdb-test.appspot.com/_ah/remote_api/ headers={'X-appcfg-api-version': '1', 'content-length': '7051', 'Content-Type': 'application/octet-stream'} body=ELIDED
LOG 0 1511429019443658 _Authenticate configuring auth; needs_auth=False
LOG 0 1511429019443702 Sending request to https://lipexdb-test.appspot.com/_ah/remote_api/ headers={'X-appcfg-api-version': '1', 'content-length': '7481', 'Content-Type': 'application/octet-stream'} body=ELIDED
LOG 0 1511429019443944 _Authenticate configuring auth; needs_auth=False
LOG 0 1511429019443990 Sending request to https://lipexdb-test.appspot.com/_ah/remote_api/ headers={'X-appcfg-api-version': '1', 'content-length': '7911', 'Content-Type': 'application/octet-stream'} body=ELIDED
LOG 0 1511429019444236 _Authenticate configuring auth; needs_auth=False
LOG 0 1511429019446394 _Authenticate configuring auth; needs_auth=False
LOG 0 1511429019446439 Sending request to https://lipexdb-test.appspot.com/_ah/remote_api/ headers={'X-appcfg-api-version': '1', 'content-length': '11354', 'Content-Type': 'application/octet-stream'} body=ELIDED
LOG 0 1511429019446692 _Authenticate configuring auth; needs_auth=False
LOG 0 1511429019446734 Sending request to https://lipexdb-test.appspot.com/_ah/remote_api/ headers={'X-appcfg-api-version': '1', 'content-length': '11785', 'Content-Type': 'application/octet-stream'} body=ELIDED
LOG 0 1511429019450336 _Authenticate configuring auth; needs_auth=False
.........
many lines later:
.........
  File "/home/fcaldas/google-cloud-sdk/platform/google_appengine/google/appengine/dist27/gae_override/httplib.py", line 522, in getresponse
    **extra_kwargs)
  File "/home/fcaldas/lipex/webapp_lipex/libs/httplib2/__init__.py", line 1106, in fixed_fetch
    validate_certificate=validate_certificate)
  File "/home/fcaldas/google-cloud-sdk/platform/google_appengine/google/appengine/api/urlfetch.py", line 293, in fetch
    return rpc.get_result()
  File "/home/fcaldas/google-cloud-sdk/platform/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 613, in get_result
    return self.__get_result_hook(self)
  File "/home/fcaldas/google-cloud-sdk/platform/google_appengine/google/appengine/api/urlfetch.py", line 413, in _get_fetch_result
    rpc.check_success()
  File "/home/fcaldas/google-cloud-sdk/platform/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 579, in check_success
    self.__rpc.CheckSuccess()
  File "/home/fcaldas/google-cloud-sdk/platform/google_appengine/google/appengine/api/apiproxy_rpc.py", line 157, in _WaitImpl
    self.request, self.response)
  File "/home/fcaldas/google-cloud-sdk/platform/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 203, in MakeSyncCall
    self._MakeRealSyncCall(service, call, request, response)
  File "/home/fcaldas/google-cloud-sdk/platform/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 228, in _MakeRealSyncCall
    encoded_response = self._server.Send(self._path, encoded_request)
  File "/home/fcaldas/google-cloud-sdk/platform/google_appengine/google/appengine/tools/appengine_rpc_httplib2.py", line 240, in Send
    self._Authenticate(self.http, auth_errors[0] > 0)
  File "/home/fcaldas/google-cloud-sdk/platform/google_appengine/google/appengine/tools/appengine_rpc_httplib2.py", line 471, in _Authenticate
    needs_auth)
  File "/usr/lib64/python2.7/logging/__init__.py", line 1155, in debug
    self._log(DEBUG, msg, args, **kwargs)
  File "/usr/lib64/python2.7/logging/__init__.py", line 1285, in _log
    record = self.makeRecord(self.name, level, fn, lno, msg, args, exc_info, func, extra)
  File "/usr/lib64/python2.7/logging/__init__.py", line 1259, in makeRecord
    rv = LogRecord(name, level, fn, lno, msg, args, exc_info, func)
  File "/usr/lib64/python2.7/logging/__init__.py", line 263, in __init__
    if (args and len(args) == 1 and isinstance(args[0], collections.Mapping)
  File "/usr/lib64/python2.7/abc.py", line 132, in __instancecheck__
    if subclass is not None and subclass in cls._abc_cache:
RuntimeError: maximum recursion depth exceeded


Thanks for your support

George (Cloud Platform Support)

unread,
Nov 27, 2017, 4:59:21 PM11/27/17
to Google App Engine
Hi Filipe, 

We'll examine the code samples provided and come back to you shortly. 

Filipe Caldas

unread,
Nov 30, 2017, 5:15:25 AM11/30/17
to Google App Engine
Thanks George,

George (Cloud Platform Support)

unread,
Nov 30, 2017, 3:14:34 PM11/30/17
to Google App Engine
Hi Filipe, 

The lines: 

INFO     2017-11-23 09:23:28,487 client.py:755] Refreshing access_token
ERROR    2017-11-23 09:23:38,957 Applicant.py:25] Could not import google.cloud.bigquery

as well as other lines, seem all to indicate that you cannot authenticate. You should verify authenticated-related code more thoroughly, for possible omissions. 

It does look as if this is a coding issue, so it may be worth mentioning: this group is meant for general questions related to the app engine, for voicing of opinions, etc.. When it comes strictly to programming and coding, you’ll be at an advantage to rather post your questions on Stackoverflow, to gain this way access to a large number of experts; it is meant for providing you specific help. 
 

Filipe Caldas

unread,
Dec 1, 2017, 5:47:58 AM12/1/17
to Google App Engine
Hi George,

 I am quite sure this is not a coding problem. The BigQuery error line is just a try catch around a import for BQ that is used in production on some CRON jobs and do not affect authentication in any way.

George (Cloud Platform Support)

unread,
Dec 1, 2017, 4:27:47 PM12/1/17
to Google App Engine
Hi Filipe, 

Some context is possibly missing here. The "Using the Remote API in a local client" sub-chapter of the "Accessing App Engine with Remote API" document indicates 7 steps to follow. Have you looked at these, if anything applicable is missing? 

Filipe Caldas

unread,
Dec 5, 2017, 5:03:30 AM12/5/17
to Google App Engine
Hi George,

  Thanks again for your support, I followed the page you sent, it is already what I am doing, which works fine for individual scripts accessing the remote Datastore, but fails when the script is invoked by the dev_appserver.py.

  I created a small github repository with a minimal project that I used to reproduce the issue:

  https://github.com/fcaldas/appengine_remote_datastore

  (You would need to deploy it first and then try to use the dev_appserver.py to launch the app.yaml)


Best,

George (Cloud Platform Support)

unread,
Dec 5, 2017, 4:36:25 PM12/5/17
to Google App Engine
Hi Filipe, 

At this stage we definitely need a representative sample of your code, that should enable reproduction of the issue, to deploy and test locally. For reasons of confidentiality, information has to be sent by way of a private message, generated from the drop-down menu at the top right of the editing windows. All necessary instructions are to be included as well. Once reproduced, the issue will get Developer attention. 

George (Cloud Platform Support)

unread,
Dec 6, 2017, 4:35:45 PM12/6/17
to Google App Engine
Hi Filipe, 

Managed to reproduce the error, following the step-by-step instructions and code provided, but it may not be the right error. What I get is: 

ImportError: cannot import name remote_api_stub
at <module> (/base/data/home/apps/.../... /appengine_config.py:2)

It this the error you receive as well? 

George (Cloud Platform Support)

unread,
Dec 7, 2017, 11:43:35 AM12/7/17
to Google App Engine
Hi Filipe, 

Does your code contain statements similar to the snippet below?  

remote_api_stub.ConfigureRemoteApiForOAuth(
    '{}.appspot.com'.format(project_id),
    '/_ah/remote_api')

This is from the "Accessing App Engine with Remote API" documentation page, that you very likely consulted for your app. 

We are definitely getting into coding, so I repeat my previous advice, to rather post your issue details on stackoverflow or other similar forum, where more in-depth help is being offered. 

Filipe Caldas

unread,
Dec 8, 2017, 4:20:08 AM12/8/17
to Google App Engine
Hi George,

  Thanks for your help, I did post it on Stack Overflow (https://stackoverflow.com/questions/47671585/dev-appserver-py-connect-to-remote-datastore) my only problem with that is that for my previous AppEngine / Datastore issues I never got any replies from anyone directly linked to these projects which is not the case for some GCloud products (eg BigQuery) where the developers are active on Stack Overflow.

  
  Did you get this error on the deployed App or on dev_appserver ?

  The error I get is different, but if you are getting this on the deployed appengine it might be because that lib is just not available on the production appengine and the code on appengine_config.py should only run on the dev_appserver. I will make the changes on the github repository and push it again.

Best,

George (Cloud Platform Support)

unread,
Dec 8, 2017, 3:54:45 PM12/8/17
to Google App Engine
The way you deployed your app in the cloud and ran it locally requires features to be retained, that would work for both. In particular, the development server bundled with the SDK (only in case of standard environment apps) doesn't support SSL at all: GAE dev_appserver.py over HTTPS. 

Filipe Caldas

unread,
Dec 11, 2017, 3:23:25 AM12/11/17
to Google App Engine
Hi George,

Does that mean there is no way to access the remote Datastore?

George Suceveanu

unread,
Dec 11, 2017, 9:08:14 AM12/11/17
to google-a...@googlegroups.com
Hi Filipe, 

There is no doubt that quite a few programmers make use of this facility. The Remote API library allows any Python client to access services available to App Engine applications. You are encouraged to follow the "Accessing App Engine with Remote API" document, more specifically "Getting started with the Google Cloud Datastore API" page, for an example.   

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/TFnk3qZ2Tc0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
 To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/6124881c-7e03-4ad2-a59d-384a298f7d93%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Reply all
Reply to author
Forward
0 new messages