upload data via appcfg

[Rotem Vilkovski]

Hey, 

I'm trying to upload data to local Datastore using the file I've downloaded from out production using appcfg download_data. 

I get an error almost every time about ProtocolBufferDecodeError: corrupted. Sometimes (on the same file) it is just working. This is really disturbing and I'd like to know if there's another way to use my local datastore with my local file and having a solution for this problem please. Everytime I update an entity in production and want to update my local Datastore it takes me for 10-20 tries.

Stack trace:

  File "/home/rotem/Desktop/google_appengine/google/appengine/tools/adaptive_thread_pool.py", line 172, in WorkOnItems

    status, instruction = item.PerformWork(self.__thread_pool)

  File "/home/rotem/Desktop/google_appengine/google/appengine/tools/bulkloader.py", line 750, in PerformWork

    transfer_time = self._TransferItem(thread_pool)

  File "/home/rotem/Desktop/google_appengine/google/appengine/tools/bulkloader.py", line 921, in _TransferItem

    self.request_manager.PostEntities(self.content)

  File "/home/rotem/Desktop/google_appengine/google/appengine/tools/bulkloader.py", line 1396, in PostEntities

    datastore.Put(entities)

  File "/home/rotem/Desktop/google_appengine/google/appengine/api/datastore.py", line 606, in Put

    return PutAsync(entities, **kwargs).get_result()

  File "/home/rotem/Desktop/google_appengine/google/appengine/datastore/datastore_rpc.py", line 923, in get_result

    results = self.__rpcs[0].get_result()

  File "/home/rotem/Desktop/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 613, in get_result

    return self.__get_result_hook(self)

  File "/home/rotem/Desktop/google_appengine/google/appengine/datastore/datastore_rpc.py", line 1881, in __put_hook

    self.check_rpc_success(rpc)

  File "/home/rotem/Desktop/google_appengine/google/appengine/datastore/datastore_rpc.py", line 1371, in check_rpc_success

    rpc.check_success()

  File "/home/rotem/Desktop/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 579, in check_success

    self.__rpc.CheckSuccess()

  File "/home/rotem/Desktop/google_appengine/google/appengine/api/apiproxy_rpc.py", line 157, in _WaitImpl

    self.request, self.response)

  File "/home/rotem/Desktop/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 286, in MakeSyncCall

    handler(request, response)

  File "/home/rotem/Desktop/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 460, in _Dynamic_Put

    'datastore_v3', 'Put', put_request, put_response)

  File "/home/rotem/Desktop/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 201, in MakeSyncCall

    self._MakeRealSyncCall(service, call, request, response)

  File "/home/rotem/Desktop/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 228, in _MakeRealSyncCall

    response_pb.ParseFromString(encoded_response)

  File "/home/rotem/Desktop/google_appengine/google/net/proto/ProtocolBuffer.py", line 140, in ParseFromString

    self.MergeFromString(s)

  File "/home/rotem/Desktop/google_appengine/google/net/proto/ProtocolBuffer.py", line 152, in MergeFromString

    self.MergePartialFromString(s)

  File "/home/rotem/Desktop/google_appengine/google/net/proto/ProtocolBuffer.py", line 168, in MergePartialFromString

    self.TryMerge(d)

  File "/home/rotem/Desktop/google_appengine/google/appengine/ext/remote_api/remote_api_pb.py", line 770, in TryMerge

    d.skipData(tt)

  File "/home/rotem/Desktop/google_appengine/google/net/proto/ProtocolBuffer.py", line 673, in skipData

    raise ProtocolBufferDecodeError, "corrupted"

ProtocolBufferDecodeError: corrupted

Thanks!

[Adam (Cloud Platform Support)]

Rather than manually trying to resync the production data locally every time an update is required, you could use the Remote API to have your development server app talk to the remote Datastore, and have both your local and deployed app use the same data source.

I don't recommend doing this for "real" production data - you should create a new project used specifically for testing / development. You can use the Datastore Backup to backup any data you need from prod and restore it to the test project.

[Adam (Cloud Platform Support)]

Regarding the 'ProtocolBufferDecodeError: corrupted' error, this can happen when using a custom URL for Remote API which isn't mapped correctly. If you would like to post your app.yaml with the sensitive bits redacted I can take a look.

[Rotem Vilkovski]

Hey Adam, thank you for the comment.

I tried to read some of your documentation about the remote api and it hasn't helped me at all.

Is there any example for something so basic? I mean this is the most basic move you need to get you local environment to run and I can't find anything that you've documented. every documentation just assume you have a local datastore that fully functions. 
Lets say I've done a DatastoreBackup, how do I restore it locally?

Thanks!

[Rotem Vilkovski]

I've done a restore on my testing project. How can I configure my local project turn to the remote datastore? I don't want to upload a version each time I make a little change without check it out locally.

Thanks!

[Rotem Vilkovski]

BTW - in my production I get 404 when I try to reach datastore admin, my project id is: 831908763256

Thanks again.

[Adam (Cloud Platform Support)]

Using the Remote API on dev_appserver.py is essentially the same as using it in a local Python client, as per the example in 'Remote API for Python'. The most straightforward way to turn it on globally for your app in the devserver is to add the setup to your 'appengine_config.py' located in your project's root:

import os
if os.environ['SERVER_SOFTWARE'].startswith('Development'):
  from google.appengine.ext.remote_api import remote_api_stub
  from google.appengine.api import apiproxy_stub_map

  local_stub = apiproxy_stub_map.apiproxy.GetStub('urlfetch')
  remote_api_stub.ConfigureRemoteApiForOAuth('myapp.appspot.com','/_ah/remote_api')
  apiproxy_stub_map.apiproxy.ReplaceStub('urlfetch', local_stub)

(Note that replacing the 'urlfetch' stub is necessary to work around a bug, if interested you can read about it more here and here).

Regarding the 404 with Datastore Admin, if you get this when following the link from the Cloud Console this usually means the 'ah-builtin-python-bundle' version got deleted from your project. The simplest way to fix this is to deploy to a new project. If creating a new project is not an option, you can file a production issue on the issue tracker and request a restore of this bundle.

[Rotem Vilkovski]

Hey Adam, Thanks again.

My ConfigureRemoteApiForOAuth code now on appengine_config looks like:

if os.environ['SERVER_SOFTWARE'].startswith('Development'):
    from google.appengine.ext.remote_api import remote_api_stub
    from google.appengine.api import 

apiproxy_stub_map
    import dev_appserver
    dev_appserver.fix_sys_path()

    local_stub = apiproxy_stub_map.apiproxy.GetStub('urlfetch')

    remote_api_stub.ConfigureRemoteApiForOAuth('storemavendev2.appspot.com', '/_ah/remote_api')
    apiproxy_stub_map.apiproxy.ReplaceStub('urlfetch', local_stub)

And this split for three scenarios.

First - when secure flag is True in ConfigureRemoteApiForOAuth (by default) I get that:

  File "D:\google_appengine\google\appengine\runtime\wsgi.py", line 240, in Handle

    handler = _config_handle.add_wsgi_middleware(self._LoadHandler())

  File "D:\google_appengine\google\appengine\api\lib_config.py", line 354, in __getattr__

    self._update_configs()

  File "D:\google_appengine\google\appengine\api\lib_config.py", line 290, in _update_configs

    self._registry.initialize()

  File "D:\google_appengine\google\appengine\api\lib_config.py", line 165, in initialize

    import_func(self._modname)

  File "D:\storemaven\code\appengine_config.py", line 12, in <module>

    remote_api_stub.ConfigureRemoteApiForOAuth('mockappstore.appspot.com', '/_ah/remote_api', True)

  File "D:\google_appengine\google\appengine\ext\remote_api\remote_api_stub.py", line 769, in ConfigureRemoteApiForOAuth

    rpc_server_factory=rpc_server_factory)

  File "D:\google_appengine\google\appengine\ext\remote_api\remote_api_stub.py", line 839, in ConfigureRemoteApi

    app_id = GetRemoteAppIdFromServer(server, path, rtok)

  File "D:\google_appengine\google\appengine\ext\remote_api\remote_api_stub.py", line 569, in GetRemoteAppIdFromServer

    response = server.Send(path, payload=None, **urlargs)

  File "D:\google_appengine\google\appengine\tools\appengine_rpc_httplib2.py", line 246, in Send

    url, method=method, body=payload, headers=headers)

  File "D:\google_appengine\lib\oauth2client\oauth2client\client.py", line 569, in new_request

    redirections, connection_type)

  File "D:\google_appengine\lib\httplib2\httplib2\__init__.py", line 1464, in request

    self.disable_ssl_certificate_validation)

  File "D:\google_appengine\lib\httplib2\httplib2\__init__.py", line 1143, in __init__

    strict, timeout, proxy_info, ca_certs, disable_ssl_certificate_validation)

  File "D:\google_appengine\lib\httplib2\httplib2\__init__.py", line 1092, in __init__

    raise NotSupportedOnThisPlatform()

NotSupportedOnThisPlatform

Second - when I'm turning off secrure by passing secure=False I get information from some other namespace, how can I pass from which namespace I wanat to work with my data?

Third - whem I'm turning off secure and try to reach to my other project (id: storemavendev2) I get that(In other projects it works just not on the wanted namespace):

Traceback (most recent call last):

  File "D:\google_appengine\google\appengine\runtime\wsgi.py", line 267, in Handle

    result = handler(dict(self._environ), self._StartResponse)

  File "D:\storemaven\code\com\lib\gaesessions\__init__.py", line 475, in __call__

    return self.app(environ, my_start_response)

  File "D:\storemaven\code/com/lib\flask\app.py", line 1836, in __call__

    return self.wsgi_app(environ, start_response)

  File "D:\storemaven\code/com/lib\flask\app.py", line 1820, in wsgi_app

    response = self.make_response(self.handle_exception(e))

  File "D:\storemaven\code/com/lib\flask\app.py", line 1403, in handle_exception

    reraise(exc_type, exc_value, tb)

  File "D:\storemaven\code/com/lib\flask\app.py", line 1817, in wsgi_app

    response = self.full_dispatch_request()

  File "D:\storemaven\code/com/lib\flask\app.py", line 1477, in full_dispatch_request

    rv = self.handle_user_exception(e)

  File "D:\storemaven\code/com/lib\flask\app.py", line 1381, in handle_user_exception

    reraise(exc_type, exc_value, tb)

  File "D:\storemaven\code/com/lib\flask\app.py", line 1475, in full_dispatch_request

    rv = self.dispatch_request()

  File "D:\storemaven\code/com/lib\flask\app.py", line 1461, in dispatch_request

    return self.view_functions[rule.endpoint](**req.view_args)

  File "D:\storemaven\code\com\scene53\api\flask\headers.py", line 20, in decorated_function

    resp = make_response(f(*args, **kwargs))

  File "D:\storemaven\code\com\scene53\api\flask\headers.py", line 62, in decorated_function

    return f(*args, **kwargs)

  File "D:\storemaven\code\com\scene53\api\flask\console\__init__.py", line 551, in listUnassignedUsers

    return json.jsonify(status="OK", text="", payload=userFacade.listUnassignedUsers())

  File "D:\storemaven\code\com\scene53\data\userfacade.py", line 175, in listUnassignedUsers

    users=self.userService.listUsers(None)

  File "D:\storemaven\code\com\scene53\data\user.py", line 180, in listUsers

    for p in iter:

  File "D:\google_appengine\google\appengine\ext\db\__init__.py", line 2330, in next

    return self.__model_class.from_entity(self.__iterator.next())

  File "D:\google_appengine\google\appengine\datastore\datastore_query.py", line 3321, in next

    next_batch = self.__batcher.next_batch(Batcher.AT_LEAST_OFFSET)

  File "D:\google_appengine\google\appengine\datastore\datastore_query.py", line 3207, in next_batch

    batch = self.__next_batch.get_result()

  File "D:\google_appengine\google\appengine\api\apiproxy_stub_map.py", line 613, in get_result

    return self.__get_result_hook(self)

  File "D:\google_appengine\google\appengine\datastore\datastore_query.py", line 2906, in __query_result_hook

    self._batch_shared.conn.check_rpc_success(rpc)

  File "D:\google_appengine\google\appengine\datastore\datastore_rpc.py", line 1373, in check_rpc_success

    raise _ToDatastoreError(err)

BadRequestError: app s~storemavendev2 cannot access app dev~storemavendev2's data

I guess that happens because of that my local app named this way?

Thanks!

Thank you very very much! 

On Tuesday, May 3, 2016 at 6:20:33 PM UTC+3, Rotem Vilkovski wrote:

[Adam (Cloud Platform Support)]

If you are getting errors related to the app's partition ('~s' vs '~dev') you can try adding the following to 'appengine_config.py' so that it matches the one used by prod:

from google.appengine.datastore.entity_pb import Reference
Reference.app = lambda *args: os.environ['APPLICATION_ID'].replace('dev~', 's~')

To set the desired namespace, you can use the namespace manager:

from google.appengine.api import namespace_manager
namespace_manager.set_namespace('my_namespace')

Let me know if you run into other problems.