Remote GAE api ndb query call results in intermittent "invalid handle" errors

[Martin Gerner]

Hi,

I use a small Python script on a GCE VM together with the google.appengine.ext.remote_api and google.appengine.ext.ndb libraries to make queries against a GAE instance (below, "Entity" is an ndb.Model):

q = Entity.query().order(Entity.creation_date)
entities, end_cursor, has_more = q.fetch_page(BATCH_SIZE, start_cursor=cursor)

Unfortunately, the query calls intermittently result in errors on the remote GAE side. With a pagesize of 250, about half of the calls fail. Doubling the pagesize to 500, almost all calls fail, so it might have some link to the amount of data returned. As can be seen from the screenshot (batchsize 250), the responses are neither particularly big or particularly time-consuming. The clientside error is just printed out as a warning and doesn't say very much:

2015-07-27 10:08:48,320 WARNING tasklets.py:409: suspended generator run_to_queue(query.py:938) raised BadRequestError(invalid handle: 14182066073298941631) 
2015-07-27 10:08:48,320 WARNING tasklets.py:409: suspended generator helper(context.py:876) raised BadRequestError(invalid handle: 14182066073298941631)

... but looking in the GAE monitoring logs, the stacktrace is a bit more detailed (not that it helps much though):

Exception while handling service_name: "datastore_v3" method: "Next" request: <request> Traceback (most recent call last): 
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/remote_api/handler.py", line 407, in post response_data = self.ExecuteRequest(request) 
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/remote_api/handler.py", line 441, in ExecuteRequest response_data) 
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 95, in MakeSyncCall return stubmap.MakeSyncCall(service, call, request, response) 
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 329, in MakeSyncCall rpc.CheckSuccess() 
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/apiproxy_rpc.py", line 134, in CheckSuccess raise self.exception ApplicationError: ApplicationError: 1 invalid handle: 14182066073298941631

I haven't been able to see any pattern to the handle numbers; they are all different, and appear random. When a failed call is retried, it usually completes successfully (and if it doesn't, the next time will usually be fine, and so on). Because of this, in practice, by simply constantly retrying my queries, I can get through anyway. However, things are clearly not working the way they're supposed to, and all these errors are clogging up logs all over the place.

As I haven't been able find a solution elsewhere or figure it out on my own, I post here. What could be going on with these remote datastore queries?

[Nick (Cloud Platform Support)]

Hi Martin,

Google Groups isn't the place to post specific technical issues, as this forum is meant more for general discussion of the platform and services. 

If you believe you've identified an issue with the platform itself, then you should proceed to open a public issue tracker [1] issue with enough detail to reproduce the issue on our side, or if possible, an attached app that can be used to directly observe the behaviour. This may not be possible, so simply providing as much relevant information as you can is helpful.

Your issue report has a decent amount of information, and posting it to the public issue tracker should give enough information for someone to get started working on the issue.

If you would like to open a thread in this forum discussing the platform or services in more broad terms, starting a discussion that would be useful for other users to join in to, feel free to do so.

Have a great day!

[1] http://code.google.com/p/google-appengine/issues/list