Getting lot of InternalError on Datastore Operations

[Muhammad Taiyab]

Hi,

I am getting lot of "InternalError" on datastore get/put operations.

Here is an exception trace.

50.134.226.108 - - [20/Mar/2015:05:12:04 -0700] "GET /profile?version=2.11&deviceOs=02_AND HTTP/1.1" 500 1113 - "Blades RPG (gzip)" "bob-live.appspot.com" ms=2416 cpu_ms=257 cpm_usd=0.000124 app_engine_release=1.9.18 instance=00c61b117c3d24569be886499b98d02451a93e
W 2015-03-20 17:12:04.916
suspended generator run_to_queue(query.py:938) raised InternalError(internal error.)
W 2015-03-20 17:12:04.917
suspended generator helper(context.py:876) raised InternalError(internal error.)
E 2015-03-20 17:12:04.917
internal error.
Traceback (most recent call last):
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 1536, in __call__
    rv = self.handle_exception(request, response, e)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 1530, in __call__
    rv = self.router.dispatch(request, response)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 1278, in default_dispatcher
    return route.handler_adapter(request, response)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 1102, in __call__
    return handler.dispatch()
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 572, in dispatch
    return self.handle_exception(e, self.app.debug)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 570, in dispatch
    return method(*args, **kwargs)
  File "/base/data/home/apps/s~bob-live/128.383018020005185273/logic/user.py", line 109, in wrapper
    fn(self, *args, **kwargs)
  File "/base/data/home/apps/s~bob-live/128.383018020005185273/handlers/ProfileHandler.py", line 42, in get
    p = profile_get(user)
  File "/base/data/home/apps/s~bob-live/128.383018020005185273/logic/profile.py", line 255, in profile_get
    profile_dict['heroes'] = hero_get_all_active(user_key)
  File "/base/data/home/apps/s~bob-live/128.383018020005185273/logic/hero.py", line 476, in hero_get_all_active
    hero_dicts.append(hero_to_dict(hero))
  File "/base/data/home/apps/s~bob-live/128.383018020005185273/logic/hero.py", line 430, in hero_to_dict
    hd['skills'] = [skill_to_dict(skill) for skill in Skill.query(ancestor=hero.key)]
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/query.py", line 1756, in next
    ent = self._fut.get_result()
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 325, in get_result
    self.check_success()
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/context.py", line 876, in helper
    batch, i, ent = yield inq.getq()
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/query.py", line 938, in run_to_queue
    batch = yield rpc
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 454, in _on_rpc_completion
    result = rpc.get_result()
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 613, in get_result
    return self.__get_result_hook(self)
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/datastore/datastore_query.py", line 2973, in __query_result_hook
    self._batch_shared.conn.check_rpc_success(rpc)
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 1342, in check_rpc_success
    raise _ToDatastoreError(err)
InternalError: internal error.

[Vinny P]

On Fri, Mar 20, 2015 at 7:31 AM, Muhammad Taiyab <m.ta...@carameltec.com> wrote:

I am getting lot of "InternalError" on datastore get/put operations.

Here is an exception trace.

50.134.226.108 - - [20/Mar/2015:05:12:04 -0700] "GET /profile?version=2.11&deviceOs=02_AND HTTP/1.1" 500 1113 - "Blades RPG (gzip)" "bob-live.appspot.com" ms=2416 cpu_ms=257 cpm_usd=0.000124 app_engine_release=1.9.18 instance=00c61b117c3d24569be886499b98d02451a93e
W 2015-03-20 17:12:04.916
suspended generator run_to_queue(query.py:938) raised InternalError(internal error.)

Hi, 

Can you  post an AppStats (https://cloud.google.com/appengine/docs/java/tools/appstats) profile of a successful version of this request? Also, can you discuss the datastore operations that take place before and during this error (for instance, using batch calls or one-entity calls, the size of the entities. etc)?

 

 

-----------------

-Vinny P

Technology & Media Consultant

Chicago, IL

App Engine Code Samples: http://www.learntogoogleit.com