Datastore Outage Question

[sandeep Koduri]

Hello Appengine Team,

We have stable application running on python runtime for long while. Today for a brief period the application went down, due to what appears to be a system outage. The application unable to connect to datastore, so all request trying to access data went into DeadlineExceededError. It restored automatically after about 40 min of downtime.

Below is the tracedump we found on out logs. Can anyone advise or help diagnose the issue and reason for its cause.


Also I wish to know, if anyone else see an outage? We have other applications working fine in the same period and also i dont see any notifications in appengine system status.

logging.info(data.count(20000))
  File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/db/__init__.py", line 2121, in count
    result = raw_query.Count(limit=limit, **kwargs)
  File "/base/python27_runtime/python27_lib/versions/1/google/appengine/api/datastore.py", line 2323, in Count
    for result in bound_query.Run(config=config):
  File "/base/python27_runtime/python27_lib/versions/1/google/appengine/datastore/datastore_query.py", line 2816, in next
    next_batch = self.__batcher.next()
  File "/base/python27_runtime/python27_lib/versions/1/google/appengine/datastore/datastore_query.py", line 2678, in next
    return self.next_batch(self.AT_LEAST_ONE)
  File "/base/python27_runtime/python27_lib/versions/1/google/appengine/datastore/datastore_query.py", line 2715, in next_batch
    batch = self.__next_batch.get_result()
  File "/base/python27_runtime/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 604, in get_result
    return self.__get_result_hook(self)
  File "/base/python27_runtime/python27_lib/versions/1/google/appengine/datastore/datastore_query.py", line 2452, in __query_result_hook
    self._batch_shared.conn.check_rpc_success(rpc)
  File "/base/python27_runtime/python27_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 1216, in check_rpc_success
    rpc.wait()
  File "/base/python27_runtime/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 545, in wait
    self.__rpc.Wait()
DeadlineExceededError

It would be really great if some one could share any info regarding this.

Thanks in advance.

Sandeep Koduri.