Datastore timeout

[Joshua Fox]

Getting a lot of these exceptions. I note that there were similar reports a month ago (below). Also, though we mostly use Java, I note that there was a similar error in Python (below).

What is happening here?

 com.google.cloud.datastore.DatastoreException: The datastore operation timed out, or the data was temporarily unavailable. at

com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.translate(HttpDatastoreRpc.java:129) at

com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.commit(HttpDatastoreRpc.java:155) at

com.google.cloud.datastore.DatastoreImpl$5.call(DatastoreImpl.java:418) at

com.google.cloud.datastore.DatastoreImpl$5.call(DatastoreImpl.java:415) at

com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:89) at

com.google.cloud.RetryHelper.run(RetryHelper.java:74) at

com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:51) at

com.google.cloud.datastore.DatastoreImpl.commit(DatastoreImpl.java:414) at

com.google.cloud.datastore.DatastoreImpl.commitMutation(DatastoreImpl.java:408) at

com.google.cloud.datastore.DatastoreImpl.put(DatastoreImpl.java:368) at

File "/base/alloc/tmpfs/dynamic_runtimes/python27/6baee03590d046f1_unzipped/python27_lib/versions/1/google/appengine/ext/datastore_admin/backup_handler.py", line 1456, in _create_kind_backup backup_info.key(), kind): File "/base/alloc/tmpfs/dynamic_runtimes/python27/6baee03590d046f1_unzipped/python27_lib/versions/1/google/appengine/ext/db/__init__.py", line 2330, in next return self.__model_class.from_entity(self.__iterator.next()) File "/base/alloc/tmpfs/dynamic_runtimes/python27/6baee03590d046f1_unzipped/python27_lib/versions/1/google/appengine/datastore/datastore_query.py", line 3321, in next next_batch = self.__batcher.next_batch(Batcher.AT_LEAST_OFFSET) File "/base/alloc/tmpfs/dynamic_runtimes/python27/6baee03590d046f1_unzipped/python27_lib/versions/1/google/appengine/datastore/datastore_query.py", line 3207, in next_batch batch = self.__next_batch.get_result() File "/base/alloc/tmpfs/dynamic_runtimes/python27/6baee03590d046f1_unzipped/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 613, in get_result return self.__get_result_hook(self) File "/base/alloc/tmpfs/dynamic_runtimes/python27/6baee03590d046f1_unzipped/python27_lib/versions/1/google/appengine/datastore/datastore_query.py", line 2906, in __query_result_hook self._batch_shared.conn.check_rpc_success(rpc) File "/base/alloc/tmpfs/dynamic_runtimes/python27/6baee03590d046f1_unzipped/python27_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 1385, in check_rpc_success raise _ToDatastoreError(err) Timeout: The datastore operation timed out, or the data was temporarily unavailable.

On Thu, Feb 15, 2018 at 10:17 PM, Justin Yip <yipj...@leap.ai> wrote:

Good to know that my team is not alone. :)

On Thursday, February 15, 2018 at 12:14:48 PM UTC-8, Kevin Pope wrote:

Currently experiencing the same issue on all datastore writes.

On Thursday, February 15, 2018 at 12:05:37 PM UTC-8, Justin Yip wrote:

Hello,

Our team noticed that all our services (across multiple projects) are not able to write anything to datastore since 11:43am PST. We are getting the following error:

err: rpc error: code = DeadlineExceeded desc = The datastore operation timed out, or the data was temporarily unavailable.

We also tried to update entities directly on the the datastore web console, it also failed.

Reading from datastore works normally.

We wonder if there is an issue going on. And if anyone has experienced the same problem?

Thanks!

Justin