Datastore Internal Error

[Vinod Kumar]

Hi,

We are facing issue with production application. It's some datastore internal error. Looks like data reads and writes are randomly failing. At start the frequency was more, but then decreased. It's been almost one hour, but still not resolved completely. Please find stacktrace below.

  File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/ext/ndb/model.py", line 3458, in _put

    return self._put_async(**ctx_options).get_result()

  File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 383, in get_result

    self.check_success()

  File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 427, in _help_tasklet_along

    value = gen.throw(exc.__class__, exc, tb)

  File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/ext/ndb/context.py", line 824, in put

    key = yield self._put_batcher.add(entity, options)

  File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 427, in _help_tasklet_along

    value = gen.throw(exc.__class__, exc, tb)

  File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/ext/ndb/context.py", line 358, in _put_tasklet

    keys = yield self._conn.async_put(options, datastore_entities)

  File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 513, in _on_rpc_completion

    result = rpc.get_result()

  File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 613, in get_result

    return self.__get_result_hook(self)

  File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 1879, in __put_hook

    self.check_rpc_success(rpc)

  File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 1371, in check_rpc_success

    raise _ToDatastoreError(err)

InternalError: internal error.

Can anyone suggest anything??

[Alex (Cloud Platform Support)]

Hi Vinod,

In case this is still happening, I will need you to provide me with the following information in order to help you further:

   - The Cloud SDK version you are using.

   - A minimal working example of your code demonstrating this issue.

Let me know if you would prefer to be contacted privately by email.

[Marcel Manz]

Hi Alex

We experienced similar DataStore errors on Feb 14, starting after 08:44 UTC in our app hosted in the European region.

Next to those DS errors we also experienced the TransientFailureExceptions as mentioned in one of the other threads.

I can send you our app ID in a PM for review, in case this helps further. But assuming as other users experienced similar issues, can you please check and let us know if there in fact have been some known problems related to this?

Thanks

Kind regards,
Marcel

[Josh Moore]

This was a known issue.  We will be following up with an incident report which will be posted on the Cloud Status Dashboard and cover all occurrences of this issue on 2017-02-14.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/322ac366-f5cb-4b2d-8c03-78bf717d8e4e%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Josh Moore |

Cloud Solutions Engineer |

josh...@google.com |

[Sandeep Koduri]

Hello Josh,

Is this issue still open, we have many services raising this error today and yesterday. This is happening with get/put/fetch calls. We are not sure what all data is lost or what needs to be done to control this. 

I don't see any status update on https://status.cloud.google.com/. Please let know if we need put services with writes disabled to avoid bad user experience.

On Wednesday, February 15, 2017 at 8:38:08 AM UTC-8, joshuamo wrote:

This was a known issue.  We will be following up with an incident report which will be posted on the Cloud Status Dashboard and cover all occurrences of this issue on 2017-02-14.

On Wed, Feb 15, 2017 at 8:14 AM, Marcel Manz <marce...@gmail.com> wrote:

Hi Alex

We experienced similar DataStore errors on Feb 14, starting after 08:44 UTC in our app hosted in the European region.

Next to those DS errors we also experienced the TransientFailureExceptions as mentioned in one of the other threads.

I can send you our app ID in a PM for review, in case this helps further. But assuming as other users experienced similar issues, can you please check and let us know if there in fact have been some known problems related to this?

Thanks

Kind regards,
Marcel

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.

Visit this group at https://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/322ac366-f5cb-4b2d-8c03-78bf717d8e4e%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.