Elevated Datastore latency, error-rates, serving errors

[Brett]

We're currently seeing elevated Datastore latency and error-rates, as
well as elevated serving error-rates. This problem began around 6:30am
Pacific time. All applications accessing the Datastore are affected.
Our team is investigating this issue and will post more information as
soon as it becomes available.

-Brett
App Engine Team

[Brett (Google)]

Now in unplanned maintenance mode. Application deployments, Datastore
writes, and memcache writes have been disabled temporarily. Please
catch CapabilityDisabledError exceptions to handle this properly:

from google.appengine.ext import db
from google.appengine.runtime.apiproxy_errors import
CapabilityDisabledError

myModel = db.Model()
try:
myModel.put()
except CapabilityDisabledError:
# fail gracefully here
pass

Will update as soon as more information becomes available.

[Brett (Google)]

Read-only mode continues. Elevated latency and error-rates persist for
Datastore reads. Memcache writes have been reenabled to better soak
read-only load. Our engineering teams are looking into the root cause
of the problem. Will post more information as soon as it's available.

[Brett (Google)]

The root of the original problem has been isolated, but additional
problems have come up. Datastore reads are now failing on all
accesses. Dynamic request and static file serving ar showing elevated
error rates and 500s. We are working our hardest to rectify the
situation.

[Brett (Google)]

Datastore reads are now working normally! Dynamic and static file
serving has returned to normal. Dynamic latency and Datastore latency
have returned to normal. However, datastore writes continue to be
disabled. We are working hard to reenable writes as soon as possible.

[Brett (Google)]

Datastore writes are reenabled and functioning normally! Overall App
Engine health is back to normal! We will update this thread if
anything else develops, but at this time we anticipate no additional
problems. Thank you for your patience.