Datastore to Firestore(datastore mode) automatic migration: request timeout while accessing datastore after REDIRECT_STRONGLY_CONSISTENT_READS step

[Sandeep]

I am using Objectify(v5) for accessing Datastore, in App Engine Standard Environment, with Java. Entities are cached by Objectify automatically, and I am also using Memcache separately.

This issue of Datastore APIs timing out started happening exactly post REDIRECT_STRONGLY_CONSISTENT_READS migration step. Strangely, it happens roughly after every hour or 2 hours, lasts for 3-4 mins and then gets back to normal. Since requests latency is going up from ~200ms to more than 60s, lots of new instances are getting created and am getting charged heavily.

here are some of the errors:

java.util.concurrent.CancellationException: Task was cancelled.    at      com.google.common.util.concurrent.AbstractFuture.cancellationExceptionWithCause(AbstractFuture.java:1550)    at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:590)    at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:467)    at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:122)    at com.google.appengine.tools.development.TimedFuture.get(TimedFuture.java:55)    at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:68)    at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:89)    at com.google.appengine.api.datastore.Batcher$ReorderingMultiFuture.get(Batcher.java:114)    at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:89)    at com.googlecode.objectify.cache.TriggerFuture.get(TriggerFuture.java:100)

and this:

com.googlecode.objectify.cache.EntityMemcache getAll: Error obtaining cache for [<dummy-entity-name>]
    java.util.concurrent.CancellationException: Task was cancelled.

and this:

java.lang.InterruptedException at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:460)

I have upgraded to latest versions of dependencies:

implementation 'com.google.appengine:appengine-api-1.0-sdk:2.0.5'
implementation 'com.googlecode.objectify:objectify:5.1.25'

and flushed cache as well. nothing has worked.

I am sure it is related to Datastore/memcache and not Objectify. Still want to know if you guys have any idea about this.

Thanks

[Jeff Schnitzer]

I'm afraid this is probably a question for someone at Google :-(

Jeff

--
You received this message because you are subscribed to the Google Groups "objectify-appengine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to objectify-appen...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/objectify-appengine/96dd74f0-3988-46fc-af86-54e46a983558n%40googlegroups.com.

[Sandeep]

Yeah that's what I assumed. Reached out to Cloud Support team, their Datastore/Firestore Product team looked at it, paused the migration, fixed something and then completed the migration. Now it's working fine again.

Thanks for your reply, and awesome work with Objectify!

Sandeep