Datastore indexing taking 30 seconds

[Patrice Bertrand]

We have been using Datastore for many years now, and yesterday we encountered a problem never met before:  indexind of newly created entities was taking an incredibly long time, up to 30 seconds.   Since there are many patterns in our programs where it is assumed that created entities will be included in upcoming queries within a few seconds, this abnormal indexing time causes many grave unintended consequences.

To make sure this was not due to a change on our side, we tested a version that had been deployed several weeks ago, and had been working fine, but found the same issue.

Fortunately, this only happened in a project that is used for testing and integration purposes.   But having the same issue in our production project would be quite bad.

Has anyone encountered such thing before ?   I see no sign of anything related to Datastore issues in Google Cloud Status Dashboard.

[Thin Rhino]

Patrice,

I am assuming after you wrote a record(s) and made a query to read the same record(s) you did not get it in the result?

This could be due to the fact that Datastore is "eventually consistent".

If you are facing the problem as I have understood, try to read the record by specifying cache and use_memcache = False,

eg: record = key.get(use_cache=False, use_memcache=False)

Cheers

ThinRhino

--
You received this message because you are subscribed to the Google Groups "Google Cloud Datastore" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gcd-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gcd-discuss/d01304f2-cb73-4e0e-9432-fd7b0b339c16n%40googlegroups.com.

--

"Ship is safe at harbor but it is not made for that."

[Patrice Bertrand]

Hello ThinRhino,

Thank you for your input.   My issue is in the indexing, and thus in the queries not in a simple get().   I don't think query results are cached in memcache/redis.  We could try a query().fetch(use_cache=False, use_memcache=False), but I think caching only relates to the fetch() part, not to the actual query.
So my issue is in the delay between creating a new datastore entity with a put(), and being able to retrieve it in a subsequent query().   I have often noticed that a small delay could exist, one or two seconds, before datastore indexing was completed and the query was OK.   So in some cases we introduced a small delay, which is not ideal but is manageable.   Yesterday however, and again today, we have huge delays.   But ultimately, if we wait for say 30 seconds, the newly created entity does end up in the query results, which confirms that it was actually created, and was "in the process of being indexed".
Does this ring a bell ?

Patrice

[Patrice Bertrand]

I have filed an issue: https://issuetracker.google.com/issues/189340139

Hope a solution is found before this happens on a critical GCP project.

[Thin Rhino]

Patrice,

Datastore is an "eventually" consistent database [1], which means queries generally run faster, but may occasionally return stale results.

NDB library internally uses a cache to read and serve data (https://cloud.google.com/appengine/docs/standard/python/ndb)

One of the main features of the NDB library is
- Integrated automatic caching, which typically gives fast and inexpensive reads via an in-context cache and Memcache.

Your issue is not that of data getting indexed, but just a case of read before the datastore became consistent. We see this issue on a regular basis.

Cheers
ThinRhino

1: https://cloud.google.com/datastore/docs/concepts/structuring_for_strong_consistency#consistency_levels

To view this discussion on the web visit https://groups.google.com/d/msgid/gcd-discuss/1456aba0-1863-429a-b728-df141a76ee17n%40googlegroups.com.

[Patrice Bertrand]

I was aware of this "eventual consistency" thing, but I thought it was a matter of seconds, not minutes.    And when it is seconds, mostly less than one, it is manageable.   When it gets to minutes (almost one minute that is), then it is not.
The strangest thing is that this phenomenon stopped this morning:  it is back to  normal, and my snippet gets the newly created entity within the query almost instantly, while it took 44 seconds yesterday.  
I don't know if this is good news however:  unpredictability is a new issue in itself.
If we could relate this to some production incident that would have happened yesterday, at least there would be an explanation other than "it may take one second or it may take one minute".

In reply to my issue report https://issuetracker.google.com/issues/189340139, I was advised to use read_consistency=ndb.STRONG in the fetch in order to enforce strong consistency, and I will probably end up doing this.   But it's hard to believe this is the final explanation, particularly since this had been working (i.e. delay of less than a second) for years now, and stopped working (i.e. delay of 44 seconds) two days ago, then went back to normal today.