Datastore eventual consistency

[Dan]

Eventual consistency means that it will take a undetermined amount of time after a datastore entity 'put' for a query on that entity to be returned.

However am I right in thinking that if I 'put' the entity then 'get' it straight away then a subsequent query will ALWAYS return it?

This behaviour seems to be the case in the dev app server and in production but I would like confirmation that this is guaranteed to happen.

[Chad Vincent]

There are multiple types of query, and each is either Eventually Consistent, or Strongly (immediately) Consistent.

Fetch-by-key is always strongly consistent.  Query-with-ancestor is also strongly consistent.  Only query by type is eventually consistent.

[Dan]

Apologies Chad, I didn't make myself clear. I was using Python ndb terminology with 'get', 'put' and 'query'.

What I want to clarify is if an entity will always be returned from a fetch by kind if it has previously been fetched by key.

This is what I experience in testing. I also remember reading that an entity's indexed properties are written to their respective indexes, if not done already, on its first fetch by key.

Therefore can you effectively get strong consistency by adding a fetch by key after adding an entity to the datastore?

Example 1:
1) Put entity A.
2) Query for entity A by kind. // This does not always return entity A due to the eventual consistency nature of the datastore.

Example 2:
1) Put entity B.
2) Fetch entity B by key.
3) Query for entity B by kind. // Always appears to return entity B.

[Jeff Schnitzer]

The answer to your question is no.

Queries are eventually consistent because index changes are replicated between datacenters asynchronously and queries are served by any datacenter. Strongly consistent get-by-key operations require a quorum of datacenters; they have no effect on query index replication.

Note that while get-by-key is strongly consistent by default, you can explicitly request an eventually consistent get-by-key. Since it returns the first result as-is (without waiting for a quorum), it is quite a bit faster.

Also: You may be experiencing cache behavior; ndb caches at both session and memcache level.

Jeff

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.

[Dan]

Thanks for the answer.

The results I am seeing are not because of ndb caching as it does not cache queries by property.

I imagine that the datastore eventual consistency has always been fast enough so the required indexes are updated by the time I run a put, then get operation making the subsequent query by property always work so far in testing. Thanks for making it clear that there is no technical reason/guarantee to rely on this.

[Jeff Schnitzer]

Just a minor nit - I don't recall offhand exactly how ndb manages this, but I believe that the session cache can affect query results in a subtle way. It won't affect which entities come back (that is determined by walking the index), but when a result comes back, I would expect ndb to replace the fetched version by a session version in the cache. So as long as the entity was a query "hit" before the change, you may see the "new" version if it was in the cache - even though the query returned the old version.

This is pretty minor, and probably not what you are seeing, but it's worth mentioning. And it's possible I have this wrong; I haven't used ndb in a long time, but this is the behavior I would expect from a session cache.

Jeff