Delete Entire Namespace ... Entity Still Persist ... What does eventual consistency really mean?

[James Gilliam]

I deleted a namespace today ... every kind and entity in the namespace, everything!  I kept fetching and deleting rows until there were no more to delete.

Ten hours later, I did a count on every kind in the previous namespace.

To my surprise, some tables are still showing entities -- not every time you do a count; but if you do the count 20 times, it will show rows 3 or 4 of those times.

Why is delete not really delete?

What does eventual consistency mean?

How long exactly do I have to wait for the records to all be deleted?

Very frustrating dealing with a datastore that doesn't seem to work reliably.

Here is the output of successive counting the entities in a kind -- notice the 4s

So, 80% of the time there are no entities, 20% there are 4 rows.

Any GAE help out there?

count 4

count 0

count 4

count 0

count 4

count 0

count 0

count 0

count 0

count 0

count 0

count 0

count 0

count 0

count 4

count 0

count 0

count 0

count 0

count 0

[Wolfram Gürlich]

Hi James, 

what you see is a combination of two things: (a) indexes are eventually consistent - meaning that there is a lag between writing and them reflecting the current state and (b) replication which also happens to be eventual consistent meaning that not all replicas have the same state. Each replica (datacenter) has a complete copy of all your data AND all your indexes.

When you do a query that query will hit exactly one (randomly choosen) datacenter (replica). So your result set might be stale because that replica hasn't yet applied all your writes. The next query may give you a very different result because it hits another replica.

I've experienced inconsistencies for more than 12 hours which isn't the regular case though. At least 99.9% of the time inconsistencies will last for less than a second. But sometimes after REALLY heavy updates (a couple 100K writes within minutes) I've seen a small portion of entities remaining stale for hours.

From what I've seen, most of the time an inconsistency will instantly vanish (on all replicas) as soon as you access the stale entity group consistently (doing a GET). The datastore detects the inconsistency and applies the writes that are still pending on some replica. Never will a consistent operation (like GET) give you a stale result (provided you haven't changed your read policy to inconsistent).

Wolfram

[Jim Gilliam]

Wolfram --

Thank you for the response.

I will do as you suggest ... access the stale group with a GET to clear up the stale replica.

Also, in rare cases, we have seen a get for a single row in the datastore, return an old result and the current result randomly,  many days after the item was updated.  You can reload the page with the data on it and literally see it flip between the two versions of the row.  I assume this is because the row is being retrieved from different replica.  Any thoughts on that and how we can resolve it?

Thanks again for your help.

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/OkCjziExt74/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[James Gilliam]

One final note ...

All of the inconsistencies that I was seeing on Sunday and Monday are now gone, including the one that I reported here regarding the deleted namespace.  It is now consistently reporting no records.

I did my read test on about a dozen other situations and they are all passing now.

Thanks for your efforts