Stale Data from App Engine Datastore

[Thomas Becker]

I am using the app engine datastore with Java JDO. According to the documentation, ancestor queries are always strongly consistent. Recently, however, I've been seeing a lot of stale data from ancestor queries. It seems to happen randomly. A query may return the most recent data, and a moment later, the same query returns stale data.

[Ryan (Cloud Platform Support)]

I would take a look at this article, specifically the second paragraph under 'Datastore Read Policy and Call Deadline'.:

'

If the query uses an ancestor filter, you can use transactions to ensure a consistent result set.) See the article Transaction Isolation in App Engine for more information on how entities and indexes are updated.
'

[Thomas Becker]

Thanks for the reply, Ryan. Actually, I need to apologize because the way I described my problem was incorrect. I'm seeing stale data not from queries, but from calls to the JDO function GetObjectById(). The calls are inside a transaction. Since the analogous low-level datastore access function get() is guaranteed to return strongly consistent data, I was under the impression that the same would be true for GetObjectById(). Am I perhaps wrong about this?

[Ryan (Cloud Platform Support)]

If you can create a sample app that shows this post on it on the PIT so I can investigate this further.

On Saturday, August 1, 2015 at 6:24:55 PM UTC-4, Thomas Becker wrote:

[Thomas Becker]

Thanks for your quick reply, Ryan. I have found several postings on the Web that describe the same problem, namely, GetObjectById returning stale data. Here are two from StackOverflow:

http://stackoverflow.com/questions/26377347/inconsistent-fetch-from-google-app-engine-datastore

http://stackoverflow.com/questions/22005349/appengine-jdo-with-multiple-server-instances-loses-transaction-updates-on-entiti

Under the circumstances, I don't think it would be a good use of my time to make a demo app. It seems clear to me that the issue exists. Besides, the problem is so intermittent that a demo app is of very limited value. I would be perfectly happy with a statement of the kind, "GetObjectById() does not guarantee strong consistency, that's how it is, and here's what you do instead to get strongly consistent data." I just need to know what's going on.

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/1re1FtqllnQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/9723734e-0b14-49c0-8c31-3407df8454ad%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jeff Schnitzer]

I'm no expert on JDO, but the low-level api does guarantee strong consistency for get-by-key operations (that is, unless you explicitly ask for eventual consistency). One of those SO responses suggested it is possible to have the GAE/JDO plugin log its low-level operations - that should at least point you in the right direction.

One other thing that might be related: In the low-level API, within a single transaction, all reads will return the original value even if you write it within the transaction. I don't know what JDO does with this; I would expect that subsequent reads would return a session-cached instance and hide this little quirk. But maybe it's related to what you are seeing?

Jeff

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.

To post to this group, send email to google-a...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CAPxj-GA3OKiGfjwgYgKmvZ%3DSi7ZXG3VLvts7%2BjWZ7c2CsC693Q%40mail.gmail.com.

[Thomas Becker]

Thanks for your detailed and thoughtful answer, Jeff. I was aware of the fact that within a transaction, all reads will return the original value even if one writes it within the transaction. That was not the problem, but thanks for pointing it out anyway, it's easy to forget!

Other than that, after reading everything on the subject of GAE datastore consistency under JDO, I have decided to fall back to low-level access in those cases where strong consistency is of the essence. At the risk of sounding acerbic or snarky, it seems clear to me that nobody knows what's going on in JDO when it comes to consistency.

So just to confirm, it is true that the low-level get-by-key guarantees strong consistency even when performed outside of a transaction, correct?

Thanks again for your thoughtful answer. Very refreshing indeed.

Thomas

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CADK-0uh4Q8kswYMhD5RcxeQJnO3rFshgkfGEbB%2BXzHVXQLbGiw%40mail.gmail.com.

[Jeff Schnitzer]

Yes, that is correct. By default, if you perform a low-level get-by-key operation, it is strongly consistent whether or not you put it in a transaction.

Note that you can, if you want, explicitly request an eventually consistent get-by-key operation in the low-level api. It does return faster since it does not require a quorum response. However, this is a pretty obscure feature.

Jeff

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CAPxj-GApGFORBdvJd88QvitX_bHLGvAtiQ943WBd0V3Ro9onZw%40mail.gmail.com.

[Thomas Becker]

Thanks, Jeff! And one more thing, it's pretty clear from the doc, but let me make sure anyway. The same is true for ancestor queries, they guarantee strong consistency even when not performed inside a transaction, right?

Thomas

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CADK-0ujMaP_xRg6Pjht%3DsCR%3Dc-3hGEJvhXGtOT0h1vcMKhQBqQ%40mail.gmail.com.

[Prashant Gupta]

I am facings exact same issue. getObjectById returns stale data even if call is made inside a transaction.

I performed an experiment - I ran two crons in two different modules, both trying to increment their execution count in an (common, shared) entity. I was shocked to see that even when I transaction,  both crons are running as if they are performing transaction in thier own copy of DataStore. Evertime, each cron will get copy of entity last updated by itself (ignoring updates made by other cron) and on top of that transaction will also commit without any exception.

--
Prashant

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CAPxj-GBwjD%2BGO9LDk5ObgTQo7w5%2B7wnN32%3D664CrVg%2BPxD_6nw%40mail.gmail.com.

[Jeff Schnitzer]

I have less direct experience here since my apps tend to be get-by-key-heavy, but the docs are very explicit about ancestor queries being strongly consistent outside of a transaction by default and I believe them. Of course, you can override this (although I would not recommend it unless you know what you are doing).

Jeff

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CAPxj-GBwjD%2BGO9LDk5ObgTQo7w5%2B7wnN32%3D664CrVg%2BPxD_6nw%40mail.gmail.com.

[Jeff Schnitzer]

Again, JDO is not my area of expertise, but if so, this seems like a shockingly obvious issue. Can you post a sample of the code you use to demonstrate the problem?

Jeff

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CAPpOf-t%3Dzp2uRUnkN9446_QoSXN6XCKdzNAe1g4GwULN8gg93A%40mail.gmail.com.

[Thomas Becker]

Thanks again for eveybody's input. It looks like I have solved my problem by falling back on the low-level datastore access when strong consistency matters. Problems like this happen, and the bottom line is GAE is a great product that's done a lot for me. So kudos to all members, former and present, of the GAE team at Google.

FWIW, here are two observations that could be helpful in understanding the behavior of the gae/JDO function GetObjectById:

1) If you set the option "Unapplied job percentage" to 100% in the SDK, that is, you force "eventually consistent" to mean "never consistent," then GetObjectById still acts strongly consistent. This means that the SDK thinks that GetObjectById should act strongly consistent, just like the low level get function.

2) Demonstrating that GetObjectById can return stale data in the production environment is difficult because the behavior is, by its very nature, intermittent. But one can increase the probability of seeing stale data vastly if one performs the write and subsequent read from two different versions of the app. As others have observed, the probability already increases when write and read are performed from different instances of  the app. But with different versions (one of them being the default version), it's been happening pretty much every time in my context. The low-level get, by contrast, has always acted strongly consistent for me, even across versions.

Thomas

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CADK-0uhSRuiH6a3Wn%3DAYYm2DF5GjvQnT1FCVAnjBkga6MF6pew%40mail.gmail.com.

[Jeff Schnitzer]

This sounds to me not like an issue with eventual consistency, but an issue with session caching. The thing about different versions is that it guarantees requests to go to different instances.

I don't remember much about how JDO works, but are you sure you are opening/closing sessions correctly?

Jeff

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CAPxj-GBWY%2BQXoN%2BS3sX4YbO%2BuhZjd8svpUMnjwzTahzOFd-%3Dyw%40mail.gmail.com.

[Thomas Becker]

I believe that session caching is specific to the Objectify interface. Since I'm not using that, I don't think there even is a way for me to open/close any sessions. I'm quite sure that there is no such a thing as a "JDO session." (There is of course the HTTP session that's provided by the Java servlet environment, but that's got nothing to do with the datastore.)

Thomas

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CADK-0uimyzpW0N5Rhv_OtQ05hrvfKxnKotnv0C_N%3D024SunE7g%40mail.gmail.com.

[Ryan (Cloud Platform Support)]

Check out this link specifically:

"Changes are persisted when the PersistenceManager is closed"


I ran some test to confirm and not closing resulted in eventual while closing resulted in strong consistency. Give it a try and let me know if that helps.

On Saturday, August 1, 2015 at 6:24:55 PM UTC-4, Thomas Becker wrote:

[Jeff Schnitzer]

JDO absolutely does have a session. You start a session by opening a persistencemanager and end by closing it. When you close the session, JDO does its dirty change detection and writes out any alterations. Also, repeated loads from an open session will return the same object. JPA works the same way. 

Objectify's session is very similar to this (minus the dirty change detection).

If you aren't closing your PM sessions... this would explain the problem.

Jeff

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CAPxj-GCTgAZ-rCchRCD-ZWaaj%3DTbZJaz9GhBHond3Acw0ZutMA%40mail.gmail.com.

[Thomas Becker]

Oh, yeah, that's right, there is the closing of the persistence manager. All my JDO calls go through one function that has all the boilerplate, and it ends with

            } finally {
                pm.close();
            }
        }

So we can safely assume that this is not the problem.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CADK-0uhKzJLtTuAS6h812cY9tsg%3DTPnfgXDJZw%2BB6mG-rabb1Q%40mail.gmail.com.

[Thomas Becker]

Thanks for the suggestion, Ryan. I checked my code, and all my JDO calls go through one function that has all the boilerplate. The last three four lines of that function are,

            } finally {
                pm.close();
            }
        }

so we can safely assume that the persistence manger always gets closed.

But let me emphasize again, my original post on this was outright wrong: it's not ancestor queries that are giving me the problem, it is specifically the JDO function GetObjectById. That begs the question, of course, if GetObjectById is even supposed to give strong consistency. Re-reading the documentation very carefully, I found that the chapter on the low-level datastore access states that getting entities by id via get() gives strong consistency. But the chapter on JDO does not make a similar statement about GetObjectById! On the other hand, when you set the option "Unapplied job percentage" to 100% in the development environment, the function GetObjectById will still act strongly consistent. So perhaps that is the only problem here, that the SDK erroneously treats GetObjectById as strongly consistent, when really it is not.

Thomas

--

You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/1re1FtqllnQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/96de2818-896a-4877-aaa0-d496cc81dc06%40googlegroups.com.