Massive HR Datastore failing. Transaction committed but old data coming back on read.

[Ray]

During the last 6-8 hours I have been experiencing massive datastore failure on my HR app.  My application behaves normally but occasionally when the data is read back, they are getting old records without all the changes in the last hour or so. 

The app has been running for 2 years and no code has changed.  It seems not affecting all datastore txn but it is happening to a lot of them.  

Anyone else experiencing the same issue?

[Jeff Schnitzer]

Is it possible you are experiencing longer eventuality than normal?
There are no guarantees about how long "eventual" will be - it should
be rare, but it could be hours if a datacenter was running behind.

Also... "running for 2 years and no code has changed" can't possibly
be right. The HRD is a little over a year old.

Jeff

[Ray]

They are all get_by_id so no.

On Tuesday, March 27, 2012 9:37:35 AM UTC+8, Jeff Schnitzer wrote:

[jon]

We've just received user reports and we haven't had a chance to
investigate fully but I think we're experiencing an identical issue.

Our users report that their recorded actions in the past 1-2 hours
aren't always visible when the data is read from the datastore.

Ray is the problem caused by get_by_id() not returning a record even
though the supplied ID is valid?

On Mar 27, 12:45 pm, Ray <windz...@gmail.com> wrote:
> They are all get_by_id so no.
>
>
>
>
>
>
>
> On Tuesday, March 27, 2012 9:37:35 AM UTC+8, Jeff Schnitzer wrote:
>

> > On Mon, Mar 26, 2012 at 8:55 PM, Ray <@gmail.com <windz...@gmail.com>>

[jon]

Yep I've confirmed that we are suffering an identical problem.

I've opened a production issue here:
http://code.google.com/p/googleappengine/issues/detail?id=7219

I wouldn't mind you starring the issue too, Ray.

[Ray]

I am not sure if it is the identical issue, but HR datastore is definitely having big problem for me.

My main concern is data is committed within transaction but after a few hours, old data would be retrieved even with get_by_id.  (Not because of cached in memcache/client/etc).  No user is reporting the problem in the last few hours but it affected a large amount of users around 10 hours before.