GAE HR commit/visibility + timestamp

[Julio Otuyama]

GAE HR is one step towards scalability, separating commit and visibility, but one step back in usability. This separation creates a problem (the "eventual consistency"?) that kills its use in some  application designs. Usually, I commit inside a http request and redirect to another page that shows the results, but I have to make a small delay (many milliseconds) to try to get the changes of that commit. This is the easiest workaround I find for this, not a fix. Unfortunately, I do not have a guarantee that I got the changes of that commit. Due to multiple computing synchronism paradigm, I think it is not possible to know how long the change updates will take, what could take milliseconds, minutes or even days (in an improbable case).

I was wondering I could "fix" this by using timestamps, what are easy pass to another request in a query string or to store in a session memcache variable. If I have the timestamp of my last commit, any later query to the database could use this timestamp to check if the retrieved data was committed after that timestamp, if not, it retries the query again and again. This way I have the guarantee of the correct visibility ("strong consistency"?), even if takes a long time. It can eventually read data committed by another user/process after that timestamp, what I do not consider a problem. Is this approach correct?

I can implement this with a timestamp field in every table, but a lib would be better (built in timestamp fields, automatic tuning of polling time, internal cache session timestamp of last commit, and maybe some low level database callback instead of the polling). Is there a third party lib like this? I know ORM libs that use GAE low level database APIs, but I have no idea how to start something like that.

[Joshua Smith]

The only thing that is “eventually consistent” is queries. If you go and fetch an item directly, you’re going to get the thing you sent to the db. Every time.

So the workaround is simply and reliable:

1. If you are just changing items, query just for keys, and then query those keys to get the actual data. In python, I use this snippet:

class HRModel(db.Model):

  @classmethod

  def gql_with_get(cls, query_string, *args, **kwds):

    return filter(None, db.get(db.GqlQuery('SELECT __key__ FROM %s %s' % (cls.kind(), query_string), *args, **kwds)))

It does the GQL query to get the keys, and then gets them, and filters out any that were missing.

2. If you are adding an item, just tell the page you redirected to “also check this one”  in a parameter in the URL.

It’s really not that big a deal.

-Joshua

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.

[Julio Otuyama]

Does it work on two different connections (two different requests)? I suppose in the same connection the database may store a cache that can give the correct visibility, but this could be lost with another connection. The documentation says that there are two phases, the commit phase and the apply phase. The apply phase writes the entity data, so I think it can not be visible just after the commit, even when you try to get it by key, I can be wrong. Of course, most of the time the get by key will work, but is it a guarantee?

[Joshua Smith]

If you get by key, you are guaranteed to get the last thing you put there.

https://cloud.google.com/appengine/docs/python/datastore/structuring_for_strong_consistency

[Julio Otuyama]

Thanks.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsub...@googlegroups.com.