Key query support in Google App Engine

[Christian Foster Howes]

Hi all,

attached is a patch for gql.py. Please review and Massimo, if people
like this, can we add it to trunk?

what does it do? it allows you to do key queries on Google App Engine.
this means that you can now perform all ID queries on tables in google
app engine. for example:

belongs = db(db.test_tbl.id.belongs([69126,69127])).select()
eq = db(db.test_tbl.id==69126).select()
neq = db(db.test_tbl.id!=69126).select()
lt = db(db.test_tbl.id<69126).select()
gt = db(db.test_tbl.id>69126).select()
lte = db(db.test_tbl.id<=69126).select()
gte = db(db.test_tbl.id>=69126).select()
all = db(db.test_tbl.id>0).select()

it also adds "__key__" to _extra on each row in the result...just in
case you really wanted to see that google key.

if i missed some test cases please let me know and i'll verify them as well.

thanks!

Christian

[mdipierro]

The belongs is fine but I have a couple of problems:

1) adding __key__ in extra is not backward compatible because it
changes the way records in Rows is acceessed

2) if you say db(db.table.id<1000) it does not actually select records
if id<1000 but Key(id)<Key(1000).

Did I misunderstand the source?

>  gql_key_query_patch.diff
> 3KViewDownload

[howesc]

1) i don't mind removing __key__ from extra. it was just something i
was using while figuring it out. (though i don't understand what it
changes in how Rows is accessed).

2) The Key on GAE is a base64 encoded string of "table_name:
id=<id>" (for the simple no-ancestor case). My understanding of how
GAE executes the query is that it decodes the key, sorts on the
numeric id, and then filters, so the transformation to the GAE Key
class should not cause problems. at least that is how i understand
it (and how it works in my limited usage so far). see
http://code.google.com/appengine/docs/python/datastore/queriesandindexes.html#Queries_on_Keys
for a little better explanation.

Does that help? shall i make a new patch w/o the __key__ in _extra?

thanks!

christian

[mdipierro]

As long as this works the same if the old queries:

id==value and id>0

yes.

If you could also avoid the trunk of changing the name='__key__' and
restoring name='id' it would be better. You could add a new attrbute
to the field object and use that for the check: _iskey=True,

On Jun 9, 11:25 pm, howesc <how...@umich.edu> wrote:
> 1) i don't mind removing __key__ from extra.  it was just something i
> was using while figuring it out.  (though i don't understand what it
> changes in how Rows is accessed).
>
> 2) The Key on GAE is a base64 encoded string of "table_name:
> id=<id>" (for the simple no-ancestor case).  My understanding of how
> GAE executes the query is that it decodes the key, sorts on the
> numeric id, and then filters, so the transformation to the GAE Key
> class should not cause problems.   at least that is how i understand

> it (and how it works in my limited usage so far).  seehttp://code.google.com/appengine/docs/python/datastore/queriesandinde...

[Christian Foster Howes]

Here is a new version of the patch to address the issues:

- no more __key__ in _extra (and in understand why it breaks backward
compatibility now....just needed to think about it)
- since it is illegal in web2py to have "_" start a field name,
"__key__" can never be a field name and never queried on. so removed
the support that i had tried to add in my previous version of the patch
- since previously web2py required that id on GAE be a long, i
continued that theme, which simplifies the code. basically the id>0 and
id==x case were already special cases in the code, if you use any other
operators on ID this version of the code detects field.type=='id' and
will use "__key__" in the query (regardless of the name you have
assigned your key, and with out my changing the name hack)

hopefully you agree that this is better, and thanks for your patience
and code review! let me know if there are any other changes that i
should make.

christian

[mdipierro]

Thanks I will apply this tonight.

>  gql_key_query_patch.diff
> 3KViewDownload

[Christian Foster Howes]

Did some more testing today...i left out the corrections for orderby.
the attached patch includes that bit as well. sorry for the extra file.

i don't think there are any other clauses that you can put ID in, is
that correct?

also, if anyone is using GAE development server with sqlite, key queries
don't work properly there (spent 4 hours trying to figure out what i
messed up to find out that others reported it as a bug to google).

christian

On 06/09/2010 08:54 PM, Christian Foster Howes wrote: