GAE NDB is there a problem with using belongs on an id field?

[Russ King]

I am struggling to use belongs on GAE+NDB it appears that:

    quests = db(db.question.level.belongs([1])).select(db.question.id,db.question.level)  is valid  however

    quests = db(db.question.id.belongs([1])).select(db.question.id, db.question.level) generates an error along the lines of 

<type 'exceptions.AttributeError'> type object 'question' has no attribute '__key__'

is this because the key field is different from the other fields and is there some sort of workaround to retrieve a set of records from a list of keys or am I missing something

Regards

Russ

[Massimo Di Pierro]

This should work, can you please open a ticket? Mind that it is pointless to select columns/fields on GAE since it always fetches them all.

[dlypka]

It looks like GAE has the 'Projection Query' feature to allow selecting a subset of columns / properties:

https://developers.google.com/appengine/docs/python/datastore/projectionqueries

[Quint]

Ok,

I did not test this but from what can see from here the line below needs to be changed. (This line does not yet consider NDB.)

(in dal.py -> GoogleDatastoreAdapter)

def BELONGS(self, first, second=None):
 if not isinstance(second, (list, tuple, set)):
   raise SyntaxError("Not supported")
 if not self.use_ndb:
   if isinstance(second, set):
     second = list(second)
 if first.type == 'id':
   second = [Key.from_path(first._tablename, int(i)) for i in second]
   second = [self.keyfunc(first._tablename, int(i)) for i in second]

 return [GAEF(first.name, 'in', second, lambda a, b:a in b)]

Quint

On Saturday, August 30, 2014 1:37:07 AM UTC+2, Russ King wrote:

[Russ King]

Many thanks for reply however this doesn't seem to be a complete fix - code is failing on the 'in'  piece of this filter function and I am not clear how this relates to the belongs code up above.  

Regards

Russ

    GAE_FILTER_OPTIONS = {

        '=': lambda q, t, p, v: q.filter(getattr(t,p) == v),

        '>': lambda q, t, p, v: q.filter(getattr(t,p) > v),

        '<': lambda q, t, p, v: q.filter(getattr(t,p) < v),

        '<=': lambda q, t, p, v: q.filter(getattr(t,p) <= v),

        '>=': lambda q, t, p, v: q.filter(getattr(t,p) >= v),

        '!=': lambda q, t, p, v: q.filter(getattr(t,p) != v),

        'in': lambda q, t, p, v: q.filter(getattr(t,p).IN(v)),

        }

    def filter(self, query, tableobj, prop, op, value):

        return self.GAE_FILTER_OPTIONS[op](query, tableobj, prop, value)

On Mon, Sep 1, 2014 at 12:32 PM, Quint <muijse...@gmail.com> wrote:

Ok,

I did not test this but from what can see from here the line below needs to be changed. (This line does not yet consider NDB.)

def BELONGS(self, first, second=None):
 if not isinstance(second, (list, tuple, set)):
   raise SyntaxError("Not supported")
 if not self.use_ndb:
   if isinstance(second, set):
     second = list(second)
 if first.type == 'id':
   second = [Key.from_path(first._tablename, int(i)) for i in second]
   second = [self.keyfunc(first._tablename, int(i)) for i in second]

 return [GAEF(first.name, 'in', second, lambda a, b:a in b)]

Quint

On Saturday, August 30, 2014 1:37:07 AM UTC+2, Russ King wrote:

--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to a topic in the Google Groups "web2py-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/web2py/_Od_UkKW05w/unsubscribe.
To unsubscribe from this group and all its topics, send an email to web2py+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Quint]

Could you post the stacktrace?

[Quint]

I will try to test this later but I can tell you now how "IN" relates to "BELONGS".

GAE does not have "belongs" and web2py solves this by using "IN". Also you must realize that GAE performs a separate subquery for every item with a IN clause. So this is very inefficient. You're also limited to 30 subqueries and thus 30  items in a "IN".

def BELONGS(self, first, second=None):
 if not isinstance(second, (list, tuple, set)):
  raise SyntaxError("Not supported")
 if not self.use_ndb:
  if isinstance(second, set):
   second = list(second)
 if first.type == 'id':

  second = [self.keyfunc(first._tablename, int(i)) for i in second]
 return [GAEF(first.name, 'in', second, lambda a, b:a in b)]

[Quint]

Looking at the code again, I see there are 5 more instances of:

Key.from_path

I think they should all be replaced by:

self.keyfunc

(Not necessarily related to the issue btw)

On Monday, September 1, 2014 1:32:33 PM UTC+2, Quint wrote:

Ok,

I did not test this but from what can see from here the line below needs to be changed. (This line does not yet consider NDB.)

(in dal.py -> GoogleDatastoreAdapter)

def BELONGS(self, first, second=None):
 if not isinstance(second, (list, tuple, set)):
   raise SyntaxError("Not supported")
 if not self.use_ndb:
   if isinstance(second, set):
     second = list(second)
 if first.type == 'id':
   second = [Key.from_path(first._tablename, int(i)) for i in second]
   second = [self.keyfunc(first._tablename, int(i)) for i in second]

 return [GAEF(first.name, 'in', second, lambda a, b:a in b)]

Quint

On Saturday, August 30, 2014 1:37:07 AM UTC+2, Russ King wrote:

[Russ King]

I have established at least a work-around to this issue by updating google.py file in adapters and  creating a new class to use instead of GAEF when ndb is being used:

class GAEFNDB(object):

    #This is a work around to belongs not working on id fields when using ndb

    def __init__(self,name,op,value,apply):

        self.name=name=='id' and '_key' or name

        self.op=op

        self.value=value

        self.apply=apply

    def __repr__(self):

        return '(%s %s %s:%s)' % (self.name, self.op, repr(self.value), type(self.value))

and then if the code for belongs is then updated to 

    def BELONGS(self,first,second=None):

        if not isinstance(second,(list, tuple, set)):

            raise SyntaxError("Not supported")

        if not self.use_ndb:

            if isinstance(second,set):

                second = list(second)

        if first.type == 'id':

            second = [self.keyfunc(first._tablename, int(i)) for i in second]

            if self.use_ndb:

                return [GAEFNDB(first.name,'in',second,lambda a,b:a in b)]

        return [GAEF(first.name,'in',second,lambda a,b:a in b)]

it seems to work again - however it's not terribly elegant and it seems to be totally beyond me to complete the testing process on GAE such that it could be submitted as a bug-fix.  However perhaps someone can review and advise.

Cheers

Russ

[Quint]

There was something else I wanted to check.

But I need the stack trace first.

Could you post it?

[Quint]

I suspect we need to add something like this in google.py.

But I cannot test anything at the moment so maybe you can try this?

def filter(self, query, tableobj, prop, op, value):

    # NDB uses "_key" in stead of "__key__"
    if prop == "__key__": prop = "_key"

 
    return self.GAE_FILTER_OPTIONS[op](query, tableobj, prop, value)

[Russ King]

Hi Quint

Below is the trace - hope it hepls.

Russ

Traceback (most recent call last):

  File "C:\web2pysrcdev\web2py\gluon\restricted.py", line 224, in restricted

    exec ccode in environment

  File "C:\web2pysrcdev\web2py\applications\gaetest\controllers/default.py", line 128, in <module>

  File "C:\web2pysrcdev\web2py\gluon\globals.py", line 393, in <lambda>

    self._caller = lambda f: f()

  File "C:\web2pysrcdev\web2py\applications\gaetest\controllers/default.py", line 56, in belongs

    recs = db(db.testtable.id.belongs(insertlist)).select()

  File "C:\web2pysrcdev\web2py\gluon\dal\objects.py", line 2064, in select

    return adapter.select(self.query,fields,attributes)

  File "C:\web2pysrcdev\web2py\gluon\dal\adapters\google.py", line 539, in select

    (items, tablename, fields) = self.select_raw(query,fields,attributes)

  File "C:\web2pysrcdev\web2py\gluon\dal\adapters\google.py", line 456, in select_raw

    items = self.filter(items, tableobj, filter.name, filter.op, filter.value)

  File "C:\web2pysrcdev\web2py\gluon\dal\adapters\google.py", line 361, in filter

    return self.GAE_FILTER_OPTIONS[op](query, tableobj, prop, value)

  File "C:\web2pysrcdev\web2py\gluon\dal\adapters\google.py", line 357, in <lambda>

    'in': lambda q, t, p, v: q.filter(getattr(t,p).IN(v)),

AttributeError: type object 'testtable' has no attribute '__key__'