ndb BadRequestError only on production

[mma]

Hi there.

I get the following error only on production: BadRequestError: BLOB, ENITY_PROTO or TEXT properties must be in a raw_property field

It happens when I put() a instance of the Receipt class (extends ndb.Model)

Below, I attach the model and the handler where the code breaks (only in production)

class Receipt(RModel):

ownerId = ndb.IntegerProperty()

houseId = ndb.IntegerProperty()

renterId = ndb.IntegerProperty()

year = ndb.IntegerProperty()

month_number = ndb.IntegerProperty()

code = ndb.StringProperty()

description = ndb.StringProperty()

value = ndb.StringProperty()

owner = ndb.ComputedProperty(lambda self: Owner.get_by_id(self.ownerId))

house = ndb.ComputedProperty(lambda self: House.get_by_id(self.houseId))

renter = ndb.ComputedProperty(lambda self: Renter.get_by_id(self.renterId))

month = ndb.ComputedProperty(lambda self: month_number_to_string(self.month_number))

class RModel(ndb.Model):

created = ndb.DateTimeProperty(auto_now_add = True)

changed = ndb.DateTimeProperty(auto_now_add = True)

creatorId = ndb.IntegerProperty()

changerId = ndb.IntegerProperty()

#def to_dict(self):

# return ndb.to_dict(self, {'id':self.key().id()})

def set_attributes(self, **attrs):

props = self.properties()

for prop in props.values():

if prop.name in attrs:

prop.__set__(self, attrs[prop.name])

class ReceiptNew(BaseHandler):

def Get(self):

user_id = self.get_user_id()

owner = Owner.get_by_id(user_id)

receipt = Receipt(value="")

houses = list(House.gql("where ownerId = :1", owner.key.id()))

renters = list(Renter.gql("where ownerId = :1", owner.key.id()))

context = {'receipt': receipt, 'houses': houses, 'renters': renters, 'new': True}

self.render_response('receipt-edit.html', **context)

def post(self):

user_id = self.get_user_id()

owner = Owner.get_by_id(user_id)

data = {

'year': self.request.get('year'),

'month': self.request.get('month'),

'house': self.request.get('house'),

'renter': self.request.get('renter'),

'value': self.request.get('value'),

'paid': self.request.get('paid')

}

receipt = Receipt()

receipt.year = int(data.get('year'))

receipt.month_number = int(data.get('month'))

receipt.houseId = int(data.get('house'))

receipt.renterId = int(data.get('renter'))

receipt.value = data.get('value')

receipt.ownerId = owner.key.id()

receipt.put() # code breaks here, only in production

self.redirect('/receipts')

[Guido van Rossum]

On Wednesday, July 4, 2012 1:59:49 AM UTC+2, mma wrote:

Hi there.

I get the following error only on production: BadRequestError: BLOB, ENITY_PROTO or TEXT properties must be in a raw_property field

It happens when I put() a instance of the Receipt class (extends ndb.Model)

Below, I attach the model and the handler where the code breaks (only in production)

class Receipt(RModel):

ownerId = ndb.IntegerProperty()

houseId = ndb.IntegerProperty()

renterId = ndb.IntegerProperty()

year = ndb.IntegerProperty()

month_number = ndb.IntegerProperty()

code = ndb.StringProperty()

description = ndb.StringProperty()

value = ndb.StringProperty()

owner = ndb.ComputedProperty(lambda self: Owner.get_by_id(self.ownerId))

house = ndb.ComputedProperty(lambda self: House.get_by_id(self.houseId))

renter = ndb.ComputedProperty(lambda self: Renter.get_by_id(self.renterId))

month = ndb.ComputedProperty(lambda self: month_number_to_string(self.month_number))

These ComputedProperties look suspicious. The owner, house and renter lambdas return entities; are you sure you don't mean to return their keys instead? E.g. Owner.get_by_id(id) loads an Owner entity (which is a blocking datastore call). If you just want to store the key, you can use ndb.Key(Owner, id) instead.

If you really want to store these as entities, it's possible that you can get away by declaring the ComputedProperty as indexed=False. But you're probably better off declaring them as e.g. StructuredProperty(Owner), and if you want them filled in automatically, you could do that in a pre-post hook.

class RModel(ndb.Model):

created = ndb.DateTimeProperty(auto_now_add = True)

changed = ndb.DateTimeProperty(auto_now_add = True)

creatorId = ndb.IntegerProperty()

changerId = ndb.IntegerProperty()

#def to_dict(self):

# return ndb.to_dict(self, {'id':self.key().id()})

Aside: What are you trying to do here? I think maybe you meant this:

def to_dict(self):

  result = super(RModel, self).to_dict()

  result['id'] = self.key.id()

  return result

???

 

def set_attributes(self, **attrs):

props = self.properties()

for prop in props.values():

if prop.name in attrs:

prop.__set__(self, attrs[prop.name])

Aside: this looks like code from old db. Its equivalent is ent.populate(**attrs), except the latter complains if you specify a keyword that has no corresponding property. 

[mma]

Guido, thanks for taking the time to respond.

You're right on ComputedProperty. I used it the wrong way. See more about that: http://stackoverflow.com/questions/11324435/ndb-badrequesterror-only-in-production/11324606#comment14917060_11324606

You're right also on the "asides". I'm not using that part and will remove it from the code.

PS: As you might have guessed, I'm a newbie on Python and on App Engine Python. First contact was thru Udacity's CS 101 and CS 253 (web apps). I get things to work, but some part of the code is not professional. I do unit testing, though -- learned it by watching Misko Hevery videos.

[mma]

By the way, theres a wiki page on Udacity about NDB vs DB:

http://forums.udacity.com/cs253/questions/30012/gae-db-vs-ndb?page=1&focusedAnswerId=30058#30058

[Primijos]

Hi,

I'm getting also this error trying to put() [to be exact, trying to do a get_or_insert] an entity with a computed property. My Model looks more or less like this (removed non-relevant code):

class DecimalProperty(ndb.StringProperty):
   def _validate(self,value):
      if not isinstance(value,(int,long,basestring,decimal.Decimal)):
         raise ndb.BadValueError("Property %s must be a decimal, string, float or int." % self.name)
      return decimal.Decimal(value).quantize(TWOPLACES)

   def _to_base_type(self,value):
      if value>=0:
         return "P%010d" % (value*100)
      else:
         new_value = MAX_DECIMAL + (value*100)
         return "N%010d" % new_value


   def _from_base_type(self,value):
      if value[0] == "P":
         return (decimal.Decimal(value[1:])/100).quantize(TWOPLACES)
      else:
         tmp = decimal.Decimal(value[1:])
         tmp = - (MAX_DECIMAL - tmp)
         return (decimal.Decimal(tmp)/100).quantize(TWOPLACES)


class Accounting(ndb.Model):
   expense           = DecimalProperty(required=False,indexed=False,default=DEC_ZERO)
   investment        = DecimalProperty(required=False,indexed=False,default=DEC_ZERO)


def accountable_basic_compute_balance(accountable):
   expense = accountable.commited.expense - accountable.invoiced.expense
   investment = accountable.invoiced.investment - accountable.invoiced.investment
   return Accounting(expense = expense, investment = investment)


class AccountableBasic(ndb.Model):
   def __init__(self, *args, **kwargs):
      # initialize structure
      super(AccountableBasic, self).__init__(*args, **kwargs)
      self.commited = Accounting()
      self.invoiced = Accounting()

   commited = ndb.StructuredProperty(Accounting, repeated = False, indexed=True)
   invoiced = ndb.StructuredProperty(Accounting, repeated = False, indexed=True)
   balance  = ndb.ComputedProperty(accountable_basic_compute_balance, repeated = False, indexed=True)


class YearAccounting(AccountableBasic):
   # id (key_name) = year
   pass

Everything works fine in the local dev server, when I try to perform a YearAccounting.get_or_insert in production, I get the following error:

BadRequestError: BLOB, ENITY_PROTO or TEXT properties must be in a raw_property field

I'm assuming that's related to the "balance" property in YearAccouting (subclass of AccountableBasic) , which in dev shows as follows in the datastore viewer:

balance (entity:proto)

<binary>

I had some problems previously in production trying to use a ComputedProperty that returned a DecimalProperty (basically, a "total" property: a ComputedProperty that returened a DecimalProperty): dev app server complained about it, so I've removed it, however it seemed to be happy with a ComputedProperty that returned a Accounting instance, but the production runtime doen't look so happy with it...

Any help will be welcome.

thanks in advance,

Jose

[Guido van Rossum]

I don't think this has anything to do with the previous thread that had the same subject.

The problem must be that your ComputedProperty is trying returning a Model instance. That's unfortunately not supported; ComputedProperty as it is currently implemented can only support the "basic" data types like int, str and the like. I'm not sure why this doesn't trigger an error in the dev appserver for you -- when I try something similar it does trigger the error in a test.(*) I'm also not sure what's the best solution; you may just have to break up the computed structured property up into two separate computed property.

(*) My session log:

>>> class A(Model):

...  x = IntegerProperty()

... 

>>> class M(Model):

...  a = ComputedProperty(lambda self: A(x=1))

... 

>>> m = M()

>>> m.a

A(x=1)

>>> m.put()

WARNING:root:suspended generator _put_tasklet(context.py:274) raised BadRequestError(BLOB, ENITY_PROTO or TEXT property a must be in a raw_property field)

WARNING:root:suspended generator put(context.py:703) raised BadRequestError(BLOB, ENITY_PROTO or TEXT property a must be in a raw_property field)

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

  File "/Users/guido/appengine-ndb-experiment/ndb/model.py", line 3151, in _put

    return self._put_async(**ctx_options).get_result()

  File "/Users/guido/appengine-ndb-experiment/ndb/tasklets.py", line 325, in get_result

    self.check_success()

  File "/Users/guido/appengine-ndb-experiment/ndb/tasklets.py", line 368, in _help_tasklet_along

    value = gen.throw(exc.__class__, exc, tb)

  File "/Users/guido/appengine-ndb-experiment/ndb/context.py", line 703, in put

    key = yield self._put_batcher.add(entity, options)

  File "/Users/guido/appengine-ndb-experiment/ndb/tasklets.py", line 368, in _help_tasklet_along

    value = gen.throw(exc.__class__, exc, tb)

  File "/Users/guido/appengine-ndb-experiment/ndb/context.py", line 274, in _put_tasklet

    keys = yield self._conn.async_put(options, datastore_entities)

  File "/Users/guido/appengine-ndb-experiment/ndb/tasklets.py", line 454, in _on_rpc_completion

    result = rpc.get_result()

  File "/usr/local/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 604, in get_result

    return self.__get_result_hook(self)

  File "/usr/local/google_appengine/google/appengine/datastore/datastore_rpc.py", line 1569, in __put_hook

    self.check_rpc_success(rpc)

  File "/usr/local/google_appengine/google/appengine/datastore/datastore_rpc.py", line 1224, in check_rpc_success

    raise _ToDatastoreError(err)

google.appengine.api.datastore_errors.BadRequestError: BLOB, ENITY_PROTO or TEXT property a must be in a raw_property field

>>> 

--Guido

[Guido van Rossum]

On Monday, October 8, 2012 1:28:57 PM UTC-7, Guido van Rossum wrote:

I don't think this has anything to do with the previous thread that had the same subject.

Apologies, I think you had the same issue, you just got there a different way.

Further experimentation suggests that making the ComputedEntity indexed=False will make this (seem to) work. However it will behave more like a LocalStructuredProperty than a StructuredProperty -- the Account entity is serialized as a single bytestring, not stored as two separate properties with a dot in their names like the StructuredProperty(Account) properties are. I'm not sure if you're happy with that or not.