NDB: Querying for a value of None on a repeated property

[Janne Savukoski]

Hi,

I couldn't find an earlier question about this, so could someone explain why: “Querying for a value of None on a repeated property has undefined behavior; don't do that.”[0]

This sounds very weird. Is this some curiosity with the datastore itself, or just some complication within the NDB library?

I've used NDB for years, but I learned about this issue just now as my friend spotted the peculiar notice. I've got no idea whether this has caused undefined behaviour in some of the systems I've worked on as I would've *never* expected this kind of behaviour. Though, if NDB raises an exception about this, then I can be sure that I haven't encountered this in practice.

[0] https://cloud.google.com/appengine/docs/standard/python/ndb/queries

Thanks,

Janne

[Jordan (Cloud Platform Support)]

It comes down to how the Datastore distinguishes if an Entity property has a value or not. If a property has no value (aka it is never set) than the Datastore is unable to filter on that specific property (since it doesn't have it) and will not return that Entity in a query. This is explained in the documentation under "Entities lacking a property named in the query are ignored"

The workaround is to assign a default value to a property such a null. Once a value is assigned it will then be visible to a filter and can be returned in a query. 

- Note that Google Groups is for general product discussions and not for technical support. If you require further Datastore technical support it is recommended to post your full detailed question to Stack Overflow using the supported Cloud tags.    

[Janne Savukoski]

Hi Jordan!

Thanks for the reply. However, I didn't quite catch how the answer is related to my question. But, maybe I can clarify the question with this quote: “To be eligible as a query result, an entity must possess a value (possibly null) for every property named in the query's filters and sort orders.” [0]

So, you can query None/null values. (Note: this has nothing to do with missing/unset values.) I'm also using this *a lot*, and never experienced any issues.

But for lists (with NDB at least), querying with None/null is undefined: “Querying for a value of None on a repeated property has undefined behavior; don't do that.” [1]

Is there an explanation for this difference?

I tested this, and NDB does seem to work as I was assuming. But, I have no idea whether I can trust that behaviour, due to the quote [1]. (If it really is undefined, as stated.)

My test procedure, very simple:

First I created an entity of

class SomeEntity(ndb.Model):

    field_abc = ndb.IntegerProperty()

with field_abc=None.

Then I created another entity of

class SomeEntity(ndb.Model):

    field_abc = SparseIntegerProperty(repeated=True)

with field_abc=[1, 2, None].

where

class SparseIntegerProperty(ndb.Property):

    def _db_set_value(self, v, unused_p, value):

        if isinstance(value, (int, long)):

            v.set_int64value(value)

    def _db_get_value(self, v, unused_p):

        if v.has_int64value():

            return int(v.int64value())

(Yes, I've used such a “sparse” property in a significant production application for many years.)

And the query works as I was expecting:

s~my-project> SomeEntity.query(SomeEntity.field_abc == None).fetch()

[SomeEntity(key=Key('SomeEntity', 3892735977), field_abc=[None]),

SomeEntity(key=Key('SomeEntity', 3896345977), field_abc=[1, 2, None])]

But, to repeat, I don't know whether I can trust this behaviour.

ps. I do not believe this is a “technical support” question.

pps. doesn't this related to ndb.StructuredProperty as well? If a sub-entity has a field with Null value, and you query by it? Aren't the sub-entities indexed simply as repeated fields? Thus, you'd query a list by a None value, in a very concrete fashion. I haven't checked this yet…

-Janne

On Wednesday, November 29, 2017 at 10:50:45 PM UTC+2, Jordan (Cloud Platform Support) wrote:

It comes down to how the Datastore distinguishes if an Entity property has a value or not. If a property has no value (aka it is never set and is None) than the Datastore is unable to filter on that specific property (since it doesn't have it) and will not return that Entity in a query. This is explained in the documentation under "Entities lacking a property named in the query are ignored"

[Janne Savukoski]

Oh, I forgot to mention: for the test, I commented out this line in ndb/model.py:

    assert b_val is not None, "Cannot wrap None"

I did a bit more testing with queries and everything seemed to work as I expected. Namely, I tested whether None values would interfere with some other kind of queries, but I found no such cases.

Also, the missing references from last post:

[0] https://cloud.google.com/datastore/docs/concepts/queries#restrictions_on_queries

[1] https://cloud.google.com/appengine/docs/standard/python/ndb/queries

-Janne

[Jordan (Cloud Platform Support)]

So you are physically setting the 'None' object to your 'field_abc'. As per the documentation, this is indeed working as intended as you have provided it with a value. If you were to instead just perform 'field_abc = ndb.IntegerProperty()' where it is optional for this Entity property to have a value, querying for 'None' will produce undefined behavior. 

Therefore it is suggested to always provide a default value for your Entity properties. You may want to have a look at existing Stack Overflow posts covering this for more technical information.