Tuples morphed into lists when loaded from mongo

[Russ Weeks]

Hi,

My app uses tuples to represent enum values.  We observe that when these tuples are referenced from a ComplexBaseField, they are converted into BaseList objects when loaded from mongo.  This breaks some common idioms, as shown in this testcase: https://gist.github.com/rweeks/4980509

The root cause is that _attach_objects in dereference.py converts all list-like sequences to BaseList.  I understand that the intent is to be able to watch for changes to the items of the list.

I'm new to Python, but it seems like tuples are meant to be the closest thing Python has to immutable data structures.  It seems worth it to preserve tuple-like behaviour, even if it means losing the ability to watch for changes to these objects.

The patch below to dereference.py causes the testcase to pass.  I think the best solution would be to implement BaseTuple alongside BaseList and BaseDict in base.py (looks like BasesTuple already exists?) but that's a little beyond me at this point.

-Russ

diff --git a/mongoengine/dereference.py b/mongoengine/dereference.py

index 386dbf4..fcb6d89 100644

--- a/mongoengine/dereference.py

+++ b/mongoengine/dereference.py

@@ -171,6 +171,7 @@ class DeReference(object):

 

         if not hasattr(items, 'items'):

             is_list = True

+            as_tuple = isinstance(items, tuple)

             iterator = enumerate(items)

             data = []

         else:

@@ -205,7 +206,7 @@ class DeReference(object):

 

         if instance and name:

             if is_list:

-                return BaseList(data, instance, name)

+                return tuple(data) if as_tuple else BaseList(data, instance, name)

             return BaseDict(data, instance, name)

         depth += 1

         return data

[Russ Weeks]

Any feedback on this?  I'd love to have an "officially-approved" way of keeping tuples as tuples.

To refresh/summarize: when BaseField.to_python returns a tuple, that tuple will be converted to a BaseList when the field is part of a ComplexBaseField.  For instance, a list of embedded documents.  It would be really great if the output of to_python was preserved regardless of where the field is in the document schema.

Thanks,

-Russ

[Ross Lawley]

Sorry for the delay, the main issue is that there is no tuple type in bson.  You could write a custom field but it is stored as a list.

Ross

--
You received this message because you are subscribed to the Google Groups "MongoEngine Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongoengine-us...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Russ Weeks]

Hi, Ross,

Thanks for the reply.  Understood that tuple isn't a native mongo type.  My problem is that if I write a custom field that converts a native mongo type to a tuple, that tuple is then re-converted to a list if the custom field is inside a ComplexBaseField.

I'm probably not explaining this well.  But my point is, I'm not trying to preserve tuple-like behaviour on the way _into_ mongo, I'm trying to preserve it after the data has come out of mongo and my custom field has already converted a native mongo type to a tuple.

-Russ

[Ross Lawley]

Hi Russ,

I got you, I'm not sure the best route to achieve your aims either patch ComplexBaseField and dereferencing (?) or perhaps a totally new field type.

Have you opened a ticket about this on github?

Ross

[Russ Weeks]

Hi, Ross,

Thanks for getting back to me.  I've opened issue #249 with proposed fix here: https://github.com/MongoEngine/mongoengine/pull/250

-Russ

[Ross Lawley]

Hi,

No problems - the main issue with the patch is the dirty data tracking is ignored - so changes to the list might not persist if you use save().

Ross