QuerySet.values() Shallow Copy

[Vitaly]

I am using django version 1.0 and ran into shallow copy limitation of
QuerySet.values() where it returns ints for ForeignKey columns. Here
is my take on deep copy implementation.

def deep_values(o, *args, **kargs):
""" Converts instance of django.db.models object into dictionary.
Uses recursive deep copy to overcome ModelManager.values() shallow
copy.
"""
if not isinstance(o, models.Model):
return None

dict = {}
lead_space = 2
c_nlevel = kargs.get('nest_level', 0)

attribs = [attr for attr in dir(o) if not attr in args and not
attr.startswith('_') and not attr == 'objects' and not callable(getattr
(o, attr)) and not attr == 'pk' and not attr.endswith('_id')]
for a in attribs:
if not repr(type(getattr(o, a))).startswith("<class"):
dict[a] = getattr(o, a)
else:
import logging
logging.debug("%srecursive call for attribute '%s' of %s", ' ' *
lead_space * c_nlevel, a, o.__class__)
tmp = deep_values(getattr(o, a), args, nest_level=c_nlevel + 1)
if not tmp is None:
dict[a] = tmp

return dict

[Malcolm Tredinnick]

On Wed, 2009-03-18 at 17:59 -0700, Vitaly wrote:
> I am using django version 1.0 and ran into shallow copy limitation of
> QuerySet.values() where it returns ints for ForeignKey columns. Here
> is my take on deep copy implementation.

Instead of leaping right into a proposed solution, can you explain the
problem you're seeing, please?

Regards,
Malcolm

[Vitaly]

I wanted json serialize a tree of django model objects: Schedule ->
Player -> django.models.User.
django.core.serializers.serialize does shallow serialization of
QuerySet but I want a deep one. Next, I looked at QuerySet.values()
plus simplejson but alas the shallow copy again.


On Mar 18, 9:55 pm, Malcolm Tredinnick <malc...@pointy-stick.com>
wrote:

[Bob Thomas]

On Mar 19, 8:17 am, Vitaly <vit...@ufairsoft.com> wrote:
> I wanted json serialize a tree of django model objects: Schedule ->
> Player -> django.models.User.
> django.core.serializers.serialize does shallow serialization of
> QuerySet but I want a deep one. Next, I looked at QuerySet.values()
> plus simplejson but alas the shallow copy again.
>

You may want to look at http://code.djangoproject.com/ticket/4656
instead, since I think that patch does what you're looking for.

-bob

[Vitaly Peressada]

Thanks, Bob. Added comment to the ticket.

On Mar 19, 12:03 pm, Bob Thomas <robert.w.tho...@gmail.com> wrote:
> On Mar 19, 8:17 am, Vitaly <vit...@ufairsoft.com> wrote:
>
> > I wanted json serialize a tree of django model objects: Schedule ->
> > Player -> django.models.User.
> > django.core.serializers.serialize does shallow serialization of
> > QuerySet but I want a deep one. Next, I looked at QuerySet.values()
> > plus simplejson but alas the shallow copy again.
>

> You may want to look athttp://code.djangoproject.com/ticket/4656

[Malcolm Tredinnick]

On Thu, 2009-03-19 at 05:17 -0700, Vitaly wrote:
> I wanted json serialize a tree of django model objects: Schedule ->
> Player -> django.models.User.
> django.core.serializers.serialize does shallow serialization of
> QuerySet but I want a deep one. Next, I looked at QuerySet.values()
> plus simplejson but alas the shallow copy again.

So it's not about "copying" -- taking one Python object and creating a
similar, but independent one -- at all. You're talking about how far
down the relation chain we descend when retrieving data.

Bob Thomas has already point out one ticket and there are some others
opened regarding pulling related models in via a values() call
(searching for tickets about values() will reveal a bunch of different
directions and proposals).

Your patch isn't particularly neat (examining the string representation
of the output of type() to determine a class when isinstance() exists,
for example). It also looks like it will fail for infinitely recursive
structures (which exist in practical situations).

Utlimately, though, I think this situation is solved by allowing
select_related() to work with values() -- ticket #5768 is one reference
to that. It's not a trivial problem to solve, but we'll fix it one day.
Multi-valued relations, in particular, require care to make them work
efficiently.

Regards,
Malcolm

[Vitaly Peressada]

@Malcolm:

I agree with you that there are some holes in code - it was a quick
hack to solve issue at hand. I did suspect that there should be some
effort to implement this feature and tickets quoted confirm that. It
is too bad that as of now it has not done yet even though tickets
appear to be 2 years old. Is there anything I could to help, please
let me know.

On Mar 19, 7:04 pm, Malcolm Tredinnick <malc...@pointy-stick.com>
wrote:

[Malcolm Tredinnick]

On Fri, 2009-03-20 at 05:08 -0700, Vitaly Peressada wrote:
> @Malcolm:
>
> I agree with you that there are some holes in code - it was a quick
> hack to solve issue at hand. I did suspect that there should be some
> effort to implement this feature and tickets quoted confirm that. It
> is too bad that as of now it has not done yet even though tickets
> appear to be 2 years old. Is there anything I could to help, please
> let me know.

They've been open for two years because nobody has fixed them yet and we
think it's worthwhile doing (plus they're not entirely trivial to fix,
so that cuts down the number of people willing to put in the effort). In
the interim we've closed, you know, a *few thousand* other tickets, so
progress has definitely been made.

I'll also note that parts of #5768 have been fixed, e.g., in r7230. Lots
of those bigger items are multi-part projects that get fixed in a few
stages.

You could work on those tickets if you want to help. I've pointed out
the difficulties in comment 4 on #5768. Whether we restrict values() to
only allowing one multi-valued relation or, preferably, constructing the
correct SQL for querying many multi-valued relations (making sure we
only return 1 + n1 +n2 rows, not n1 * n2 rows, in the notation in that
comment). The latter situation is best, but hard to implement.

So start working on that if you want this solved. We aren't going to
commit a hack to work around something when the real problem is known.
If you get stuck, ask as many questions as you like on this list.

Regards,
Malcolm

[Vitaly Peressada]

Malcolm, I might take a stab on this later. Do you know if queryset-
refactor branch was merged into trunk?

On Mar 20, 10:52 pm, Malcolm Tredinnick <malc...@pointy-stick.com>
wrote:

[Alex Gaynor]

qs-rf was merged into trunk months ago, several months before 1.0.

Alex
--
"I disapprove of what you say, but I will defend to the death your right to say it." --Voltaire
"The people's good is the highest law."--Cicero

[Vitaly Peressada]

After spending some time looking at the code I found out that my
original problem has been solved by Malcolm with #5768 by using "__"
to traverse foreign key relations. Too bad that this was not reflected
in django online doc at http://docs.djangoproject.com/en/dev/ref/models/querysets/#values-fields

Hence, working on solving outstanding multi-value relation won't bring
any benefit to my current work. Plus, getting up to speed with django
ORM code will take some spare time which I don't have :-(

Thanks everyone for their constructive comments.