I am using django version 1.0 and ran into shallow copy limitation of
QuerySet.values() where it returns ints for ForeignKey columns. Here
is my take on deep copy implementation.
def deep_values(o, *args, **kargs):
""" Converts instance of django.db.models object into dictionary.
Uses recursive deep copy to overcome ModelManager.values() shallow
copy.
"""
if not isinstance(o, models.Model):
return None
attribs = [attr for attr in dir(o) if not attr in args and not
attr.startswith('_') and not attr == 'objects' and not callable(getattr
(o, attr)) and not attr == 'pk' and not attr.endswith('_id')]
for a in attribs:
if not repr(type(getattr(o, a))).startswith("<class"):
dict[a] = getattr(o, a)
else:
import logging
logging.debug("%srecursive call for attribute '%s' of %s", ' ' *
lead_space * c_nlevel, a, o.__class__)
tmp = deep_values(getattr(o, a), args, nest_level=c_nlevel + 1)
if not tmp is None:
dict[a] = tmp
On Wed, 2009-03-18 at 17:59 -0700, Vitaly wrote: > I am using django version 1.0 and ran into shallow copy limitation of > QuerySet.values() where it returns ints for ForeignKey columns. Here > is my take on deep copy implementation.
Instead of leaping right into a proposed solution, can you explain the problem you're seeing, please?
I wanted json serialize a tree of django model objects: Schedule ->
Player -> django.models.User.
django.core.serializers.serialize does shallow serialization of
QuerySet but I want a deep one. Next, I looked at QuerySet.values()
plus simplejson but alas the shallow copy again.
On Mar 18, 9:55 pm, Malcolm Tredinnick <malc...@pointy-stick.com>
wrote:
> On Wed, 2009-03-18 at 17:59 -0700, Vitaly wrote:
> > I am using django version 1.0 and ran into shallow copy limitation of
> > QuerySet.values() where it returns ints for ForeignKey columns. Here
> > is my take on deep copy implementation.
> Instead of leaping right into a proposed solution, can you explain the
> problem you're seeing, please?
On Mar 19, 8:17 am, Vitaly <vit...@ufairsoft.com> wrote:
> I wanted json serialize a tree of django model objects: Schedule ->
> Player -> django.models.User.
> django.core.serializers.serialize does shallow serialization of
> QuerySet but I want a deep one. Next, I looked at QuerySet.values()
> plus simplejson but alas the shallow copy again.
> On Mar 19, 8:17 am, Vitaly <vit...@ufairsoft.com> wrote:
> > I wanted json serialize a tree of django model objects: Schedule ->
> > Player -> django.models.User.
> > django.core.serializers.serialize does shallow serialization of
> > QuerySet but I want a deep one. Next, I looked at QuerySet.values()
> > plus simplejson but alas the shallow copy again.
On Thu, 2009-03-19 at 05:17 -0700, Vitaly wrote: > I wanted json serialize a tree of django model objects: Schedule -> > Player -> django.models.User. > django.core.serializers.serialize does shallow serialization of > QuerySet but I want a deep one. Next, I looked at QuerySet.values() > plus simplejson but alas the shallow copy again.
So it's not about "copying" -- taking one Python object and creating a similar, but independent one -- at all. You're talking about how far down the relation chain we descend when retrieving data.
Bob Thomas has already point out one ticket and there are some others opened regarding pulling related models in via a values() call (searching for tickets about values() will reveal a bunch of different directions and proposals).
Your patch isn't particularly neat (examining the string representation of the output of type() to determine a class when isinstance() exists, for example). It also looks like it will fail for infinitely recursive structures (which exist in practical situations).
Utlimately, though, I think this situation is solved by allowing select_related() to work with values() -- ticket #5768 is one reference to that. It's not a trivial problem to solve, but we'll fix it one day. Multi-valued relations, in particular, require care to make them work efficiently.
I agree with you that there are some holes in code - it was a quick
hack to solve issue at hand. I did suspect that there should be some
effort to implement this feature and tickets quoted confirm that. It
is too bad that as of now it has not done yet even though tickets
appear to be 2 years old. Is there anything I could to help, please
let me know.
On Mar 19, 7:04 pm, Malcolm Tredinnick <malc...@pointy-stick.com>
wrote:
> On Thu, 2009-03-19 at 05:17 -0700, Vitaly wrote:
> > I wanted json serialize a tree of django model objects: Schedule ->
> > Player -> django.models.User.
> > django.core.serializers.serialize does shallow serialization of
> > QuerySet but I want a deep one. Next, I looked at QuerySet.values()
> > plus simplejson but alas the shallow copy again.
> So it's not about "copying" -- taking one Python object and creating a
> similar, but independent one -- at all. You're talking about how far
> down the relation chain we descend when retrieving data.
> Bob Thomas has already point out one ticket and there are some others
> opened regarding pulling related models in via a values() call
> (searching for tickets about values() will reveal a bunch of different
> directions and proposals).
> Your patch isn't particularly neat (examining the string representation
> of the output of type() to determine a class when isinstance() exists,
> for example). It also looks like it will fail for infinitely recursive
> structures (which exist in practical situations).
> Utlimately, though, I think this situation is solved by allowing
> select_related() to work with values() -- ticket #5768 is one reference
> to that. It's not a trivial problem to solve, but we'll fix it one day.
> Multi-valued relations, in particular, require care to make them work
> efficiently.
On Fri, 2009-03-20 at 05:08 -0700, Vitaly Peressada wrote: > @Malcolm:
> I agree with you that there are some holes in code - it was a quick > hack to solve issue at hand. I did suspect that there should be some > effort to implement this feature and tickets quoted confirm that. It > is too bad that as of now it has not done yet even though tickets > appear to be 2 years old. Is there anything I could to help, please > let me know.
They've been open for two years because nobody has fixed them yet and we think it's worthwhile doing (plus they're not entirely trivial to fix, so that cuts down the number of people willing to put in the effort). In the interim we've closed, you know, a *few thousand* other tickets, so progress has definitely been made.
I'll also note that parts of #5768 have been fixed, e.g., in r7230. Lots of those bigger items are multi-part projects that get fixed in a few stages.
You could work on those tickets if you want to help. I've pointed out the difficulties in comment 4 on #5768. Whether we restrict values() to only allowing one multi-valued relation or, preferably, constructing the correct SQL for querying many multi-valued relations (making sure we only return 1 + n1 +n2 rows, not n1 * n2 rows, in the notation in that comment). The latter situation is best, but hard to implement.
So start working on that if you want this solved. We aren't going to commit a hack to work around something when the real problem is known. If you get stuck, ask as many questions as you like on this list.
> On Fri, 2009-03-20 at 05:08 -0700, Vitaly Peressada wrote:
> > @Malcolm:
> > I agree with you that there are some holes in code - it was a quick
> > hack to solve issue at hand. I did suspect that there should be some
> > effort to implement this feature and tickets quoted confirm that. It
> > is too bad that as of now it has not done yet even though tickets
> > appear to be 2 years old. Is there anything I could to help, please
> > let me know.
> They've been open for two years because nobody has fixed them yet and we
> think it's worthwhile doing (plus they're not entirely trivial to fix,
> so that cuts down the number of people willing to put in the effort). In
> the interim we've closed, you know, a *few thousand* other tickets, so
> progress has definitely been made.
> I'll also note that parts of #5768 have been fixed, e.g., in r7230. Lots
> of those bigger items are multi-part projects that get fixed in a few
> stages.
> You could work on those tickets if you want to help. I've pointed out
> the difficulties in comment 4 on #5768. Whether we restrict values() to
> only allowing one multi-valued relation or, preferably, constructing the
> correct SQL for querying many multi-valued relations (making sure we
> only return 1 + n1 +n2 rows, not n1 * n2 rows, in the notation in that
> comment). The latter situation is best, but hard to implement.
> So start working on that if you want this solved. We aren't going to
> commit a hack to work around something when the real problem is known.
> If you get stuck, ask as many questions as you like on this list.
> Malcolm, I might take a stab on this later. Do you know if queryset-
> refactor branch was merged into trunk?
> On Mar 20, 10:52 pm, Malcolm Tredinnick <malc...@pointy-stick.com>
> wrote:
> > On Fri, 2009-03-20 at 05:08 -0700, Vitaly Peressada wrote:
> > > @Malcolm:
> > > I agree with you that there are some holes in code - it was a quick
> > > hack to solve issue at hand. I did suspect that there should be some
> > > effort to implement this feature and tickets quoted confirm that. It
> > > is too bad that as of now it has not done yet even though tickets
> > > appear to be 2 years old. Is there anything I could to help, please
> > > let me know.
> > They've been open for two years because nobody has fixed them yet and we
> > think it's worthwhile doing (plus they're not entirely trivial to fix,
> > so that cuts down the number of people willing to put in the effort). In
> > the interim we've closed, you know, a *few thousand* other tickets, so
> > progress has definitely been made.
> > I'll also note that parts of #5768 have been fixed, e.g., in r7230. Lots
> > of those bigger items are multi-part projects that get fixed in a few
> > stages.
> > You could work on those tickets if you want to help. I've pointed out
> > the difficulties in comment 4 on #5768. Whether we restrict values() to
> > only allowing one multi-valued relation or, preferably, constructing the
> > correct SQL for querying many multi-valued relations (making sure we
> > only return 1 + n1 +n2 rows, not n1 * n2 rows, in the notation in that
> > comment). The latter situation is best, but hard to implement.
> > So start working on that if you want this solved. We aren't going to
> > commit a hack to work around something when the real problem is known.
> > If you get stuck, ask as many questions as you like on this list.
> > Regards,
> > Malcolm
qs-rf was merged into trunk months ago, several months before 1.0.
Alex
-- "I disapprove of what you say, but I will defend to the death your right to
say it." --Voltaire
"The people's good is the highest law."--Cicero
After spending some time looking at the code I found out that my
original problem has been solved by Malcolm with #5768 by using "__"
to traverse foreign key relations. Too bad that this was not reflected
in django online doc at http://docs.djangoproject.com/en/dev/ref/models/querysets/#values-fields
Hence, working on solving outstanding multi-value relation won't bring
any benefit to my current work. Plus, getting up to speed with django
ORM code will take some spare time which I don't have :-(
Thanks everyone for their constructive comments.
On Mar 22, 9:53 pm, Alex Gaynor <alex.gay...@gmail.com> wrote:
> On Sun, Mar 22, 2009 at 9:51 PM, Vitaly Peressada <vit...@ufairsoft.com>wrote:
> > Malcolm, I might take a stab on this later. Do you know if queryset-
> > refactor branch was merged into trunk?
> > On Mar 20, 10:52 pm, Malcolm Tredinnick <malc...@pointy-stick.com>
> > wrote:
> > > On Fri, 2009-03-20 at 05:08 -0700, Vitaly Peressada wrote:
> > > > @Malcolm:
> > > > I agree with you that there are some holes in code - it was a quick
> > > > hack to solve issue at hand. I did suspect that there should be some
> > > > effort to implement this feature and tickets quoted confirm that. It
> > > > is too bad that as of now it has not done yet even though tickets
> > > > appear to be 2 years old. Is there anything I could to help, please
> > > > let me know.
> > > They've been open for two years because nobody has fixed them yet and we
> > > think it's worthwhile doing (plus they're not entirely trivial to fix,
> > > so that cuts down the number of people willing to put in the effort). In
> > > the interim we've closed, you know, a *few thousand* other tickets, so
> > > progress has definitely been made.
> > > I'll also note that parts of #5768 have been fixed, e.g., in r7230. Lots
> > > of those bigger items are multi-part projects that get fixed in a few
> > > stages.
> > > You could work on those tickets if you want to help. I've pointed out
> > > the difficulties in comment 4 on #5768. Whether we restrict values() to
> > > only allowing one multi-valued relation or, preferably, constructing the
> > > correct SQL for querying many multi-valued relations (making sure we
> > > only return 1 + n1 +n2 rows, not n1 * n2 rows, in the notation in that
> > > comment). The latter situation is best, but hard to implement.
> > > So start working on that if you want this solved. We aren't going to
> > > commit a hack to work around something when the real problem is known.
> > > If you get stuck, ask as many questions as you like on this list.
> > > Regards,
> > > Malcolm
> qs-rf was merged into trunk months ago, several months before 1.0.
> Alex
> --
> "I disapprove of what you say, but I will defend to the death your right to
> say it." --Voltaire
> "The people's good is the highest law."--Cicero
