foreignKey + manipulator...performance problem

9 views
Skip to first unread message

gabor

unread,
Aug 30, 2006, 3:08:08 PM8/30/06
to django...@googlegroups.com
hi,

imagine 2 simple models like:


=====================
class Thing(Model):
name = CharField(maxlength=500)
=====================


=====================
class Owner(Model):
name = CharField(maxlength=500)
thing = ForeignKey(Thing)
=====================

now let's say you have 20000 Thing objects.

everything is ok, until you call

Owner.AddManipulator()... this will take a looooooooong time.

i think it's because somehow the AddManipulator goes through all the
20000 Thing objects. i have no idea why btw. why would he need it at
this point? i can understand him needing it when the formwrapper wants
to render the field as a selectbox, but for the rest of time...why?

i tried to do something like Owner.AddManipulator(follow={'thing':False}),

and this helps, but from that point on, of course the manipulator does
not validate the thing-field in the Owner object.

i could live with that (i can validate that one input data quite easily),
but then i cannot save that object using the manipulator...

is there some way to speed up this situation?

to somehow put 'back' that thing-field in manipulator-save..?

the other way for me is to completely bypass the manipulator, but the
object with which i work is quite complex, and i'd rather not do that.

any ideas?

gabor

Gábor Farkas

unread,
Aug 31, 2006, 6:23:55 AM8/31/06
to django...@googlegroups.com
gabor wrote:
> hi,
>
> imagine 2 simple models like:
>
>
> =====================
> class Thing(Model):
> name = CharField(maxlength=500)
> =====================
>
>
> =====================
> class Owner(Model):
> name = CharField(maxlength=500)
> thing = ForeignKey(Thing)
> =====================
>
> now let's say you have 20000 Thing objects.
>
> everything is ok, until you call
>
> Owner.AddManipulator()... this will take a looooooooong time.
>
> i think it's because somehow the AddManipulator goes through all the
> 20000 Thing objects. i have no idea why btw. why would he need it at
> this point? i can understand him needing it when the formwrapper wants
> to render the field as a selectbox, but for the rest of time...why?
>


ok, to make a long story short,
you need to use the raw_id_admin = True flag in the ForeignKey,
and all will be fine.

it's also in the documentation (model_api), but what's not mentioned is
that not only works in the admin-interface, but also globally.

apologies to the django-team for assuming that they did not solve this
isse :-)

gabor

Lau...@gmail.com

unread,
Aug 31, 2006, 8:53:06 AM8/31/06
to Django users
What do you mean with "2000 thing objects"? Are there 2000 records in
the database for that table, or did you have 2000 records in memory at
runtime?

Gábor Farkas

unread,
Aug 31, 2006, 11:06:20 AM8/31/06
to django...@googlegroups.com

if you don't use raw_id_admin, then (from my understanding at least),
the manipulator loads all of them into the memory.
so you have all the objects from the database-table in the memory.

gabor

josep...@gmail.com

unread,
Sep 1, 2006, 12:39:35 AM9/1/06
to Django users

You know, I really couldn't believe it - so I went home this evening
and made a test case to work it through. Sure enough, it did *exactly*
that... I've posted a short sample of the performance degredation graph
on my blog at


http://www.rhonabwy.com/wp/2006/08/31/a-subtle-django-performance-hit/

If anyone wants the "test case" project, it is available at
http://www.rhonabwy.com/django_perf_test.zip

-joe

gabor

unread,
Aug 31, 2006, 1:16:28 PM8/31/06
to django...@googlegroups.com
josep...@gmail.com wrote:
>
> Gábor Farkas wrote:
>> Lau...@gmail.com wrote:
>>> What do you mean with "2000 thing objects"? Are there 2000 records in
>>> the database for that table, or did you have 2000 records in memory at
>>> runtime?
>>>
>> if you don't use raw_id_admin, then (from my understanding at least),
>> the manipulator loads all of them into the memory.
>> so you have all the objects from the database-table in the memory.
>
> You know, I really couldn't believe it - so I went home this evening
> and made a test case to work it through. Sure enough, it did *exactly*
> that... I've posted a short sample of the performance degredation graph
> on my blog at
>
>
> http://www.rhonabwy.com/wp/2006/08/31/a-subtle-django-performance-hit/
>

nice :)

i'm not sure if i agree with the idea of having raw_id_admin=True by
default (hmmm..maybe i do... i don't know), but i definitively think
that this flag should be mentioned in a more "frequently read" place
than the model_api docs... maybe in the tutorials even.

gabor

Jorge Gajon

unread,
Sep 2, 2006, 9:18:19 PM9/2/06
to django...@googlegroups.com
Hi,

On 8/30/06, gabor <ga...@nekomancer.net> wrote:
> everything is ok, until you call
>
> Owner.AddManipulator()... this will take a looooooooong time.

I got this same problem yesterday, however I'm short in time so I
couldn't look deeper on what to do. I ended passing
follow={'field':False} and then saving the object manually but I don't
like that and will try to do something better later.

I think this should be put on a ticket as an enhancement, the
manipulator should load the foreign data only (and just only) when you
actually put a {{ form.foreign_field }} in your template and the html
select tag needs to be rendered.

I can't write up the ticket because I need to go in 5 minutes :(

Regards,
Jorge

Jorge Gajon

unread,
Sep 2, 2006, 9:21:55 PM9/2/06
to django...@googlegroups.com
On 9/2/06, Jorge Gajon <jorge...@gmail.com> wrote:
> I think this should be put on a ticket as an enhancement, the
> manipulator should load the foreign data only (and just only) when you
> actually put a {{ form.foreign_field }} in your template and the html
> select tag needs to be rendered.

Or maybe just wait for the impending removal of manipulators :)

James Bennett

unread,
Sep 2, 2006, 9:28:23 PM9/2/06
to django...@googlegroups.com
On 9/2/06, Jorge Gajon <jorge...@gmail.com> wrote:
> Or maybe just wait for the impending removal of manipulators :)

The class django.forms.Manipulator is going away, as are classes
derived from it. The functionality they provide, however, is not going
away -- it's just going to be handled differently and more cleanly.


--
"May the forces of evil become confused on the way to your house."
-- George Carlin

Joseph Heck

unread,
Sep 3, 2006, 6:02:25 PM9/3/06
to django...@googlegroups.com
Is there a wiki page that outlines where the current thinking is leading?

-joe

Jorge Gajon

unread,
Sep 4, 2006, 12:22:04 PM9/4/06
to django...@googlegroups.com
Hi,

On 8/31/06, Gábor Farkas <ga...@nekomancer.net> wrote:
> you need to use the raw_id_admin = True flag in the ForeignKey,
> and all will be fine.

Great!, Thank you Gábor :)
I was having the same problem too.

Cheers,
Jorge

Reply all
Reply to author
Forward
0 new messages