On returning appropriate subclass instances when querying a model with subclasses

[Andrea Zilio]

Hi all,

Here's what I'm talking about:
You have 3 models: Place, Restaurant and Cinema where Restaurant and
Cinema both inherits from Place.
So in the database you have some simple places, some places which
really are Cinemas and some places which really are Restaurants.

Well, I think that this should be possible:
>>> Place.objects.all()
[<Place>, <Cinema>, <Restaurant>, <Place>]
(Current implementation only returns Places objects)

This seems to have been taken into account here:
http://code.djangoproject.com/wiki/ModelInheritance#a2.ModelingjoinsinSQL
In that wiki page it's said that this should be possible using LEFT
JOINS, but that this approach could drive to a lot of overhead as the
subclass count increase.

But... What about allowing this kind of query-including-subclasses by
requesting it in an explicit way?
Something like this?
>>> Place.objects.select_subclasses(Cinema,Restaurant).all()
[<Place>, <Cinema>, <Restaurant>, <Place>]
>>> Place.objects.select_subclasses(Cinema).all() # Here we ask to join ONLY with Cinema
[<Place>, <Cinema>, <Place>, <Place>]

Maybe this has been previously discussed but I couldn't find any
ticket or any discussion about it.
So take this as a proposal if it wasn't already discussed or as a
request to have some info about the previous discussion.

Thanks in advance

P.S.
SqlObject seems to have implemented something like this
http://www.sqlobject.org/Inheritance.html

[Russell Keith-Magee]

On Sun, Aug 23, 2009 at 9:18 AM, Andrea Zilio<m.e...@gmail.com> wrote:
>
> Hi all,
>
> Here's what I'm talking about:
> You have 3 models: Place, Restaurant and Cinema where Restaurant and
> Cinema both inherits from Place.
> So in the database you have some simple places, some places which
> really are Cinemas and some places which really are Restaurants.
>
> Well, I think that this should be possible:
>>>> Place.objects.all()
> [<Place>, <Cinema>, <Restaurant>, <Place>]
> (Current implementation only returns Places objects)

...

> Maybe this has been previously discussed but I couldn't find any
> ticket or any discussion about it.
> So take this as a proposal if it wasn't already discussed or as a
> request to have some info about the previous discussion.

This has been discussed, at length, and rejected. If you search the
django-dev archives, the discussion happened around the time that
model inheritance was being added to the ORM. For the most part,
you're looking for a series of discussions between myself and Malcolm.
Using "CORBA" as a keyword in your search will help (and if you know
anything about CORBA, will also explain why the idea was ultimately
rejected).

The techniques to achieve this sort of thing are well known and well
established, but they do come at a cost. We opted not to wear that
cost by default in inherited models. If you want this capability, it
(or something functionally equivalent) can be added as an end-user
add-on to models.

Yours,
Russ Magee %-)

[Andrea Zilio]

Thanks! I think I've found the discussion you're talking about:
http://groups.google.com/group/django-developers/browse_thread/thread/7d40ad373ebfa912/a20fabc661b7035d

So seems that this idea was in fact rejected because of the O(# of
nodes in the inheritance tree) joins
needed to get all the fields from subclass tables.

But the idea of letting this as an optional way to retrieve data (so
not the default way, but by using something like
QuerySet.select_subclasses() as suggested on my previous email) could
be a solution to this problem or not?

Moreover I've dug a bit on how Hibernate handles inheritance and
indeed it always make outer joins to subclasses (it has to since it
doesn't use a discriminator column and returns, by default, instances
of subclasses when querying for a base class).

Anyway, what do you mean by "end-user add-on to models"?
Where can I find info to make such an add-on? Or are you just talking
about editing the django source code on my installation?

Thanks again ;)
Andrea

On Aug 23, 10:25 am, Russell Keith-Magee <freakboy3...@gmail.com>
wrote:

[Marc Fargas]

On Sun, Aug 23, 2009 at 1:54 PM, Andrea Zilio <m.e...@gmail.com> wrote:

So seems that this idea was in fact rejected because of the O(# of
nodes in the inheritance tree) joins
needed to get all the fields from subclass tables.

You may want to loook at Alex's post on the subject: http://lazypython.blogspot.com/2009/02/second-look-at-inheritance-and.html for a solution to apply on your own projects.

Note that his code was written before Malcolm wrote proxy classes, Alex's code could now be improved to not query the subclasses but return proxy classes to them effectively avoiding the O(#..) queries untill you try to access subclass data ;)

Regards,
Marc

[Andrea Zilio]

Great, but that wasn't exactly what I was looking for.

What I need is a way to get the right instances from a *single* query.

This means that I should do LEFT JOINS on direct subclasses of the
base class.

What I need to know is if this is somehow possible without editing the
Django core, but using some signals ore something else.
Or if this is planned for some future release.

I need to do something like this:
http://groups.google.com/group/django-users/browse_thread/thread/2d1c286b37a153d3

Thanks in advance ;)
Andrea

On Aug 23, 3:06 pm, Marc Fargas <teleni...@telenieko.com> wrote:

> On Sun, Aug 23, 2009 at 1:54 PM, Andrea Zilio <m.ep...@gmail.com> wrote:
>
> > So seems that this idea was in fact rejected because of the O(# of
> > nodes in the inheritance tree) joins
> > needed to get all the fields from subclass tables.
>

> You may want to loook at Alex's post on the subject:http://lazypython.blogspot.com/2009/02/second-look-at-inheritance-and...

[Russell Keith-Magee]

On Sun, Aug 23, 2009 at 11:00 PM, Andrea Zilio<m.e...@gmail.com> wrote:
>
> Great, but that wasn't exactly what I was looking for.
>
> What I need is a way to get the right instances from a *single* query.
>
> This means that I should do LEFT JOINS on direct subclasses of the
> base class.
>
> What I need to know is if this is somehow possible without editing the
> Django core, but using some signals ore something else.

When I said before that it was possible to emulate this in user space,
I meant you could do something like the following:

* Add a klass column to your base model.
* On subclasses, populate this column with a class-specific identifying value.
* Add a narrow() function (naming borrowed from CORBA) on the base
class that uses the value of klass to determine the subclass instance
that needs to be retrieved
* Whenever you need a list of subclasses, use a list comprehension
that incorporates narrow:
[ obj.narrow() for obj in BaseClass.objects.all() ]

It's not spectacularly efficient - it does lots of joins and lots of
database traffic, but it works.

> Or if this is planned for some future release.

As I said previously, this isn't planned for a future release - in
fact, it's planned that we _wont_ do this.

Yours,
Russ Magee %-)

[Andrea Zilio]

So the answer to the question "Can I get the right instances with one
single and only DB query" seems to be:
"No, you cannot do so without touching the Django orm code".

Am I right?

On Aug 24, 1:52 am, Russell Keith-Magee <freakboy3...@gmail.com>
wrote:
>

> > Or if this is planned for some future release.
>
> As I said previously, this isn't planned for a future release -  in
> fact, it's planned that we _wont_ do this.

I tried :)

[Russell Keith-Magee]

On Mon, Aug 24, 2009 at 9:14 AM, Andrea Zilio<m.e...@gmail.com> wrote:
>
> So the answer to the question "Can I get the right instances with one
> single and only DB query" seems to be:
> "No, you cannot do so without touching the Django orm code".
>
> Am I right?

If, by "one single and only DB query", you mean a single Django ORM
query - you are correct.

Yours,
Russ Magee %-)

[Carl Meyer]

On Aug 23, 9:14 pm, Andrea Zilio <m.ep...@gmail.com> wrote:
> So the answer to the question "Can I get the right instances with one
> single and only DB query" seems to be:
> "No, you cannot do so without touching the Django orm code".
>
> Am I right?

Actually, I think the ORM's extensibility mechanisms are sufficient
that you could provide your QuerySet.select_subclasses() method, which
would perform a single query with n LEFT JOIN's, with custom QuerySet/
Query subclasses. Writing it would certainly require a non-trivial
level of familiarity with the ORM internals, and you might have to use
private APIs - but I don't think you'd have to modify Django code.

Carl

[Alex Gaynor]

Actually there's already a ticket to cover the ability to do that type
of joining. Ticket 7270 works to add select_related support to
reverse 1-1 relations, which is all that inheritance is.

Alex

--
"I disapprove of what you say, but I will defend to the death your
right to say it." -- Voltaire
"The people's good is the highest law." -- Cicero
"Code can always be simpler than you think, but never as simple as you
want" -- Me