Django & django-polymorphic

[Hugo Osvaldo Barrera]

Hi there,

I'd love to see some (most?) of the features from django-polymorphic on
django itself. I'm very much be willing to work on this myself. Of
course, there's several details that need to be discussed too, HOWEVER,
but before I even start, I'd like to know if they'd even be acceptable
in django, or, for some reason, the functionality has been kept out of
django-core.

Would a well designed PR to incorporate this be acceptable? Is it worth
opening a discussion? Or has this been rejected before, and my searching
skills failed me?

Also on this topic; should this discussion be opened here, or rather as
an issue over at code.djangoproject.com?

Thanks,

--
Hugo Osvaldo Barrera

[Tim Graham]

I'm not sure. Did you find this past discussion: https://groups.google.com/d/topic/django-developers/tSLuqu0UFrY/discussion ? The idea was also mentioned a month ago with no replies: https://groups.google.com/d/topic/django-developers/Ov91x7AXwmM/discussion

Are you a maintainer of django-polymorphic? What's the advantage of bringing it into core as opposed to keeping it as an external project?

This is the correct place for the discussion (non-trivial design decisions).

[Cristiano Coelho]

How's the performance of django-polymorphic compared to using select_related of childs of interest and then handle them manually in the code with hasattr ? Does it resolve to multiple joins as the select_related option, or does it perform multiple queries? Inheritance using multiple tables is always a big headache :(

[Curtis Maloney]

On 11/02/16 09:22, 'Hugo Osvaldo Barrera' via Django developers

(Contributions to Django itself) wrote:
> Hi there,
>
> I'd love to see some (most?) of the features from django-polymorphic on
> django itself. I'm very much be willing to work on this myself. Of
> course, there's several details that need to be discussed too, HOWEVER,
> but before I even start, I'd like to know if they'd even be acceptable
> in django, or, for some reason, the functionality has been kept out of
> django-core.

As I recall it, the general approach here is: is there a reason this
can't live as a 3rd party app?

Would it benefit significantly by being more deeply integrated into
core, and if so, does it bring sufficient benefit to users to warrant
the changes?

There was some research by Sebastian Vetter on "single-child
auto-resolve" solutions for MTI and how they scaled (from memory,
model-utils' InheritanceManager, polymorphic, and generic-m2m)
https://github.com/elbaschid/mti-lightbulb

--
Curtis

[Hugo Osvaldo Barrera]

Hi all,

 

On Wed, Feb 10, 2016, at 19:38, Tim Graham wrote:

I'm not sure. Did you find this past discussion: https://groups.google.com/d/topic/django-developers/tSLuqu0UFrY/discussion ? The idea was also mentioned a month ago with no replies: https://groups.google.com/d/topic/django-developers/Ov91x7AXwmM/discussion

 

Sorry, I failed to find that. Most external links seem to be broken, sadly, but, more importantly: the discussion seemed to reach an end with no final decision.

 

 

Are you a maintainer of django-polymorphic? What's the advantage of bringing it into core as opposed to keeping it as an external project?

 

I am not. I have submited a few PRs though, and did the django1.9-compatibility work, but I'm not a maintainer.

 

On Wed, Feb 10, 2016, at 21:39, Cristiano Coelho wrote:

How's the performance of django-polymorphic compared to using select_related of childs of interest and then handle them manually in the code with hasattr ? Does it resolve to multiple joins as the select_related option, or does it perform multiple queries? Inheritance using multiple tables is always a big headache :(

 

django_polymorphic executes 1+N queries, where N is the amount of subclasses all the selected objects have (NOT the amount of objects).

 

So if you had:

 

    class Venue(models.Model):

        pass

 

    class Restaurant(Venue):

        pass

 

    class ParkingLot(Venue):

        pass

 

Selecting objects would perform *at most* 3 queries (and would perform only 1 if all objects are of class Venue). (I know, it's a bad real-life example).

 

I'd avoid touching the default queryset, and actually add an option `.polymorphic` like:

 

    Venue.objects.polymorphic.filter(is_real=False)

 

so the default behaviour does not change.

 

In regards to performance of django_polymorphic vs `select_related`, the latter is probably more performance, but the code would probably be a lot cleaner for the former. Note that it's not just about object fields, but also it's class (which can include additional methods on each child, or overloaded methods).

 

On Wed, Feb 10, 2016, at 22:13, Curtis Maloney wrote:

>

> [...]

>

> As I recall it, the general approach here is:  is there a reason this

> can't live as a 3rd party app?

>

 

Mostly, increased simplicity in maintenance, code, and following the latest django release. The current approach extends lots of django classes in an external  module to alter them, which isn't generally very clean.

 

I also get a feeling that this is a very sought after feature in django.

 

 

> Would it benefit significantly by being more deeply integrated into

> core, and if so, does it bring sufficient benefit to users to warrant

> the changes?

>

 

I'm interested in performing the changes myself (in case you wonder about the effort in it). Benefits to end-user are the fact that polymorphic features won't ever lag behind master/latest-stable (BEING part of it), and more simplicity in dealing with polymorphic models (even simpler than configuring a separate app).

 

> There was some research by Sebastian Vetter on "single-child

> auto-resolve" solutions for MTI and how they scaled (from memory,

> model-utils' InheritanceManager, polymorphic, and generic-m2m)

> https://github.com/elbaschid/mti-lightbulb

>

 

Django already uses MTI. The only difference I'm proposing is, basically, making life simpler for those wanting to get all objects as their native class, rather than parent class (while not affecting those that don't use them).

 

--

Hugo Osvaldo Barrera

 

[Curtis Maloney]

On 11/02/16 15:05, 'Hugo Osvaldo Barrera' via Django developers

(Contributions to Django itself) wrote:

> > There was some research by Sebastian Vetter on "single-child
> > auto-resolve" solutions for MTI and how they scaled (from memory,
> > model-utils' InheritanceManager, polymorphic, and generic-m2m)
> > https://github.com/elbaschid/mti-lightbulb

> Django already uses MTI. The only difference I'm proposing is,
> basically, making life simpler for those wanting to get all objects as
> their native class, rather than parent class (while not affecting those
> that don't use them).

I know Django uses MTI. This thread is, after all, talking about a tool
for a _specific_ use case of MTI.

This investigation was specifically comparing the _scalability_ of 3
solutions to the same problem.

The implementations differ as:

polymorphic:
- has a "hidden" content_type field to indicate which child model to
link for this "parent" record
- uses prefetch_related to resolve
- results in 1+N queries

InheritanceManager (admin-tools):-
- uses select_related to grab _all_ the child models
- selects the first one that exists
- always results in 1 query
- can blow your DB server's brains out with the number of joins (if you
have a couple of hundred child models).
- can select the "wrong" child if you somehow made two for the same parent

Generic M2M:
- quite a different solution, but could be used in the same way for
Seb's purposes.
- don't recall specifics
- scaled dramatically better than the other two


--
C