[Django] #30844: Add after_db_init() hook method to model
Django
-----------------------------------------+------------------------
Reporter: rsinger86 | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 2.2
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 1
UI/UX: 0 |
-----------------------------------------+------------------------
I've encountered a need in my personal projects, and when writing
[https://github.com/rsinger86/django-lifecycle django-lifecycle], to hook
into the moment when a model has been fully initialized from the database.
Overriding the model's __init__ method does NOT work here because the
`select_related` relationships have not yet been added in/cached in the
model's FK fields. It would be useful to do things right after the model
is fully loaded and initialized from the database. For example, if you
have a "type" foreign key field, you may want to apply some polymorphic
behavior based on the value of that field. Doing this in `__init__` will
cause a n+1 query explosion if you do it when iterating over a `QuerySet`.
Causes n+1 issue when iterating a QuerySet:
{{{
class CatMixin(object):
def greet(self):
return "Meow"
class DogMixin(object):
def greet(self):
return "Woof"
class PolymorphicModel(models.Model):
type = models.ForeignKey(PetType)
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
if type.name == 'cat':
self.__class__ = CatMixin
else:
self.__class__ = DogMixin
}}}
The fix: Add a call to `obj.post_db_init()` to this line:
https://github.com/django/django/blob/master/django/db/models/query.py#L91
Then this code will eliminate the n+1 problem (assuming
`select_related('type')` is used):
{{{
class PolymorphicModel(models.Model):
type = models.ForeignKey(PetType)
def post_db_init(self,):
if type.name == 'cat':
self.__class__ = CatMixin
else:
self.__class__ = DogMixin
}}}
I realize there are other ways to achieve the behavior in this example --
I'm bringing up polymorphism as a general use case. In django-lifecycle,
this hook would allow me to track a model instance's initial state across
a foreign key relationship without causing users to experience the n+1
problem.
--
Ticket URL: <https://code.djangoproject.com/ticket/30844>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: rsinger86 | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 2.2
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
Description changed by rsinger86:
Old description:
New description:
I've encountered a need in my personal projects, and when writing
[https://github.com/rsinger86/django-lifecycle django-lifecycle], to hook
into the moment when a model has been fully initialized from the database.
Overriding the model's __init__ method does NOT work here because the
`select_related` relationships have not yet been added in/cached in the
model's FK fields. It would be useful to do things right after the model
is fully loaded and initialized from the database. For example, if you
have a "type" foreign key field, you may want to apply some polymorphic
behavior based on the value of that field. Doing this in `__init__` will
cause a n+1 query explosion if you load multiple models and iterate over
the `QuerySet`.
== Current Problem
This will cause an n+1 issue when iterating a QuerySet:
{{{
class CatMixin(object):
def greet(self):
return "Meow"
class DogMixin(object):
def greet(self):
return "Woof"
class PolymorphicModel(models.Model):
type = models.ForeignKey(PetType)
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
if type.name == 'cat':
self.__class__ = CatMixin
else:
self.__class__ = DogMixin
}}}
== The Fix
Add a call to `obj.post_db_init()` to this line:
https://github.com/django/django/blob/master/django/db/models/query.py#L91
Then this code will eliminate the n+1 problem (assuming
`select_related('type')` is used):
{{{
class PolymorphicModel(models.Model):
type = models.ForeignKey(PetType)
def post_db_init(self,):
if type.name == 'cat':
self.__class__ = CatMixin
else:
self.__class__ = DogMixin
}}}
I realize there are other ways to achieve the behavior in this example --
I'm bringing up polymorphism as a general use case. In django-lifecycle,
this hook would allow me to track a model instance's initial state across
a foreign key relationship without causing users to experience the n+1
problem.
--
--
Ticket URL: <https://code.djangoproject.com/ticket/30844#comment:1>
Django
Reporter: rsinger86 | Owner: nobody
Type: Uncategorized | Status: new
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
Changes (by rsinger86):
* component: Uncategorized => Database layer (models, ORM)
--
Ticket URL: <https://code.djangoproject.com/ticket/30844#comment:2>
Django
Reporter: rsinger86 | Owner: nobody
Type: New feature | Status: new
Component: Database layer | Version: 2.2
(models, ORM) |
Severity: Normal | Resolution:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
Changes (by rsinger86):
* type: Uncategorized => New feature
--
Ticket URL: <https://code.djangoproject.com/ticket/30844#comment:3>
Django
Reporter: rsinger86 | Owner: nobody
Type: New feature | Status: new
Component: Database layer | Version: 2.2
(models, ORM) |
Severity: Normal | Resolution:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
Description changed by rsinger86:
Old description:
> I've encountered a need in my personal projects, and when writing
> [https://github.com/rsinger86/django-lifecycle django-lifecycle], to hook
> into the moment when a model has been fully initialized from the
> database.
>
> Overriding the model's __init__ method does NOT work here because the
> `select_related` relationships have not yet been added in/cached in the
> model's FK fields. It would be useful to do things right after the model
> is fully loaded and initialized from the database. For example, if you
> have a "type" foreign key field, you may want to apply some polymorphic
> behavior based on the value of that field. Doing this in `__init__` will
> cause a n+1 query explosion if you load multiple models and iterate over
> the `QuerySet`.
>
> == Current Problem
> This will cause an n+1 issue when iterating a QuerySet:
> {{{
> class CatMixin(object):
> def greet(self):
> return "Meow"
>
> class DogMixin(object):
> def greet(self):
> return "Woof"
>
> class PolymorphicModel(models.Model):
> type = models.ForeignKey(PetType)
>
> def __init__(self, *args, **kwargs):
> super().__init__(*args, **kwargs)
>
> if type.name == 'cat':
> self.__class__ = CatMixin
> else:
> self.__class__ = DogMixin
> }}}
>
> == The Fix
> Add a call to `obj.post_db_init()` to this line:
> https://github.com/django/django/blob/master/django/db/models/query.py#L91
>
> Then this code will eliminate the n+1 problem (assuming
> `select_related('type')` is used):
> {{{
> class PolymorphicModel(models.Model):
> type = models.ForeignKey(PetType)
>
> def post_db_init(self,):
> if type.name == 'cat':
> self.__class__ = CatMixin
> else:
> self.__class__ = DogMixin
> }}}
>
> I realize there are other ways to achieve the behavior in this example --
> I'm bringing up polymorphism as a general use case. In django-lifecycle,
> this hook would allow me to track a model instance's initial state across
> a foreign key relationship without causing users to experience the n+1
> problem.
New description:
I've encountered a need in my personal projects, and when writing
[https://github.com/rsinger86/django-lifecycle django-lifecycle], to hook
into the moment when a model has been fully initialized from the database.
Overriding the model's __init__ method does NOT work here because the
`select_related` relationships have not yet been added in/cached in the
model's FK fields. It would be useful to do things right after the model
is fully loaded and initialized from the database. For example, if you
have a "type" foreign key field, you may want to apply some polymorphic
behavior based on the value of that field. Doing this in `__init__` will
cause a n+1 query explosion if you load multiple models and iterate over
the `QuerySet`.
== Current Problem
This will cause an n+1 issue when iterating a QuerySet:
{{{
class CatMixin(object):
def greet(self):
return "Meow"
class DogMixin(object):
def greet(self):
return "Woof"
class PolymorphicModel(models.Model):
type = models.ForeignKey(PetType)
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
# another db query is triggered b/c select_related has populated
the instance yet
if self.type.name == 'cat':
self.__class__ = CatMixin
else:
self.__class__ = DogMixin
}}}
== The Fix
Add a call to `obj.post_db_init()` to this line:
https://github.com/django/django/blob/master/django/db/models/query.py#L91
Then this code will eliminate the n+1 problem (assuming
`select_related('type')` is used):
{{{
class PolymorphicModel(models.Model):
type = models.ForeignKey(PetType)
def post_db_init(self,):
if type.name == 'cat':
self.__class__ = CatMixin
else:
self.__class__ = DogMixin
}}}
I realize there are other ways to achieve the behavior in this example --
I'm bringing up polymorphism as a general use case. In django-lifecycle,
this hook would allow me to track a model instance's initial state across
a foreign key relationship without causing users to experience the n+1
problem.
--
--
Ticket URL: <https://code.djangoproject.com/ticket/30844#comment:4>
Django
Reporter: rsinger86 | Owner: nobody
Type: New feature | Status: closed
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution: wontfix
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
-------------------------------------+-------------------------------------
Changes (by felixxm):
* status: new => closed
* version: 2.2 => master
* resolution: => wontfix
* easy: 1 => 0
Comment:
`select_related()` is not related with `Model`'s initialization it's a
`QuerySet` method, so I cannot imagine how you would like to connect them.
As you pointed out there are many ways to solve your issues in the current
Django (e.g. you can
[https://docs.djangoproject.com/en/2.2/topics/db/managers/#modifying-a-manager-s
-initial-queryset add select_related() to the initial QuerySet] and use
`@cached_property`, there is also
[https://docs.djangoproject.com/en/2.2/ref/signals/#post-init post_init]
signal). You can use one of
[https://code.djangoproject.com/wiki/TicketClosingReasons/UseSupportChannels
support channels] to describe your real use case and get help.
--
Ticket URL: <https://code.djangoproject.com/ticket/30844#comment:5>
Django
Reporter: rsinger86 | Owner: nobody
Type: New feature | Status: closed
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution: wontfix
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
-------------------------------------+-------------------------------------
Comment (by rsinger86):
Replying to [comment:5 felixxm]:
> `select_related()` is not related with `Model`'s initialization it's a
`QuerySet` method, so I cannot imagine how you would like to connect them.
As you pointed out there are many ways to solve your issues in the current
Django (e.g. you can
[https://docs.djangoproject.com/en/2.2/topics/db/managers/#modifying-a-manager-s
-initial-queryset add select_related() to the initial QuerySet] and use
`@cached_property`, there is also
[https://docs.djangoproject.com/en/2.2/ref/signals/#post-init post_init]
signal). You can use one of
[https://code.djangoproject.com/wiki/TicketClosingReasons/UseSupportChannels
support channels] to describe your real use case and get help.
I appreciate the quick response to the ticket, but I feel I may have
failed to convey what the issue is. The problem is that a model's
initialization lifecycle currently lacks a way to hook into the moment
right after the ORM populates FK relations that are specified in
`select_related`. The point where that happens is right here:
https://github.com/django/django/blob/master/django/db/models/query.py#L91
The nearest point to this moment is `__init__`, but it occurs before FK
relations are populated, so examining FK related field values at that
point will cause an n+1 query problem. There is no workaround/support for
hooking into this moment currently: modifying an initial queryset or using
the @cached_property won't work. I am glad to submit a pull request to add
this feature. But it's probably something only advanced users/library
authors need, so it may be out of scope. My use case is enhancing my
library, [https://github.com/rsinger86/django-lifecycle django-lifecycle],
to more efficiently support state transitions based on FK values.
--
Ticket URL: <https://code.djangoproject.com/ticket/30844#comment:6>
Django
Reporter: Robert Singer | Owner: nobody
Type: New feature | Status: closed
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution: wontfix
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
-------------------------------------+-------------------------------------
Comment (by felixxm):
Adding a hook after each iteration doesn't sound like a good idea to me.
You need to take into account that we have different iterable classes e.g.
`ValuesIterable`, `ModelIterable`, `NamedValuesListIterable`, etc. and
that `Queryset ` is lazy, so you will not be able to add any useful hook
without forcing users to iterate through all rows. This looks quite niche
and I still cannot imagine any real use case based on a post iteration
hook. You can start a discussion on DevelopersMailingList if you don't
agree.
--
Ticket URL: <https://code.djangoproject.com/ticket/30844#comment:7>