[Django] #28692: QuerySet.bulk_create() combine with select/prefetch_related()
Django
-------------------------------------+-------------------------------------
Reporter: Дилян | Owner: nobody
Палаузов |
Type: | Status: new
Uncategorized |
Component: Database | Version: 1.11
layer (models, ORM) | Keywords: bulk_create
Severity: Normal | select_related prefetch_related
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
When several objects are created with Models.objects.bulk_create() the
called might want to prefetch/JOIN some related models to the new objects,
so that additional SELECTs are avoided.
{{{
for m in ModelA.objects.select_related('b').bulk_create([ModelA(...),
ModelA(...), ModelA(...)]):
print(m.b.description) # this line shall now cause a new SELECT on each
iteration
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/28692>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
-------------------------------------+-------------------------------------
Type: Uncategorized | Status: new
Component: Database layer | Version: 1.11
(models, ORM) |
Severity: Normal | Resolution:
Keywords: bulk_create | Triage Stage:
select_related prefetch_related | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Tomer Chachamu):
https://docs.djangoproject.com/en/dev/internals/contributing/bugs-and-
features/
> First request the feature on the django-developers list, not in the
ticket tracker. It’ll get read more closely if it’s on the mailing list.
This is even more important for large-scale feature requests. We like to
discuss any big changes to Django’s core on the mailing list before
actually working on them.
Please send the feature request again to the mailing list. However, here's
my opinion:
I don't think it will be possible to implement `select_related` as the
behaviour of `INSERT .. RETURNING ..` varies a lot between databases. It
would be possible to implement `prefetch_related`.
There are already two ways to reduce the number of queries, is either good
for you? One is:
{{{#!python
from django.db.models import prefetch_related_objects
new_objects = [ModelA(..., b_id=3), ModelA(..., b_id=4), ModelA(...,
b_id=5)]
ModelA.objects.bulk_create(new_objects)
prefetch_related_objects(new_objects, 'b')
for m in new_objects:
print m.b.description
}}}
Another is setting the `b` attribute on your models, instead of `b_id`
{{{#!python
bs_by_id = ModelB.objects.in_bulk(relevant_b_ids)
new_objects = [ModelA(..., b=bs_by_id[3]), ModelA(..., b=bs_by_id[4]),
ModelA(..., b=bs_by_id[5])]
ModelA.objects.bulk_create(new_objects)
for m in new_objects:
print m.b.description
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/28692#comment:1>
Django
-------------------------------------+-------------------------------------
Component: Database layer | Version: 1.11
(models, ORM) |
Keywords: bulk_create | Triage Stage:
select_related prefetch_related | Unreviewed
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: new => closed
* resolution: => needsinfo
--
Ticket URL: <https://code.djangoproject.com/ticket/28692#comment:2>
Django
-------------------------------------+-------------------------------------
Component: Database layer | Version: 1.11
(models, ORM) |
Severity: Normal | Resolution:
select_related prefetch_related | Unreviewed
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: closed => new
* resolution: needsinfo =>
Comment:
#28600 asks for adding .prefetch_related() support to RawQuerySet . I
don't see why that ticket is clear, but this one, asking for adding
prefetch_related() support to bulk_create() is anyhow different and hence
unclear.
Despite there are already two other ways to reduce the number of queries,
prefetch_related() is more coherent, as this what users are used to in
other circumstances.
--
Ticket URL: <https://code.djangoproject.com/ticket/28692#comment:3>
Django
-------------------------------------+-------------------------------------
Type: Uncategorized | Status: new
Component: Database layer | Version: 1.11
(models, ORM) |
Severity: Normal | Resolution:
Keywords: bulk_create | Triage Stage:
select_related prefetch_related | Unreviewed
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Tomer Chachamu):
I think it will be impossible to implement on backends other than
postgres, as you would want to do {{{SELECT b.* FROM modela JOIN modelb ON
modela.b_id = modelb.id WHERE modela.id IN (...)}}}, however the value of
{{{(...)}}} is only available in postgres.
--
Ticket URL: <https://code.djangoproject.com/ticket/28692#comment:4>
Django
-------------------------------------+-------------------------------------
Type: | Status: new
Cleanup/optimization |
Component: Database layer | Version: 1.11
(models, ORM) |
Severity: Normal | Resolution:
select_related prefetch_related |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* type: Uncategorized => Cleanup/optimization
* stage: Unreviewed => Accepted
Comment:
I guess a patch could be evaluated.
--
Ticket URL: <https://code.djangoproject.com/ticket/28692#comment:5>
Django
-------------------------------------+-------------------------------------
Type: | Status: new
Cleanup/optimization |
Component: Database layer | Version: 1.11
(models, ORM) |
Severity: Normal | Resolution:
Keywords: bulk_create | Triage Stage: Accepted
select_related prefetch_related |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Дилян Палаузов):
I guess a patch will be provided for prefetch_related() once #28600 gets
integrated, as stated in the fourth comment of that ticket.
--
Ticket URL: <https://code.djangoproject.com/ticket/28692#comment:6>
Django
-------------------------------------+-------------------------------------
Cleanup/optimization |
Component: Database layer | Version: 1.11
(models, ORM) |
Keywords: bulk_create | Triage Stage: Accepted
select_related prefetch_related |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: new => closed
* resolution: => wontfix
Comment:
No patch was provided and I agree that this will be very hard to implement
`select_related` in a backend agnostic way. Now that
[https://docs.djangoproject.com/en/4.1/ref/models/querysets/#django.db.models.prefetch_related_objects
prefetch_related_objects] is a documented public API it can be used it to
achieve the same results as what `prefetch_related` would have done.
Closing as ''wontfix'' for now, I wouldn't be opposed to reopening once we
get better support for `RETURNING` handling at the `sql.Query` level if a
patch is provided.
--
Ticket URL: <https://code.djangoproject.com/ticket/28692#comment:7>