[Django] #35904: Speed up fixture loading by bulk insert

[Django]

#35904: Speed up fixture loading by bulk insert
-------------------------------------+-------------------------------------
Reporter: JorisBenschop | Type:
| Uncategorized
Status: new | Component: Testing
| framework
Version: 5.0 | Severity: Normal
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
I noticed that loading many fixtures is fairly inefficient, in that it
tries an update and then an insert for each record. I would propose an
additional option to the command line (—bulk-insert) that only runs the
insert queries, but also detects if subsequent items in the fixtures are
from the same model, so they may be inserted using bulk_insert. If there
is genuine interest in this feature i will develop it and submit. I have
this running in my own codebase already.
Please also indicate if there is no interest, and i spend my energy
elsewhere =)
--
Ticket URL: <https://code.djangoproject.com/ticket/35904>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#35904: Speed up fixture loading by bulk insert

-----------------------------------+--------------------------------------
Reporter: JorisBenschop | Owner: (none)
Type: Uncategorized | Status: closed

Component: Testing framework | Version: 5.0

Severity: Normal | Resolution: wontfix

Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

-----------------------------------+--------------------------------------
Changes (by Simon Charette):

* resolution: => wontfix
* status: new => closed

Comment:

Hello Joris,

This sounds interesting particularly given features like
[https://docs.djangoproject.com/en/5.1/topics/testing/overview/#test-case-
serialized-rollback test case serialized rollbacks] (which are quite slow)
are based on top of model serialization. It would have to be a distinct
option as `bulk_create` doesn't fire signals which some setup might
require.

Just like any new feature requests though
[https://docs.djangoproject.com/en/5.1/internals/contributing/bugs-and-
features/#requesting-features they should be discussed on the forum to
reach a consensus] before being accepted. Given this is a performance
related new feature I suggest your proposal come equipped with some
details about what kind of improvements users should expect backed by step
to reproduce as well as a PoC that properly deals with other features of
serde framework such as natural keys and a plan on how to deal with
backends that don't support `ignore_conflicts`.

Assuming there is interest in moving forward we can then re-open this
issue.
--
Ticket URL: <https://code.djangoproject.com/ticket/35904#comment:1>