[Django] #35975: New feature: faster fixture loading via loaddata command

[Django]

#35975: New feature: faster fixture loading via loaddata command
-------------------------------+-----------------------------------------
Reporter: JorisBenschop | Type: Uncategorized
Status: new | Component: Uncategorized
Version: 5.1 | Severity: Normal
Keywords: loaddata | Triage Stage: Unreviewed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+-----------------------------------------
As per [https://forum.djangoproject.com/t/feature-proposal-faster-fixture-
loading-via-loaddata-command/36972/4 this forum discussion], I have
created a patch to improve load times for the loaddata command under some
circumstances.

Currently the “loaddata” management command uses the obj.save() method for
each deserialized object within a fixture. This function first tries an
UPDATE statement and, if that fails, tries an INSERT statement.

I propose to add two optional flags to the loaddata command:

–force-insert adds the “force_insert=True” to the save() method. This
reduces load time by ~50% but has an increased risk of the upload failing
in case the record already exists.

–bulk_create: This option groups records in the fixture by the model,
and insert the group as a single bulk_create statement. For large
fixtures, I have achieved a 1000-fold improvement in loading time. It has
a number of risks (already described in the bulk_create section), notably
that it skips some routines covered in the save() method of the model.

Both these flags are supposed to be run by people who know that above-
mentioned issues will not be a problem for them, and should not be enabled
by default.
--
Ticket URL: <https://code.djangoproject.com/ticket/35975>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#35975: New feature: faster fixture loading via loaddata command
-------------------------------+-----------------------------------------

Reporter: JorisBenschop | Owner: JorisBenschop
Type: Uncategorized | Status: assigned
Component: Uncategorized | Version: 5.1
Severity: Normal | Resolution:

Keywords: loaddata | Triage Stage: Unreviewed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+-----------------------------------------

Changes (by JaeHyuckSa):

* owner: (none) => JorisBenschop
* status: new => assigned

--
Ticket URL: <https://code.djangoproject.com/ticket/35975#comment:1>

[Django]

#35975: New feature: faster fixture loading via loaddata command

-------------------------------------+-------------------------------------

Reporter: JorisBenschop | Owner:
| JorisBenschop
Type: Uncategorized | Status: assigned

Component: Core (Management | Version: 5.1
commands) |

Severity: Normal | Resolution:
Keywords: loaddata | Triage Stage:
| Unreviewed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

-------------------------------------+-------------------------------------
Changes (by JorisBenschop):

* component: Uncategorized => Core (Management commands)

--
Ticket URL: <https://code.djangoproject.com/ticket/35975#comment:2>

[Django]

#35975: New feature: faster fixture loading via loaddata command
-------------------------------------+-------------------------------------
Reporter: JorisBenschop | Owner:
| JorisBenschop

Type: Uncategorized | Status: closed

Component: Core (Management | Version: 5.1
commands) |

Severity: Normal | Resolution: duplicate

Keywords: loaddata | Triage Stage:
| Unreviewed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Sarah Boyce):

* resolution: => duplicate
* status: assigned => closed

Comment:

Duplicate of #35904
--
Ticket URL: <https://code.djangoproject.com/ticket/35975#comment:3>