[Django] #33460: Change SQLite backend to generate INSERT statements using VALUES instead of UNION.

[Django]

#33460: Change SQLite backend to generate INSERT statements using VALUES instead of
UNION.
-------------------------------------+-------------------------------------
Reporter: Keryn | Owner: Keryn Knight
Knight |
Type: | Status: assigned
Cleanup/optimization |
Component: Database | Version: dev
layer (models, ORM) | Keywords: sqlite orm
Severity: Normal | bulk_create batch
Triage Stage: | Has patch: 1
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
This is a ticket for tracking this
[https://github.com/django/django/pull/15354 pre-emptive PR].

Django's minimum supported version of SQLite is currently documented as
3.9.

Using `INSERT INTO ... VALUES (...)` instead of `INSERT INTO ... SELECT
UNION ALL ...` for multiple rows (i.e. `bulk_create`) was added to SQLite
in [http://www.sqlite.org/releaselog/3_7_11.html 3.7.11] released in 2012.

[https://www.sqlite.org/releaselog/3_8_8.html SQLite 3.8.8], released in
2015, further changed it so that `VALUES(...)` was not subject to the
`SQLITE_LIMIT_COMPOUND_SELECT` value, which should subsequently allow for
improvements in batch sizes (e.g. investigating changes to
`django.db.backends.sqlite3.operations.DatabaseOperations.bulk_batch_size`
and
`django.db.backends.sqlite3.features.DatabaseFeatures.max_query_params`,
the latter of which supports `32766 ` values instead of `999` as of
[https://www.sqlite.org/releaselog/3_32_0.html 3.32.0])

The documentation for current SQLite (3.37.2) says, of using
`VALUES(...)`:

> The phrase "VALUES(expr-list)" means the same thing as "SELECT expr-
list". The phrase "VALUES(expr-list-1),...,(expr-list-N)" means the same
thing as "SELECT expr-list-1 UNION ALL ... UNION ALL SELECT expr-list-N".
Both forms are the same, except that the number of SELECT statements in a
compound is limited by SQLITE_LIMIT_COMPOUND_SELECT whereas the number of
rows in a VALUES clause has no arbitrary limit.
> There are some restrictions on the use of a VALUES clause that are not
shown on the syntax diagrams:
> A VALUES clause cannot be followed by ORDER BY.
> A VALUES clause cannot be followed by LIMIT.

I have run the test suite against `3.37.2` and whatever the CI versions
are (I've not checked) ... so far so good.

--
Ticket URL: <https://code.djangoproject.com/ticket/33460>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#33460: Change SQLite backend to generate INSERT statements using VALUES instead of
UNION.
-------------------------------------+-------------------------------------

Reporter: Keryn Knight | Owner: Keryn
Type: | Knight
Cleanup/optimization | Status: assigned
Component: Database layer | Version: dev
(models, ORM) |
Severity: Normal | Resolution:
Keywords: sqlite orm | Triage Stage: Accepted
bulk_create batch |
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Mariusz Felisiak):

* stage: Unreviewed => Accepted

Comment:

Thanks!

--
Ticket URL: <https://code.djangoproject.com/ticket/33460#comment:1>

[Django]

#33460: Change SQLite backend to generate INSERT statements using VALUES instead of
UNION.
-------------------------------------+-------------------------------------

Reporter: Keryn Knight | Owner: Keryn
Type: | Knight
Cleanup/optimization | Status: assigned
Component: Database layer | Version: dev
(models, ORM) |
Severity: Normal | Resolution:

Keywords: sqlite orm | Triage Stage: Ready for
bulk_create batch | checkin
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Mariusz Felisiak):

* stage: Accepted => Ready for checkin

--
Ticket URL: <https://code.djangoproject.com/ticket/33460#comment:2>

[Django]

#33460: Change SQLite backend to generate INSERT statements using VALUES instead of
UNION.
-------------------------------------+-------------------------------------

Reporter: Keryn Knight | Owner: Keryn
Type: | Knight

Cleanup/optimization | Status: closed

Component: Database layer | Version: dev
(models, ORM) |

Severity: Normal | Resolution: fixed

Keywords: sqlite orm | Triage Stage: Ready for
bulk_create batch | checkin

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Mariusz Felisiak <felisiak.mariusz@…>):

* status: assigned => closed
* resolution: => fixed

Comment:

In [changeset:"c27932ec938217d4fbb0adad23c0d0708f83f690" c27932ec]:
{{{
#!CommitTicketReference repository=""
revision="c27932ec938217d4fbb0adad23c0d0708f83f690"
Fixed #33460 -- Used VALUES clause for insert in bulk on SQLite.

SQLite 3.7.11 introduced the ability to use multiple values directly.
SQLite 3.8.8 made multiple values not subject to the
SQLITE_LIMIT_COMPOUND_SELECT (500).
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/33460#comment:3>