[Django] #27639: Add a chunk size argument to QuerySet.iterator()
Django
-------------------------------------+-------------------------------------
Reporter: François | Owner: nobody
Freitag |
Type: New | Status: new
feature |
Component: Database | Version: master
layer (models, ORM) |
Severity: Normal | Keywords: cursors database
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
Django currently fetches results from the database in batches of
{{{GET_ITERATOR_CHUNK_SIZE}}} (currently 100). When {{{.iterator()}}} is
used, usually for a large query, specifying the batch size would allow
more control over the number of back-and-forth communications between
Django and the database.
[https://www.python.org/dev/peps/pep-0249/#fetchmany PEP249] define the
{{{size}}} argument for the {{{.fetchmany()}}} method.
* MySQL-python provides the [https://dev.mysql.com/doc/connector-python/en
/connector-python-api-mysqlcursor-fetchmany.html size] argument for
{{{.fetchmany()}}}.
* SQLite provides the
[https://docs.python.org/3.6/library/sqlite3.html#sqlite3.Cursor.fetchmany
size] argument for {{{.fetchmany()}}}.
* cx_Oracle provides [http://cx-
oracle.readthedocs.io/en/latest/cursor.html?highlight=arraysize#Cursor.arraysize
arraysize] attribute on the cursor and the [http://cx-
oracle.readthedocs.io/en/latest/cursor.html#Cursor.fetchmany numRows]
argument for {{{.fetchmany()}}}.
* Psycopg2 (used for PostgreSQL) provides the
[http://initd.org/psycopg/docs/cursor.html?highlight=itersize#cursor.itersize
itersize] attribute (used by server-side cursors, see discussion on
[https://github.com/django/django/pull/7587 This PR -- Added server side
cursors support for QuerySet iteration in PostgreSQL].
Note:
In [https://code.djangoproject.com/ticket/26530#comment:3 #26530], Anssi
Kääriäinen proposed the name {{{cursor_size}}} for this argument.
--
Ticket URL: <https://code.djangoproject.com/ticket/27639>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
-------------------------------------+-------------------------------------
Type: New feature | Status: new
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: cursors database | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* stage: Unreviewed => Accepted
--
Ticket URL: <https://code.djangoproject.com/ticket/27639#comment:1>
Django
-------------------------------------+-------------------------------------
Type: New feature | Status: new
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: me@… (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/27639#comment:2>
Django
-------------------------------------+-------------------------------------
| Freitag
Type: New feature | Status: assigned
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* owner: nobody => François Freitag
* status: new => assigned
--
Ticket URL: <https://code.djangoproject.com/ticket/27639#comment:3>
Django
-------------------------------------+-------------------------------------
| Freitag
Type: New feature | Status: assigned
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* has_patch: 0 => 1
Comment:
[https://github.com/django/django/pull/7836/ PR #7836]
--
Ticket URL: <https://code.djangoproject.com/ticket/27639#comment:4>
Django
-------------------------------------+-------------------------------------
| Freitag
Type: New feature | Status: assigned
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/27639#comment:5>
Django
-------------------------------------+-------------------------------------
| Freitag
Type: New feature | Status: assigned
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_better_patch: 0 => 1
* stage: Ready for checkin => Accepted
Comment:
Some test failures on Oracle must be fixed.
--
Ticket URL: <https://code.djangoproject.com/ticket/27639#comment:6>
Django
-------------------------------------+-------------------------------------
| Freitag
Type: New feature | Status: assigned
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_better_patch: 1 => 0
--
Ticket URL: <https://code.djangoproject.com/ticket/27639#comment:7>
Django
-------------------------------------+-------------------------------------
| Freitag
Type: New feature | Status: assigned
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/27639#comment:8>
Django
-------------------------------------+-------------------------------------
| Freitag
Type: New feature | Status: assigned
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_better_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/27639#comment:8>
Django
-------------------------------------+-------------------------------------
| Freitag
Type: New feature | Status: assigned
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_better_patch: 1 => 0
--
Ticket URL: <https://code.djangoproject.com/ticket/27639#comment:9>
Django
-------------------------------------+-------------------------------------
| Freitag
Type: New feature | Status: assigned
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_better_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/27639#comment:10>
Django
-------------------------------------+-------------------------------------
| Freitag
Type: New feature | Status: assigned
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_better_patch: 1 => 0
--
Ticket URL: <https://code.djangoproject.com/ticket/27639#comment:11>
Django
-------------------------------------+-------------------------------------
| Freitag
Type: New feature | Status: assigned
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Has patch: 1 | Needs documentation: 0
Needs tests: 1 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_tests: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/27639#comment:12>
Django
-------------------------------------+-------------------------------------
| Freitag
Type: New feature | Status: assigned
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_tests: 1 => 0
--
Ticket URL: <https://code.djangoproject.com/ticket/27639#comment:13>
Django
-------------------------------------+-------------------------------------
| Freitag
Type: New feature | Status: assigned
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/27639#comment:14>
Django
-------------------------------------+-------------------------------------
| Freitag
Type: New feature | Status: assigned
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_better_patch: 0 => 1
* stage: Ready for checkin => Accepted
Comment:
Since it was marked as RFC, the PR received some updates to allow
`chunk_size=None` to disable server-side cursors (use case in #28062). A
few review comments remain.
--
Ticket URL: <https://code.djangoproject.com/ticket/27639#comment:15>
Django
-------------------------------------+-------------------------------------
| Freitag
Type: New feature | Status: assigned
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by François Freitag):
After reworking this PR to allow {{{chunk_size=None}}}, I'm not convinced
with {{{chunk_size=None}}}:
- It is introduced to workaround an issue with PostgreSQL, I do not
believe iterator should change for that reason.
- It's a PostgreSQL-only parameter, because server-side cursors cannot be
disabled on Oracle, and it'll be ignored on databases that don't support
server-side cursors.
- Its meaning is ambiguous. I would expect chunk_size=None to signify "do
not use chunked fetch and fetch all the results at once", i.e. use
{{{fetchall}}}. Django uses {{{fetchmany}}}.
- In most use cases I can think of, server-side cursors need to be either
globally enabled/disabled, not on a per-query basis. If a end-user really
want to do so, it's possible to setup an other connection and use
{{{using()}}} to disable server-side cursors for that query. I have a hard
time coming up with a use-case where third party libraries want to use
iterator, but not with a server-side cursor.
--
Ticket URL: <https://code.djangoproject.com/ticket/27639#comment:16>
Django
-------------------------------------+-------------------------------------
| Freitag
Type: New feature | Status: assigned
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Tim Graham):
Until a use case arises, omitting support for `chunk_size=None` is fine
with me.
--
Ticket URL: <https://code.djangoproject.com/ticket/27639#comment:17>
Django
-------------------------------------+-------------------------------------
| Freitag
Type: New feature | Status: assigned
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_better_patch: 1 => 0
--
Ticket URL: <https://code.djangoproject.com/ticket/27639#comment:18>
Django
-------------------------------------+-------------------------------------
| Freitag
Component: Database layer | Version: master
(models, ORM) |
Keywords: cursors database | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"edee5a8de6afb34a9247de2bb73ab66397a6e573" edee5a8d]:
{{{
#!CommitTicketReference repository=""
revision="edee5a8de6afb34a9247de2bb73ab66397a6e573"
Fixed #27639 -- Added chunk_size parameter to QuerySet.iterator().
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/27639#comment:19>