[Django] #33143: Block import-time queries

Django

unread,

Sep 25, 2021, 3:42:41 AM9/25/21

to django-...@googlegroups.com

{{{
class Form(forms.Form):
country = forms.ChoiceField(choices=[c.name for c in
Country.objects.all()])
}}}

I see import time queries fairly frequently, and every time I have
encountered them they were a mistake. They end up querying the database
once when the web server starts and caching the data forever, which is
never desired.

Python allows imports to happen at any time, so there's no way to detect
when "import time" is over, and the app is really running. An inner import
may cause a module to be first loaded during a web request.

I therefore propose we block queries until *after* the `AppConfig.ready()`
phase. This would protect against most problems.

We could block until *before* `AppConfig.ready()`. Queries inside a
`ready()` method kind-of imply the user knows that the query is one-off at
startup. But we do encourage users to perform some imports within
`ready()` methods such as for signal registration.

--
Ticket URL: <https://code.djangoproject.com/ticket/33143>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

Django

unread,

Sep 25, 2021, 9:07:54 AM9/25/21

to django-...@googlegroups.com

#33143: Block import-time queries
-------------------------------------+-------------------------------------

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Tim Graham):

[https://groups.google.com/g/django-developers/c/7JwWatLfP44/ django-
developers discussion]

--
Ticket URL: <https://code.djangoproject.com/ticket/33143#comment:1>

Django

unread,

Sep 25, 2021, 10:54:45 AM9/25/21

to django-...@googlegroups.com

#33143: Block import-time queries
-------------------------------------+-------------------------------------

Reporter: Adam Johnson | Owner: nobody
Type: | Status: new
Cleanup/optimization |

Has patch: 0 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Simon Charette):

I'd be in favor of blocking until *after* `AppConfig.ready()` phase as any
queries performed at `.ready()` time has the potential to crash due to
still unapplied migrations (missing table, schema mismatch) and can get
users in a pickle where they can't even run `migrate`.
[https://docs.djangoproject.com/en/3.2/ref/applications/#django.apps.AppConfig.ready
This is something we've been warning against for a while] and that I've
had to point to a fair amount of time when reviewing Django project
changes.

--
Ticket URL: <https://code.djangoproject.com/ticket/33143#comment:2>

Django

unread,

Sep 25, 2021, 3:03:49 PM9/25/21

to django-...@googlegroups.com

#33143: Block import-time queries
-------------------------------------+-------------------------------------

Reporter: Adam Johnson | Owner: nobody
Type: | Status: new
Cleanup/optimization |

Has patch: 0 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Adam Johnson):

Thanks for the developers discussion link Tim. I like the proposal of
warning and then if there are no serious reports, deprecating from the
next version. A `RuntimeWarning` sounds appropriate.

And I see you have some history with this Simon.

Aymeric was previously "+0" but he may have changed his mind. I will post
again on the thread with the link to this ticket just to see if there are
any dissenting opinions.

--
Ticket URL: <https://code.djangoproject.com/ticket/33143#comment:3>

Django

unread,

Sep 28, 2021, 2:47:23 AM9/28/21

to django-...@googlegroups.com

#33143: Block import-time queries
-------------------------------------+-------------------------------------

Reporter: Adam Johnson | Owner: nobody
Type: | Status: new
Cleanup/optimization |

Component: Database layer | Version: dev
(models, ORM) |
Severity: Normal | Resolution:

Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Carlton Gibson):

* stage: Unreviewed => Accepted

Comment:

I think there's enough consensus to Accept.

Right or wrong, I suspect there are plenty of folks using `ready()` as a
home for running some kind of query at startup. We should have some advice
for them.

--
Ticket URL: <https://code.djangoproject.com/ticket/33143#comment:4>

Django

unread,

Jun 1, 2023, 8:20:04 AM6/1/23

to django-...@googlegroups.com

#33143: Block import-time queries
-------------------------------------+-------------------------------------

Reporter: Adam Johnson | Owner: Florian
Type: | Zimmermann
Cleanup/optimization | Status: assigned

Component: Database layer | Version: dev
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted

Has patch: 0 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Florian Zimmermann):

* owner: nobody => Florian Zimmermann
* status: new => assigned

--
Ticket URL: <https://code.djangoproject.com/ticket/33143#comment:5>

Django

unread,

Jun 1, 2023, 11:55:39 AM6/1/23

to django-...@googlegroups.com

#33143: Block import-time queries
-------------------------------------+-------------------------------------

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Florian Zimmermann):

* has_patch: 0 => 1

Comment:

[https://github.com/django/django/pull/16923 PR]

--
Ticket URL: <https://code.djangoproject.com/ticket/33143#comment:6>

Django

unread,

Jun 2, 2023, 2:54:48 PM6/2/23

to django-...@googlegroups.com

#33143: Block import-time queries
-------------------------------------+-------------------------------------

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Natalia Bidart):

* needs_better_patch: 0 => 1

--
Ticket URL: <https://code.djangoproject.com/ticket/33143#comment:7>

Django

unread,

Jun 18, 2023, 11:33:51 AM6/18/23

to django-...@googlegroups.com

#33143: Block import-time queries
-------------------------------------+-------------------------------------

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Florian Zimmermann):

* needs_better_patch: 1 => 0

--
Ticket URL: <https://code.djangoproject.com/ticket/33143#comment:8>

Django

unread,

Jul 12, 2023, 6:01:16 AM7/12/23

to django-...@googlegroups.com

#33143: Block import-time queries
-------------------------------------+-------------------------------------

Has patch: 1 | Needs documentation: 0
Needs tests: 1 | Patch needs improvement: 1

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Mariusz Felisiak):

* needs_better_patch: 0 => 1

* needs_tests: 0 => 1

--
Ticket URL: <https://code.djangoproject.com/ticket/33143#comment:9>

Django

unread,

Jul 15, 2023, 4:28:28 AM7/15/23

to django-...@googlegroups.com

#33143: Block import-time queries
-------------------------------------+-------------------------------------

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Florian Zimmermann):

* needs_better_patch: 1 => 0

* needs_tests: 1 => 0

--
Ticket URL: <https://code.djangoproject.com/ticket/33143#comment:10>

Django

unread,

Jul 31, 2023, 5:13:08 AM7/31/23

to django-...@googlegroups.com

#33143: Block import-time queries
-------------------------------------+-------------------------------------

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Mariusz Felisiak):

* needs_better_patch: 0 => 1

Comment:

Marking as need improvement per Adam's
[https://github.com/django/django/pull/16923#discussion_r1278985276
comment].

--
Ticket URL: <https://code.djangoproject.com/ticket/33143#comment:11>

Django

unread,

Aug 1, 2023, 5:30:58 PM8/1/23

to django-...@googlegroups.com

#33143: Block import-time queries
-------------------------------------+-------------------------------------

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Florian Zimmermann):

* needs_better_patch: 1 => 0

--
Ticket URL: <https://code.djangoproject.com/ticket/33143#comment:12>

Django

unread,

Aug 2, 2023, 1:51:32 AM8/2/23

to django-...@googlegroups.com

#33143: Block import-time queries
-------------------------------------+-------------------------------------

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Mariusz Felisiak):

* needs_better_patch: 0 => 1

--
Ticket URL: <https://code.djangoproject.com/ticket/33143#comment:13>

Django

unread,

Aug 21, 2023, 7:46:55 AM8/21/23

to django-...@googlegroups.com

#33143: Block import-time queries
-------------------------------------+-------------------------------------

Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Mariusz Felisiak):

* needs_better_patch: 1 => 0
* stage: Accepted => Ready for checkin

--
Ticket URL: <https://code.djangoproject.com/ticket/33143#comment:14>

Django

unread,

Aug 21, 2023, 1:27:55 PM8/21/23

to django-...@googlegroups.com

#33143: Block import-time queries
-------------------------------------+-------------------------------------

Reporter: Adam Johnson | Owner: Florian
Type: | Zimmermann

Cleanup/optimization | Status: closed

Component: Database layer | Version: dev
(models, ORM) |

Severity: Normal | Resolution: fixed

Keywords: | Triage Stage: Ready for
| checkin

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Mariusz Felisiak <felisiak.mariusz@…>):

* status: assigned => closed
* resolution: => fixed

Comment:

In [changeset:"fbd16438f46bc2128926958ad24331da5d1b406f" fbd16438]:
{{{
#!CommitTicketReference repository=""
revision="fbd16438f46bc2128926958ad24331da5d1b406f"
Fixed #33143 -- Raised RuntimeWarning when performing import-time queries.
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/33143#comment:15>

Reply all

Reply to author

Forward