[Django] #33107: ImportError: partially initialized module '...' has no attribute '...' (most likely due to a circular import)
Django
(most likely due to a circular import)
-------------------------------------------+------------------------
Reporter: Collin Anderson | Owner: nobody
Type: Bug | Status: new
Component: Uncategorized | Version: dev
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------------+------------------------
Today I've been getting some occasional "ImportError"s when reloading the
runserver. I have a feeling it's a race condition when importing my custom
auth backend when runserver is reloading, likely caused by #33099 cached
imports, committed on Friday. It doesn't happen every time which makes it
harder to track down.
Maybe the `cached_import` logic needs to somehow check for a "partially
initialized module" and use the slow-path in that case?
I'm not 100% it's not just an issue on my end, but I at least wanted to
raise this here in case other people have issues and want to diagnose
further.
Here's an example (ImportError when importing my custom auth backend):
{{{
Request Method: GET
Request URL: /cart/dropdown/
Django Version: 4.0.dev20210913065016
Python Version: 3.8.10
Template error:
In template templates/order/cart_dropdown.html, error at line 3
Module "authbackend" does not define a "Bknd" attribute/class
3 : {% if request.order.ops %}
Traceback (most recent call last):
File "django/template/base.py", line 862, in _resolve_lookup
current = current[bit]
During handling of the above exception ('WSGIRequest' object is not
subscriptable), another exception occurred:
File "django/utils/module_loading.py", line 26, in import_string
return cached_import(module_path, class_name)
File "django/utils/module_loading.py", line 12, in cached_import
return getattr(modules[module_path], class_name)
The above exception (partially initialized module 'authbackend' has no
attribute 'Bknd' (most likely due to a circular import)) was the direct
cause of the following exception:
File "django/core/handlers/exception.py", line 47, in inner
response = get_response(request)
File "django/core/handlers/base.py", line 181, in _get_response
response = wrapped_callback(request, *callback_args,
**callback_kwargs)
File "django/shortcuts.py", line 19, in render
content = loader.render_to_string(template_name, context, request,
using=using)
File "django/template/loader.py", line 62, in render_to_string
return template.render(context, request)
File "django/template/backends/django.py", line 61, in render
return self.template.render(context)
File "django/template/base.py", line 176, in render
return self._render(context)
File "django/template/base.py", line 168, in _render
return self.nodelist.render(context)
File "django/template/base.py", line 977, in render
return SafeString(''.join([
File "django/template/base.py", line 978, in <listcomp>
node.render_annotated(context) for node in self
File "django/template/base.py", line 938, in render_annotated
return self.render(context)
File "django/template/defaulttags.py", line 386, in render
return
strip_spaces_between_tags(self.nodelist.render(context).strip())
File "django/template/base.py", line 977, in render
return SafeString(''.join([
File "django/template/base.py", line 978, in <listcomp>
node.render_annotated(context) for node in self
File "django/template/base.py", line 938, in render_annotated
return self.render(context)
File "django/template/defaulttags.py", line 288, in render
match = condition.eval(context)
File "django/template/defaulttags.py", line 829, in eval
return self.value.resolve(context, ignore_failures=True)
File "django/template/base.py", line 701, in resolve
obj = self.var.resolve(context)
File "django/template/base.py", line 829, in resolve
value = self._resolve_lookup(context)
File "django/template/base.py", line 870, in _resolve_lookup
current = getattr(current, bit)
File "order/middleware.py", line 20, in __get__
if not request.user.pk: # if not logged in
File "django/utils/functional.py", line 248, in inner
self._setup()
File "django/utils/functional.py", line 384, in _setup
self._wrapped = self._setupfunc()
File "django/contrib/auth/middleware.py", line 25, in <lambda>
request.user = SimpleLazyObject(lambda: get_user(request))
File "django/contrib/auth/middleware.py", line 11, in get_user
request._cached_user = auth.get_user(request)
File "django/contrib/auth/__init__.py", line 183, in get_user
backend = load_backend(backend_path)
File "django/contrib/auth/__init__.py", line 21, in load_backend
return import_string(path)()
File "django/utils/module_loading.py", line 28, in import_string
raise ImportError('Module "%s" does not define a "%s" attribute/class'
% (
Exception Type: ImportError at /cart/dropdown/
Exception Value: Module "authbackend" does not define a "Bknd"
attribute/class
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/33107>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: Collin Anderson | Owner: nobody
Type: Bug | Status: new
Component: Uncategorized | Version: dev
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
--
Ticket URL: <https://code.djangoproject.com/ticket/33107#comment:1>
Django
Reporter: Collin Anderson | Owner: nobody
Type: Bug | Status: new
Component: Uncategorized | Version: dev
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Keryn Knight):
* cc: Keryn Knight (added)
* stage: Unreviewed => Accepted
Comment:
Seems like a legitimate problem, that either requires working around
(fastpath vs slowpath as mentioned) or reverting the change made in
#33099.
It'd be interesting to see the middlewares & modules in play, even if
they're reduced for privacy. My ''guess'' is circular imports are
occurring possibly?
--
Ticket URL: <https://code.djangoproject.com/ticket/33107#comment:2>
Django
Reporter: Collin Anderson | Owner: nobody
Type: Bug | Status: new
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Mariusz Felisiak):
* version: dev => 4.0
* component: Uncategorized => Utilities
* severity: Normal => Release blocker
Comment:
Regression in ecf87ad513fd8af6e4a6093ed918723a7d88d5ca.
--
Ticket URL: <https://code.djangoproject.com/ticket/33107#comment:3>
Django
Reporter: Collin Anderson | Owner: nobody
Type: Bug | Status: new
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Mariusz Felisiak):
The easiest (but hacky) way to fix it is to check Python's internals and
reload the module again:
{{{
diff --git a/django/utils/module_loading.py
b/django/utils/module_loading.py
index 1df82b1c32..39cb784f72 100644
--- a/django/utils/module_loading.py
+++ b/django/utils/module_loading.py
@@ -7,7 +7,11 @@ from importlib.util import find_spec as importlib_find
def cached_import(module_path, class_name):
modules = sys.modules
- if module_path not in modules:
+ if module_path not in modules or (
+ # Module it not fully initialize.
+ getattr(modules[module_path], '__spec__', None) is not None and
+ getattr(modules[module_path].__spec__, '_initializing', False) is
True
+ ):
import_module(module_path)
return getattr(modules[module_path], class_name)
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/33107#comment:4>
Django
Reporter: Collin Anderson | Owner: nobody
Type: Bug | Status: new
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Mariusz Felisiak):
`importlib` now uses the same hook, see
https://github.com/python/cpython/commit/03648a2a91f9f1091cd21bd4cd6ca092ddb25640.
--
Ticket URL: <https://code.djangoproject.com/ticket/33107#comment:5>
Django
Reporter: Collin Anderson | Owner: nobody
Type: Bug | Status: new
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Keryn Knight):
Serendipitous find :) For historical reference, it looks like
`_initializing` is set via `importlib._bootstrap._load_unlocked`:
{{{
# This must be done before putting the module in sys.modules
# (otherwise an optimization shortcut in import.c becomes
# wrong).
spec._initializing = True
}}}
Possibly worth noting however is `importlib.util._module_to_load` which
has equivalent but different (at least in `3.9.5`):
{{{
# This must be done before putting the module in sys.modules
# (otherwise an optimization shortcut in import.c becomes wrong)
module.__initializing__ = True
}}}
If we're going to try and keep the `cached_import`, perhaps we need to
accommodate both, or at least investigate the `__initializing__`
variation?
FWIW, the investigation/patch proposed above does change the performance
profile (it's `5.64 µs` for me now) but is still world's better than
repeatedly doing `import_module` (at least until Python `3.11` which might
improve things via the linked bpo-43392)
--
Ticket URL: <https://code.djangoproject.com/ticket/33107#comment:6>
Django
Reporter: Collin Anderson | Owner: Mariusz
| Felisiak
Type: Bug | Status: assigned
Component: Utilities | Version: 4.0
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Mariusz Felisiak):
* owner: nobody => Mariusz Felisiak
* status: new => assigned
* has_patch: 0 => 1
Comment:
[https://github.com/django/django/pull/14858 PR]
> Possibly worth noting however is `importlib.util._module_to_load` which
has equivalent but different (at least in `3.9.5`):
As far as I'm aware we shouldn't reach `_module_to_load()` using
`import_module()`. Also Python uses the same strategy (checking
`__spec__._initialized`) in few places, e.g.
[https://github.com/python/cpython/blob/40d2ac92f9a28a486156dafdbb613016bb1f6b98/Python/import.c#L353-L370
import_ensure_initialized()].
--
Ticket URL: <https://code.djangoproject.com/ticket/33107#comment:7>
Django
Reporter: Collin Anderson | Owner: Mariusz
| Felisiak
Type: Bug | Status: assigned
Component: Utilities | Version: 4.0
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Collin Anderson):
By the way, I figured out I can just put `time.sleep(5)` at the top-level
of my custom auth backend to reproduce the issue the patch does fix the
issue. Thanks!
--
Ticket URL: <https://code.djangoproject.com/ticket/33107#comment:8>
Django
Reporter: Collin Anderson | Owner: Mariusz
| Felisiak
Component: Utilities | Version: 4.0
Severity: Release blocker | Resolution: fixed
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by GitHub <noreply@…>):
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"6426c3077c8048bb3fa5bfcec6be80f73476b534" 6426c307]:
{{{
#!CommitTicketReference repository=""
revision="6426c3077c8048bb3fa5bfcec6be80f73476b534"
Fixed #33107 -- Fixed import_string() crash on not fully initialized
modules.
Regression in ecf87ad513fd8af6e4a6093ed918723a7d88d5ca.
Thanks Collin Anderson for the report.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/33107#comment:9>
Django
Reporter: Collin Anderson | Owner: Mariusz
| Felisiak
Type: Bug | Status: closed
Component: Utilities | Version: 4.0
Severity: Release blocker | Resolution: fixed
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by GitHub <noreply@…>):
In [changeset:"a3185a670169eb67f1ce5c774fe7af555f80163b" a3185a6]:
{{{
#!CommitTicketReference repository=""
revision="a3185a670169eb67f1ce5c774fe7af555f80163b"
Refs #33107 -- Optimized cached_import() helper.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/33107#comment:10>