[Django] #28241: module_has_submodule behavior in Python 3
Django
-------------------------------------+-------------------------------------
Reporter: Thomas | Owner: nobody
Khyn |
Type: Bug | Status: new
Component: Utilities | Version: 1.11
Severity: Normal | Keywords: module_loading
Triage Stage: | python3
Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 1
UI/UX: 0 |
-------------------------------------+-------------------------------------
Hello,
Porting a django project from python 2.7 to 3.6, I noticed an issue with
``utils.module_loading.module_has_submodule``. At some stage my project
makes use of ``autodiscover_modules('module.submodule')`` to try and
discover modules that are nested in my app.
I'm using django 1.11.1, but this bug probably also affects 1.8 as the
code of ``module_has_submodule`` is the same.
So here we go, with python 2.7 we have the expected behavior (taking
contenttypes as an example app):
{{{
python2.7
>>> from django.utils.module_loading import module_has_submodule
>>> from django.contrib import contenttypes
>>> module_has_submodule('invalid_module.submodule')
False
>>> module_has_submodule('checks')
True
>>> module_has_submodule('checks.submodule')
False
}}}
But, with python 3.6
{{{
python3.6
>>> from django.utils.module_loading import module_has_submodule
>>> from django.contrib import contenttypes
>>> module_has_submodule('invalid_module.submodule')
File "<console>", line 1, in <module>
File
"d:\dev\.env\buildout\eggs\django-1.11.1-py3.6.egg\django\utils\module_loading.py",
line 79, in module_has_submodule
return importlib_find(full_module_name, package_path) is not None
File "D:\dev\.env\venv\buildout\lib\importlib\util.py", line 88, in
find_spec
parent = __import__(parent_name, fromlist=['__path__'])
ModuleNotFoundError: No module named
'django.contrib.contenttypes.invalid_module'
>>> module_has_submodule('checks')
True
>>> module_has_submodule('checks.submodule')
File "<console>", line 1, in <module>
File
"d:\dev\.env\buildout\eggs\django-1.11.1-py3.6.egg\django\utils\module_loading.py",
line 79, in module_has_submodule
return importlib_find(full_module_name, package_path) is not None
File "D:\dev\.env\venv\buildout\lib\importlib\util.py", line 89, in
find_spec
return _find_spec(fullname, parent.__path__)
AttributeError: module 'django.contrib.contenttypes.checks' has no
attribute '__path__'
}}}
From the replies I got on the python bug tracker
(http://bugs.python.org/issue30436) the `AttributeError` will be converted
to `ModuleNotFoundError` only in Python 3.7, but that behavior is
apparently not expected to be fixed in previous versions.
We'll definitely need to catch `ModuleNotFoundError`, and to have a proper
fix for python < 3.7 we'll need to:
- either catch `AttributeError` as well. The problem I see is
``AttributeError`` is too broad and may result in 'false catches'
- or convert a dotted path to [package, name] (basically do what
``find_spec`` actually does) but detect if the module has a `__path__`
attribute before calling `find_spec` so that it can not raise exceptions
What are your thoughts ?
--
Ticket URL: <https://code.djangoproject.com/ticket/28241>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
-------------------------------------+-------------------------------------
Reporter: Thomas Khyn | Owner: nobody
Type: Bug | Status: new
Component: Utilities | Version: 1.11
Keywords: module_loading | Triage Stage:
python3 | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
-------------------------------------+-------------------------------------
--
Ticket URL: <https://code.djangoproject.com/ticket/28241#comment:1>
Django
-------------------------------------+-------------------------------------
Reporter: Thomas Khyn | Owner: nobody
Type: Bug | Status: new
Component: Utilities | Version: 1.11
Keywords: module_loading | Triage Stage:
python3 | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
-------------------------------------+-------------------------------------
Old description:
New description:
Hello,
Porting a django project from python 2.7 to 3.6, I noticed an issue with
``utils.module_loading.module_has_submodule``. At some stage my project
makes use of ``autodiscover_modules('module.submodule')`` to try and
discover modules that are nested in my app.
I'm using django 1.11.1, but this bug probably also affects 1.8 as the
code of ``module_has_submodule`` is the same.
So here we go, with python 2.7 we have the expected behavior (taking
contenttypes as an example app):
{{{
python2.7
>>> from django.utils.module_loading import module_has_submodule
>>> from django.contrib import contenttypes
>>> module_has_submodule(contenttypes, 'invalid_module.submodule')
False
>>> module_has_submodule(contenttypes, 'checks')
True
>>> module_has_submodule(contenttypes, 'checks.submodule')
False
}}}
But, with python 3.6
{{{
python3.6
>>> from django.utils.module_loading import module_has_submodule
>>> from django.contrib import contenttypes
>>> module_has_submodule(contenttypes, 'invalid_module.submodule')
File "<console>", line 1, in <module>
File
"d:\dev\.env\buildout\eggs\django-1.11.1-py3.6.egg\django\utils\module_loading.py",
line 79, in module_has_submodule
return importlib_find(full_module_name, package_path) is not None
File "D:\dev\.env\venv\buildout\lib\importlib\util.py", line 88, in
find_spec
parent = __import__(parent_name, fromlist=['__path__'])
ModuleNotFoundError: No module named
'django.contrib.contenttypes.invalid_module'
>>> module_has_submodule(contenttypes, 'checks')
True
>>> module_has_submodule(contenttypes, 'checks.submodule')
File "<console>", line 1, in <module>
File
"d:\dev\.env\buildout\eggs\django-1.11.1-py3.6.egg\django\utils\module_loading.py",
line 79, in module_has_submodule
return importlib_find(full_module_name, package_path) is not None
File "D:\dev\.env\venv\buildout\lib\importlib\util.py", line 89, in
find_spec
return _find_spec(fullname, parent.__path__)
AttributeError: module 'django.contrib.contenttypes.checks' has no
attribute '__path__'
}}}
From the replies I got on the python bug tracker
(http://bugs.python.org/issue30436) the `AttributeError` will be converted
to `ModuleNotFoundError` only in Python 3.7, but that behavior is
apparently not expected to be fixed in previous versions.
We'll definitely need to catch `ModuleNotFoundError`, and to have a proper
fix for python < 3.7 we'll need to:
- either catch `AttributeError` as well. The problem I see is
``AttributeError`` is too broad and may result in 'false catches'
- or convert a dotted path to [package, name] (basically do what
``find_spec`` actually does) but detect if the module has a `__path__`
attribute before calling `find_spec` so that it can not raise exceptions
What are your thoughts ?
--
--
Ticket URL: <https://code.djangoproject.com/ticket/28241#comment:2>
Django
is a dotted path
-------------------------------------+-------------------------------------
Reporter: Thomas Khyn | Owner: nobody
Component: Utilities | Version: 1.11
Keywords: module_loading | Triage Stage:
python3 | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
-------------------------------------+-------------------------------------
Changes (by Tim Graham):
* easy: 1 => 0
Old description:
> Hello,
>
> Porting a django project from python 2.7 to 3.6, I noticed an issue with
> ``utils.module_loading.module_has_submodule``. At some stage my project
> makes use of ``autodiscover_modules('module.submodule')`` to try and
> discover modules that are nested in my app.
>
> I'm using django 1.11.1, but this bug probably also affects 1.8 as the
> code of ``module_has_submodule`` is the same.
>
> So here we go, with python 2.7 we have the expected behavior (taking
> contenttypes as an example app):
>
> {{{
> python2.7
> >>> from django.utils.module_loading import module_has_submodule
> >>> from django.contrib import contenttypes
> >>> module_has_submodule(contenttypes, 'invalid_module.submodule')
> False
> >>> module_has_submodule(contenttypes, 'checks')
> True
> >>> module_has_submodule(contenttypes, 'checks.submodule')
> False
> }}}
>
> But, with python 3.6
>
> {{{
> python3.6
> >>> from django.utils.module_loading import module_has_submodule
> >>> from django.contrib import contenttypes
> >>> module_has_submodule(contenttypes, 'invalid_module.submodule')
> File "<console>", line 1, in <module>
> File
> "d:\dev\.env\buildout\eggs\django-1.11.1-py3.6.egg\django\utils\module_loading.py",
> line 79, in module_has_submodule
> return importlib_find(full_module_name, package_path) is not None
> File "D:\dev\.env\venv\buildout\lib\importlib\util.py", line 88, in
> find_spec
> parent = __import__(parent_name, fromlist=['__path__'])
> ModuleNotFoundError: No module named
> 'django.contrib.contenttypes.invalid_module'
> >>> module_has_submodule(contenttypes, 'checks')
> True
> >>> module_has_submodule(contenttypes, 'checks.submodule')
> File "<console>", line 1, in <module>
> File
> "d:\dev\.env\buildout\eggs\django-1.11.1-py3.6.egg\django\utils\module_loading.py",
> line 79, in module_has_submodule
> return importlib_find(full_module_name, package_path) is not None
> File "D:\dev\.env\venv\buildout\lib\importlib\util.py", line 89, in
> find_spec
> return _find_spec(fullname, parent.__path__)
> AttributeError: module 'django.contrib.contenttypes.checks' has no
> attribute '__path__'
> }}}
>
> From the replies I got on the python bug tracker
> (http://bugs.python.org/issue30436) the `AttributeError` will be
> converted to `ModuleNotFoundError` only in Python 3.7, but that behavior
> is apparently not expected to be fixed in previous versions.
>
> We'll definitely need to catch `ModuleNotFoundError`, and to have a
> proper fix for python < 3.7 we'll need to:
> - either catch `AttributeError` as well. The problem I see is
> ``AttributeError`` is too broad and may result in 'false catches'
> - or convert a dotted path to [package, name] (basically do what
> ``find_spec`` actually does) but detect if the module has a `__path__`
> attribute before calling `find_spec` so that it can not raise exceptions
>
> What are your thoughts ?
New description:
Hello,
Porting a django project from python 2.7 to 3.6, I noticed an issue with
`utils.module_loading.module_has_submodule`. At some stage my project
makes use of `autodiscover_modules('module.submodule')` to try and
discover modules that are nested in my app.
I'm using django 1.11.1, but this bug probably also affects 1.8 as the
code of `module_has_submodule` is the same.
So here we go, with python 2.7 we have the expected behavior (taking
contenttypes as an example app):
{{{
python2.7
>>> from django.utils.module_loading import module_has_submodule
>>> from django.contrib import contenttypes
>>> module_has_submodule(contenttypes, 'invalid_module.submodule')
False
>>> module_has_submodule(contenttypes, 'checks')
True
>>> module_has_submodule(contenttypes, 'checks.submodule')
False
}}}
But, with python 3.6
{{{
python3.6
>>> from django.utils.module_loading import module_has_submodule
>>> from django.contrib import contenttypes
>>> module_has_submodule(contenttypes, 'invalid_module.submodule')
File "<console>", line 1, in <module>
File
"d:\dev\.env\buildout\eggs\django-1.11.1-py3.6.egg\django\utils\module_loading.py",
line 79, in module_has_submodule
return importlib_find(full_module_name, package_path) is not None
File "D:\dev\.env\venv\buildout\lib\importlib\util.py", line 88, in
find_spec
parent = __import__(parent_name, fromlist=['__path__'])
ModuleNotFoundError: No module named
'django.contrib.contenttypes.invalid_module'
>>> module_has_submodule(contenttypes, 'checks')
True
>>> module_has_submodule(contenttypes, 'checks.submodule')
File "<console>", line 1, in <module>
File
"d:\dev\.env\buildout\eggs\django-1.11.1-py3.6.egg\django\utils\module_loading.py",
line 79, in module_has_submodule
return importlib_find(full_module_name, package_path) is not None
File "D:\dev\.env\venv\buildout\lib\importlib\util.py", line 89, in
find_spec
return _find_spec(fullname, parent.__path__)
AttributeError: module 'django.contrib.contenttypes.checks' has no
attribute '__path__'
}}}
From the replies I got on the python bug tracker
(http://bugs.python.org/issue30436) the `AttributeError` will be converted
to `ModuleNotFoundError` only in Python 3.7, but that behavior is
apparently not expected to be fixed in previous versions.
We'll definitely need to catch `ModuleNotFoundError`, and to have a proper
fix for python < 3.7 we'll need to:
- either catch `AttributeError` as well. The problem I see is
``AttributeError`` is too broad and may result in 'false catches'
- or convert a dotted path to [package, name] (basically do what
``find_spec`` actually does) but detect if the module has a `__path__`
attribute before calling `find_spec` so that it can not raise exceptions
What are your thoughts ?
--
Comment:
I don't think the `module_name` argument of `module_has_submodule()` is
expected to be a dotted path. For example, this test (incorrectly, as far
as I see) passes on Python 2:
{{{#!diff
diff --git a/tests/utils_tests/test_module_loading.py
b/tests/utils_tests/test_module_loading.py
index 2a524a2..70047b2 100644
--- a/tests/utils_tests/test_module_loading.py
+++ b/tests/utils_tests/test_module_loading.py
@@ -26,6 +26,8 @@ class DefaultLoader(unittest.TestCase):
test_no_submodule = import_module(
'utils_tests.test_no_submodule')
+ self.assertTrue(module_has_submodule(test_module,
'invalid.good_module'))
+
# An importable child
self.assertTrue(module_has_submodule(test_module, 'good_module'))
mod = import_module('utils_tests.test_module.good_module')
}}}
Is Django making any calls like that or is this function used like this
outside of Django?
--
Ticket URL: <https://code.djangoproject.com/ticket/28241#comment:3>
Django
is a dotted path
-------------------------------------+-------------------------------------
Reporter: Thomas Khyn | Owner: nobody
Component: Utilities | Version: 1.11
Keywords: module_loading | Triage Stage:
python3 | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
-------------------------------------+-------------------------------------
Comment (by Thomas Khyn):
That's what I said, it works on Python 2, not on Python 3.
The problem I have with that is I need to use
`autodiscover_modules('path_to.my_module')`, which calls
`module_has_submodule` in every app. In Python 2 it attempts to discover
this module in all apps. In Python 3 it raises an exception the first time
`path_to` is not a package, preventing loading any module in subsequent
apps.
I know one solution is to re-organize my apps structure to put `my_module`
at the apps' root level, but:
- that does mess up with the organization of my code
- this used to work in Python 2 and could easily be made to work in Python
3, and I would like to avoid having to monkey-patch django just to add a
try / catch block ...
--
Ticket URL: <https://code.djangoproject.com/ticket/28241#comment:4>
Django
is a dotted path
-------------------------------------+-------------------------------------
Reporter: Thomas Khyn | Owner: nobody
Component: Utilities | Version: 1.11
Keywords: module_loading | Triage Stage:
python3 | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
-------------------------------------+-------------------------------------
Comment (by Thomas Khyn):
In addition, trying to reproduce your example with python 2.7.13 I get:
{{{
>>> from importlib import import_module
>>> from django.utils.module_loading import module_has_submodule
>>> test_module = import_module('utils_tests.test_module')
>>> module_has_submodule(test_module, 'invalid.good_module')
False
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/28241#comment:5>
Django
is a dotted path
-------------------------------------+-------------------------------------
Reporter: Thomas Khyn | Owner: nobody
Component: Utilities | Version: 1.11
Keywords: module_loading | Triage Stage: Accepted
python3 |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
-------------------------------------+-------------------------------------
Changes (by Tim Graham):
* stage: Unreviewed => Accepted
Comment:
I don't know if `module_has_submodule()` should not accept dotted paths,
however, as far as I can tell, the behavior is undocumented and untested.
We could either fix it or raise a more helpful error message.
--
Ticket URL: <https://code.djangoproject.com/ticket/28241#comment:6>
Django
is a dotted path
-------------------------------------+-------------------------------------
Type: Bug | Status: assigned
Component: Utilities | Version: 1.11
Severity: Normal | Resolution:
Keywords: module_loading | Triage Stage: Accepted
python3 |
Needs tests: 0 | Patch needs improvement: 0
-------------------------------------+-------------------------------------
* owner: nobody => tkhyn
* status: new => assigned
Comment:
Great, I'll submit a PR with testcases + solution for all currently
supported versions.
--
Ticket URL: <https://code.djangoproject.com/ticket/28241#comment:7>
Django
is a dotted path
-------------------------------------+-------------------------------------
Reporter: tkhyn | Owner: tkhyn
Type: Bug | Status: assigned
Severity: Normal | Resolution:
Keywords: module_loading | Triage Stage: Accepted
python3 |
Needs tests: 0 | Patch needs improvement: 0
-------------------------------------+-------------------------------------
Changes (by tkhyn):
* has_patch: 0 => 1
Comment:
Here is a patch: [https://github.com/django/django/pull/8611 PR 8611]
--
Ticket URL: <https://code.djangoproject.com/ticket/28241#comment:8>
Django
is a dotted path
-------------------------------------+-------------------------------------
Reporter: tkhyn | Owner: tkhyn
Component: Utilities | Version: 1.11
Severity: Normal | Resolution: fixed
Keywords: module_loading | Triage Stage: Accepted
python3 |
Needs tests: 0 | Patch needs improvement: 0
-------------------------------------+-------------------------------------
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"f6bd00131e687aedf2719ad31e84b097562ca5f2" f6bd0013]:
{{{
#!CommitTicketReference repository=""
revision="f6bd00131e687aedf2719ad31e84b097562ca5f2"
Fixed #28241 -- Allowed module_has_submodule()'s module_name arg to be a
dotted path.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/28241#comment:9>