[Django] #33099: Improve performance of import_string().
Django
-------------------------------------+-------------------------------------
Reporter: Mariusz | Owner: piaoxue1949
Felisiak |
Type: | Status: assigned
Cleanup/optimization |
Component: Utilities | Version: 3.2
Severity: Normal | Keywords:
Triage Stage: Accepted | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
We can improve performance of `import_string()` by avoiding multiple
imports for the same path. Proposed by `piaoxue1949` in
[https://github.com/django/django/pull/14815 PR].
--
Ticket URL: <https://code.djangoproject.com/ticket/33099>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
-------------------------------------+-------------------------------------
Type: | piaoxue1949
Cleanup/optimization | Status: assigned
Component: Utilities | Version: 3.2
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* has_patch: 0 => 1
Comment:
[https://github.com/django/django/pull/14850 PR]
--
Ticket URL: <https://code.djangoproject.com/ticket/33099#comment:1>
Django
-------------------------------------+-------------------------------------
Type: | piaoxue1949
Cleanup/optimization | Status: assigned
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Keryn Knight):
Here's some back of the napkin cprofile output on my laptop, in case
`piaoxue1949` is unable to provide before/after based on the proposed
changes:
Before, as of `46c8df640cfed5dd525ac1bcf5ad7e57b7ff2571`:
{{{
In [2]: get_backends()
Out[2]:
[<django.contrib.auth.backends.ModelBackend at 0x105307e50>,
<django.contrib.auth.backends.AllowAllUsersModelBackend at 0x1053bf790>,
<django.contrib.auth.backends.RemoteUserBackend at 0x105307940>,
<django.contrib.auth.backends.AllowAllUsersRemoteUserBackend at
0x105307790>]
In [3]: %timeit get_backends()
45.2 µs ± 1.02 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
In [4]: %prun for _ in range(10000): get_backends()
1940003 function calls in 0.911 seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
80000 0.119 0.000 0.228 0.000 <frozen
importlib._bootstrap>:166(_get_module_lock)
40000 0.118 0.000 0.718 0.000 <frozen
importlib._bootstrap>:1002(_find_and_load)
80000 0.079 0.000 0.100 0.000 <frozen
importlib._bootstrap>:87(acquire)
80000 0.077 0.000 0.097 0.000 <frozen
importlib._bootstrap>:112(release)
80000 0.058 0.000 0.089 0.000 <frozen
importlib._bootstrap>:58(__init__)
80000 0.054 0.000 0.080 0.000 <frozen
importlib._bootstrap>:185(cb)
40000 0.034 0.000 0.243 0.000 <frozen
importlib._bootstrap>:203(_lock_unlock_module)
40000 0.032 0.000 0.856 0.000
module_loading.py:15(import_string)
160000 0.030 0.000 0.030 0.000 {built-in method
_thread.allocate_lock}
40000 0.029 0.000 0.800 0.000
__init__.py:109(import_module)
40000 0.028 0.000 0.194 0.000 <frozen
importlib._bootstrap>:156(__enter__)
40000 0.024 0.000 0.761 0.000 <frozen
importlib._bootstrap>:1018(_gcd_import)
... many more lines, snipped for brevity
}}}
and after, using the `cached_import` to peek directly into `sys.modules`
{{{
In [3]: %timeit get_backends()
3.78 µs ± 24.1 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
In [4]: %prun for _ in range(10000): get_backends()
260003 function calls in 0.115 seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
40000 0.024 0.000 0.061 0.000
module_loading.py:15(import_string)
10000 0.023 0.000 0.105 0.000
__init__.py:24(_get_backends)
40000 0.018 0.000 0.028 0.000
module_loading.py:8(cached_import)
40000 0.017 0.000 0.078 0.000 __init__.py:20(load_backend)
40000 0.010 0.000 0.010 0.000 {built-in method
builtins.getattr}
40000 0.010 0.000 0.010 0.000 {method 'rsplit' of 'str'
objects}
1 0.005 0.005 0.115 0.115 <string>:1(<module>)
40000 0.004 0.000 0.004 0.000 {method 'append' of 'list'
objects}
10000 0.004 0.000 0.109 0.000 __init__.py:37(get_backends)
1 0.000 0.000 0.115 0.115 {built-in method
builtins.exec}
1 0.000 0.000 0.000 0.000 {method 'disable' of
'_lsprof.Profiler' objects}
}}}
that's the whole list. Not stripped for brevity.
--
Ticket URL: <https://code.djangoproject.com/ticket/33099#comment:2>
Django
-------------------------------------+-------------------------------------
Type: | piaoxue1949
Component: Utilities | Version: 3.2
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"ecf87ad513fd8af6e4a6093ed918723a7d88d5ca" ecf87ad5]:
{{{
#!CommitTicketReference repository=""
revision="ecf87ad513fd8af6e4a6093ed918723a7d88d5ca"
Fixed #33099 -- Improved performance of import_string().
This improves performance of import_string() by avoiding multiple
imports for the same path.
Thanks Andrew Godwin and Keryn Knight for the implementation idea.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/33099#comment:3>