[Django] #25491: django.utils.regex_helper.normalize should cache results locally
Django
--------------------------------------+--------------------
Reporter: atl-gmathews | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Utilities | Version: 1.8
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
--------------------------------------+--------------------
This function is used to process URL regexes for
django.core.urlresolvers.reverse, and can get called a lot in reverse-
heavy code.
This function is relatively expensive, has no side effects, and always
returns the same output for a given input. Therefore it makes sense to
memoize it with a local cache.
--
Ticket URL: <https://code.djangoproject.com/ticket/25491>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: atl-gmathews | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Utilities | Version: 1.8
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Easy pickings: 0
UI/UX: 0 |
Changes (by atl-gmathews):
* Attachment "0001-Cache-results-from-regex_util.normalize.patch" added.
Proposed patch to cache results from normalize()
Django
Reporter: atl-gmathews | Owner: nobody
Type: | Status: closed
Cleanup/optimization |
Component: Utilities | Version: 1.8
Severity: Normal | Resolution: wontfix
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by knbk):
* status: new => closed
* needs_docs: => 0
* resolution: => wontfix
* needs_tests: => 0
* needs_better_patch: => 0
Comment:
`normalize()` is called exactly once for each pattern, in `_populate()`
(during the first call to `resolve()` or `reverse()`), and then cached in
the `RegexURLResolver`. I don't see any sense in adding an additional
layer of caching. Even if it provides a performance boost for patterns
that are reused multiple times, the difference will be minimal at best,
and it won't affect any requests after the `RegexURLResolver` has cached
the normalized patterns.
1:
https://github.com/django/django/blob/master/django/core/urlresolvers.py#L260
--
Ticket URL: <https://code.djangoproject.com/ticket/25491#comment:1>
Django
Reporter: atl-gmathews | Owner: nobody
Type: | Status: closed
Cleanup/optimization |
Component: Utilities | Version: 1.8
Severity: Normal | Resolution: wontfix
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by erikvanzijst):
* cc: erik.van.zijst, at, gmail (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/25491#comment:2>
Django
Reporter: atl-gmathews | Owner: nobody
Type: | Status: closed
Cleanup/optimization |
Component: Utilities | Version: 1.8
Severity: Normal | Resolution: wontfix
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by atl-gmathews):
Replying to [comment:1 knbk]:
> Even if it provides a performance boost for patterns that are reused
multiple times, the difference will be minimal at best, and it won't
affect any requests after the `RegexURLResolver` has cached the normalized
patterns.
You're correct that `_populate()` only gets called once, and I was basing
my assertion that `normalize()` is called for each `reverse()` on old
code.
But I'm seeing a large performance difference between cached and uncached
`normalize()` in that one `_populate()` call during a normal request:
Uncached: 1000ms (57.2% of total time); 1400 calls
Cached: 5ms (0.89% of total time); 1400 calls
--
Ticket URL: <https://code.djangoproject.com/ticket/25491#comment:3>
Django
Reporter: atl-gmathews | Owner: nobody
Type: | Status: closed
Cleanup/optimization |
Component: Utilities | Version: 1.8
Severity: Normal | Resolution: wontfix
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by atl-gmathews):
* Attachment "uncached.png" added.
Profiling a request with uncached normalize()
Django
Reporter: atl-gmathews | Owner: nobody
Type: | Status: closed
Cleanup/optimization |
Component: Utilities | Version: 1.8
Severity: Normal | Resolution: wontfix
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by atl-gmathews):
* Attachment "cached.png" added.
Profiling a request with cached normalize()
Django
Reporter: atl-gmathews | Owner: nobody
Type: | Status: closed
Cleanup/optimization |
Component: Utilities | Version: 1.8
Severity: Normal | Resolution: wontfix
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by atl-gmathews):
I think my data is off for this issue: I made several requests to warm up
caches before profiling, but those requests were with an un-authenticated
user. I then profiled with requests from an authenticated user, which
instantiated a new `RegexURLResolver` and thus made a new `_populate()`
call.
So this change doesn't make most things faster, but it does make the
initial request much faster (31% faster for unauthorized, 37% for
authorized).
--
Ticket URL: <https://code.djangoproject.com/ticket/25491#comment:4>
Django
Reporter: atl-gmathews | Owner: nobody
Type: | Status: closed
Cleanup/optimization |
Component: Utilities | Version: 1.8
Severity: Normal | Resolution: wontfix
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by knbk):
* cc: marten.knbk@… (added)
Comment:
Could you share the urlpatterns you're using? Do you get a similar result
timing `_populate()` instead of profiling?
--
Ticket URL: <https://code.djangoproject.com/ticket/25491#comment:5>