[Django] #35250: Stop URL system checks from compiling regular expressions

[Django]

#35250: Stop URL system checks from compiling regular expressions
-------------------------------------+-------------------------------------
Reporter: Adam | Owner: Adam Johnson
Johnson |
Type: | Status: assigned
Cleanup/optimization |
Component: Core | Version: dev
(System checks) |
Severity: Normal | Keywords:
Triage Stage: | Has patch: 1
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
Continuing my project to optimize the system checks, I found some good
optimizations under `django.core.checks.urls.check_url_config()`, which
showed up as quite expensive in profiling.

Looking down the call tree, it seems the most expensive part of this
process is compiling the each URL pattern’s regular expression. This is
unnecessary work though, as the checks only need *uncompiled* regular
expression patterns. Using the compiled versions “undoes” the lazy-compile
optimization that `LocaleRegexDescriptor` was created for in #27453 /
6e222dae5636f875c19ec66f730a4241abe33faa, at least for any process that
runs checks.

The checks were fetching the uncompiled pattern with `self.regex.pattern`,
which makse `LocaleRegexDescriptor` compile the pattern only to then read
the uncompiled pattern from
[https://docs.python.org/3.12/library/re.html#re.Pattern.pattern its
pattern attribute].

Additionally, `RoutePattern` was calling `_route_to_regex()` twice to
fetch its two result variables in different places: once in `__init__()`
and again in `_compile()`. This function is non-trivial.

Before optimization stats:

* `check_url_config` took 67ms, or ~10% of the time for checks.
* `LocaleRegexDescriptor.__get__()` showed 965 calls taking ~60ms, ~9% of
the total runtime of checks.
* `re.compile()` showed 741 calls for 94ms.
* `_route_to_regex()` had 1900 calls taking 18ms (~2.6% of the total
runtime).

After optimization:

* `check_url_config()` took 5ms, ~0.9% of the new total runtime.
* The calls to `LocaleRegexDescriptor.__get__` are gone.
* `re.compile()` drops to 212 calls from other sites, for a total of 51ms.
* `_route_to_regex()` drops to the expected 950 calls, taking half the
time at 9ms.
--
Ticket URL: <https://code.djangoproject.com/ticket/35250>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#35250: Stop URL system checks from compiling regular expressions
-------------------------------------+-------------------------------------

Reporter: Adam Johnson | Owner: Adam
Type: | Johnson
Cleanup/optimization | Status: assigned
Component: Core (System | Version: dev
checks) |
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Description changed by Adam Johnson:

Old description:

New description:

Continuing my project to optimize the system checks, I found some good
optimizations under `django.core.checks.urls.check_url_config()`, which
showed up as quite expensive in profiling.

Looking down the call tree, it seems the most expensive part of this
process is compiling the each URL pattern’s regular expression. This is
unnecessary work though, as the checks only need *uncompiled* regular
expression patterns. Using the compiled versions “undoes” the lazy-compile
optimization that `LocaleRegexDescriptor` was created for in #27453 /
6e222dae5636f875c19ec66f730a4241abe33faa, at least for any process that
runs checks.

The checks were fetching the uncompiled pattern with `self.regex.pattern`,
which makse `LocaleRegexDescriptor` compile the pattern only to then read
the uncompiled pattern from
[https://docs.python.org/3.12/library/re.html#re.Pattern.pattern its
pattern attribute].

Additionally, `RoutePattern` was calling `_route_to_regex()` twice to
fetch its two result variables in different places: once in `__init__()`

and again in `_compile()` (in the non-translated case). This function has
non-trivial cost so avoiding double execution is worth it.

Before optimization stats:

* `check_url_config` took 67ms, or ~10% of the time for checks.
* `LocaleRegexDescriptor.__get__()` showed 965 calls taking ~60ms, ~9% of
the total runtime of checks.
* `re.compile()` showed 741 calls for 94ms.
* `_route_to_regex()` had 1900 calls taking 18ms (~2.6% of the total
runtime).

After optimization:

* `check_url_config()` took 5ms, ~0.9% of the new total runtime.
* The calls to `LocaleRegexDescriptor.__get__` are gone.
* `re.compile()` drops to 212 calls from other sites, for a total of 51ms.
* `_route_to_regex()` drops to the expected 950 calls, taking half the
time at 9ms.

--

--
Ticket URL: <https://code.djangoproject.com/ticket/35250#comment:1>

[Django]

(I also tried benchmarking with django-asv but got inconclusive results
where change was within the error margins.)

--
--
Ticket URL: <https://code.djangoproject.com/ticket/35250#comment:2>

[Django]

#35250: Stop URL system checks from compiling regular expressions
-------------------------------------+-------------------------------------
Reporter: Adam Johnson | Owner: Adam
Type: | Johnson
Cleanup/optimization | Status: assigned
Component: Core (System | Version: dev
checks) |
Severity: Normal | Resolution:

Keywords: | Triage Stage: Accepted

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Mariusz Felisiak):

* stage: Unreviewed => Accepted

Comment:

Tentatively accepted.
--
Ticket URL: <https://code.djangoproject.com/ticket/35250#comment:3>

[Django]

#35250: Stop URL system checks from compiling regular expressions
-------------------------------------+-------------------------------------
Reporter: Adam Johnson | Owner: Adam
Type: | Johnson
Cleanup/optimization | Status: assigned
Component: Core (System | Version: dev
checks) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 1

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Mariusz Felisiak):

* needs_better_patch: 0 => 1

--
Ticket URL: <https://code.djangoproject.com/ticket/35250#comment:4>

[Django]

#35250: Stop URL system checks from compiling regular expressions
-------------------------------------+-------------------------------------
Reporter: Adam Johnson | Owner: Adam
Type: | Johnson
Cleanup/optimization | Status: assigned
Component: Core (System | Version: dev
checks) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Adam Johnson):

* needs_better_patch: 1 => 0

--
Ticket URL: <https://code.djangoproject.com/ticket/35250#comment:5>

[Django]

#35250: Stop URL system checks from compiling regular expressions
-------------------------------------+-------------------------------------
Reporter: Adam Johnson | Owner: Adam
Type: | Johnson
Cleanup/optimization | Status: assigned
Component: Core (System | Version: dev
checks) |
Severity: Normal | Resolution:

Keywords: | Triage Stage: Ready for
| checkin

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Mariusz Felisiak):

* stage: Accepted => Ready for checkin

--
Ticket URL: <https://code.djangoproject.com/ticket/35250#comment:6>

[Django]

#35250: Stop URL system checks from compiling regular expressions
-------------------------------------+-------------------------------------
Reporter: Adam Johnson | Owner: Adam
Type: | Johnson
Cleanup/optimization | Status: assigned
Component: Core (System | Version: dev
checks) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Mariusz Felisiak <felisiak.mariusz@…>):

In [changeset:"5dfcf343cd414d3f7a33dabb763b4478fa081d72" 5dfcf34]:
{{{#!CommitTicketReference repository=""
revision="5dfcf343cd414d3f7a33dabb763b4478fa081d72"
Refs #35250 -- Avoided double conversion in RoutePattern.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/35250#comment:7>

[Django]

#35250: Stop URL system checks from compiling regular expressions
-------------------------------------+-------------------------------------
Reporter: Adam Johnson | Owner: Adam
Type: | Johnson

Cleanup/optimization | Status: closed

Component: Core (System | Version: dev
checks) |

Severity: Normal | Resolution: fixed

Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Mariusz Felisiak <felisiak.mariusz@…>):

* resolution: => fixed
* status: assigned => closed

Comment:

In [changeset:"71d5eafb05a19287d954847beb33e4eeca65b4c4" 71d5eaf]:
{{{#!CommitTicketReference repository=""
revision="71d5eafb05a19287d954847beb33e4eeca65b4c4"
Fixed #35250 -- Made URL system checks use uncompiled regexes.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/35250#comment:8>