[Django] #36866: Optimize template compilation with intelligent caching for repeated renders

[Django]

#36866: Optimize template compilation with intelligent caching for repeated renders
-------------------------------------+-------------------------------------
Reporter: TheAshutoshMishra | Type: New
| feature
Status: new | Component: Template
| system
Version: 6.0 | Severity: Normal
Keywords: templates, | Triage Stage:
performance, caching, | Unreviewed
optimization |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Currently, Django's template system performs expensive compilation
operations (tokenization and filter expression parsing) on every template
render, even when rendering the same template repeatedly with different
contexts. This causes unnecessary CPU overhead in production.

=== Problem ===

The template compilation process involves:
1. **Tokenization**: Regex splitting of template strings via
`tag_re.split()`
2. **Filter expression compilation**: Parsing filter syntax like `{{
var|filter:"arg" }}`

Both operations are repeated identically for every render, even though the
results are deterministic based on the template source.

=== Proposed Solution ===

Add bounded, memory-safe caching for:
1. **Tokenized template strings** (max 1000 entries, hash-based keys)
2. **Compiled FilterExpression objects** (max 500 entries)

=== Implementation Details ===

- Hash-based cache keys for O(1) lookups
- Smart eviction: removes 20-25% of oldest entries when limit reached
- Skips caching for stateful verbatim templates
- Graceful fallbacks for edge cases
- New `clear_template_caches()` utility for production cache management

=== Performance Impact (Measured) ===

- **13-47% faster** template rendering on repeated renders
- **1.15M tokenizations/second** throughput with caching
- Zero configuration required
- Fully backward compatible

=== PR ===

Implementation available in PR #20540:
https://github.com/django/django/pull/20540

Looking for feedback on this approach and guidance on adding comprehensive
test coverage.
--
Ticket URL: <https://code.djangoproject.com/ticket/36866>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#36866: Optimize template compilation with intelligent caching for repeated renders
-------------------------------------+-------------------------------------

Reporter: TheAshutoshMishra | Owner: (none)
Type: New feature | Status: closed

Component: Template system | Version: 6.0

Severity: Normal | Resolution:
Keywords: templates, | needsnewfeatureprocess
performance, caching, | Triage Stage:
optimization | Unreviewed

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Natalia Bidart):

* resolution: => needsnewfeatureprocess
* status: new => closed

Comment:

Hello TheAshutoshMishra,

New features proposals must first go through Django's new feature proposal
process and be discussed with the community. To do that, please raise this
on the [https://github.com/django/new-features/issues ​new feature
tracker].

Also note that when using an LLM to assist with a contribution, you are
responsible for ensuring it fully reads and follows Django's
documentation, including `docs/internals/contributing/` in the repo, or
[https://docs.djangoproject.com/en/6.0/internals/contributing/ the online
docs]. Submissions that do not meet those requirements cannot be reviewed
in Trac nor in GitHub.
--
Ticket URL: <https://code.djangoproject.com/ticket/36866#comment:1>