Re: [Django] #5815: Adds per-view cache refreshing (clearing)
Django
-------------------------------------+-------------------------------------
Reporter: k0001 | Owner: nobody
Type: New feature | Status: new
Component: Core (Cache system) | Version: master
Severity: Normal | Resolution:
Keywords: cache refresh clear | Triage Stage: Design
Has patch: 1 | decision needed
Needs tests: 1 | Needs documentation: 0
Easy pickings: 0 | Patch needs improvement: 1
| UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by aaugustin):
* stage: Accepted => Design decision needed
Comment:
Setting to DDN since there's some disagreement between Jacob and Chris.
Explicit cache invalidation can be useful in some scenarios, it'd be nice
to support this.
--
Ticket URL: <https://code.djangoproject.com/ticket/5815#comment:8>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
-------------------------------------+------------------------------------
Type: New feature | Status: new
Component: Core (Cache system) | Version: master
Severity: Normal | Resolution:
Has patch: 1 | Needs documentation: 0
Needs tests: 1 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+------------------------------------
Changes (by jacob):
* stage: Design decision needed => Accepted
Comment:
If there's really not a better implementation (perhaps there isn't) then
I'm OK with this going in as-is.
--
Ticket URL: <https://code.djangoproject.com/ticket/5815#comment:9>
Django
-------------------------------------+------------------------------------
Reporter: k0001 | Owner: nobody
Type: New feature | Status: new
Component: Core (Cache system) | Version: master
Severity: Normal | Resolution:
Keywords: cache refresh clear | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 1 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+------------------------------------
Comment (by Carlton Gibson):
#31938 was a duplicate request similar functionality.
--
Ticket URL: <https://code.djangoproject.com/ticket/5815#comment:10>
Django
-------------------------------------+------------------------------------
Reporter: k0001 | Owner: nobody
Type: New feature | Status: new
Severity: Normal | Resolution:
Keywords: cache refresh clear | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 1 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+------------------------------------
Comment (by Mike Lissner):
Just to add a data points here, I spent a bunch of time today trying to
figure out which key to delete to nuke a cached page and eventually gave
up and just nuked a larger part of the cache in hopes of getting lucky (I
did).
Having a helper util just to figure out what the cache key value is would
be really nice.
Here's a summary of my failed research:
- "Expire a view-cache in Django?"
(https://stackoverflow.com/q/2268417/64911) — Says it works with < django
1.7. I didn't try it, but I don't think it'd work.
- "How to invalidate cache_page in Django?"
(https://stackoverflow.com/q/33749369/64911) — Accepted answer just says
to cache the queryset instead of using cache_page.
- "How to manually clear/update a cached view in django"
(https://stackoverflow.com/q/47040438/64911) — Has a solution, but it's
quite old and doesn't work. This is when I gave up.
The reason I need this (and couldn't wait for the cache to just expire) is
because I use the db cache to cache really big sitemaps that don't change
very often. The cache is usually on the order of a couple of weeks.
--
Ticket URL: <https://code.djangoproject.com/ticket/5815#comment:11>
Django
Reporter: k0001 | Owner: Ahter
| Sönmez
Type: New feature | Status: assigned
Component: Core (Cache system) | Version: dev
Severity: Normal | Resolution:
Keywords: cache refresh clear | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 1 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
Changes (by Ahter Sönmez):
* owner: nobody => Ahter Sönmez
* status: new => assigned
--
Ticket URL: <https://code.djangoproject.com/ticket/5815#comment:12>
Django
-------------------------------------+-------------------------------------
Reporter: k0001 | Owner: Ahter
| Sönmez
Type: New feature | Status: assigned
Component: Core (Cache system) | Version: dev
Severity: Normal | Resolution:
Keywords: cache refresh clear | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 1 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Carlton Gibson):
#34271 was a duplicate with a suggestion to make the cache key just depend
on the request path:
{{{
from pathlib import Path
from django.conf import settings
from django.core.cache import caches
default_cache = caches[settings.CACHE_MIDDLEWARE_ALIAS]
cache_key = ".".join(Path(request.path[1:]).parts)
if not (response := default_cache.get(cache_key)):
response = view_func(request, *args, **kwargs)
response.add_post_render_callback(
lambda r: default_cache.set(cache_key, r, timeout)
)
}}}
That would be easy to invalidate outside the request context.
Allowing/documenting easier make_key customisation might allow us to move
this forward? 🤔
--
Ticket URL: <https://code.djangoproject.com/ticket/5815#comment:13>
Django
-------------------------------------+-------------------------------------
Reporter: k0001 | Owner: Ahter
| Sönmez
Type: New feature | Status: assigned
Component: Core (Cache system) | Version: dev
Severity: Normal | Resolution:
Keywords: cache refresh clear | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 1 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: Petr Přikryl (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/5815#comment:14>
Django
-------------------------------------+-------------------------------------
Reporter: k0001 | Owner: Ahter
| Sönmez
Type: New feature | Status: assigned
Component: Core (Cache system) | Version: dev
Severity: Normal | Resolution:
Keywords: cache refresh clear | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 1 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Ahter Sönmez):
I haven't had a chance to look at this since the sprint in DjangoCon
Europe (nearly 6 months ago). As far as I remember, I already "something"
working.
In the next few days, let me check what was done, and share some progress
here.
--
Ticket URL: <https://code.djangoproject.com/ticket/5815#comment:15>
Django
-------------------------------------+-------------------------------------
Reporter: k0001 | Owner: Ahter
| Sönmez
Type: New feature | Status: assigned
Component: Core (Cache system) | Version: dev
Severity: Normal | Resolution:
Keywords: cache refresh clear | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 1 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Ahter Sönmez):
Here's a quick update on this:
I think this can be done. However, there are a few things to take into
consideration.
- Django's core caching code is quite old, 18+ years. I believe big part
of it should be redesigned and refactored so it's easy to work on it and
extend it in non-hacky ways. While this refactor is out of scope for this
PR, I think it might be beneficial. I think this might be why this ticket
is quite old.
- Even when we solve this problem, due to the way cache keys are
generated, this will be only supported by some cache backends (ie. Redis)
which allow filtering by keys or part of keys.
- There are better design patterns that would allow more intelligent
caching in the projects/apps that use Django. Especially projects that use
layered architecture. However, this is not applicable to Django as a
library as it cannot be opinionated about how devs would like to structure
their code in their apps.
I finally now have a chance to have a look at this again in DjangoCon
Europe 2023 sprints. This time we've also paired with Marco Silva and
we're tackling the issues together. I'm hoping to work on this more (given
we're now two) and get something out as a PR soon.
--
Ticket URL: <https://code.djangoproject.com/ticket/5815#comment:16>
Django
-------------------------------------+-------------------------------------
Reporter: k0001 | Owner: Ahter
| Sönmez
Type: New feature | Status: assigned
Component: Core (Cache system) | Version: dev
Severity: Normal | Resolution:
Keywords: cache refresh clear | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 1 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
I would like to my use case to widen the perspective.
I am running a website with ~4k RPM and clearing cache for some view could
result in timeouts due to cache hammering.
Multiple requests would bump into the cleared cache at the same time and
it would result in too many DB requests which would slow down the single
database significantly (the requests would take tens of seconds).
So clearing the cache even for single page/view is not an option for me.
The solution for me would be to directly inject new keys.
--
Ticket URL: <https://code.djangoproject.com/ticket/5815#comment:17>