[Django] #33252: CacheMiddleware is not thread safe
Django
------------------------------------------+------------------------
Reporter: Iuri de Silvio | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 3.2
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
------------------------------------------+------------------------
CacheMiddleware persist `self.cache = caches[cache_alias]` on startup and
it is not thread safe.
https://github.com/django/django/blob/main/django/middleware/cache.py#L186
I found that after some production errors with pylibmc and uwsgi threaded.
Created a small project to reproduce it. Nothing fancy, just pylibmc cache
and a `@cache_page` cached view. It fails even with development server,
with concurrent requests.
{{{
Traceback (most recent call last):
File "versions/pylibmcbug/lib/python3.9/site-
packages/django/core/handlers/exception.py", line 47, in inner
response = get_response(request)
File "versions/pylibmcbug/lib/python3.9/site-
packages/django/core/handlers/base.py", line 181, in _get_response
response = wrapped_callback(request, *callback_args,
**callback_kwargs)
File "versions/pylibmcbug/lib/python3.9/site-
packages/django/utils/decorators.py", line 122, in _wrapped_view
result = middleware.process_request(request)
File "versions/pylibmcbug/lib/python3.9/site-
packages/django/middleware/cache.py", line 145, in process_request
cache_key = get_cache_key(request, self.key_prefix, 'GET',
cache=self.cache)
File "versions/pylibmcbug/lib/python3.9/site-
packages/django/utils/cache.py", line 362, in get_cache_key
headerlist = cache.get(cache_key)
File "versions/pylibmcbug/lib/python3.9/site-
packages/django/core/cache/backends/memcached.py", line 77, in get
return self._cache.get(key, default)
pylibmc.ConnectionError: error 3 from
memcached_get(:1:views.decorators.cache.cache_): (0x7f290400bd60) FAILURE,
poll() returned a value that was not dealt with, host: localhost:11211 ->
libmemcached/io.cc:254
}}}
Looking for git history, it is this way since 2010.
https://github.com/django/django/commit/673e6fc7fb243ed44841b9969d26a161c25733b3
--
Ticket URL: <https://code.djangoproject.com/ticket/33252>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: Iuri de Silvio | Owner: Iuri de Silvio
Type: Uncategorized | Status: assigned
Component: Uncategorized | Version: 3.2
Severity: Normal | Resolution:
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Iuri de Silvio):
* owner: nobody => Iuri de Silvio
* status: new => assigned
Comment:
PR https://github.com/django/django/pull/15044
--
Ticket URL: <https://code.djangoproject.com/ticket/33252#comment:1>
Django
Reporter: Iuri de Silvio | Owner: Iuri de Silvio
Type: Uncategorized | Status: assigned
Severity: Normal | Resolution:
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Iuri de Silvio):
Possibly related to #33092.
--
Ticket URL: <https://code.djangoproject.com/ticket/33252#comment:2>
Django
Reporter: Iuri de Silvio | Owner: Iuri de Silvio
Type: Uncategorized | Status: assigned
Severity: Normal | Resolution:
Keywords: | Triage Stage: Unreviewed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Iuri de Silvio):
* has_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/33252#comment:3>
Django
Reporter: Iuri de Silvio | Owner: Iuri de
| Silvio
Type: Uncategorized | Status: assigned
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Iuri de Silvio):
* component: Uncategorized => Core (Cache system)
--
Ticket URL: <https://code.djangoproject.com/ticket/33252#comment:4>
Django
-------------------------------------+-------------------------------------
Reporter: Iuri de Silvio | Owner: Iuri de
| Silvio
Type: Bug | Status: assigned
Component: Core (Cache system) | Version: 3.2
Severity: Normal | Resolution:
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Mariusz Felisiak):
* cc: Nick Pope (added)
* type: Uncategorized => Bug
* stage: Unreviewed => Accepted
Comment:
Thanks for the report!
--
Ticket URL: <https://code.djangoproject.com/ticket/33252#comment:5>
Django
-------------------------------------+-------------------------------------
Reporter: Iuri de Silvio | Owner: Iuri de
| Silvio
Type: Bug | Status: assigned
Component: Core (Cache system) | Version: 3.2
Severity: Normal | Resolution:
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Mariusz Felisiak):
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/33252#comment:6>
Django
-------------------------------------+-------------------------------------
Reporter: Iuri de Silvio | Owner: Iuri de
| Silvio
Component: Core (Cache system) | Version: 3.2
Keywords: | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Mariusz Felisiak <felisiak.mariusz@…>):
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"3ff7b15bb79f2ee5b7af245c55ae14546243bb77" 3ff7b15]:
{{{
#!CommitTicketReference repository=""
revision="3ff7b15bb79f2ee5b7af245c55ae14546243bb77"
Fixed #33252 -- Made cache middlewares thread-safe.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/33252#comment:8>
Django
-------------------------------------+-------------------------------------
Reporter: Iuri de Silvio | Owner: Iuri de
| Silvio
Component: Core (Cache system) | Version: 3.2
Severity: Normal | Resolution:
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Mariusz Felisiak <felisiak.mariusz@…>):
In [changeset:"0c05c183e414fcf83763ca83f97de873e613c162" 0c05c18]:
{{{
#!CommitTicketReference repository=""
revision="0c05c183e414fcf83763ca83f97de873e613c162"
Refs #33252 -- Used @override_settings in
BaseCacheTests.test_cache_write_unpicklable_object()
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/33252#comment:7>