Re: [Django] #35757: memcached.PyMemcacheCache reentrancy problem with ASGI-based runserver
42 views
Skip to first unread message
Django
Sep 12, 2024, 10:20:55 AM9/12/24
to django-...@googlegroups.com
#35757: memcached.PyMemcacheCache reentrancy problem with ASGI-based runserver
-------------------------------+--------------------------------------
Reporter: Harm Verhagen | Owner: (none)
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 5.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+--------------------------------------
Description changed by Harm Verhagen:
Old description:
> I found that using `memcached.PyMemcacheCache` with runserver has
> reentrancy problems.
>
> I implemented a json api in django, when issuing multiple request
> simultaneously from javascript, the requests sometimes crash.
>
> I have urls that do something like this.
>
> {{{
> from django.core.cache import cache
>
> def example_url(request):
> cache.incr("some key", 1) # count requests
>
> return JsonResponse(data=data, status=201)
> }}}
>
> When issuing 5-10 requests simultaneously on a runserver, it often (not
> always) get this crash
>
> {{{
>
> File "..../views/api.py", line 2512, in example_url
> cache.incr("some key", 1)
> File "./venv/lib/python3.12/site-
> packages/django/core/cache/backends/memcached.py", line 110, in incr
> val = self._cache.incr(key, delta)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "./venv/lib/python3.12/site-packages/pymemcache/client/hash.py",
> line 350, in incr
> return self._run_cmd("incr", key, False, *args, **kwargs)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "./venv/lib/python3.12/site-packages/pymemcache/client/hash.py",
> line 322, in _run_cmd
> return self._safely_run_func(client, func, default_val, *args,
> **kwargs)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "./venv/lib/python3.12/site-packages/pymemcache/client/hash.py",
> line 211, in _safely_run_func
> result = func(*args, **kwargs)
> ^^^^^^^^^^^^^^^^^^^^^
> File "./venv/lib/python3.12/site-packages/pymemcache/client/base.py",
> line 827, in incr
> results = self._misc_cmd([cmd], b"incr", noreply)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "./venv/lib/python3.12/site-packages/pymemcache/client/base.py",
> line 1291, in _misc_cmd
> buf, line = _reader(self.sock, buf)
> ^^^^^^^^^^^^^^^^^^^^^^^
> File "./venv/lib/python3.12/site-packages/pymemcache/client/base.py",
> line 1658, in _readline
> buf = _recv(sock, RECV_SIZE)
> ^^^^^^^^^^^^^^^^^^^^^^
> File "./venv/lib/python3.12/site-packages/pymemcache/client/base.py",
> line 1750, in _recv
> return sock.recv(size)
>
> OSError: [Errno 9] Bad file descriptor
> }}}
>
>
> The following code looked a bit suspicious.
>
> `cache.backends.memcached.py:99 BaseMemcachedCache`
>
> {{{
> def close(self, **kwargs):
> # Many clients don't clean up connections properly.
> self._cache.disconnect_all()
> }}}
>
> Doesn't this race? closing a all connections, might close a connection
> of a parallel request that was just opened but not yet finished.
>
> == Notes
> I'm not putting a heavy load on runserver, its a simple javascript app,
> that happens to do 10 requests on a single page. Django should be able
> to handle that, also in a dev environment
>
> == Workaround
> Adding the `--noasgi` option seems to help.
> {{{
> ./manage.py runserver --noasgi
> }}}
>
> Weird enough the move obvious, `--nothreading` option does _not_ help.
> That crashes in the same way.
>
>
> == Version info
>
> {{{
> Django==5.1.1
> pymemcache==4.0.0
> python 3.12.6
> }}}
New description:
I found that using `memcached.PyMemcacheCache` with runserver has
reentrancy problems.
I implemented a json api in django, when issuing multiple request
simultaneously from javascript, the requests sometimes crash.
I have urls that do something like this.
{{{
from django.core.cache import cache
def example_url(request):
cache.incr("some key", 1) # count requests
return JsonResponse(data=data, status=201)
}}}
When issuing 5-10 requests simultaneously on a runserver, it often (not
always) get this crash
{{{
File "..../views/api.py", line 2512, in example_url
cache.incr("some key", 1)
File "./venv/lib/python3.12/site-
packages/django/core/cache/backends/memcached.py", line 110, in incr
val = self._cache.incr(key, delta)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "./venv/lib/python3.12/site-packages/pymemcache/client/hash.py",
line 350, in incr
return self._run_cmd("incr", key, False, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "./venv/lib/python3.12/site-packages/pymemcache/client/hash.py",
line 322, in _run_cmd
return self._safely_run_func(client, func, default_val, *args,
**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "./venv/lib/python3.12/site-packages/pymemcache/client/hash.py",
line 211, in _safely_run_func
result = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "./venv/lib/python3.12/site-packages/pymemcache/client/base.py",
line 827, in incr
results = self._misc_cmd([cmd], b"incr", noreply)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "./venv/lib/python3.12/site-packages/pymemcache/client/base.py",
line 1291, in _misc_cmd
buf, line = _reader(self.sock, buf)
^^^^^^^^^^^^^^^^^^^^^^^
File "./venv/lib/python3.12/site-packages/pymemcache/client/base.py",
line 1658, in _readline
buf = _recv(sock, RECV_SIZE)
^^^^^^^^^^^^^^^^^^^^^^
File "./venv/lib/python3.12/site-packages/pymemcache/client/base.py",
line 1750, in _recv
return sock.recv(size)
OSError: [Errno 9] Bad file descriptor
}}}
The following code looked a bit suspicious.
`cache.backends.memcached.py:99 BaseMemcachedCache`
{{{
def close(self, **kwargs):
# Many clients don't clean up connections properly.
self._cache.disconnect_all()
}}}
Doesn't this race? closing a all connections, might close a connection
of a parallel request that was just opened but not yet finished.
== Notes
I'm not putting a heavy load on runserver, its a simple javascript app,
that happens to do 10 requests on a single page. Django should be able
to handle that, also in a dev environment
== Workaround
Adding the `--noasgi` option seems to help.
{{{
./manage.py runserver --noasgi
}}}
Weird enough the (to me) more obvious, `--nothreading` option does _not_
help. That crashes in the same way.
== Version info
{{{
Django==5.1.1
pymemcache==4.0.0
python 3.12.6
}}}
--
--
Ticket URL: <https://code.djangoproject.com/ticket/35757#comment:5>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
-------------------------------+--------------------------------------
Reporter: Harm Verhagen | Owner: (none)
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 5.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+--------------------------------------
Description changed by Harm Verhagen:
Old description:
> I found that using `memcached.PyMemcacheCache` with runserver has
> reentrancy problems.
>
> I implemented a json api in django, when issuing multiple request
> simultaneously from javascript, the requests sometimes crash.
>
> I have urls that do something like this.
>
> {{{
> from django.core.cache import cache
>
> def example_url(request):
> cache.incr("some key", 1) # count requests
>
> return JsonResponse(data=data, status=201)
> }}}
>
> When issuing 5-10 requests simultaneously on a runserver, it often (not
> always) get this crash
>
> {{{
>
> File "..../views/api.py", line 2512, in example_url
> cache.incr("some key", 1)
> File "./venv/lib/python3.12/site-
> packages/django/core/cache/backends/memcached.py", line 110, in incr
> val = self._cache.incr(key, delta)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "./venv/lib/python3.12/site-packages/pymemcache/client/hash.py",
> line 350, in incr
> return self._run_cmd("incr", key, False, *args, **kwargs)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "./venv/lib/python3.12/site-packages/pymemcache/client/hash.py",
> line 322, in _run_cmd
> return self._safely_run_func(client, func, default_val, *args,
> **kwargs)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "./venv/lib/python3.12/site-packages/pymemcache/client/hash.py",
> line 211, in _safely_run_func
> result = func(*args, **kwargs)
> ^^^^^^^^^^^^^^^^^^^^^
> File "./venv/lib/python3.12/site-packages/pymemcache/client/base.py",
> line 827, in incr
> results = self._misc_cmd([cmd], b"incr", noreply)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "./venv/lib/python3.12/site-packages/pymemcache/client/base.py",
> line 1291, in _misc_cmd
> buf, line = _reader(self.sock, buf)
> ^^^^^^^^^^^^^^^^^^^^^^^
> File "./venv/lib/python3.12/site-packages/pymemcache/client/base.py",
> line 1658, in _readline
> buf = _recv(sock, RECV_SIZE)
> ^^^^^^^^^^^^^^^^^^^^^^
> File "./venv/lib/python3.12/site-packages/pymemcache/client/base.py",
> line 1750, in _recv
> return sock.recv(size)
>
> OSError: [Errno 9] Bad file descriptor
> }}}
>
>
> The following code looked a bit suspicious.
>
> `cache.backends.memcached.py:99 BaseMemcachedCache`
>
> {{{
> def close(self, **kwargs):
> # Many clients don't clean up connections properly.
> self._cache.disconnect_all()
> }}}
>
> Doesn't this race? closing a all connections, might close a connection
> of a parallel request that was just opened but not yet finished.
>
> == Notes
> I'm not putting a heavy load on runserver, its a simple javascript app,
> that happens to do 10 requests on a single page. Django should be able
> to handle that, also in a dev environment
>
> == Workaround
> Adding the `--noasgi` option seems to help.
> {{{
> ./manage.py runserver --noasgi
> }}}
>
> Weird enough the move obvious, `--nothreading` option does _not_ help.
> That crashes in the same way.
>
>
> == Version info
>
> {{{
> Django==5.1.1
> pymemcache==4.0.0
> python 3.12.6
> }}}
New description:
I found that using `memcached.PyMemcacheCache` with runserver has
reentrancy problems.
I implemented a json api in django, when issuing multiple request
simultaneously from javascript, the requests sometimes crash.
I have urls that do something like this.
{{{
from django.core.cache import cache
def example_url(request):
cache.incr("some key", 1) # count requests
return JsonResponse(data=data, status=201)
}}}
When issuing 5-10 requests simultaneously on a runserver, it often (not
always) get this crash
{{{
File "..../views/api.py", line 2512, in example_url
cache.incr("some key", 1)
File "./venv/lib/python3.12/site-
packages/django/core/cache/backends/memcached.py", line 110, in incr
val = self._cache.incr(key, delta)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "./venv/lib/python3.12/site-packages/pymemcache/client/hash.py",
line 350, in incr
return self._run_cmd("incr", key, False, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "./venv/lib/python3.12/site-packages/pymemcache/client/hash.py",
line 322, in _run_cmd
return self._safely_run_func(client, func, default_val, *args,
**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "./venv/lib/python3.12/site-packages/pymemcache/client/hash.py",
line 211, in _safely_run_func
result = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "./venv/lib/python3.12/site-packages/pymemcache/client/base.py",
line 827, in incr
results = self._misc_cmd([cmd], b"incr", noreply)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "./venv/lib/python3.12/site-packages/pymemcache/client/base.py",
line 1291, in _misc_cmd
buf, line = _reader(self.sock, buf)
^^^^^^^^^^^^^^^^^^^^^^^
File "./venv/lib/python3.12/site-packages/pymemcache/client/base.py",
line 1658, in _readline
buf = _recv(sock, RECV_SIZE)
^^^^^^^^^^^^^^^^^^^^^^
File "./venv/lib/python3.12/site-packages/pymemcache/client/base.py",
line 1750, in _recv
return sock.recv(size)
OSError: [Errno 9] Bad file descriptor
}}}
The following code looked a bit suspicious.
`cache.backends.memcached.py:99 BaseMemcachedCache`
{{{
def close(self, **kwargs):
# Many clients don't clean up connections properly.
self._cache.disconnect_all()
}}}
Doesn't this race? closing a all connections, might close a connection
of a parallel request that was just opened but not yet finished.
== Notes
I'm not putting a heavy load on runserver, its a simple javascript app,
that happens to do 10 requests on a single page. Django should be able
to handle that, also in a dev environment
== Workaround
Adding the `--noasgi` option seems to help.
{{{
./manage.py runserver --noasgi
}}}
Weird enough the (to me) more obvious, `--nothreading` option does _not_
help. That crashes in the same way.
== Version info
{{{
Django==5.1.1
pymemcache==4.0.0
python 3.12.6
}}}
--
--
Ticket URL: <https://code.djangoproject.com/ticket/35757#comment:5>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Sep 12, 2024, 10:44:19 AM9/12/24
to django-...@googlegroups.com
your app, then this `workaround` is not usable.
== Version info
{{{
Django==5.1.1
pymemcache==4.0.0
python 3.12.6
}}}
--
--
Django
Sep 12, 2024, 10:44:40 AM9/12/24
to django-...@googlegroups.com
Django
Sep 12, 2024, 3:12:33 PM9/12/24
to django-...@googlegroups.com
#35757: memcached.PyMemcacheCache reentrancy problem with ASGI-based runserver
-------------------------------+--------------------------------------
Reporter: Harm Verhagen | Owner: (none)
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 5.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+--------------------------------------
Comment (by Harm Verhagen):
-------------------------------+--------------------------------------
Reporter: Harm Verhagen | Owner: (none)
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 5.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+--------------------------------------
Its not just a reentrancy problem. Wrapping the calls to `cache.incr()`
in a global mutex (`threading.Lock()`, just for testing purposes) does not
seem to make any difference.
--
Ticket URL: <https://code.djangoproject.com/ticket/35757#comment:8>
Django
Sep 12, 2024, 5:30:39 PM9/12/24
to django-...@googlegroups.com
#35757: memcached.PyMemcacheCache reentrancy problem with ASGI-based runserver
-------------------------------+--------------------------------------
Reporter: Harm Verhagen | Owner: (none)
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 5.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+--------------------------------------
-------------------------------+--------------------------------------
Reporter: Harm Verhagen | Owner: (none)
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 5.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+--------------------------------------
channels-redis==3.4.1
}}}
--
--
Ticket URL: <https://code.djangoproject.com/ticket/35757#comment:9>
Django
Sep 13, 2024, 4:55:00 AM9/13/24
to django-...@googlegroups.com
#35757: memcached.PyMemcacheCache reentrancy problem with ASGI-based runserver
-------------------------------+--------------------------------------
Reporter: Harm Verhagen | Owner: (none)
Type: Uncategorized | Status: closed
-------------------------------+--------------------------------------
Reporter: Harm Verhagen | Owner: (none)
Component: Uncategorized | Version: 5.1
Severity: Normal | Resolution: invalid
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+--------------------------------------
Changes (by Sarah Boyce):
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+--------------------------------------
* resolution: => invalid
* status: new => closed
Comment:
This report seems better suited to be a support request. The best place to
get answers to your issue is using any of the user support channels from
[https://docs.djangoproject.com/en/dev/faq/help/#how-do-i-do-x-why-
doesn-t-y-work-where-can-i-go-to-get-help this link].
Trac is for issues that are confirmed to be in Django. If, after
debugging, you find out that this is indeed a bug in Django, please re-
open with the specific details and please be sure to include a small
Django project to reproduce or a failing test case.
Thank you!
--
Ticket URL: <https://code.djangoproject.com/ticket/35757#comment:10>
Django
Sep 14, 2024, 10:26:45 AM9/14/24
to django-...@googlegroups.com
#35757: memcached.PyMemcacheCache reentrancy problem with ASGI-based runserver
-------------------------------+--------------------------------------
Reporter: Harm Verhagen | Owner: (none)
Type: Uncategorized | Status: closed
Component: Uncategorized | Version: 5.1
Severity: Normal | Resolution: invalid
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+--------------------------------------
Comment (by Harm Verhagen):
-------------------------------+--------------------------------------
Reporter: Harm Verhagen | Owner: (none)
Type: Uncategorized | Status: closed
Component: Uncategorized | Version: 5.1
Severity: Normal | Resolution: invalid
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+--------------------------------------
Sarah Boyce wrote:
> This report seems better suited to be a support request. The best place
I actually intended to submit a bug report, not a support request or to
get answer.
I'll try to cook up a small(er) test that demonstrates the problem.
--
Ticket URL: <https://code.djangoproject.com/ticket/35757#comment:11>
Django
Sep 30, 2024, 9:38:42 AM9/30/24
to django-...@googlegroups.com
#35757: memcached.PyMemcacheCache reentrancy problem with ASGI-based runserver
-------------------------------+--------------------------------------
Reporter: Harm Verhagen | Owner: (none)
Type: Uncategorized | Status: closed
Component: Uncategorized | Version: 5.1
Severity: Normal | Resolution: invalid
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+--------------------------------------
Comment (by Harm Verhagen):
I reproduced this problem with a small Django project.
-------------------------------+--------------------------------------
Reporter: Harm Verhagen | Owner: (none)
Type: Uncategorized | Status: closed
Component: Uncategorized | Version: 5.1
Severity: Normal | Resolution: invalid
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+--------------------------------------
Comment (by Harm Verhagen):
It seems to be related with channels (v3)
Please advise, if Django related (if so, could you reopen the ticket?)
I created a new project according the to polls example. The problem occurs
when adding 'channels' to `INSTALLED_APPS` (without even using channels).
polls.views.py
{{{#!python
from django.http import HttpResponse
from django.core.cache import cache
def incr():
key = 'example:key'
try:
val = cache.incr(key, 1)
except ValueError:
cache.set(key, 1)
val = 1
return val
def index(request):
v = incr()
return HttpResponse("Hello, world. You're at the polls index. %d" % v)
}}}
settings.py
{{{#!python
CACHES = {
'default': {
'BACKEND': 'django.core.cache.backends.memcached.PyMemcacheCache',
# 'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',
# using LocMemCache 'fixes' the problem
'LOCATION': "localhost:11211"
},
}
INSTALLED_APPS = [
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'channels', # adding this line shows the problem, removing
this line 'fixes' the problem
]
ASGI_APPLICATION = "mysite.asgi.application" # required for
channels
}}}
Steps to reproduce, run the following script get_parallel.sh
{{{#!bash
#!/bin/bash
#
# do requests in parallel
#https://code.djangoproject.com/ticket/35757
url='http://localhost:8000/polls/'
curl $url&
curl $url&
curl $url&
curl $url
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/35757#comment:12>
Django
Sep 30, 2024, 12:56:50 PM9/30/24
to django-...@googlegroups.com
#35757: memcached.PyMemcacheCache reentrancy problem with ASGI-based runserver
-------------------------------+--------------------------------------
Reporter: Harm Verhagen | Owner: (none)
Type: Uncategorized | Status: new
-------------------------------+--------------------------------------
Reporter: Harm Verhagen | Owner: (none)
Component: Uncategorized | Version: 5.1
Severity: Normal | Resolution:
Severity: Normal | Resolution:
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+--------------------------------------
Changes (by Harm Verhagen):
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+--------------------------------------
* resolution: invalid =>
* status: closed => new
--
Ticket URL: <https://code.djangoproject.com/ticket/35757#comment:13>
Django
Oct 1, 2024, 9:35:59 AM10/1/24
to django-...@googlegroups.com
#35757: memcached.PyMemcacheCache reentrancy problem with ASGI-based runserver
-------------------------------------+-------------------------------------
Reporter: Harm Verhagen | Owner: (none)
Type: Bug | Status: closed
Component: Core (Cache system) | Version: 5.1
Severity: Normal | Resolution: duplicate
Keywords: asgi, async | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Natalia Bidart):
* component: Uncategorized => Core (Cache system)
* keywords: => asgi, async
* resolution: => duplicate
* status: new => closed
* type: Uncategorized => Bug
Comment:
Replying to [comment:13 Harm Verhagen]:
Hello Harm! Thank you for the extra details. After some investigation, I
believe this is a duplicate of #33625, so I'll be closing as such.
--
Ticket URL: <https://code.djangoproject.com/ticket/35757#comment:14>
Django
Oct 2, 2024, 8:27:21 AM10/2/24
to django-...@googlegroups.com
#35757: memcached.PyMemcacheCache reentrancy problem with ASGI-based runserver
-------------------------------------+-------------------------------------
Reporter: Harm Verhagen | Owner: (none)
Type: Bug | Status: closed
Component: Core (Cache system) | Version: 5.1
Severity: Normal | Resolution: duplicate
Keywords: asgi, async | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Harm Verhagen):
-------------------------------------+-------------------------------------
Reporter: Harm Verhagen | Owner: (none)
Type: Bug | Status: closed
Component: Core (Cache system) | Version: 5.1
Severity: Normal | Resolution: duplicate
Keywords: asgi, async | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
I don't think its a duplicate of #33625 (although it seems related)
I reopened for reconsideration.
=== Why this is different?
#33625 is an optimization problem. All works fine, but could be optimized
for performance. The connection is opened/closed which adds (a bit of)
latency to each request. (which might be acceptable, or not depending on
your usecase)
This problem (#35757) however is about that the system actually breaks.
When running the test above. runserver stops working completely, until
restarted.
If you are developing a javascript app, that issues multiple requests
simultaniously, then runserver (in this configuration) is unusabe.
Another reason why its no duplicate. I can reproduce the problem also with
`django.core.cache.backends.memcached.PyLibMCCache` instead of
`'....memcached.PyMemcacheCache`.
PyLibMCCache already does provide connection pooling, so the optimization
requested in #33625 is not applicable for PyLibMCCache.
=== Reproducibility
Where you able to reproduce the problem with the example attached? (the
problem that runserver actually gives 500's and eventually stops working
completely)
=== Results with PyLibMCCache
NB: this might be a different problem, triggered by the same testcase
==== Result (PyLibMCCache)
When running requests in parallel, some of the requests get a 500.
{{{
Internal Server Error: /polls/
Traceback (most recent call last):
File "venv/lib/python3.12/site-packages/asgiref/sync.py", line 518, in
thread_handler
raise exc_info[1]
File "venv/lib/python3.12/site-
packages/django/core/handlers/exception.py", line 42, in inner
response = await get_response(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "venv/lib/python3.12/site-packages/asgiref/sync.py", line 518, in
thread_handler
raise exc_info[1]
File "venv/lib/python3.12/site-packages/django/core/handlers/base.py",
line 253, in _get_response_async
response = await wrapped_callback(
^^^^^^^^^^^^^^^^^^^^^^^
File "venv/lib/python3.12/site-packages/asgiref/sync.py", line 468, in
__call__
ret = await asyncio.shield(exec_coro)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/opt/homebrew/Cellar/pyt...@3.12/3.12.6/Frameworks/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/thread.py",
line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "venv/lib/python3.12/site-packages/asgiref/sync.py", line 522, in
thread_handler
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "django-35757/mysite/polls/views.py", line 22, in index
v = incr()
^^^^^^
File "/tmp/dbg/django-35757/mysite/polls/views.py", line 11, in incr
val = cache.incr(key, 1)
^^^^^^^^^^^^^^^^^^
File
"/Users/harmverhagen/Documents/symeon/intemo/sensoren/cityznz/cityzenz/api/venv/lib/python3.12
/site-packages/django/core/cache/backends/memcached.py", line 110, in incr
val = self._cache.incr(key, delta)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pylibmc.Error: 1 keys failed
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
HTTP GET /polls/ 500 [5.13, 127.0.0.1:63568]
}}}
NB: Sometimes I even noticed a python crash !
{{{
HTTP GET /polls/ 200 [0.11, 127.0.0.1:63819]
Python(51613,0x17d037000) malloc: double free for ptr 0x13980a400
Python(51613,0x17d037000) malloc: *** set a breakpoint in
malloc_error_break to debug
}}}
So, maybe the `PyLibMCCache` issue is something different
--
Ticket URL: <https://code.djangoproject.com/ticket/35757#comment:15>
Django
Oct 2, 2024, 8:28:45 AM10/2/24
to django-...@googlegroups.com
#35757: memcached.PyMemcacheCache reentrancy problem with ASGI-based runserver
-------------------------------------+-------------------------------------
Reporter: Harm Verhagen | Owner: (none)
Type: Bug | Status: new
-------------------------------------+-------------------------------------
Reporter: Harm Verhagen | Owner: (none)
Component: Core (Cache system) | Version: 5.1
Severity: Normal | Resolution:
Severity: Normal | Resolution:
Keywords: asgi, async | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Harm Verhagen):
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* resolution: duplicate =>
* status: closed => new
--
Ticket URL: <https://code.djangoproject.com/ticket/35757#comment:16>
Django
Oct 2, 2024, 8:48:39 AM10/2/24
to django-...@googlegroups.com
#35757: memcached.PyMemcacheCache reentrancy problem with ASGI-based runserver
-------------------------------------+-------------------------------------
Reporter: Harm Verhagen | Owner: (none)
Type: Bug | Status: new
Component: Core (Cache system) | Version: 5.1
Severity: Normal | Resolution:
Keywords: asgi, async | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Natalia Bidart):
-------------------------------------+-------------------------------------
Reporter: Harm Verhagen | Owner: (none)
Type: Bug | Status: new
Component: Core (Cache system) | Version: 5.1
Severity: Normal | Resolution:
Keywords: asgi, async | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Replying to [comment:16 Harm Verhagen]:
Hello Harm, thanks for the extra details. I haven't been able to reproduce
yet. While I re-try to do that, could you please confirm if this is
reproducible with a production web server? (that means, without using the
development server provided by `runserver`).
Thank you!
--
Ticket URL: <https://code.djangoproject.com/ticket/35757#comment:17>
Django
Oct 3, 2024, 2:48:19 AM10/3/24
to django-...@googlegroups.com
#35757: memcached.PyMemcacheCache reentrancy problem with ASGI-based runserver
-------------------------------------+-------------------------------------
Reporter: Harm Verhagen | Owner: (none)
Type: Bug | Status: new
Component: Core (Cache system) | Version: 5.1
Severity: Normal | Resolution:
Keywords: asgi, async | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Harm Verhagen):
-------------------------------------+-------------------------------------
Reporter: Harm Verhagen | Owner: (none)
Type: Bug | Status: new
Component: Core (Cache system) | Version: 5.1
Severity: Normal | Resolution:
Keywords: asgi, async | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
I've not seen this issue on production, only `runserver`.
We run production with
{{{daphne <ourappname>.asgi:application -b 0.0.0.0 -p 8000}}}
In a container, behind nginx
--
Ticket URL: <https://code.djangoproject.com/ticket/35757#comment:18>
Django
Oct 3, 2024, 3:37:51 AM10/3/24
to django-...@googlegroups.com
#35757: memcached.PyMemcacheCache reentrancy problem with ASGI-based runserver
-------------------------------------+-------------------------------------
Reporter: Harm Verhagen | Owner: (none)
Type: Bug | Status: new
Component: Core (Cache system) | Version: 5.1
Severity: Normal | Resolution:
Keywords: asgi, async | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Carlton Gibson):
-------------------------------------+-------------------------------------
Reporter: Harm Verhagen | Owner: (none)
Type: Bug | Status: new
Component: Core (Cache system) | Version: 5.1
Severity: Normal | Resolution:
Keywords: asgi, async | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Removing `daphne` (or `channels` in older versions) restores use of
Django’s own `runserver` command, so at that point you’re no longer
running ASGI.
I would say the comment about the race condition looks most likely, but
I’m confused by the last remark that it doesn’t occur in production.
(Daphe’s `runserver` command does very little except run `daphne`)
My memcached chops are not what they could be. (I haven’t used it in
production since before 2010 IIRC so… 🤷) Short of a clear diagnosis, it’s
hard to know what to suggest.
--
Ticket URL: <https://code.djangoproject.com/ticket/35757#comment:19>
Django
Oct 3, 2024, 7:57:51 AM10/3/24
to django-...@googlegroups.com
#35757: memcached.PyMemcacheCache reentrancy problem with ASGI-based runserver
-------------------------------------+-------------------------------------
Reporter: Harm Verhagen | Owner: (none)
Type: Bug | Status: new
Component: Core (Cache system) | Version: 5.1
Severity: Normal | Resolution:
Keywords: asgi, async | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Harm Verhagen):
-------------------------------------+-------------------------------------
Reporter: Harm Verhagen | Owner: (none)
Type: Bug | Status: new
Component: Core (Cache system) | Version: 5.1
Severity: Normal | Resolution:
Keywords: asgi, async | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
> Removing daphne (or channels in older versions) restores use of Django’s
own runserver command, so at that point you’re no longer running ASGI.
--
Ticket URL: <https://code.djangoproject.com/ticket/35757#comment:20>
Django
Oct 10, 2024, 3:04:14 PM10/10/24
to django-...@googlegroups.com
#35757: memcached.PyMemcacheCache reentrancy problem with ASGI-based runserver
-------------------------------------+-------------------------------------
Reporter: Harm Verhagen | Owner: (none)
Type: Bug | Status: closed
-------------------------------------+-------------------------------------
Reporter: Harm Verhagen | Owner: (none)
Component: Core (Cache system) | Version: 5.1
Severity: Normal | Resolution: needsinfo
Keywords: asgi, async | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Carlton Gibson):
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* resolution: => needsinfo
* status: new => closed
Comment:
> I've not seen this issue on production, only runserver.
That still doesn’t make 100% sense to me.
Can I ask you to open an issue on the Daphne repo, providing a full
reproduce project there please?
https://github.com/django/daphne
I’ll try and take a look, but short of something more concrete I can’t say
that this is an issue in Django per se.
I’m going to mark this as needsinfo in the meantime.
--
Ticket URL: <https://code.djangoproject.com/ticket/35757#comment:21>
Django
Oct 24, 2024, 8:51:38 AM10/24/24
to django-...@googlegroups.com
#35757: memcached.PyMemcacheCache reentrancy problem with ASGI-based runserver
-------------------------------------+-------------------------------------
Reporter: Harm Verhagen | Owner: (none)
Type: Bug | Status: closed
Component: Core (Cache system) | Version: 5.1
Severity: Normal | Resolution: needsinfo
Keywords: asgi, async | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Harm Verhagen):
-------------------------------------+-------------------------------------
Reporter: Harm Verhagen | Owner: (none)
Type: Bug | Status: closed
Component: Core (Cache system) | Version: 5.1
Severity: Normal | Resolution: needsinfo
Keywords: asgi, async | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Filed in daphne repo as https://github.com/django/daphne/issues/534
--
Ticket URL: <https://code.djangoproject.com/ticket/35757#comment:22>
Reply all
Reply to author
Forward
0 new messages