[Django] #34747: Django hangs on async views with asycio.gather and an async ORM call
Django
-------------------------------------+-------------------------------------
Reporter: rasca | Owner: nobody
Type: Bug | Status: new
Component: Database | Version: 4.2
layer (models, ORM) | Keywords: async
Severity: Normal | asyncio.gather
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
This simple view:
{{{
import asyncio
from django.http import HttpResponse
from myapp.models import MyModel
async def test_hang(request):
await asyncio.gather(MyModel.objects.acreate())
return HttpResponse('OK')
}}}
Hangs when called from daphne or uvicorn. When called from the django
shell (with ipdb) like `await test_hang(None)` it works with no problem.
I created this minimal view to reproduce the issue, but in my project I'm
creating a lot of tasks that have long external API requests saving the
results to the db. Notice that with only one task it already hangs.
I haven't found anything in the async documentation / channels / daphne
stating that I cannot do something like this.
When setting a trace with ipdb I found that the line that gets stuck is:
{{{
>
/usr/local/Cellar/pyt...@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/selectors.py:561
kev_list = self._selector.control(None, max_ev, timeout)
}}}
And when running with python -m ipdb and then ctrl c it gets stuck in this
line:
{{{
File
"/usr/local/Cellar/pyt...@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/threading.py",
line 1132, in _wait_for_tstate_lock
if lock.acquire(block, timeout):
}}}
Shouldn't this simple view be working? If not, we should document it
somehow. Any pointers much appreciated!
--
Ticket URL: <https://code.djangoproject.com/ticket/34747>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
-------------------------------------+-------------------------------------
Reporter: rasca | Owner: nobody
Type: Bug | Status: new
(models, ORM) |
Severity: Normal | Resolution:
Keywords: async | Triage Stage:
asyncio.gather | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Old description:
New description:
This simple view:
{{{
import asyncio
from django.http import HttpResponse
from myapp.models import MyModel
async def test_hang(request):
await asyncio.gather(MyModel.objects.acreate())
return HttpResponse('OK')
}}}
Hangs when called from daphne or uvicorn. When called from the django
shell (with ipdb) like `await test_hang(None)` it works with no problem.
I created this minimal view to reproduce the issue, but in my project I'm
creating a lot of tasks that have long external API requests saving the
results to the db. Notice that with only one task it already hangs.
I haven't found anything in the async documentation / channels / daphne
stating that I cannot do something like this.
When setting a trace with ipdb I found that the line that gets stuck is:
{{{
>
/usr/local/Cellar/pyt...@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/selectors.py:561
kev_list = self._selector.control(None, max_ev, timeout)
}}}
And when running with python -m ipdb and then ctrl c it gets stuck in this
line:
{{{
File
"/usr/local/Cellar/pyt...@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/threading.py",
line 1132, in _wait_for_tstate_lock
if lock.acquire(block, timeout):
}}}
Tried with Python 3.11.4 in OS X and in docker (alpine).
Shouldn't this simple view be working? If not, we should document it
somehow. Any pointers much appreciated!
--
--
Ticket URL: <https://code.djangoproject.com/ticket/34747#comment:1>
Django
-------------------------------------+-------------------------------------
Reporter: rasca | Owner: nobody
Component: Database layer | Version: 4.2
Severity: Normal | worksforme
Keywords: async | Triage Stage:
asyncio.gather | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: new => closed
* resolution: => worksforme
Comment:
> Shouldn't this simple view be working?
Yes, and it works for me, with and without `daphne`, this may be some
misconfiguration of your server. You can try to use `asyncio.TaskGroup()`
which is [https://fly.io/django-beats/running-tasks-concurrently-in-
django-asynchronous-views/ available on Python 3.11+] and check out
TicketClosingReasons/UseSupportChannels for ways to get help.
--
Ticket URL: <https://code.djangoproject.com/ticket/34747#comment:2>
Django
-------------------------------------+-------------------------------------
Reporter: rasca | Owner: nobody
Type: Bug | Status: closed
Component: Database layer | Version: 4.2
(models, ORM) | Resolution:
Severity: Normal | worksforme
Keywords: async | Triage Stage:
asyncio.gather | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by rasca):
Thanks a lot Mariusz for taking the time to test the issue and unblocking
me!!
I've fixed it. I'm writing my results here for anyone having this problem
in the future.
I first created a bare bones project to test that the simple view worked.
After that I started removing everything from my project till it started
working.
My problem was that I had a middleware not prepared for async and that
somehow broke this usage:
My middleware was this one:
{{{
from django.http import HttpResponse
class HealthCheckMiddleware:
def __init__(self, get_response):
self.get_response = get_response
def __call__(self, request):
if request.path == '/health':
return HttpResponse('ok')
return self.get_response(request)
}}}
and I changed it to this one and now it works.
{{{
from asgiref.sync import iscoroutinefunction
from django.http import HttpResponse
from django.utils.decorators import sync_and_async_middleware
@sync_and_async_middleware
def health_check_middleware(get_response):
if iscoroutinefunction(get_response):
async def middleware(request):
if request.path == '/health':
return HttpResponse('ok')
response = await get_response(request)
return response
else:
def middleware(request):
if request.path == '/health':
return HttpResponse('ok')
response = get_response(request)
return response
return middleware
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/34747#comment:3>
Django
-------------------------------------+-------------------------------------
Reporter: rasca | Owner: nobody
Type: Bug | Status: closed
Component: Database layer | Version: 4.2
(models, ORM) |
Keywords: async | Triage Stage:
asyncio.gather | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* resolution: worksforme => fixed
Comment:
I was having the same issue, transforming the sync middlewares to async
worked for me
Replying to [comment:3 rasca]:
--
Ticket URL: <https://code.djangoproject.com/ticket/34747#comment:4>
Django
-------------------------------------+-------------------------------------
Reporter: rasca | Owner: nobody
Type: Bug | Status: closed
Component: Database layer | Version: 4.2
Severity: Normal | worksforme
Keywords: async | Triage Stage:
asyncio.gather | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* resolution: fixed => worksforme
--
Ticket URL: <https://code.djangoproject.com/ticket/34747#comment:5>