[Django] #36714: Async signals lose ContextVar state due to use of asyncio.gather
20 views
Skip to first unread message
Django
Nov 6, 2025, 5:54:33 AM (6 days ago) Nov 6
to django-...@googlegroups.com
#36714: Async signals lose ContextVar state due to use of asyncio.gather
-------------------------------------+-------------------------------------
Reporter: Mykhailo Havelia | Type:
| Uncategorized
Status: new | Component: HTTP
| handling
Version: dev | Severity: Normal
Keywords: asyncio, signals | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
The natural way to share global, per-request state in asyncio is through
contextvars. In Django, this is typically used via `asgiref.local.Local`.
However, Django's async signal dispatch currently uses `asyncio.gather`,
which internally creates new tasks (`asyncio.create_task`). This breaks
context propagation, since each task gets its own copy of the context. As
a result, it's impossible to set a global (context-based) variable inside
a signal handler and have it shared with other signal handlers or parts of
the same request/response cycle.
Example
{{{
from django.core import signals
from django.http import (
HttpRequest,
HttpResponse,
)
import contextvars
from django.urls import path
request_id = contextvars.ContextVar('request_id', default=None)
async def set_global_variable(*args, **kwargs):
# set global variable
request_id.set('request_id_value')
print('get value', request_id.get())
signals.request_started.connect(set_global_variable)
async def index(request: HttpRequest) -> HttpResponse:
# get global variable
print('request_id', request_id.get())
return HttpResponse(content=request_id.get())
urlpatterns = [path("", index), ]
}}}
result
{{{
get value request_id_value
request_id None
}}}
The value set inside the signal handler is lost, because the handler runs
in a separate task with its own context.
If we are talking exactly about `signals.request_started` and
`signals.request_finished`, they are typically used for setting up and
cleaning up per-request resources. With `asyncio.gather`, cleanup logic
that relies on `ContextVar` cannot work properly.
{{{
from django.core import signals
from django.http import (
HttpRequest,
HttpResponse,
)
import contextvars
from django.urls import path
db_connection = contextvars.ContextVar('db_connection', default=None)
async def get_or_create_connection():
if not db_connection.get():
db_connection.set('connection')
return db_connection.get()
async def close_connection(*args, **kwargs):
connection = db_connection.get()
if not connection:
print('cannot clean - connection does not exist')
return
print('close connection')
connection.set(None)
signals.request_finished.connect(close_connection)
async def index(request: HttpRequest) -> HttpResponse:
# create connection inside handler
connection = await get_or_create_connection()
# await get_data(connection)
return HttpResponse(content="ok")
urlpatterns = [path("", index), ]
}}}
result
{{{
cannot clean - connection does not exist
}}}
**Expected behavior**
Signal handlers should run in the same async context as the request,
preserving `ContextVar` and `asgiref.local.Local` state.
**Proposed solution**
Signal:
Dispatch async signal handlers sequentially (or via direct await) instead
of using `asyncio.gather`, so that the existing execution context is
preserved throughout the request lifecycle. Yes, this change removes
parallelism, but that shouldn’t be a major concern. The only real benefit
of running signal handlers in parallel would be for IO-bound operations -
yet in most cases, these handlers interact with the same database
connection. Since database operations aren’t truly parallel under the
hood, the performance gain from `asyncio.gather` is negligible.
ASGIHandler:
{{{
async def handle(self, scope, receive, send):
...
await signals.request_started.asend(sender=self.__class__,
scope=scope)
tasks = [
asyncio.create_task(self.listen_for_disconnect(receive)),
asyncio.create_task(process_request(request, send)),
]
...
await signals.request_finished.asend(sender=self.__class__)
}}}
Global variables created inside `process_request` are not visible to
`request_finished`, because each task runs in a separate context. We can
try using `contextvars.copy_context()` to preserve and share the same
context between tasks and signal handlers.
{{{
async def handle(self, scope, receive, send):
...
await signals.request_started.asend(sender=self.__class__,
scope=scope)
ctx = contextvars.copy_context()
tasks = [
asyncio.create_task(self.listen_for_disconnect(receive)),
asyncio.create_task(process_request(request, send), context=ctx),
]
...
await
asyncio.create_task(signals.request_finished.asend(sender=self.__class__),
context=ctx)
}}}
Here is a simple example
{{{
import asyncio
import contextvars
global_state = contextvars.ContextVar('stage', default=0)
async def inc():
value = global_state.get()
print('value: ', value)
global_state.set(value + 1)
async def main():
await asyncio.create_task(inc())
await asyncio.create_task(inc())
await asyncio.create_task(inc())
print('first: ', global_state.get())
ctx = contextvars.copy_context()
await asyncio.create_task(inc(), context=ctx)
await asyncio.create_task(inc(), context=ctx)
await asyncio.create_task(inc(), context=ctx)
print('second: ', ctx.get(global_state))
await main()
}}}
result
{{{
value: 0
value: 0
value: 0
first: 0
value: 0
value: 1
value: 2
second: 3
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/36714>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
-------------------------------------+-------------------------------------
Reporter: Mykhailo Havelia | Type:
| Uncategorized
Status: new | Component: HTTP
| handling
Version: dev | Severity: Normal
Keywords: asyncio, signals | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
The natural way to share global, per-request state in asyncio is through
contextvars. In Django, this is typically used via `asgiref.local.Local`.
However, Django's async signal dispatch currently uses `asyncio.gather`,
which internally creates new tasks (`asyncio.create_task`). This breaks
context propagation, since each task gets its own copy of the context. As
a result, it's impossible to set a global (context-based) variable inside
a signal handler and have it shared with other signal handlers or parts of
the same request/response cycle.
Example
{{{
from django.core import signals
from django.http import (
HttpRequest,
HttpResponse,
)
import contextvars
from django.urls import path
request_id = contextvars.ContextVar('request_id', default=None)
async def set_global_variable(*args, **kwargs):
# set global variable
request_id.set('request_id_value')
print('get value', request_id.get())
signals.request_started.connect(set_global_variable)
async def index(request: HttpRequest) -> HttpResponse:
# get global variable
print('request_id', request_id.get())
return HttpResponse(content=request_id.get())
urlpatterns = [path("", index), ]
}}}
result
{{{
get value request_id_value
request_id None
}}}
The value set inside the signal handler is lost, because the handler runs
in a separate task with its own context.
If we are talking exactly about `signals.request_started` and
`signals.request_finished`, they are typically used for setting up and
cleaning up per-request resources. With `asyncio.gather`, cleanup logic
that relies on `ContextVar` cannot work properly.
{{{
from django.core import signals
from django.http import (
HttpRequest,
HttpResponse,
)
import contextvars
from django.urls import path
db_connection = contextvars.ContextVar('db_connection', default=None)
async def get_or_create_connection():
if not db_connection.get():
db_connection.set('connection')
return db_connection.get()
async def close_connection(*args, **kwargs):
connection = db_connection.get()
if not connection:
print('cannot clean - connection does not exist')
return
print('close connection')
connection.set(None)
signals.request_finished.connect(close_connection)
async def index(request: HttpRequest) -> HttpResponse:
# create connection inside handler
connection = await get_or_create_connection()
# await get_data(connection)
return HttpResponse(content="ok")
urlpatterns = [path("", index), ]
}}}
result
{{{
cannot clean - connection does not exist
}}}
**Expected behavior**
Signal handlers should run in the same async context as the request,
preserving `ContextVar` and `asgiref.local.Local` state.
**Proposed solution**
Signal:
Dispatch async signal handlers sequentially (or via direct await) instead
of using `asyncio.gather`, so that the existing execution context is
preserved throughout the request lifecycle. Yes, this change removes
parallelism, but that shouldn’t be a major concern. The only real benefit
of running signal handlers in parallel would be for IO-bound operations -
yet in most cases, these handlers interact with the same database
connection. Since database operations aren’t truly parallel under the
hood, the performance gain from `asyncio.gather` is negligible.
ASGIHandler:
{{{
async def handle(self, scope, receive, send):
...
await signals.request_started.asend(sender=self.__class__,
scope=scope)
tasks = [
asyncio.create_task(self.listen_for_disconnect(receive)),
asyncio.create_task(process_request(request, send)),
]
...
await signals.request_finished.asend(sender=self.__class__)
}}}
Global variables created inside `process_request` are not visible to
`request_finished`, because each task runs in a separate context. We can
try using `contextvars.copy_context()` to preserve and share the same
context between tasks and signal handlers.
{{{
async def handle(self, scope, receive, send):
...
await signals.request_started.asend(sender=self.__class__,
scope=scope)
ctx = contextvars.copy_context()
tasks = [
asyncio.create_task(self.listen_for_disconnect(receive)),
asyncio.create_task(process_request(request, send), context=ctx),
]
...
await
asyncio.create_task(signals.request_finished.asend(sender=self.__class__),
context=ctx)
}}}
Here is a simple example
{{{
import asyncio
import contextvars
global_state = contextvars.ContextVar('stage', default=0)
async def inc():
value = global_state.get()
print('value: ', value)
global_state.set(value + 1)
async def main():
await asyncio.create_task(inc())
await asyncio.create_task(inc())
await asyncio.create_task(inc())
print('first: ', global_state.get())
ctx = contextvars.copy_context()
await asyncio.create_task(inc(), context=ctx)
await asyncio.create_task(inc(), context=ctx)
await asyncio.create_task(inc(), context=ctx)
print('second: ', ctx.get(global_state))
await main()
}}}
result
{{{
value: 0
value: 0
value: 0
first: 0
value: 0
value: 1
value: 2
second: 3
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/36714>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Nov 6, 2025, 6:19:31 AM (6 days ago) Nov 6
to django-...@googlegroups.com
#36714: Async signals lose ContextVar state due to use of asyncio.gather
----------------------------------+------------------------------------
Reporter: Mykhailo Havelia | Owner: (none)
Type: Uncategorized | Status: new
Component: HTTP handling | Version: dev
Severity: Normal | Resolution:
Component: HTTP handling | Version: dev
Keywords: asyncio, signals | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Carlton Gibson):
* cc: Carlton Gibson (added)
* stage: Unreviewed => Accepted
Comment:
Hi Mykhailo.
Thanks for the report. (And thanks for writing it up so nicely.)
> We can try using contextvars.copy_context() to preserve and share the
same context between tasks and signal handlers.
thought. The whole idea of that API is to allow control over this, when
trying to keep things ''structured'', no?
Then we should resolve #36315 first no? TaskGroup is the newer preferred
API. Do you fancy reviewing that?
I'd be a bit sad if we had to lose the concurrent async dispatch on
signals. (I grant your point that it may not be a big gain in performance
necessarily, but ... 🙂 )
--
Ticket URL: <https://code.djangoproject.com/ticket/36714#comment:1>
Django
Nov 6, 2025, 7:50:21 AM (6 days ago) Nov 6
to django-...@googlegroups.com
#36714: Async signals lose ContextVar state due to use of asyncio.gather
-------------------------------------+-------------------------------------
Reporter: Mykhailo Havelia | Owner: Varun
| Kasyap Pentamaraju
Type: Uncategorized | Status: assigned
Component: HTTP handling | Version: dev
Severity: Normal | Resolution:
Keywords: asyncio, signals | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Severity: Normal | Resolution:
Keywords: asyncio, signals | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Varun Kasyap Pentamaraju):
* owner: (none) => Varun Kasyap Pentamaraju
* status: new => assigned
--
Ticket URL: <https://code.djangoproject.com/ticket/36714#comment:2>
Django
Nov 6, 2025, 9:54:42 AM (6 days ago) Nov 6
to django-...@googlegroups.com
#36714: Async signals lose ContextVar state due to use of asyncio.gather
----------------------------------+------------------------------------
Reporter: Mykhailo Havelia | Owner: (none)
Type: Uncategorized | Status: new
Reporter: Mykhailo Havelia | Owner: (none)
Type: Uncategorized | Status: new
Component: HTTP handling | Version: dev
Severity: Normal | Resolution:
Keywords: asyncio, signals | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Severity: Normal | Resolution:
Keywords: asyncio, signals | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Varun Kasyap Pentamaraju):
* owner: Varun Kasyap Pentamaraju => (none)
* status: assigned => new
--
Ticket URL: <https://code.djangoproject.com/ticket/36714#comment:3>
Django
Nov 6, 2025, 10:05:30 AM (6 days ago) Nov 6
to django-...@googlegroups.com
#36714: Async signals lose ContextVar state due to use of asyncio.gather
----------------------------------+------------------------------------
Reporter: Mykhailo Havelia | Owner: (none)
Type: Uncategorized | Status: new
Component: HTTP handling | Version: dev
Severity: Normal | Resolution:
Keywords: asyncio, signals | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Comment (by Mykhailo Havelia):
----------------------------------+------------------------------------
Reporter: Mykhailo Havelia | Owner: (none)
Type: Uncategorized | Status: new
Component: HTTP handling | Version: dev
Severity: Normal | Resolution:
Keywords: asyncio, signals | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Replying to [comment:1 Carlton Gibson]:
> The whole idea of that API is to allow control over this, when trying to
🙂 If we’re talking about `ASGIHandler.handler`, then yes. It’s easy to
implement and would unblock my current work on the async backend.
> Then we should resolve #36315 first no?
- database cursor
- ORM
- middlewares,
- signals
- cache
to ensure proper async handling first. After that, we can focus on
optimization and improvements since optimization often adds complexity,
and there are still some core challenges to solve in the async port 😔.
That said, if I can prepare a small MR with minimal changes that could be
reviewed and merged quickly, I’d really prefer that approach, especially
since `#36315` has been pending for over six months. It would help unblock
my current work.
If that’s not possible, I'm more than happy to contribute to `#36315` to
help move it forward and unblock these changes together.
What do you think?
> The whole idea of that API is to allow control over this, when trying to
If we're talking about using `asyncio.gather` inside signals, things get a
bit more complicated. I'm not entirely sure how context sharing behaves
with parallel tasks yet (I'll need to look into it). We might need to do
something similar to what `asgiref` does, for example:
{{{
def _restore_context(context: contextvars.Context) -> None:
cvalue = context.get(cvar)
try:
if cvar.get() != cvalue:
cvar.set(cvalue)
except LookupError:
cvar.set(cvalue)
}}}
Manually handling context like this can easily lead to unexpected
behavior. For instance, overwriting a global variable when we shouldn't.
So we'll need to test this carefully, especially with `sync_to_async` and
`async_to_sync`. It might take a fair bit of time to review and verify
that everything works as expected.
It's much easier to delete it. What do you think?
--
Ticket URL: <https://code.djangoproject.com/ticket/36714#comment:4>
Django
Nov 6, 2025, 10:16:47 AM (6 days ago) Nov 6
to django-...@googlegroups.com
#36714: Async signals lose ContextVar state due to use of asyncio.gather
----------------------------------+------------------------------------
Reporter: Mykhailo Havelia | Owner: (none)
Type: Uncategorized | Status: new
Component: HTTP handling | Version: dev
Severity: Normal | Resolution:
Keywords: asyncio, signals | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Comment (by Carlton Gibson):
----------------------------------+------------------------------------
Reporter: Mykhailo Havelia | Owner: (none)
Type: Uncategorized | Status: new
Component: HTTP handling | Version: dev
Severity: Normal | Resolution:
Keywords: asyncio, signals | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
There's already two small PRs for #36315. They look good, and just need a
review, and confirmation. (They're sat on my list. 🤹) — My point was that
we should resolve those and then implement the change here on top of that,
rather than making a separate change here, and then having to redo it.
--
Ticket URL: <https://code.djangoproject.com/ticket/36714#comment:5>
Django
Nov 6, 2025, 10:42:20 AM (6 days ago) Nov 6
to django-...@googlegroups.com
#36714: Async signals lose ContextVar state due to use of asyncio.gather
----------------------------------+------------------------------------
Reporter: Mykhailo Havelia | Owner: (none)
Type: Uncategorized | Status: new
Component: HTTP handling | Version: dev
Severity: Normal | Resolution:
Keywords: asyncio, signals | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Comment (by Mykhailo Havelia):
----------------------------------+------------------------------------
Reporter: Mykhailo Havelia | Owner: (none)
Type: Uncategorized | Status: new
Component: HTTP handling | Version: dev
Severity: Normal | Resolution:
Keywords: asyncio, signals | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Replying to [comment:5 Carlton Gibson]:
> There's already two small PRs for #36315. They look good, and just need
a review, and confirmation. (They're sat on my list. 🤹) — My point was
that we should resolve those and then implement the change here on top of
that, rather than making a separate change here, and then having to redo
it.
Got it. I’ll take a look at them tonight and leave a review. Hopefully,
a review, and confirmation. (They're sat on my list. 🤹) — My point was
that we should resolve those and then implement the change here on top of
that, rather than making a separate change here, and then having to redo
it.
that helps speed up the process.
--
Ticket URL: <https://code.djangoproject.com/ticket/36714#comment:6>
Django
Nov 6, 2025, 4:26:57 PM (6 days ago) Nov 6
to django-...@googlegroups.com
#36714: Async signals lose ContextVar state due to use of asyncio.gather
----------------------------------+------------------------------------
Reporter: Mykhailo Havelia | Owner: (none)
Type: Bug | Status: new
----------------------------------+------------------------------------
Reporter: Mykhailo Havelia | Owner: (none)
Component: HTTP handling | Version: dev
Severity: Normal | Resolution:
Keywords: asyncio, signals | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Changes (by Jacob Walls):
Severity: Normal | Resolution:
Keywords: asyncio, signals | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
* type: Uncategorized => Bug
--
Ticket URL: <https://code.djangoproject.com/ticket/36714#comment:7>
Django
Nov 7, 2025, 5:51:02 AM (5 days ago) Nov 7
to django-...@googlegroups.com
#36714: Async signals lose ContextVar state due to use of asyncio.gather
----------------------------------+------------------------------------
Reporter: Mykhailo Havelia | Owner: (none)
Type: Bug | Status: new
Component: HTTP handling | Version: dev
Severity: Normal | Resolution:
Keywords: asyncio, signals | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Comment (by Carlton Gibson):
----------------------------------+------------------------------------
Reporter: Mykhailo Havelia | Owner: (none)
Type: Bug | Status: new
Component: HTTP handling | Version: dev
Severity: Normal | Resolution:
Keywords: asyncio, signals | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
I reviewed #36315 and marked it as Ready for checkin, so hopefully that
can progress to clear the path for work here.
--
Ticket URL: <https://code.djangoproject.com/ticket/36714#comment:8>
Django
Nov 9, 2025, 6:31:40 PM (3 days ago) Nov 9
to django-...@googlegroups.com
#36714: Async signals lose ContextVar state due to use of asyncio.gather
----------------------------------+------------------------------------
Reporter: Mykhailo Havelia | Owner: (none)
Type: Bug | Status: new
Component: HTTP handling | Version: dev
Severity: Normal | Resolution:
Keywords: asyncio, signals | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Comment (by Mykhailo Havelia):
----------------------------------+------------------------------------
Reporter: Mykhailo Havelia | Owner: (none)
Type: Bug | Status: new
Component: HTTP handling | Version: dev
Severity: Normal | Resolution:
Keywords: asyncio, signals | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Replying to [comment:8 Carlton Gibson]:
> I reviewed #36315 and marked it as Ready for checkin, so hopefully that
can progress to clear the path for work here.
>
Test MR: https://github.com/Arfey/django/pull/3/files
can progress to clear the path for work here.
>
I used the latest changes and prepared a fix for context sharing. At this
point, we have:
- Successful context sharing between async handlers for asend/send
- Successful context sharing between sync handlers for asend/send
- Successful cross-sharing between async/sync handlers for send
Current issue:
- Cross-sharing between async/sync handlers for asend
The difference is in how send and asend execute handlers. `send` runs sync
handlers first, and then runs async handlers in parallel and this works
fine. `asend` tries to run all handlers "in parallel". In this case,
`sync_to_async` copies the context instead of using the existing one, so
context sharing breaks. I propose splitting the execution of sync and
async handlers for asend as well.
Current code:
{{{
await _gather(
sync_send(),
_gather(
*(
receiver(signal=self, sender=sender, **named)
for receiver in async_receivers
)
),
)
}}}
Proposed adjustment:
{{{
await sync_send()
await _gather(
*(
receiver(signal=self, sender=sender, **named)
for receiver in async_receivers
)
)
}}}
What do you think?
--
Ticket URL: <https://code.djangoproject.com/ticket/36714#comment:9>
Django
Nov 10, 2025, 4:09:57 AM (2 days ago) Nov 10
to django-...@googlegroups.com
#36714: Async signals lose ContextVar state due to use of asyncio.gather
----------------------------------+------------------------------------
Reporter: Mykhailo Havelia | Owner: (none)
Type: Bug | Status: new
Component: HTTP handling | Version: dev
Severity: Normal | Resolution:
Keywords: asyncio, signals | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Comment (by Carlton Gibson):
----------------------------------+------------------------------------
Reporter: Mykhailo Havelia | Owner: (none)
Type: Bug | Status: new
Component: HTTP handling | Version: dev
Severity: Normal | Resolution:
Keywords: asyncio, signals | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
I continue to think that it would be a shame to not at least try to
maintain the concurrent signal dispatch here. We should be able to pass
the current context to the sync_send task, using the context parameter,
AFAICS. I'd like us to at least try that. If there's some blocking reason
why that's not feasible then (sure) we can look at running the sync
signals first.
--
Ticket URL: <https://code.djangoproject.com/ticket/36714#comment:10>
Django
Nov 10, 2025, 5:29:30 AM (2 days ago) Nov 10
to django-...@googlegroups.com
#36714: Async signals lose ContextVar state due to use of asyncio.gather
----------------------------------+------------------------------------
Reporter: Mykhailo Havelia | Owner: (none)
Type: Bug | Status: new
Component: HTTP handling | Version: dev
Severity: Normal | Resolution:
Keywords: asyncio, signals | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Comment (by Mykhailo Havelia):
----------------------------------+------------------------------------
Reporter: Mykhailo Havelia | Owner: (none)
Type: Bug | Status: new
Component: HTTP handling | Version: dev
Severity: Normal | Resolution:
Keywords: asyncio, signals | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Replying to [comment:10 Carlton Gibson]:
> I continue to think that it would be a shame to not at least try to
maintain the concurrent signal dispatch here. We should be able to pass
the current context to the sync_send task, using the context parameter,
AFAICS. I'd like us to at least try that. If there's some blocking reason
why that's not feasible then (sure) we can look at running the sync
signals first.
This would require changes to `sync_to_async`. It might look something
maintain the concurrent signal dispatch here. We should be able to pass
the current context to the sync_send task, using the context parameter,
AFAICS. I'd like us to at least try that. If there's some blocking reason
why that's not feasible then (sure) we can look at running the sync
signals first.
like:
{{{
context = contextvars.copy_context()
@sync_to_async(context=ctx)
def foo():
...
}}}
If this approach is acceptable, I can prepare an MR.
--
Ticket URL: <https://code.djangoproject.com/ticket/36714#comment:11>
Django
Nov 10, 2025, 5:43:53 AM (2 days ago) Nov 10
to django-...@googlegroups.com
#36714: Async signals lose ContextVar state due to use of asyncio.gather
----------------------------------+------------------------------------
Reporter: Mykhailo Havelia | Owner: (none)
Type: Bug | Status: new
Component: HTTP handling | Version: dev
Severity: Normal | Resolution:
Keywords: asyncio, signals | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Comment (by Carlton Gibson):
----------------------------------+------------------------------------
Reporter: Mykhailo Havelia | Owner: (none)
Type: Bug | Status: new
Component: HTTP handling | Version: dev
Severity: Normal | Resolution:
Keywords: asyncio, signals | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
sync_to_async already propagates the parent context.
[https://github.com/django/asgiref/blob/2138f0317d79cedd065571447ae0a7571989550e/tests/test_sync_contextvars.py#L37-L55
See tests]. Is it not the case that the `asyncio.TaskGroup()` call in
`_gather` needs to take the context parameter? 🤔
I think this is going to be easier to look at in a PR than a series of
comments on the issue here.
--
Ticket URL: <https://code.djangoproject.com/ticket/36714#comment:12>
Django
Nov 10, 2025, 6:20:26 AM (2 days ago) Nov 10
to django-...@googlegroups.com
#36714: Async signals lose ContextVar state due to use of asyncio.gather
----------------------------------+------------------------------------
Reporter: Mykhailo Havelia | Owner: (none)
Type: Bug | Status: new
Component: HTTP handling | Version: dev
Severity: Normal | Resolution:
Keywords: asyncio, signals | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Comment (by Mykhailo Havelia):
----------------------------------+------------------------------------
Reporter: Mykhailo Havelia | Owner: (none)
Type: Bug | Status: new
Component: HTTP handling | Version: dev
Severity: Normal | Resolution:
Keywords: asyncio, signals | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Replying to [comment:12 Carlton Gibson]:
> sync_to_async already propagates the parent context.
[https://github.com/django/asgiref/blob/2138f0317d79cedd065571447ae0a7571989550e/tests/test_sync_contextvars.py#L37-L55
See tests]. Is it not the case that the `asyncio.TaskGroup()` call in
`_gather` needs to take the context parameter? 🤔
>
> I think this is going to be easier to look at in a PR than a series of
comments on the issue here.
I’ve already attached the link to the MR above:
[https://github.com/django/asgiref/blob/2138f0317d79cedd065571447ae0a7571989550e/tests/test_sync_contextvars.py#L37-L55
See tests]. Is it not the case that the `asyncio.TaskGroup()` call in
`_gather` needs to take the context parameter? 🤔
>
> I think this is going to be easier to look at in a PR than a series of
comments on the issue here.
https://github.com/Arfey/django/pull/3/files. Did you have a chance to
look at it? 🙂
`sync_to_async` does support modifying contextvars, but what we need is
the ability to share the same context between tasks running in parallel.
There’s a test included in my MR that demonstrates exactly what behavior
we have to achieve
{{{
async def test_asend_correct_contextvars_sharing_mix_receivers(self):
handler1 = self.CtxSyncHandler(self.ctx_var)
handler2 = self.CtxAsyncHandler(self.ctx_var)
signal = dispatch.Signal()
signal.connect(handler1)
signal.connect(handler2)
# set custom value outer signal
self.ctx_var.set(1)
await signal.asend(self.__class__)
self.assertEqual(len(handler1.values), 1)
self.assertEqual(len(handler2.values), 1)
self.assertEqual(
sorted([*handler1.values, *handler2.values]),
[2, 3]
)
self.assertEqual(self.ctx_var.get(), 3)
}}}
Right now we don't end up with all 3. We only get 2, because `handler1`
and `handler2` don't share context with each other. The reason is that
`sync_to_async` copies the context (see:
https://github.com/django/asgiref/blob/2138f0317d79cedd065571447ae0a7571989550e/asgiref/sync.py#L483),
but in our case we need them to share it.
--
Ticket URL: <https://code.djangoproject.com/ticket/36714#comment:13>
Django
Nov 10, 2025, 6:37:29 AM (2 days ago) Nov 10
to django-...@googlegroups.com
#36714: Async signals lose ContextVar state due to use of asyncio.gather
----------------------------------+------------------------------------
Reporter: Mykhailo Havelia | Owner: (none)
Type: Bug | Status: new
Component: HTTP handling | Version: dev
Severity: Normal | Resolution:
Keywords: asyncio, signals | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Comment (by Carlton Gibson):
----------------------------------+------------------------------------
Reporter: Mykhailo Havelia | Owner: (none)
Type: Bug | Status: new
Component: HTTP handling | Version: dev
Severity: Normal | Resolution:
Keywords: asyncio, signals | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
OK, so first step is an issue on asgiref showing the necessary adjust
there, yes?
--
Ticket URL: <https://code.djangoproject.com/ticket/36714#comment:14>
Django
Nov 10, 2025, 6:39:29 AM (2 days ago) Nov 10
to django-...@googlegroups.com
#36714: Async signals lose ContextVar state due to use of asyncio.gather
----------------------------------+------------------------------------
Reporter: Mykhailo Havelia | Owner: (none)
Type: Bug | Status: new
Component: HTTP handling | Version: dev
Severity: Normal | Resolution:
Keywords: asyncio, signals | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Comment (by Mykhailo Havelia):
----------------------------------+------------------------------------
Reporter: Mykhailo Havelia | Owner: (none)
Type: Bug | Status: new
Component: HTTP handling | Version: dev
Severity: Normal | Resolution:
Keywords: asyncio, signals | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Replying to [comment:14 Carlton Gibson]:
> OK, so first step is an issue on asgiref showing the necessary adjust
there, yes?
If we want to try keep async running then yes.
there, yes?
--
Ticket URL: <https://code.djangoproject.com/ticket/36714#comment:15>
Django
Nov 10, 2025, 6:58:38 AM (2 days ago) Nov 10
to django-...@googlegroups.com
#36714: Async signals lose ContextVar state due to use of asyncio.gather
----------------------------------+------------------------------------
Reporter: Mykhailo Havelia | Owner: (none)
Type: Bug | Status: new
Component: HTTP handling | Version: dev
Severity: Normal | Resolution:
Keywords: asyncio, signals | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Comment (by Carlton Gibson):
----------------------------------+------------------------------------
Reporter: Mykhailo Havelia | Owner: (none)
Type: Bug | Status: new
Component: HTTP handling | Version: dev
Severity: Normal | Resolution:
Keywords: asyncio, signals | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Being able to share the context between multiple concurrent sync_to_async
tasks seems like something that's generally useful, and in-line with the
create_task and TaskGroup API. So, that's a useful step independently I'd
think. Thanks!
--
Ticket URL: <https://code.djangoproject.com/ticket/36714#comment:16>
Django
Nov 10, 2025, 7:01:16 AM (2 days ago) Nov 10
to django-...@googlegroups.com
#36714: Async signals lose ContextVar state due to use of asyncio.gather
----------------------------------+------------------------------------
Reporter: Mykhailo Havelia | Owner: (none)
Type: Bug | Status: new
Component: HTTP handling | Version: dev
Severity: Normal | Resolution:
Keywords: asyncio, signals | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Comment (by Mykhailo Havelia):
----------------------------------+------------------------------------
Reporter: Mykhailo Havelia | Owner: (none)
Type: Bug | Status: new
Component: HTTP handling | Version: dev
Severity: Normal | Resolution:
Keywords: asyncio, signals | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Replying to [comment:16 Carlton Gibson]:
> Being able to share the context between multiple concurrent
sync_to_async tasks seems like something that's generally useful, and in-
line with the create_task and TaskGroup API. So, that's a useful step
independently I'd think. Thanks!
Got it. I'll prepare the MR soon.
sync_to_async tasks seems like something that's generally useful, and in-
line with the create_task and TaskGroup API. So, that's a useful step
independently I'd think. Thanks!
--
Ticket URL: <https://code.djangoproject.com/ticket/36714#comment:17>
Django
Nov 10, 2025, 6:41:48 PM (2 days ago) Nov 10
to django-...@googlegroups.com
#36714: Async signals lose ContextVar state due to use of asyncio.gather
----------------------------------+------------------------------------
Reporter: Mykhailo Havelia | Owner: (none)
Type: Bug | Status: new
Component: HTTP handling | Version: dev
Severity: Normal | Resolution:
Keywords: asyncio, signals | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Comment (by Mykhailo Havelia):
Replying to [comment:16 Carlton Gibson]:
> Being able to share the context between multiple concurrent
sync_to_async tasks seems like something that's generally useful, and in-
line with the create_task and TaskGroup API. So, that's a useful step
independently I'd think. Thanks!
Done https://github.com/django/asgiref/pull/536 😌
----------------------------------+------------------------------------
Reporter: Mykhailo Havelia | Owner: (none)
Type: Bug | Status: new
Component: HTTP handling | Version: dev
Severity: Normal | Resolution:
Keywords: asyncio, signals | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Comment (by Mykhailo Havelia):
Replying to [comment:16 Carlton Gibson]:
> Being able to share the context between multiple concurrent
sync_to_async tasks seems like something that's generally useful, and in-
line with the create_task and TaskGroup API. So, that's a useful step
independently I'd think. Thanks!
--
Ticket URL: <https://code.djangoproject.com/ticket/36714#comment:18>
Django
Nov 11, 2025, 11:48:56 AM (19 hours ago) Nov 11
to django-...@googlegroups.com
#36714: Async signals lose ContextVar state due to use of asyncio.gather
----------------------------------+------------------------------------
Reporter: Mykhailo Havelia | Owner: (none)
Type: Bug | Status: new
Component: HTTP handling | Version: dev
Severity: Normal | Resolution:
Keywords: asyncio, signals | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Comment (by Mykhailo Havelia):
Replying to [comment:18 Mykhailo Havelia]:
----------------------------------+------------------------------------
Reporter: Mykhailo Havelia | Owner: (none)
Type: Bug | Status: new
Component: HTTP handling | Version: dev
Severity: Normal | Resolution:
Keywords: asyncio, signals | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Comment (by Mykhailo Havelia):
> Replying to [comment:16 Carlton Gibson]:
> > Being able to share the context between multiple concurrent
sync_to_async tasks seems like something that's generally useful, and in-
line with the create_task and TaskGroup API. So, that's a useful step
independently I'd think. Thanks!
>
> Done https://github.com/django/asgiref/pull/536 😌
I tested the updated asgiref with Django signals using my branch
> > Being able to share the context between multiple concurrent
sync_to_async tasks seems like something that's generally useful, and in-
line with the create_task and TaskGroup API. So, that's a useful step
independently I'd think. Thanks!
>
> Done https://github.com/django/asgiref/pull/536 😌
{{{
pip install git+https://github.com/Arfey/asgiref.git@feat/added-custom-
context-parameter-for-sync-to-async#egg=asgiref
}}}
Everything works correctly:
{{{
./runtests.py signals
Testing against Django installed in ... with up to 10 processes
Found 28 test(s).
............................
----------------------------------------------------------------------
Ran 28 tests in 0.333s
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/36714#comment:19>
Reply all
Reply to author
Forward
0 new messages