[Django] #32815: Failed to reset ContextVars in sync/async middlewares
Django
-----------------------------------------------+------------------------
Reporter: Michael Manganiello | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 3.2
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-----------------------------------------------+------------------------
When using a middleware that can process both sync and async requests, and
trying to set and reset a ContextVar (in different methods of its request
lifecycle), Python fails with error:
`ValueError: <Token var=<ContextVar name='current_context' at
0x7f9a8b9ad900> at 0x7f9a68575180> was created in a different Context`
This is a simple middleware example to reproduce the mentioned issue:
{{{
@sync_and_async_middleware
class TemplateResponseMiddleware(BaseMiddleware):
def process_view(self, request, view_func, view_args, view_kwargs):
request.META['_CONTEXT_RESET_TOKEN'] =
current_context.set(id(request))
def process_template_response(self, request, response):
current_context.reset(request.META['_CONTEXT_RESET_TOKEN'])
response.context_data['mw'].append(self.__class__.__name__)
return response
}}}
This use case is what the OpenTelemetry integration uses for spans to be
traced in Django: https://github.com/open-telemetry/opentelemetry-python-
contrib/blob/main/instrumentation/opentelemetry-instrumentation-
django/src/opentelemetry/instrumentation/django/middleware.py
* In `process_request`, a `ContextVar` is set, and the generated token is
persisted in the `request.META` object.
* In `process_response`, the `ContextVar` is reset, by using the persisted
token.
This approach works correctly for synchronous requests. However, as part
of adding ASGI support to the Django integration for OpenTelemetry (in
https://github.com/open-telemetry/opentelemetry-python-contrib/pull/391),
we found that the `ContextVar` triggers the mentioned error when we want
to reset it to its previous value. OpenTelemetry inherits from
`MiddlewareMixin`, but I'm attaching a diff for a simple test scenario
that reproduces the issue, using the new Middleware format.
The main suspects here are the calls to `sync_to_async`, which adapt the
middleware methods to the async flow. However, both those calls explicitly
set `thread_sensitive=True`.
Traceback for the attached test scenario:
{{{
$ ./runtests.py -k
MiddlewareSyncAsyncTests.test_async_process_template_response
# ...
ERROR: test_async_process_template_response
(middleware_exceptions.tests.MiddlewareSyncAsyncTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/mike/.virtualenvs/django/lib/python3.9/site-
packages/asgiref/sync.py", line 222, in __call__
return call_result.result()
File "/usr/lib/python3.9/concurrent/futures/_base.py", line 438, in
result
return self.__get_result()
File "/usr/lib/python3.9/concurrent/futures/_base.py", line 390, in
__get_result
raise self._exception
File "/home/mike/.virtualenvs/django/lib/python3.9/site-
packages/asgiref/sync.py", line 287, in main_wrap
result = await self.awaitable(*args, **kwargs)
File "/mnt/data/Proyectos/third_party/django/django/test/utils.py", line
423, in inner
return await func(*args, **kwargs)
File
"/mnt/data/Proyectos/third_party/django/tests/middleware_exceptions/tests.py",
line 319, in test_async_process_template_response
response = await self.async_client.get(
File "/mnt/data/Proyectos/third_party/django/django/test/client.py",
line 911, in request
self.check_exception(response)
File "/mnt/data/Proyectos/third_party/django/django/test/client.py",
line 580, in check_exception
raise exc_value
File "/home/mike/.virtualenvs/django/lib/python3.9/site-
packages/asgiref/sync.py", line 458, in thread_handler
raise exc_info[1]
File
"/mnt/data/Proyectos/third_party/django/django/core/handlers/exception.py",
line 38, in inner
response = await get_response(request)
File
"/mnt/data/Proyectos/third_party/django/django/core/handlers/base.py",
line 249, in _get_response_async
response = await middleware_method(request, response)
File "/home/mike/.virtualenvs/django/lib/python3.9/site-
packages/asgiref/sync.py", line 423, in __call__
ret = await asyncio.wait_for(future, timeout=None)
File "/usr/lib/python3.9/asyncio/tasks.py", line 442, in wait_for
return await fut
File "/home/mike/.virtualenvs/django/lib/python3.9/site-
packages/asgiref/current_thread_executor.py", line 22, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/mike/.virtualenvs/django/lib/python3.9/site-
packages/asgiref/sync.py", line 462, in thread_handler
return func(*args, **kwargs)
File
"/mnt/data/Proyectos/third_party/django/tests/middleware_exceptions/middleware.py",
line 135, in process_template_response
current_context.reset(request.META['_CONTEXT_RESET_TOKEN'])
ValueError: <Token var=<ContextVar name='current_context' at
0x7f1dd4ec2720> at 0x7f1db129a880> was created in a different Context
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/32815>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: Michael Manganiello | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 3.2
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Michael Manganiello):
* Attachment "django-issue-32815.diff" added.
Test scenario (diff over main commit f10c52afab)
Django
Reporter: Michael Manganiello | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 3.2
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Description changed by Michael Manganiello:
Old description:
New description:
When using a middleware that can process both sync and async requests, and
trying to set and reset a ContextVar (in different methods of its request
lifecycle), Python fails with error:
`ValueError: <Token var=<ContextVar name='current_context' at
0x7f9a8b9ad900> at 0x7f9a68575180> was created in a different Context`
This is a simple middleware example to reproduce the mentioned issue:
{{{
import contextvars
current_context = contextvars.ContextVar('current_context')
--
--
Ticket URL: <https://code.djangoproject.com/ticket/32815#comment:1>
Django
Reporter: Michael Manganiello | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 3.2
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Description changed by Michael Manganiello:
Old description:
> When using a middleware that can process both sync and async requests,
> and trying to set and reset a ContextVar (in different methods of its
> request lifecycle), Python fails with error:
> `ValueError: <Token var=<ContextVar name='current_context' at
> 0x7f9a8b9ad900> at 0x7f9a68575180> was created in a different Context`
>
> This is a simple middleware example to reproduce the mentioned issue:
>
> {{{
> import contextvars
>
> current_context = contextvars.ContextVar('current_context')
>
New description:
When using a middleware that can process both sync and async requests, and
trying to set and reset a ContextVar (in different methods of its request
lifecycle), Python fails with error:
`ValueError: <Token var=<ContextVar name='current_context' at
0x7f9a8b9ad900> at 0x7f9a68575180> was created in a different Context`
This is a simple middleware example to reproduce the mentioned issue:
{{{
import contextvars
current_context = contextvars.ContextVar('current_context')
@sync_and_async_middleware
class TemplateResponseMiddleware(BaseMiddleware):
def process_view(self, request, view_func, view_args, view_kwargs):
request.META['_CONTEXT_RESET_TOKEN'] =
current_context.set(id(request))
def process_template_response(self, request, response):
current_context.reset(request.META['_CONTEXT_RESET_TOKEN'])
return response
}}}
--
--
Ticket URL: <https://code.djangoproject.com/ticket/32815#comment:2>
Django
Reporter: Michael Manganiello | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 3.2
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Michael Manganiello):
It seems this is not an issue only related to middlewares, but to how
`sync_to_async` works in general? I am able to reproduce the same issue
with this simple endpoint (when running Django using Gunicorn with Uvicorn
workers):
{{{
import contextvars
from asgiref.sync import sync_to_async
from django.http import HttpResponse
from django.urls import path
current_context = contextvars.ContextVar('current_context')
async def healthcheck(request):
token = await sync_to_async(current_context.set,
thread_sensitive=True)(id(request))
await sync_to_async(current_context.reset,
thread_sensitive=True)(token)
return HttpResponse('OK')
urlpatterns = [path('', healthcheck)]
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/32815#comment:3>
Django
Reporter: Michael Manganiello | Owner: nobody
Component: Uncategorized | Version: 3.2
Severity: Normal | Resolution: invalid
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Mariusz Felisiak):
* status: new => closed
* resolution: => invalid
Comment:
Thanks for this report, however it looks like a support question and Trac
is not a support channel. Moreover you're discussing `asgiref` behavior
not Django itself. I would recommend to open
[https://github.com/django/asgiref/issues an issue] in `asgiref`.
--
Ticket URL: <https://code.djangoproject.com/ticket/32815#comment:4>
Django
Reporter: Michael Manganiello | Owner: nobody
Component: Uncategorized | Version: 3.2
Severity: Normal | Resolution: invalid
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Michael Manganiello):
Thanks for the feedback. I've filed
https://github.com/django/asgiref/issues/267 to find the root cause.
--
Ticket URL: <https://code.djangoproject.com/ticket/32815#comment:5>