[Django] #32173: Autoreloader doesn't work with asyncio.

[Django]

#32173: Autoreloader doesn't work with asyncio.
------------------------------------------+------------------------
Reporter: Carlton Gibson | Owner: nobody
Type: New feature | Status: new
Component: Utilities | Version: 3.1
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
------------------------------------------+------------------------
This is a follow-up to a [https://forum.djangoproject.com/t/feasibility-
of-using-subprocess-for-auto-reloader/4970/7 discussion on the forum about
the feasibility of using subprocess, rather than threads for the
autoreloader].

The summary is that because of the way the autoreloader uses threading,
it's not compatible with (most.many uses) of asyncio. (We end up trying to
use the event loop from the main thread on the child thread, which Doesn't
Work™).

This [https://github.com/django/channels/issues/1374 originally surfaced
on Channels], we [https://github.com/django/daphne/pull/294 tried to fix
it in Daphne], but this has lead to other issues, and we're going to have
to revert that.

Other Channels users, looking at removing the dependency on Daphne there,
have [https://github.com/django/channels/issues/490#issuecomment-720252418
reported needing to re-implement reloading due to the threading
behaviour].

Ultimately I think we have to resolve the threading issue in Django
itself.

On the forum thread, Tom suggested that we might look into leveraging
''Hupper'' here. Assuming that would solve the threading issues (which
seems like an empirical question just yet) I guess there'd need to a
discussion about making that a required dependency or not, but it might be
nice to settle on a solution here that's support more widely than just in
Django.

--
Ticket URL: <https://code.djangoproject.com/ticket/32173>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#32173: Autoreloader doesn't work with asyncio.

--------------------------------+--------------------------------------

Reporter: Carlton Gibson | Owner: nobody
Type: New feature | Status: new
Component: Utilities | Version: 3.1

Severity: Normal | Resolution:
Keywords: async | Triage Stage: Unreviewed

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

--------------------------------+--------------------------------------
Changes (by Carlton Gibson):

* keywords: => async

--
Ticket URL: <https://code.djangoproject.com/ticket/32173#comment:1>

[Django]

#32173: Autoreloader doesn't work with asyncio.

--------------------------------+------------------------------------

Reporter: Carlton Gibson | Owner: nobody
Type: New feature | Status: new
Component: Utilities | Version: 3.1

Severity: Normal | Resolution:
Keywords: async | Triage Stage: Accepted

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

--------------------------------+------------------------------------
Changes (by Mariusz Felisiak):

* stage: Unreviewed => Accepted

--
Ticket URL: <https://code.djangoproject.com/ticket/32173#comment:2>

[Django]

#32173: Autoreloader doesn't work with asyncio.

--------------------------------+------------------------------------

Reporter: Carlton Gibson | Owner: nobody
Type: New feature | Status: new
Component: Utilities | Version: 3.1

Severity: Normal | Resolution:
Keywords: async | Triage Stage: Accepted

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

--------------------------------+------------------------------------

Comment (by Carlton Gibson):

Part of the issue here is the structure of the auto-reloading: as it is
currently,
[https://github.com/django/django/blob/18c5ba07cc81be993941ecc2ecc17923b401b66f/django/utils/autoreload.py#L270-L276
`runserver` is restarted in a subprocess] but then runs Django, in
[https://github.com/django/django/blob/18c5ba07cc81be993941ecc2ecc17923b401b66f/django/utils/autoreload.py#L649-L661
`start_django()` in a thread], watching for changes in the main thread.
(The multiple threads is what causes the issue here, with event loops
being accessed on the wrong thread.)

A different structure would be start Django as a subprocess from the
parent process, watch for changes in that same parent process, and restart
the child process on a change. That's quite a big restructuring (but I
think it might remove the need for the
[https://github.com/django/django/blob/18c5ba07cc81be993941ecc2ecc17923b401b66f/django/utils/autoreload.py#L331
apps `ready_event`] 🤔).

--
Ticket URL: <https://code.djangoproject.com/ticket/32173#comment:3>

[Django]

#32173: Autoreloader doesn't work with asyncio.

--------------------------------+------------------------------------

Reporter: Carlton Gibson | Owner: nobody

Type: New feature | Status: closed
Component: Utilities | Version: 3.1
Severity: Normal | Resolution: wontfix

Keywords: async | Triage Stage: Accepted

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

--------------------------------+------------------------------------
Changes (by Carlton Gibson):

* status: new => closed
* resolution: => wontfix

Comment:

Over time this has faded away — I think dropping Python 3.6 support when
that went EOL was the key. (As `asyncio` moved from `get_event_loop` to
`get_running_loop` the tendency to get a reference to an event loop from a
different thread faded.)

Auto-reloading works with the latest Daphne versions (>4). There may be
lingering issues, but I think they can be resolved there.

--
Ticket URL: <https://code.djangoproject.com/ticket/32173#comment:4>