[Django] #37218: Fetch modes aren't async-compatible

[Django]

#37218: Fetch modes aren't async-compatible
-------------------------------------+-------------------------------------
Reporter: Jacob | Owner: Jacob Walls
Walls |
Type: Bug | Status: assigned
Component: Database | Version: 6.1
layer (models, ORM) |
Severity: Normal | Keywords: fetch modes, async
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
See [https://dryorm.xterm.info/fetch-mode-async-failure fiddle]:

{{{#!py
from asgiref.sync import async_to_sync
from django.contrib.auth.models import User
from django.db import models


class Person(models.Model):
name = models.CharField(max_length=100)
user = models.ForeignKey(User, models.CASCADE)


@async_to_sync
async def async_view():
qs = Person.objects.fetch_mode(models.FETCH_PEERS)
async for el in qs:
print(el.user)


def run():
admin = User.objects.create(username="admin")
instance = Person.objects.create(name='John Doe', user=admin)
instance2 = Person.objects.create(name='Jane Doe', user=admin)
async_view()

}}}
{{{#!py
File "/app/app/models.py", line 15, in async_view
print(el.user)
^^^^^^^
File "/django-pr/django/db/models/fields/related_descriptors.py", line
266, in __get__
instance._state.fetch_mode.fetch(self, instance)
File "/django-pr/django/db/models/fetch_modes.py", line 38, in fetch
fetcher.fetch_many(instances)
File "/django-pr/django/db/models/fields/related_descriptors.py", line
291, in fetch_many
prefetch_related_objects(missing_instances, self.field.name)
File "/django-pr/django/db/models/query.py", line 2701, in
prefetch_related_objects
obj_list, additional_lookups = prefetch_one_level(
^^^^^^^^^^^^^^^^^^^
File "/django-pr/django/db/models/query.py", line 2876, in
prefetch_one_level
all_related_objects = list(rel_qs)
^^^^^^^^^^^^
File "/django-pr/django/db/models/query.py", line 434, in __iter__
self._fetch_all()
File "/django-pr/django/db/models/query.py", line 2239, in _fetch_all
self._result_cache = list(self._iterable_class(self))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/django-pr/django/db/models/query.py", line 98, in __iter__
results = compiler.execute_sql(
^^^^^^^^^^^^^^^^^^^^^
File "/django-pr/django/db/models/sql/compiler.py", line 1622, in
execute_sql
cursor = self.connection.cursor()
^^^^^^^^^^^^^^^^^^^^^^^^
File "/django-pr/django/utils/asyncio.py", line 24, in inner
raise SynchronousOnlyOperation(message)
django.core.exceptions.SynchronousOnlyOperation: You cannot call this from
an async context - use a thread or sync_to_async.
}}}

We probably need to add `AFETCH_PEERS` and `AFETCH_ONE` fetch modes, and
duplicate the `fetch_{one,many}()` methods into `afetch_{one,many}()` on
the descriptors.

I can hustle this in before the release candidate if it seems sane.
--
Ticket URL: <https://code.djangoproject.com/ticket/37218>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#37218: Fetch modes aren't async-compatible
-------------------------------------+-------------------------------------

Reporter: Jacob Walls | Owner: Jacob

| Walls
Type: Bug | Status: assigned

Component: Database layer | Version: 6.1
(models, ORM) |
Severity: Release blocker | Resolution:

Keywords: fetch modes, async | Triage Stage:

| Unreviewed
Has patch: 0 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Jacob Walls):

* severity: Normal => Release blocker


Old description:

New description:

--
Ticket URL: <https://code.djangoproject.com/ticket/37218#comment:1>

[Django]

#37218: Fetch modes aren't async-compatible
-------------------------------------+-------------------------------------
Reporter: Jacob Walls | Owner: Jacob
| Walls
Type: Bug | Status: assigned
Component: Database layer | Version: 6.1
(models, ORM) |
Severity: Release blocker | Resolution:
Keywords: fetch modes, async | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Simon Charette):

I'm not sure I understand the need to introduce a fetch mode for this
case?

Isn't the problem already present [https://dryorm.xterm.info/related-attr-
async-failire without fetch mode involved]

{{{#!python
@async_to_sync
async def async_view():
qs = Person.objects.all()

async for el in qs:
print(el.user)
}}}

{{{#!python

File "/app/app/models.py", line 15, in async_view
print(el.user)
^^^^^^^
File "/django-pr/django/db/models/fields/related_descriptors.py", line
266, in __get__
instance._state.fetch_mode.fetch(self, instance)

File "/django-pr/django/db/models/fetch_modes.py", line 17, in fetch
fetcher.fetch_one(instance)
File "/django-pr/django/db/models/fields/related_descriptors.py", line
279, in fetch_one
rel_obj = self.get_object(instance)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/django-pr/django/db/models/fields/related_descriptors.py", line
231, in get_object
return qs.get(self.field.get_reverse_related_filter(instance))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/django-pr/django/db/models/query.py", line 679, in get
num = len(clone)
^^^^^^^^^^
File "/django-pr/django/db/models/query.py", line 416, in __len__

self._fetch_all()
File "/django-pr/django/db/models/query.py", line 2239, in _fetch_all
self._result_cache = list(self._iterable_class(self))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/django-pr/django/db/models/query.py", line 98, in __iter__
results = compiler.execute_sql(
^^^^^^^^^^^^^^^^^^^^^
File "/django-pr/django/db/models/sql/compiler.py", line 1622, in
execute_sql
cursor = self.connection.cursor()
^^^^^^^^^^^^^^^^^^^^^^^^
File "/django-pr/django/utils/asyncio.py", line 24, in inner
raise SynchronousOnlyOperation(message)
django.core.exceptions.SynchronousOnlyOperation: You cannot call this from
an async context - use a thread or sync_to_async.
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/37218#comment:2>

[Django]

#37218: Fetch modes aren't async-compatible
-------------------------------------+-------------------------------------
Reporter: Jacob Walls | Owner: Jacob
| Walls
Type: Bug | Status: assigned
Component: Database layer | Version: 6.1
(models, ORM) |
Severity: Release blocker | Resolution:
Keywords: fetch modes, async | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Carlton Gibson):

* cc: Carlton Gibson (added)

Comment:

Yes... this is *exactly* the No Deferred Queries constraint. Totally
expected behaviour IMO.
--
Ticket URL: <https://code.djangoproject.com/ticket/37218#comment:3>

[Django]

#37218: Fetch modes aren't async-compatible
-------------------------------------+-------------------------------------
Reporter: Jacob Walls | Owner: Jacob
| Walls

Type: Bug | Status: closed

Component: Database layer | Version: 6.1
(models, ORM) |

Severity: Release blocker | Resolution: invalid

Keywords: fetch modes, async | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Jacob Walls):

* resolution: => invalid
* status: assigned => closed

Comment:

I was aware of that limitation, and the prescription to use
`select_related()` to avoid deferred queries, but as the spiritual
successor to `select_related()`, I guess I assumed an analogous
afetch_peers mode would have a way to swap in `aprefetch_related_objects`
in place of `prefetch_related_objects`, but looking at that stacktrace, we
have that pesky `__get__()` right there at the top, so, ack--of course
not! Thanks for confirming.

I'll have a look around to see if there's a sentence or two we should add
about this.
--
Ticket URL: <https://code.djangoproject.com/ticket/37218#comment:4>

[Django]

#37218: Fetch modes aren't async-compatible
-------------------------------------+-------------------------------------
Reporter: Jacob Walls | Owner: Jacob

Type: | Walls
Cleanup/optimization | Status: closed
Component: Documentation | Version: 6.1
Severity: Normal | Resolution: invalid

Keywords: fetch modes, async | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Jacob Walls):

* component: Database layer (models, ORM) => Documentation
* severity: Release blocker => Normal
* type: Bug => Cleanup/optimization

Comment:

I was surprised to see we only have information about "beware `defer()`
and `only()`" rather than also "beware lazy relation access" in the async
queries section.

I did a few tweaks here, let me know if you think any of it is useful?
[https://github.com/django/django/pull/21635 PR]
--
Ticket URL: <https://code.djangoproject.com/ticket/37218#comment:5>