[Django] #37311: In/Range lookups silently return no results when rhs is an iterator (regression in 6.1)

[Django]

#37311: In/Range lookups silently return no results when rhs is an iterator
(regression in 6.1)
-------------------------------------+-------------------------------------
Reporter: Oleksandr Tatarinov | Type: Bug
Status: new | Component: Database
| layer (models, ORM)
Version: 6.1 | Severity: Release
| blocker
Keywords: regression | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Passing an iterator to `__in` silently produces an empty result on Django
6.1. Regression in [https://github.com/django/django/commit/7b54ddd5e6
7b54ddd5e6] (refs #36025).
`FieldGetDbPrepValueIterableMixin.get_prep_lookup()` now iterates
`self.rhs` twice: the `any()` call exhausts it, the second iteration sees
nothing, and the lookup compiles to an empty `IN ()`.
On 6.0 the method made a single pass, and the `__in` docs say "in a given
iterable". One possible fix is to materialize the `rhs` before the
`any()` check.
--
Ticket URL: <https://code.djangoproject.com/ticket/37311>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#37311: In/Range lookups silently return no results when rhs is an iterator
(regression in 6.1)
-------------------------------------+-------------------------------------

Reporter: Oleksandr Tatarinov | Owner: (none)
Type: Bug | Status: new
Component: Database layer | Version: 6.1
(models, ORM) |
Severity: Release blocker | Resolution:
Keywords: regression | Triage Stage: Accepted

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Yassin Bahri):

* stage: Unreviewed => Accepted

Comment:

I reproduced this on current `main`, although the issue is narrower than
the current description suggests.

Iterator right-hand sides for ordinary field lookups already work because
`Query.build_filter()` materializes them:

{{{#!python
Article.objects.filter(id__in=iter([article_1.id, article_2.id]))
}}}

The regression occurs when the left-hand side resolves to an annotation or
alias:

{{{#!python
queryset = Article.objects.alias(article_id=F("id"))

queryset.filter(
article_id__in=iter([article_1.id, article_2.id]),
)
# Incorrectly returns no rows.

queryset.filter(
article_id__range=iter([article_1.id, article_2.id]),
)
# Raises ValueError: not enough values to unpack.
}}}

In this path, the iterator reaches
`FieldGetDbPrepValueIterableMixin.get_prep_lookup()` directly. The `any()`
expression consumes it, leaving no values for the following iteration.

I confirmed the regression boundary:

* Both regression tests pass immediately before `7b54ddd5e6`.
* Both fail at `7b54ddd5e6`.
* `in` returns an empty result, while `range` raises `ValueError`.

The ticket is valid and actionable. I suggest narrowing the summary to:

{{{
In/Range lookups on annotations consume iterator rhs (regression in 6.1)
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/37311#comment:1>

[Django]

#37311: In/Range lookups silently return no results when rhs is an iterator
(regression in 6.1)
-------------------------------------+-------------------------------------

Reporter: Oleksandr Tatarinov | Owner: Yassin
| Bahri
Type: Bug | Status: assigned

Component: Database layer | Version: 6.1
(models, ORM) |
Severity: Release blocker | Resolution:
Keywords: regression | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Yassin Bahri):

* owner: (none) => Yassin Bahri
* status: new => assigned

--
Ticket URL: <https://code.djangoproject.com/ticket/37311#comment:2>

[Django]

#37311: In/Range lookups silently return no results when rhs is an iterator
(regression in 6.1)
-------------------------------------+-------------------------------------
Reporter: Oleksandr Tatarinov | Owner: Yassin
| Bahri
Type: Bug | Status: assigned
Component: Database layer | Version: 6.1
(models, ORM) |
Severity: Release blocker | Resolution:
Keywords: regression | Triage Stage: Accepted

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Yassin Bahri):

* has_patch: 0 => 1

--
Ticket URL: <https://code.djangoproject.com/ticket/37311#comment:3>