[Django] #29396: Using date__year comparisons together with OuterRef causes an IndexError
Django
-------------------------------------+-------------------------------------
Reporter: Dmitry | Owner: nobody
Shachnev |
Type: Bug | Status: new
Component: Database | Version:
layer (models, ORM) |
Severity: Normal | Keywords:
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
With the following models:
{{{
#!python
class ModelWithDate(models.Model):
id = models.AutoField(primary_key=True)
date = models.DateField()
class ModelWithYear(models.Model):
id = models.AutoField(primary_key=True)
year = models.IntegerField()
date_ref = models.ForeignKey(to=ModelWithDate,
on_delete=models.CASCADE)
}}}
the following code:
{{{
#!python
>>> dates = ModelWithDate.objects.filter(date__year__gte=OuterRef("year"))
>>> dates_subq = Subquery(dates.values("id"))
>>> ModelWithYear.objects.filter(date_ref__in=dates_subq)
}}}
causes an exception:
{{{
#!pytb
Traceback (most recent call last):
...
File "/usr/lib/python3/dist-packages/django/db/models/lookups.py", line
90, in process_rhs
sql, params = compiler.compile(value)
File "/usr/lib/python3/dist-packages/django/db/models/sql/compiler.py",
line 391, in compile
sql, params = node.as_sql(self, self.connection)
File "/usr/lib/python3/dist-packages/django/db/models/expressions.py",
line 1041, in as_sql
template_params['subquery'], sql_params =
self.queryset.query.get_compiler(connection=connection).as_sql()
File "/usr/lib/python3/dist-packages/django/db/models/sql/compiler.py",
line 459, in as_sql
where, w_params = self.compile(self.where) if self.where is not None
else ("", [])
File "/usr/lib/python3/dist-packages/django/db/models/sql/compiler.py",
line 391, in compile
sql, params = node.as_sql(self, self.connection)
File "/usr/lib/python3/dist-packages/django/db/models/sql/where.py",
line 80, in as_sql
sql, params = compiler.compile(child)
File "/usr/lib/python3/dist-packages/django/db/models/sql/compiler.py",
line 391, in compile
sql, params = node.as_sql(self, self.connection)
File "/usr/lib/python3/dist-packages/django/db/models/lookups.py", line
523, in as_sql
start, finish = self.year_lookup_bounds(connection, rhs_params[0])
IndexError: list index out of range
}}}
The same happens if I create an annotation using ExtractYear and use it
directly.
A small test project is attached. The bug can be reproduced with
“DJANGO_SETTINGS_MODULE=settings python3 ./test.py”. Tested with Django
2.0.5 and 1.11.13.
--
Ticket URL: <https://code.djangoproject.com/ticket/29396>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: Database layer | Version:
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* Attachment "test-project.tar.gz" added.
Test project to reproduce the bug
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: Database layer | Version:
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Dmitry Shachnev):
I should add that it happens only with `gte`/`lte` comparisons. If I
replace `date__year__gte` with `date__year__exact`, then the code works as
expected.
--
Ticket URL: <https://code.djangoproject.com/ticket/29396#comment:1>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* version: => master
* stage: Unreviewed => Accepted
Comment:
Reproduced on master at 9c4ea63e878c053600c284e32d5f32d27a59b63a.
--
Ticket URL: <https://code.djangoproject.com/ticket/29396#comment:2>
Django
-------------------------------------+-------------------------------------
| yang
Type: Bug | Status: assigned
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: new => assigned
* owner: nobody => Windson yang
--
Ticket URL: <https://code.djangoproject.com/ticket/29396#comment:3>
Django
-------------------------------------+-------------------------------------
| yang
Type: Bug | Status: assigned
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Windson yang):
Please tell me if I'm doing wrong, Just like
[https://github.com/django/django/blob/master/django/db/models/lookups.py#L517
YearExact], We should adding the same try-except block to handle when
rhs_params[0] doesn't exist. In the except part, should we just raise an
error like below? Or we have to overwrite GreaterThanOrEqual and
LessThanOrEqual class?
{{{
raise ValueError(
'The QuerySet value for an exact lookup must be
limited to '
'one result using slicing.'
)
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/29396#comment:4>
Django
-------------------------------------+-------------------------------------
| yang
Type: Bug | Status: assigned
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Windson yang):
pull request https://github.com/django/django/pull/10023
--
Ticket URL: <https://code.djangoproject.com/ticket/29396#comment:5>
Django
-------------------------------------+-------------------------------------
| yang
Type: Bug | Status: assigned
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_better_patch: 0 => 1
* has_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/29396#comment:6>
Django
values/expression.
-------------------------------------+-------------------------------------
Reporter: Dmitry Shachnev | Owner: Windson
| yang
Type: Bug | Status: assigned
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Simon Charette):
As I commented on the PR the bug has little to do with subquery usage;
`__year` lookup simply never supported non-direct values and `OuterRef`
happens to implement the expression API in this particular case.
Both this issue and #30494 which was for the `year__exact` case are
handled by https://github.com/django/django/pull/11393.
--
Ticket URL: <https://code.djangoproject.com/ticket/29396#comment:7>
Django
values/expression.
-------------------------------------+-------------------------------------
Reporter: Dmitry Shachnev | Owner: Windson
| yang
Component: Database layer | Version: master
(models, ORM) |
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"2b582a7b84348aee0667cd3f9dc9793c04d958e0" 2b582a7]:
{{{
#!CommitTicketReference repository=""
revision="2b582a7b84348aee0667cd3f9dc9793c04d958e0"
Fixed #29396 -- Added indirect values support to __year lookups.
Thanks Windson Yang for the initial patch.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/29396#comment:8>
Django
values/expression.
-------------------------------------+-------------------------------------
Reporter: Dmitry Shachnev | Owner: Windson
| yang
Type: Bug | Status: closed
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Mariusz Felisiak <felisiak.mariusz@…>):
In [changeset:"514104cf236c1039644b70c0c0f128cecd42b233" 514104cf]:
{{{
#!CommitTicketReference repository=""
revision="514104cf236c1039644b70c0c0f128cecd42b233"
Refs #29396, #30494 -- Reduced code duplication in year lookups.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/29396#comment:9>