[Django] #28560: distinct() on None values
Django
-------------------------------------+-------------------------------------
Reporter: felixxm | Owner: felixxm
Type: Bug | Status: assigned
Component: Database | Version: master
layer (models, ORM) |
Severity: Normal | Keywords: distinct values
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
`distinct()` doesn't work properly on `None` values. If we select columns
with `null` values then `distinct()` returns each of them instead of
single `None`, e.g. (based on `queries/tests.py`):
{{{
>>>
list(Item.objects.filter(modified__isnull=True).values_list('modified',
flat=True).distinct())
[None, None, None]
}}}
instead of `[None]`.
--
Ticket URL: <https://code.djangoproject.com/ticket/28560>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
-------------------------------------+-------------------------------------
Reporter: felixxm | Owner: felixxm
Type: Bug | Status: assigned
(models, ORM) |
Severity: Normal | Resolution:
Keywords: distinct values | Triage Stage:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* Attachment "28560.diff" added.
Django
-------------------------------------+-------------------------------------
Reporter: felixxm | Owner: felixxm
Type: Bug | Status: assigned
(models, ORM) |
Severity: Normal | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by felixxm):
Issue is related with ordering. After `distinct()` columns from `ORDER BY`
clause are added to a `SELECT`. It doesn't take into account restricted
list of columns from `values` or `values_list`. `None` doesn't matter.
--
Ticket URL: <https://code.djangoproject.com/ticket/28560#comment:1>
Django
incorrect result
-------------------------------------+-------------------------------------
Reporter: felixxm | Owner: felixxm
Type: Bug | Status: assigned
(models, ORM) |
Severity: Normal | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
--
Ticket URL: <https://code.djangoproject.com/ticket/28560#comment:2>
Django
incorrect result
Reporter: felixxm | Owner: felixxm
Type: Bug | Status: assigned
(models, ORM) |
Severity: Normal | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Old description:
> `distinct()` doesn't work properly on `None` values. If we select columns
> with `null` values then `distinct()` returns each of them instead of
> single `None`, e.g. (based on `queries/tests.py`):
>
> {{{
> >>>
> list(Item.objects.filter(modified__isnull=True).values_list('modified',
> flat=True).distinct())
> [None, None, None]
> }}}
>
> instead of `[None]`.
New description:
I think we should raise `NotSupportedError` when `distinct()` is used with
`values()` (or `values_list()`) on ordered queryset and a list of fields
in `values()` doesn't contain all fields from `ORDER BY`. It cannot return
correct result because columns from `ORDER BY` clause must be included in
`SELECT`.
--
--
Ticket URL: <https://code.djangoproject.com/ticket/28560#comment:3>
Django
incorrect result
Reporter: felixxm | Owner: felixxm
Type: Bug | Status: assigned
(models, ORM) |
Severity: Normal | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Simon Charette):
Couldn't we wrap the query in a subquery like we did in #24254 in this
case?
--
Ticket URL: <https://code.djangoproject.com/ticket/28560#comment:4>
Django
incorrect result
Reporter: felixxm | Owner: felixxm
Type: Bug | Status: assigned
(models, ORM) |
Severity: Normal | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by felixxm):
Good idea! it should be feasible. I will try to prepare patch in this
week.
--
Ticket URL: <https://code.djangoproject.com/ticket/28560#comment:5>
Django
incorrect result
Reporter: felixxm | Owner: felixxm
Type: Bug | Status: assigned
(models, ORM) |
Severity: Normal | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Simon Charette):
By the way Mariusz, do you have an opinion on
https://code.djangoproject.com/ticket/14357#comment:11?
--
Ticket URL: <https://code.djangoproject.com/ticket/28560#comment:6>
Django
incorrect result
Reporter: felixxm | Owner: felixxm
Type: Bug | Status: assigned
(models, ORM) |
Severity: Normal | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Old description:
> I think we should raise `NotSupportedError` when `distinct()` is used
> with `values()` (or `values_list()`) on ordered queryset and a list of
> fields in `values()` doesn't contain all fields from `ORDER BY`. It
> cannot return correct result because columns from `ORDER BY` clause must
> be included in `SELECT`.
New description:
When `distinct()` is used with `values()` (or `values_list()`) on ordered
queryset and a list of fields in `values()` doesn't contain all fields
from `ORDER BY`, then it doesn't return correct result because columns
from `ORDER BY` clause must be included in `SELECT`. As Simon suggested we
should wrap a query in a subquery.
--
--
Ticket URL: <https://code.djangoproject.com/ticket/28560#comment:7>
Django
incorrect result
Reporter: felixxm | Owner: felixxm
Type: Bug | Status: assigned
(models, ORM) |
Severity: Normal | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Old description:
> When `distinct()` is used with `values()` (or `values_list()`) on ordered
> queryset and a list of fields in `values()` doesn't contain all fields
> from `ORDER BY`, then it doesn't return correct result because columns
> from `ORDER BY` clause must be included in `SELECT`. As Simon suggested
> we should wrap a query in a subquery.
New description:
When `distinct()` is used with `values()` (or `values_list()`) on ordered
queryset and a list of fields in `values()` doesn't contain all fields
from `ORDER BY`, then it doesn't return correct result because columns
from `ORDER BY` clause must be included in `SELECT`. As Simon suggested we
should wrap a query in a subquery (see related tickets #7070, #5321).
--
--
Ticket URL: <https://code.djangoproject.com/ticket/28560#comment:8>
Django
incorrect result
Reporter: felixxm | Owner: felixxm
Type: Bug | Status: assigned
(models, ORM) |
Severity: Normal | Resolution:
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* has_patch: 0 => 1
Comment:
[https://github.com/django/django/pull/9055 PR]
--
Ticket URL: <https://code.djangoproject.com/ticket/28560#comment:9>
Django
incorrect result
Reporter: felixxm | Owner: felixxm
Type: Bug | Status: assigned
(models, ORM) |
Severity: Normal | Resolution:
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* stage: Unreviewed => Accepted
--
Ticket URL: <https://code.djangoproject.com/ticket/28560#comment:10>
Django
incorrect result
Reporter: felixxm | Owner: felixxm
Type: Bug | Status: assigned
(models, ORM) |
Severity: Normal | Resolution:
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_better_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/28560#comment:11>
Django
incorrect result
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* owner: felixxm => (none)
* needs_better_patch: 1 => 0
* has_patch: 1 => 0
* status: assigned => new
--
Ticket URL: <https://code.djangoproject.com/ticket/28560#comment:12>
Django
incorrect result
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Tobias McNulty):
After some discussion on the mailing list (https://groups.google.com/g
/django-developers/c/DNVRFqVBsfk/m/xDUvaq3DAAAJ) it looks like the
consensus is to require an explicit opt-in or raise an error (i.e., never
add a column implicitly if the user specified a list of columns via
values() or values_list()).
I am not too familiar with the ORM. I believe these columns are added in
`get_extra_select()` (`django/db/models/sql/compiler.py`), which could be
adapted initially to raise a deprecation warning and eventually an
exception. Does that sound right?
I may not have time right away so if anyone feels like picking this up, go
for it.
--
Ticket URL: <https://code.djangoproject.com/ticket/28560#comment:14>
Django
incorrect result
-------------------------------------+-------------------------------------
Reporter: Mariusz Felisiak | Owner: (none)
Type: Bug | Status: new
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Old description:
> When `distinct()` is used with `values()` (or `values_list()`) on ordered
> queryset and a list of fields in `values()` doesn't contain all fields
> from `ORDER BY`, then it doesn't return correct result because columns
> from `ORDER BY` clause must be included in `SELECT`. As Simon suggested
> we should wrap a query in a subquery (see related tickets #7070, #5321).
New description:
When `distinct()` is used with `values()` (or `values_list()`) on ordered
queryset and a list of fields in `values()` doesn't contain all fields
from `ORDER BY`, then it doesn't return correct result because columns
from `ORDER BY` clause must be included in `SELECT`.
After some discussion on the mailing list (https://groups.google.com/g
/django-developers/c/DNVRFqVBsfk/m/xDUvaq3DAAAJ) it looks like the
consensus is to require an explicit opt-in or raise an error (i.e., never
add a column implicitly if the user specified a list of columns via
values() or values_list()).
--
--
Ticket URL: <https://code.djangoproject.com/ticket/28560#comment:15>