Re: [Django] #5763: Queryset doesn't have a "not equal" filter operator
Django
-------------------------------------+-------------------------------------
Reporter: jdetaeye | Owner: nobody
Type: New | Status: closed
feature | Component: Database layer
Milestone: | (models, ORM)
Version: SVN | Severity: Normal
Resolution: wontfix | Keywords: qs-rf
Triage Stage: Design | Has patch: 0
decision needed | Needs tests: 0
Needs documentation: 0 | Easy pickings: 0
Patch needs improvement: 0 |
UI/UX: 0 |
-------------------------------------+-------------------------------------
Changes (by carljm):
* status: reopened => closed
* ui_ux: => 0
* resolution: => wontfix
* easy: => 0
Comment:
Based on discussion with Alex and Andrew at the sprint, closing this
wontfix on the basis that it doesn't make sense to have `__ne` without any
other negated queries, when we already have .exclude() to express
negatives.
Alex promises that he really will fix the SQL generation to not be stupid
with .exclude().
--
Ticket URL: <https://code.djangoproject.com/ticket/5763#comment:13>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
-------------------------------------+-------------------------------------
Reporter: jdetaeye | Owner: nobody
Component: Database layer | Version: SVN
(models, ORM) | Resolution:
Severity: Normal | Triage Stage: Design
Keywords: qs-rf | decision needed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by asmoore82):
* status: closed => reopened
* resolution: wontfix =>
* easy: 0 => 1
Comment:
Apologies for the re-open - but I wasn't sure what would be worse, a re-
open or a new ticket for the same old discussion...
I'm going to have to take the position that the absence of a `__ne`
operator //still//
represents a functional hole in the querying API, even with consideration
of exclude()
> it doesn't make sense to have `__ne` without any other negated queries,
Ah but there are indeed other negated queries.
`filter(__gte)` and `exclude(__lt)` are **almost** equivalent.
Similarly, you can **almost** approximate `__ne` with a hokey `Q(__lt) |
Q(__gt)` contraption.
But the real issue lies within that "almost" -- this is what could be
cause for a separate ticket, but a quick and dirty `__ne` would head the
issue off altogether ;). Chaining multiple `filter()` and `exclude()`
calls on multi-valued relationships yields not-so-surprising but
nonetheless undesired results. To quote from the Django docs:
> Django has a consistent way of processing `filter()` and `exclude()`
calls. Everything inside a single `filter()` call is applied
simultaneously to filter out items matching all those requirements.
Successive `filter()` calls further restrict the set of objects, but for
multi-valued relations, they apply to any object linked to the primary
model, not necessarily those objects that were selected by an earlier
`filter()` call.
This consistency is a good thing but it combines with the lack of `__ne`
to form a problem.
Say you have blogs with entries with tags and author_counts
If you want to get all entries that are tagged 'django' with more than 2
co-authors:
{{{ Entry.objects.filter(tag__name='django', author_count__gt=2) }}}
It is similarly easy to get blogs with entries with the above criteria:
{{{ Blog.objects.filter(entry__tag__name='django',
entry__author_count__gt=2) }}}
But what if you want entries tagged 'django' that are **not** co-authored
by 2:
{{{ Entry.objects.filter(tag__name='django').exclude(author_count=2) }}}
But if you want the blogs with those entries, it can't easily be done:
{{{
Blog.objects.filter(entry__tag__name='django').exclude(entry__author_count=2)
}}}
is **not** the equivalent of the imaginary query:
{{{ Blog.objects.filter(entry__tag__name='django',
entry__author_count__ne=2) }}}
which can currently be approximated with:
{{{ Blog.objects.filter(Q(entry__author_count__lt=2) |
Q(entry__author_count__gt=2), entry__tag__name='django') }}}
You can also get the desired results with a `.extra()` call but let's not
go there :P.
This brings us to the most surprising aspect, using the negation operator
`~` on `Q()` objects that select on multi-valued relations is technically
possible but can yield the undesired results on the unsuspecting, which
actually runs sort of contrary to the Documentation quote above.
Bad Surprise!!:
{{{ Blog.objects.filter(~Q(entry__author_count=2),
entry__tag__name='django') }}}
`^`The more I look at this, the more I think it is cause for a ticket in
its own right. The fix for this would be to smarten up the `Q()` objects
so that a NOT operator on `__gt` becomes `__lte`, a NOT on `__lt` becomes
`__gte`, and so on. But you will ultimately be missing the `__ne` and
other NOT primitives to fall back on.
In other words, this ticket is a "could-go-either-way" blocker for `^`that
more important ticket. I'll do some research on that and make a ticket if
it hasn't already been addressed.
Thanks to all who make Django awesome! I'm a database newbie and loving
it!
~Adam sM
--
Ticket URL: <https://code.djangoproject.com/ticket/5763#comment:14>
Django
-------------------------------------+-------------------------------------
Reporter: jdetaeye | Owner: nobody
Keywords: qs-rf | decision needed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: reopened => closed
* resolution: => wontfix
Comment:
Thanks for your thoughts, Adam, but reopening a ticket closed by a core
developer is a no-no (as mentioned in the
[https://docs.djangoproject.com/en/1.3/internals/contributing/#ticket-
resolutions contributing documentation]).
Try bringing this up in the django-developers mailing list.
--
Ticket URL: <https://code.djangoproject.com/ticket/5763#comment:15>
Django
-------------------------------------+-------------------------------------
Reporter: jdetaeye | Owner: nobody
Type: New feature | Status: closed
Component: Database layer | Version: SVN
(models, ORM) | Resolution: wontfix
Severity: Normal | Triage Stage: Design
Keywords: qs-rf | decision needed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: Kronuz (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/5763#comment:16>
Django
-------------------------------------+-------------------------------------
Reporter: jdetaeye | Owner: nobody
Type: New feature | Status: closed
(models, ORM) | Resolution: wontfix
Severity: Normal | Triage Stage: Design
Keywords: qs-rf | decision needed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by PhiR_42):
Just for the record, this is very important for a specific use case: when
you want to add extra select columns from the joined tables. I have an
ORM-based search form and using exclude will break in many cases because
it uses more subqueries. Using filter(related__field__ne = something) will
work as expected while exclude(related__field = something) will generate a
subquery and remove the possibility of selecting tables.
--
Ticket URL: <https://code.djangoproject.com/ticket/5763#comment:17>
Django
-------------------------------------+-------------------------------------
Reporter: jdetaeye | Owner: nobody
Type: New feature | Status: closed
Component: Database layer | Version: master
(models, ORM) | Resolution: wontfix
Severity: Normal | Triage Stage: Design
Keywords: qs-rf | decision needed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by mjtamlyn):
It is worth noting that in 1.7 this can be written as a custom lookup if
you need it. I think it might even be the example in the docs.
--
Ticket URL: <https://code.djangoproject.com/ticket/5763#comment:18>
Django
-------------------------------------+-------------------------------------
Reporter: jdetaeye | Owner: nobody
Type: New feature | Status: closed
Component: Database layer | Version: master
(models, ORM) | Resolution: wontfix
Severity: Normal | Triage Stage: Design
Keywords: qs-rf | decision needed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by akaariai):
Be warned - not using subqueries might actually produce incorrect results.
Specifically, .exclude() guarantees that you get the complement of what
you get with .filter(). There are cases where `field__ne` will not produce
the complement of `field__exact`. This happens for example if the field
has NULL values. Both `__exact` and `__ne` will miss such rows, as in SQL
terms both 'val = NULL' and 'val != NULL' have unknown result.
As mjtamlyn noted a `__ne` lookup will be fairly straightforward to
implement in 1.7. Check https://docs.djangoproject.com/en/dev/ref/models
/custom-lookups/ for details.
--
Ticket URL: <https://code.djangoproject.com/ticket/5763#comment:19>
Django
-------------------------------------+-------------------------------------
Reporter: jdetaeye | Owner: nobody
Type: New feature | Status: closed
Component: Database layer | Version: master
(models, ORM) | Resolution: wontfix
Severity: Normal | Triage Stage: Design
Keywords: qs-rf | decision needed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by PhiR_42):
Thanks for that clarification.
In my case it seems the patch mitigates #14645 though, so I will probably
take my chances in production.
--
Ticket URL: <https://code.djangoproject.com/ticket/5763#comment:20>
Django
-------------------------------------+-------------------------------------
Reporter: jdetaeye | Owner: nobody
Type: New feature | Status: closed
Component: Database layer | Version: master
(models, ORM) |
Keywords: qs-rf | Triage Stage: Design
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by merutak):
Since this is an issue that keeps coming up for me, and I'm seeing it in
discussions on the web as well as on this ticket, and its usefulness has
been demonstrated by asmoore82 (and I can give more examples) --
is there an explanation why this shouldn't be implemented? Is there some
consideration by which this ticket isn't a good idea?
--
Ticket URL: <https://code.djangoproject.com/ticket/5763#comment:21>
Django
-------------------------------------+-------------------------------------
Reporter: jdetaeye | Owner: nobody
Type: New feature | Status: closed
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution: wontfix
Keywords: qs-rf | Triage Stage: Design
| decision needed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by timgraham):
Please first read through the ticket history as well as the mailing list
threads linked from the the ticket. If you disagree with the conclusions
that have been reached so far, you are welcome to argue your case on the
DevelopersMailingList.
--
Ticket URL: <https://code.djangoproject.com/ticket/5763#comment:22>
Django
-------------------------------------+-------------------------------------
Reporter: jdetaeye | Owner: nobody
Type: New feature | Status: closed
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution: wontfix
Keywords: qs-rf | Triage Stage: Design
| decision needed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: carsten.fuchs@… (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/5763#comment:23>
Django
-------------------------------------+-------------------------------------
Reporter: jdetaeye | Owner: nobody
Type: New feature | Status: closed
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution: wontfix
Keywords: qs-rf | Triage Stage: Design
| decision needed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: unai@… (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/5763#comment:24>
Django
-------------------------------------+-------------------------------------
Reporter: jdetaeye | Owner: nobody
Type: New feature | Status: closed
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution: wontfix
Keywords: qs-rf | Triage Stage: Design
| decision needed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by cpbotha):
Is there a way to work around the absence of the `ne` operator when one
wants to do a `get_or_create()` or an `update_or_create()` where one wants
to select records with a non-blank text field value?
In other words, how would I do for example
`Blog.objects.update_or_create(text_contents__ne='',
other_field_indicating_sort=42, defaults={yet_another_field: True})` ?
--
Ticket URL: <https://code.djangoproject.com/ticket/5763#comment:25>
Django
-------------------------------------+-------------------------------------
Reporter: jdetaeye | Owner: nobody
Type: New feature | Status: closed
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution: wontfix
Keywords: qs-rf | Triage Stage: Design
| decision needed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Simon Charette):
You should be able to use `exclude` for that.
{{{#!python
Blog.objects.exclude(
text_contents='',
).update_or_create(
other_field_indicating_sort=42,
defaults={yet_another_field: True},
)
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/5763#comment:26>
Django
-------------------------------------+-------------------------------------
Reporter: jdetaeye | Owner: nobody
Type: New feature | Status: closed
(models, ORM) |
Severity: Normal | Resolution: wontfix
Keywords: qs-rf | Triage Stage: Design
| decision needed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by dvg):
The "... spirit of not needlessly offering multiple ways to do the same
thing ..." makes a lot of sense to me.
However, there's this question on StackOverflow:
[https://stackoverflow.com/questions/687295 How do I do a not equal in
Django queryset filtering?]
Current stats:
- **555246 views**
- 859 upvotes
- bookmarked 179 times
Just to be clear, this is a **really** high view count: top 0.02% (top
5500 out of 23 million questions, according to
https://data.stackexchange.com/stackoverflow)
Apparently, the alternatives to an `__ne` lookup are not that obvious to
many people.
Perhaps it's worth re-opening this issue and, for example, implementing
the `ne` from the [https://docs.djangoproject.com/en/4.0/howto/custom-
lookups/#a-lookup-example custom lookup example]?
--
Ticket URL: <https://code.djangoproject.com/ticket/5763#comment:27>