__ne, #5763

[Carsten Fuchs]

Hi all,

sorry if this is a stupid question, but after having read https://code.djangoproject.com/ticket/5763 and the discussions linked from it, why should __ne not be implemented?

Without __ne, I'm experiencing the same problems that asmoore82 described at https://code.djangoproject.com/ticket/5763#comment:23
That is, afaics, some queries cannot be formulated without __ne, as exclude() is not an equivalent. Or am I missing or misunderstanding something?
I can live with the existing work-arounds, so I'm mostly curious.   :-)

Best regards,
Carsten

[Marcin Nowak]

On Friday, November 20, 2015 at 8:37:02 PM UTC+1, Carsten Fuchs wrote:

sorry if this is a stupid question, but after having read https://code.djangoproject.com/ticket/5763 and the discussions linked from it, why should __ne not be implemented?

Because Django ORM __ne SQLAlchemy. 

BR,

Marcin

[Aaron C. de Bruyn]

With all due respect, looking through the ticket and reading responses shows me that the 'pro' side has good use cases for __ne, and the 'con' side basically says "use exclude" or "a core committer close the ticket, stop opening it" with no good reasoning or rationale behind it.

-A

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/5cb90d5f-8683-4a34-9c15-f06bb23e6eee%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[James Bennett]

8 years later, I still think we should figure out how to make exclude() do what people expect it to do, rather than implement another lookup type that overlaps with it.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAEE%2BrGopLJaRL6%3D8uafjBoAjkcFO9DuRaDXeBD6M6oWNFoADiQ%40mail.gmail.com.

[Carsten Fuchs]

Hi Marcin,

Can you please tell how the examples at https://code.djangoproject.com/ticket/5763#comment:14  (sorry I gave a wrong comment number in my initial post, comment 14 is the right one) can be implemented with exclude() ?

Oh, and what is SQLAlchemy?   ;-)
(Really, I heard about it, but have never even looked at it.)

Best regards,
Carsten

[Carsten Fuchs]

Hi James,

Am Samstag, 21. November 2015 02:56:45 UTC+1 schrieb James Bennett:

8 years later, I still think we should figure out how to make exclude() do what people expect it to do, rather than implement another lookup type that overlaps with it.

What really confuses me is how exclude() is supposed to make __ne redundant?

Please see the examples at https://code.djangoproject.com/ticket/5763#comment:14  (sorry I gave a wrong comment number in my initial post, comment 14 is the right one). The comment seems to explain clearly how exclude() can not be used where __ne can.

Best regards,
Carsten

[Carsten Fuchs]

Am Samstag, 21. November 2015 02:27:42 UTC+1 schrieb Aaron C. de Bruyn:

With all due respect, looking through the ticket and reading responses shows me that the 'pro' side has good use cases for __ne, and the 'con' side basically says "use exclude" or "a core committer close the ticket, stop opening it" with no good reasoning or rationale behind it.

Dito.

Best regards,
Carsten

[Marten Kenbeek]

The 'con' side argument is that it would create in inconsistency in the API, since we don't have any other negated lookups either. If we can get the same behaviour by fixing the current API, Django should not introduce an unnecessary consistency. ("Closed as wontfix by core dev" is not an argument against the ticket, it is a simple rule that you must discuss such a ticket on the mailing list before reopening it.)

Anyway, with the new lookup API, it has become trivial for any project to implement the __ne lookup. It is the first example in the how-to guide: https://docs.djangoproject.com/en/dev/howto/custom-lookups/#a-simple-lookup-example.

[Carsten Fuchs]

Hi Marten,

Am 2015-11-21 um 11:53 schrieb Marten Kenbeek:
> The 'con' side argument is that it would create in inconsistency in the
> API, since we don't have any other negated lookups either. If we can get
> the same behaviour by fixing the current API, Django should not
> introduce an unnecessary consistency.

I think that this is where the communication problem is:

The "con" side seems to see a problem in the current API regarding
exclude() (personally, about what it is I only have a vague idea from
older linked discussions: NOT(...) vs. "proper" negation) and argues
that fixing the problem solves the issue and avoids inconsistency in the
API brought by the introduction of __ne.

The "pro" side sees a problem in functionality when __ne is lacking, and
cannot see how the "con" side's arguments would help. The problem in
functionality is detailed in
https://code.djangoproject.com/ticket/5763#comment:14, which essentially
quotes (possibly an older version of)
https://docs.djangoproject.com/en/1.8/topics/db/queries/#spanning-multi-valued-relationships
and shows a related example where exclude() cannot replace __ne. As we/I
cannot see how fixing/changing exclude() would help in this situation,
we keep asking about __ne. ;-)

In summary, the imaginary query of comment 14

Blog.objects.filter(entry__tag__name='django',
entry__author_count__ne=2)

seems quasi not be possible to implement in today's Django (but probably
with your suggestion below).

> Anyway, with the new lookup API, it has become trivial for any project
> to implement the __ne lookup. It is the first example in the how-to
> guide: https://docs.djangoproject.com/en/dev/howto/custom-lookups/#a-simple-lookup-example.

Thanks, I'll definitively check and try this out, I didn't know it before!

Best regards,
Carsten

[Anssi Kääriäinen]

On Saturday, November 21, 2015, Carsten Fuchs <carste...@cafu.de> wrote:

Hi Marten,

Am 2015-11-21 um 11:53 schrieb Marten Kenbeek:

The 'con' side argument is that it would create in inconsistency in the
API, since we don't have any other negated lookups either. If we can get
the same behaviour by fixing the current API, Django should not
introduce an unnecessary consistency.

I think that this is where the communication problem is:

The "con" side seems to see a problem in the current API regarding exclude() (personally, about what it is I only have a vague idea from older linked discussions: NOT(...) vs. "proper" negation) and argues that fixing the problem solves the issue and avoids inconsistency in the API brought by the introduction of __ne.

The "pro" side sees a problem in functionality when __ne is lacking, and cannot see how the "con" side's arguments would help. The problem in functionality is detailed in https://code.djangoproject.com/ticket/5763#comment:14, which essentially quotes (possibly an older version of) https://docs.djangoproject.com/en/1.8/topics/db/queries/#spanning-multi-valued-relationships and shows a related example where exclude() cannot replace __ne. As we/I cannot see how fixing/changing exclude() would help in this situation, we keep asking about __ne.  ;-)

In summary, the imaginary query of comment 14

    Blog.objects.filter(entry__tag__name='django', entry__author_count__ne=2)

This isn't a real query. There isn't a field author_count, the query needs an annotation somewhere. So, I don't think this argument is convincing without the annotation (note that the place of the annotation matters). In addition, providing example data and the expected results would be great, too.

The discussion seems to miss a real definition of what exactly the ne lookup should do. There are two ways to implement ne, one is as a complement of exact, another is as the != operator. In SQL the first one is "col != val OR col IS NULL", the latter one is just "col != val".

The complement is already achieved with .exclude(col__exact=val), so we don't need a duplicate API for that. But we don't have any way of doing the != operator alone for nullable fields, and I don't see a fundamental reason why we couldn't add it. The biggest problem is that the new __ne lookup isn't actually a complement of __exact, and that could be confusing. But this is the case for lt and gte, too.

The underlying problem here is SQL's three-valued boolean logic. Unsuprisingly, this feature of SQL causes a lot of confusion.

- Anssi

seems quasi not be possible to implement in today's Django (but probably with your suggestion below).

Anyway, with the new lookup API, it has become trivial for any project
to implement the __ne lookup. It is the first example in the how-to
guide: https://docs.djangoproject.com/en/dev/howto/custom-lookups/#a-simple-lookup-example.

Thanks, I'll definitively check and try this out, I didn't know it before!

Best regards,
Carsten

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/565054EA.40508%40cafu.de.

[Carsten Fuchs]

Hi Anssi,

Am 2015-11-21 um 12:50 schrieb Anssi Kääriäinen:
> In summary, the imaginary query of comment 14
>
> Blog.objects.filter(entry__tag__name='django',
> entry__author_count__ne=2)
>
>
> This isn't a real query. There isn't a field author_count, the query
> needs an annotation somewhere. So, I don't think this argument is
> convincing without the annotation (note that the place of the annotation
> matters). In addition, providing example data and the expected results
> would be great, too.

Well, yes, this is a fictional example, but if you replace author_count
by some valid field that doesn't require an annotation, e.g.
author__name, the (imaginary) query should be valid (for the purpose of
demonstrating the problem)?

The key issue is really this, quoted from the linked Django documentation:

> "To handle both of these situations, Django has a consistent way of processing
> filter() calls. Everything inside a single filter() call is applied
> simultaneously to filter out items matching all those requirements. Successive
> filter() calls further restrict the set of objects, but for multi-valued
> relations, they apply to any object linked to the primary model, not
> necessarily those objects that were selected by an earlier filter() call."

That is, sometimes we *have* to put several filters into a single
filter() call to obtain the desired set. If such a situation requires a
negation, exclude() cannot help, because "[...], they apply to *any*
object linked to the primary model, not necessarily those objects that
were selected by an earlier filter() call".

> The discussion seems to miss a real definition of what exactly the ne
> lookup should do. There are two ways to implement ne, one is as a
> complement of exact, another is as the != operator. In SQL the first one
> is "col != val OR col IS NULL", the latter one is just "col != val".

Thanks for pointing this out, I wasn't aware of this (in this context)
before. It seems to be another facet in the overall problem, but
independent from the above, isn't it? (In my normal, "non-negated"
queries, where required I account for NULLs explicitly all the time...)

Best regards,
Carsten

[Aaron C. de Bruyn]

I hadn't come across custom lookups in the docs.  Thanks for the explanation and a fix Marten, I appreciate it.

-A

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/8d31a6bf-f435-4579-936b-3206c2c7c2e7%40googlegroups.com.

[bliy...@rentlytics.com]

I have to say, the lack of __ne is very inconsistent for people new to django, but veterans at sql.  It's just one more frustration point when you're trying to learn a new ORM.

-Ben

[Anssi Kääriäinen]

On Saturday, November 21, 2015, Carsten Fuchs <carste...@cafu.de> wrote:

Hi Anssi,

Am 2015-11-21 um 12:50 schrieb Anssi Kääriäinen:

    In summary, the imaginary query of comment 14

         Blog.objects.filter(entry__tag__name='django',
    entry__author_count__ne=2)


This isn't a real query. There isn't a field author_count, the query
needs an annotation somewhere. So, I don't think this argument is
convincing without the annotation (note that the place of the annotation
matters). In addition, providing example data and the expected results
would be great, too.

Well, yes, this is a fictional example, but if you replace author_count by some valid field that doesn't require an annotation, e.g. author__name, the (imaginary) query should be valid (for the purpose of demonstrating the problem)?

The author_count name suggested this was an aggregation. If this is just a regular field, then things are a bit simpler. Note that negated Q-object + filter and exclude() queries are the same thing.

The key issue is really this, quoted from the linked Django documentation:

"To handle both of these situations, Django has a consistent way of processing
filter() calls. Everything inside a single filter() call is applied
simultaneously to filter out items matching all those requirements. Successive
filter() calls further restrict the set of objects, but for multi-valued
relations, they apply to any object linked to the primary model, not
necessarily those objects that were selected by an earlier filter() call."

That is, sometimes we *have* to put several filters into a single filter() call to obtain the desired set. If such a situation requires a negation, exclude() cannot help, because "[...], they apply to *any* object linked to the primary model, not necessarily those objects that were selected by an earlier filter() call".

There is a fix for exactly this issue in pr https://github.com/django/django/pull/4385. After the pr, you could just use .filter(Q(entry__tags__name='django')&~Q(entry__author_count=2)).

The discussion seems to miss a real definition of what exactly the ne
lookup should do. There are two ways to implement ne, one is as a
complement of exact, another is as the != operator. In SQL the first one
is "col != val OR col IS NULL", the latter one is just "col != val".

Thanks for pointing this out, I wasn't aware of this (in this context) before. It seems to be another facet in the overall problem, but independent from the above, isn't it? (In my normal, "non-negated" queries, where required I account for NULLs explicitly all the time...)

Yeah. We should correct negated filtering to work as documented when combined with other conditions.

So, to fix the issue in comment 14 of the ticket, the above mentioned PR is the right fix. If you want the SQL != operator, then we need a new lookup.

 - Anssi

 

Best regards,
Carsten

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/56506174.4050609%40cafu.de.

[Carsten Fuchs]

Dear Anssi,

Am 2015-11-22 um 13:53 schrieb Anssi Kääriäinen:
> The author_count name suggested this was an aggregation. If this is just
> a regular field, then things are a bit simpler. Note that negated
> Q-object + filter and exclude() queries are the same thing.
>

> [...]

>
> There is a fix for exactly this issue in pr
> https://github.com/django/django/pull/4385. After the pr, you could just
> use .filter(Q(entry__tags__name='django')&~Q(entry__author_count=2)).

Sorry for my late reply, and many thanks for yours, it's very much
appreciated! :-)

Best regards,
Carsten