[Django] #34538: Incorrect query generated with on subquery WHERE depending on the order of the Q() objects

[Django]

#34538: Incorrect query generated with on subquery WHERE depending on the order of
the Q() objects
-------------------------------------+-------------------------------------
Reporter: Alex | Owner: nobody
Type: Bug | Status: new
Component: Database | Version: 4.2
layer (models, ORM) |
Severity: Normal | Keywords:
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
I have reproduced this error on
Django 4.1.9 and Mysql 5.7.
Django 4.2.1 and Mysql 8.0, Postgres 14.1 and Sqlite 3.

Models:
{{{
#!python
from django.db import models

class Child(models.Model):
pass

class ParentLink(models.Model):
enabled = models.BooleanField(db_index=True)

class ChildLink(models.Model):
enabled = models.BooleanField(db_index=True)
origin_child = models.ForeignKey(
Child,
on_delete=models.PROTECT,
related_name='origin_children',
)
parent_link = models.ForeignKey(
ParentLink,
on_delete=models.PROTECT,
related_name='child_links',
)
}}}

Data
{{{
#!python
Child.objects.create(id=1)
Child.objects.create(id=2)

ParentLink.objects.create(id=1, enabled=True)
ParentLink.objects.create(id=2, enabled=True)

ChildLink.objects.create(id=1, enabled=True, origin_child_id=1,
parent_link_id=1)
ChildLink.objects.create(id=2, enabled=True, origin_child_id=2,
parent_link_id=2)
ChildLink.objects.create(id=3, enabled=True, origin_child_id=2,
parent_link_id=1)
ChildLink.objects.create(id=4, enabled=True, origin_child_id=1,
parent_link_id=2)
}}}

This code generates the correct SQL query and returns the correct results
(0 results for the test data)
{{{
#!python
(
ParentLink.objects
.filter(
~Q(child_links__origin_child_id=1) |
Q(child_links__origin_child_id=1, child_links__enabled=False),
enabled=True
)
)
}}}
SQL query generated in Postgres
{{{
#!sql
SELECT "testquery_parentlink"."id", "testquery_parentlink"."enabled" FROM
"testquery_parentlink" LEFT OUTER JOIN "testquery_childlink" ON
("testquery_parentlink"."id" = "testquery_childlink"."parent_link_id")
WHERE ((NOT (EXISTS(SELECT 1 AS "a" FROM "testquery_childlink" U1 WHERE
(U1."origin_child_id" = 1 AND U1."parent_link_id" =
("testquery_parentlink"."id")) LIMIT 1)) OR (NOT
"testquery_childlink"."enabled" AND
"testquery_childlink"."origin_child_id" = 1)) AND
"testquery_parentlink"."enabled")
}}}

This one returns incorrect results (Returns both ParentLinks in the test
data)
{{{
#!python
(
ParentLink.objects
.filter(
Q(child_links__origin_child_id=1, child_links__enabled=False) |
~Q(child_links__origin_child_id=1),
enabled=True
)
)
}}}

SQL query generated in Postgres
{{{
#!sql
SELECT "testquery_parentlink"."id", "testquery_parentlink"."enabled" FROM
"testquery_parentlink" LEFT OUTER JOIN "testquery_childlink" ON
("testquery_parentlink"."id" = "testquery_childlink"."parent_link_id")
WHERE (((NOT "testquery_childlink"."enabled" AND
"testquery_childlink"."origin_child_id" = 1) OR NOT (EXISTS(SELECT 1 AS
"a" FROM "testquery_childlink" U1 WHERE (U1."origin_child_id" = 1 AND
U1."id" = ("testquery_childlink"."id") AND
"testquery_childlink"."parent_link_id" = ("testquery_parentlink"."id"))
LIMIT 1))) AND "testquery_parentlink"."enabled")
}}}

The WHERE in the subquery in the second case is incorrect.

--
Ticket URL: <https://code.djangoproject.com/ticket/34538>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#34538: Incorrect query generated with on subquery WHERE depending on the order of
the Q() objects
-------------------------------------+-------------------------------------
Reporter: Alex | Owner: nobody
Type: Bug | Status: new

Component: Database layer | Version: 4.2
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by David Sanders):

* stage: Unreviewed => Accepted

Comment:

Thanks for the report 👍

--
Ticket URL: <https://code.djangoproject.com/ticket/34538#comment:1>

[Django]

#34538: Incorrect query generated with on subquery WHERE depending on the order of
the Q() objects
-------------------------------------+-------------------------------------
Reporter: Alex | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: 4.2
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Old description:

> class Child(models.Model):
> pass
>

New description:

class Child(models.Model):
pass

Data
{{{
#!python
Child.objects.create(id=1)
Child.objects.create(id=2)

ParentLink.objects.create(id=1, enabled=True)
ParentLink.objects.create(id=2, enabled=True)

WHERE ((NOT (exists
(SELECT 1 AS "a"
FROM "testquery_childlink" u1
WHERE (u1."origin_child_id" = 1
AND u1."parent_link_id" =

OR NOT (exists
(SELECT 1 AS "a"
FROM "testquery_childlink" u1
WHERE (u1."origin_child_id" = 1
AND u1."id" = ("testquery_childlink"."id")

AND "testquery_childlink"."parent_link_id" =
("testquery_parentlink"."id"))
LIMIT 1)))
AND "testquery_parentlink"."enabled")
}}}

The WHERE in the subquery in the second case is incorrect.

--

Comment (by David Sanders):

(formatting sql in description to make it a little clearer for others)

--
Ticket URL: <https://code.djangoproject.com/ticket/34538#comment:2>

[Django]

#34538: Incorrect query generated with on subquery WHERE depending on the order of
the Q() objects
-------------------------------------+-------------------------------------
Reporter: Alex | Owner: nobody

Type: Bug | Status: closed

Component: Database layer | Version: 4.2
(models, ORM) |

Severity: Normal | Resolution: duplicate
Keywords: | Triage Stage:
| Unreviewed

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Mariusz Felisiak):

* status: new => closed
* resolution: => duplicate
* stage: Accepted => Unreviewed

Comment:

I'm pretty sure it's a duplicate of #25245.

--
Ticket URL: <https://code.djangoproject.com/ticket/34538#comment:3>

[Django]

#34538: Incorrect query generated with on subquery WHERE depending on the order of
the Q() objects
-------------------------------------+-------------------------------------
Reporter: Alex | Owner: nobody
Type: Bug | Status: closed
Component: Database layer | Version: 4.2
(models, ORM) |
Severity: Normal | Resolution: duplicate
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Hana Burtin):

Looks like the issue is the priorities of operators in postgres:

{{{
postgres=# select false and false or true;
?column?
t
}}}


{{{
postgres=# select false and true or false;
?column?
f
}}}

And I think we are expecting the same operation priority as python:

{{{
>>> 1 & 0 | 1
<<< 1
>>> 1 & 1 | 0
<<< 1
}}}

To make things simpler: in python OR operators (`|` and `or`) are
evaluated before AND operators (`&` and `and`), while for postgres they
have the same priority order.

To solve this issue and similar ones, we might want to always add brackets
around `OR` operator.
--
Ticket URL: <https://code.djangoproject.com/ticket/34538#comment:4>