[Django] #28101: Wrong field used for sub query lookup on nested query using to_field ForeignKey
Django
ForeignKey
-------------------------------------+-------------------------------------
Reporter: klette | Owner: nobody
Type: Bug | Status: new
Component: Database | Version: 1.11
layer (models, ORM) |
Severity: Normal | Keywords:
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
Given the following models:
{{{#!python
class CustomerUser(models.Model):
name = models.CharField(max_length=100)
class Customer(models.Model):
id = models.UUIDField(primary_key=True, default=uuid.uuid4,
editable=False)
customer_number = models.CharField(unique=True, max_length=100)
users = models.ManyToManyField(CustomerUser)
class CustomerProduct(models.Model):
id = models.UUIDField(primary_key=True, default=uuid.uuid4,
editable=False)
customer = models.ForeignKey(
Customer, to_field='customer_number', on_delete=models.PROTECT)
name = models.CharField(max_length=100)
class CustomerPayment(models.Model):
id = models.UUIDField(primary_key=True, default=uuid.uuid4,
editable=False)
product = models.ForeignKey(CustomerProduct, on_delete=models.PROTECT)
}}}
And the tests:
{{{#!python
class SubQueryM2MWithToFieldFkTests(TestCase):
def test_regression(self):
user = CustomerUser.objects.create(name='my user')
customer = Customer.objects.create(customer_number='A123')
customer.users.add(user)
product = CustomerProduct.objects.create(customer=customer,
name='Foo')
payment = CustomerPayment.objects.create(product=product)
products = CustomerProduct.objects.filter(
customer__in=user.customer_set.all())
result = CustomerPayment.objects.filter(product__in=products)
self.assertEquals(result.count(), 1)
self.assertEquals(list(result), [payment])
}}}
One should get the query:
{{{#!sql
SELECT "queries_customerpayment"."id",
"queries_customerpayment"."product_id"
FROM "queries_customerpayment"
WHERE "queries_customerpayment"."product_id" IN (
SELECT V0."id" AS Col1 FROM "queries_customerproduct" V0
WHERE V0."customer_id" IN (
SELECT U0."customer_number" AS Col1
FROM "queries_customer" U0 INNER JOIN "queries_customer_users" U1 ON
(U0."id" = U1."customer_id")
WHERE U1."customeruser_id" = 1))
}}}
But at least 1.11 and master generates:
{{{#!sql
SELECT "queries_customerpayment"."id",
"queries_customerpayment"."product_id"
FROM "queries_customerpayment"
WHERE "queries_customerpayment"."product_id" IN (
SELECT V0."id" AS Col1 FROM "queries_customerproduct" V0
WHERE V0."customer_id" IN (
SELECT U0."id" AS Col1 FROM "queries_customer" U0 INNER JOIN
"queries_customer_users" U1 ON (U0."id" = U1."customer_id")
WHERE U1."customeruser_id" = 1))
}}}
which attempts to match the wrong `Customer`-field to
`CustomerProduct.customer_id`.
I tried to figure out what was going on and the test passes if I add
`print(self)` to `Queryset._prepare_as_filter_value` in the forced pk
block. Quite arbitrary placement of the print, and probably not related to
`_prepare_as_filter_value`. My guess would be that the evaluation of the
inner queryset fixes some field matching some other place - but I'm way
out of my depth here :)
The tests were added directly to `tests/queries/tests.py` and the models
to `tests/queries/models.py` and run using Django's test suite.
Might be related: [https://code.djangoproject.com/ticket/26196]
--
Ticket URL: <https://code.djangoproject.com/ticket/28101>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
ForeignKey
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: Database layer | Version: 1.11
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Old description:
New description:
Given the following models:
And the tests:
products = CustomerProduct.objects.filter(
customer__in=user.customer_set.all())
Running git bisect found the following change as the first bad one, so
seems like a regression.
{{{
7a2c27112d1f804f75191e9bf45a96a89318a684 is the first bad commit
commit 7a2c27112d1f804f75191e9bf45a96a89318a684
Author: Jani Tiainen <jani.t...@tintti.net>
Date: Wed Aug 31 21:16:39 2016 +0300
Fixed #27159 -- Prevented pickling a query with an __in=inner_qs
lookup from evaluating inner_qs.
}}}
--
--
Ticket URL: <https://code.djangoproject.com/ticket/28101#comment:1>
Django
ForeignKey
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: Database layer | Version: 1.11
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: stein.magnus@… (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/28101#comment:2>
Django
ForeignKey
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: Database layer | Version: 1.11
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Old description:
> Given the following models:
> Running git bisect found the following change as the first bad one, so
> seems like a regression.
>
> {{{
> 7a2c27112d1f804f75191e9bf45a96a89318a684 is the first bad commit
> commit 7a2c27112d1f804f75191e9bf45a96a89318a684
> Author: Jani Tiainen <jani.t...@tintti.net>
> Date: Wed Aug 31 21:16:39 2016 +0300
>
> Fixed #27159 -- Prevented pickling a query with an __in=inner_qs
> lookup from evaluating inner_qs.
> }}}
New description:
Given the following models:
And the tests:
products = CustomerProduct.objects.filter(
customer__in=user.customer_set.all())
Running git bisect found the following change as the first bad one, so
seems like a regression.
{{{
7a2c27112d1f804f75191e9bf45a96a89318a684 is the first bad commit
commit 7a2c27112d1f804f75191e9bf45a96a89318a684
Author: Jani Tiainen <jani.t...@tintti.net>
Date: Wed Aug 31 21:16:39 2016 +0300
Fixed #27159 -- Prevented pickling a query with an __in=inner_qs
lookup from evaluating inner_qs.
}}}
while it was first fixed in:
{{{
commit 46ecfb9b3a11a360724e3375ba78c33c46d6a992
Author: Anssi Kääriäinen <anssi.ka...@thl.fi>
Date: Thu Feb 11 08:39:37 2016 +0200
Fixed #26196 -- Made sure __in lookups use to_field as default.
Thanks Simon Charette for the test.
}}}
--
--
Ticket URL: <https://code.djangoproject.com/ticket/28101#comment:3>
Django
-------------------------------------+-------------------------------------
Reporter: Kristian Klette | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: 1.11
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Old description:
> Given the following models:
> Running git bisect found the following change as the first bad one, so
> seems like a regression.
>
> {{{
> 7a2c27112d1f804f75191e9bf45a96a89318a684 is the first bad commit
> commit 7a2c27112d1f804f75191e9bf45a96a89318a684
> Author: Jani Tiainen <jani.t...@tintti.net>
> Date: Wed Aug 31 21:16:39 2016 +0300
>
> Fixed #27159 -- Prevented pickling a query with an __in=inner_qs
> lookup from evaluating inner_qs.
> }}}
>
> while it was first fixed in:
> {{{
> commit 46ecfb9b3a11a360724e3375ba78c33c46d6a992
> Author: Anssi Kääriäinen <anssi.ka...@thl.fi>
> Date: Thu Feb 11 08:39:37 2016 +0200
>
> Fixed #26196 -- Made sure __in lookups use to_field as default.
>
> Thanks Simon Charette for the test.
>
> }}}
New description:
Given the following models:
And the tests:
products = CustomerProduct.objects.filter(
customer__in=user.customer_set.all())
block. This causes the inner_qs to be evaluated, and thus simulating the
the behavior before 7a2c27112d1f804f75191e9bf45a96a89318a684 was applied.
The tests were added directly to `tests/queries/tests.py` and the models
to `tests/queries/models.py` and run using Django's test suite.
Related: #26196
Running git bisect found the following change as the first bad one, so
seems like a regression.
{{{
7a2c27112d1f804f75191e9bf45a96a89318a684 is the first bad commit
commit 7a2c27112d1f804f75191e9bf45a96a89318a684
Author: Jani Tiainen <jani.t...@tintti.net>
Date: Wed Aug 31 21:16:39 2016 +0300
Fixed #27159 -- Prevented pickling a query with an __in=inner_qs
lookup from evaluating inner_qs.
}}}
while it was first fixed in:
{{{
commit 46ecfb9b3a11a360724e3375ba78c33c46d6a992
Author: Anssi Kääriäinen <anssi.ka...@thl.fi>
Date: Thu Feb 11 08:39:37 2016 +0200
Fixed #26196 -- Made sure __in lookups use to_field as default.
Thanks Simon Charette for the test.
}}}
--
--
Ticket URL: <https://code.djangoproject.com/ticket/28101#comment:4>
Django
-------------------------------------+-------------------------------------
Reporter: Kristian Klette | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: 1.11
(models, ORM) |
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* severity: Normal => Release blocker
* stage: Unreviewed => Accepted
--
Ticket URL: <https://code.djangoproject.com/ticket/28101#comment:5>
Django
-------------------------------------+-------------------------------------
Reporter: Kristian Klette | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: 1.11
(models, ORM) |
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* has_patch: 0 => 1
Comment:
[https://github.com/django/django/pull/8392 PR].
Kristian, could you confirm the proposed patch solves your issue? Thanks.
--
Ticket URL: <https://code.djangoproject.com/ticket/28101#comment:6>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: Database layer | Version: 1.11
(models, ORM) |
Severity: Release blocker | Resolution:
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/28101#comment:7>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: closed
Component: Database layer | Version: 1.11
(models, ORM) |
Keywords: | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: new => closed
* resolution: => fixed
Comment:
In [changeset:"8ef35468b660e1c25af67a8299202b8bc108679f" 8ef35468]:
{{{
#!CommitTicketReference repository=""
revision="8ef35468b660e1c25af67a8299202b8bc108679f"
Fixed #28101 -- Fixed a regression with nested __in subquery lookups and
to_field.
Thanks Kristian Klette for the report and Tim for the help.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/28101#comment:8>
Django
-------------------------------------+-------------------------------------
Reporter: Kristian Klette | Owner: nobody
Type: Bug | Status: closed
Component: Database layer | Version: 1.11
(models, ORM) |
Severity: Release blocker | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Simon Charette <charette.s@…>):
In [changeset:"0ad16934940fdbf0984f5255c7e5fff085593737" 0ad16934]:
{{{
#!CommitTicketReference repository=""
revision="0ad16934940fdbf0984f5255c7e5fff085593737"
[1.11.x] Fixed #28101 -- Fixed a regression with nested __in subquery
lookups and to_field.
Thanks Kristian Klette for the report and Tim for the help.
Backport of 8ef35468b660e1c25af67a8299202b8bc108679f from master
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/28101#comment:9>