[Django] #24525: AssertionError at `Query.change_aliases`
Django
----------------------------------------------+--------------------
Reporter: coolRR | Owner: nobody
Type: Bug | Status: new
Component: Database layer (models, ORM) | Version: 1.7
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------------------+--------------------
(Django version 1.7.7)
I'm getting an assertion error on the first line of
`Query.change_aliases`:
assert
set(change_map.keys()).intersection(set(change_map.values())) == set()
Problem is, I can't post more data, because the queryset has confidential
client information :(
I can post the value of `change_map`, with identifying details renamed:
{'T5': 'T10',
'T6': 'T11',
'T8': 'T13',
'my_app_follow': 'my_app_follow',
'my_app_stack__readers': 'T7',
'my_app_stack__writers': 'T9',
'my_app_user': 'T8'}
This happened when doing `&` on two querysets, one of which having
`.distinct()` applied on it, among other things.
--
Ticket URL: <https://code.djangoproject.com/ticket/24525>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
-------------------------------------+-------------------------------------
Reporter: coolRR | Owner: nobody
Type: Bug | Status: new
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by coolRR):
* needs_better_patch: => 0
* needs_tests: => 0
* needs_docs: => 0
Old description:
> (Django version 1.7.7)
>
> I'm getting an assertion error on the first line of
> `Query.change_aliases`:
>
> assert
> set(change_map.keys()).intersection(set(change_map.values())) == set()
>
> Problem is, I can't post more data, because the queryset has confidential
> client information :(
>
> I can post the value of `change_map`, with identifying details renamed:
>
> {'T5': 'T10',
> 'T6': 'T11',
> 'T8': 'T13',
> 'my_app_follow': 'my_app_follow',
> 'my_app_stack__readers': 'T7',
> 'my_app_stack__writers': 'T9',
> 'my_app_user': 'T8'}
>
> This happened when doing `&` on two querysets, one of which having
> `.distinct()` applied on it, among other things.
New description:
(Django version 1.7.7)
I'm getting an assertion error on the first line of
`Query.change_aliases`:
{{{
assert
set(change_map.keys()).intersection(set(change_map.values())) == set()
}}}
Problem is, I can't post more data, because the queryset has confidential
client information :(
I can post the value of `change_map`, with identifying details renamed:
{{{
{'T5': 'T10',
'T6': 'T11',
'T8': 'T13',
'my_app_follow': 'my_app_follow',
'my_app_stack__readers': 'T7',
'my_app_stack__writers': 'T9',
'my_app_user': 'T8'}
}}}
This happened when doing `&` on two querysets, one of which having
`.distinct()` applied on it, among other things.
--
--
Ticket URL: <https://code.djangoproject.com/ticket/24525#comment:1>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: new
(models, ORM) |
Severity: Normal | Resolution:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by timgraham):
Couldn't you change the names of the models and values of the data and
offer a test case to reproduce the error?
--
Ticket URL: <https://code.djangoproject.com/ticket/24525#comment:2>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: new
(models, ORM) |
Severity: Normal | Resolution:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by coolRR):
These querysets are passed through several functions in different files so
it'll be hard to construct them in straight code. It'll be about an hour
of work for me which is too much.
--
Ticket URL: <https://code.djangoproject.com/ticket/24525#comment:3>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: closed
Component: Database layer | Version: 1.7
(models, ORM) |
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* status: new => closed
* resolution: => needsinfo
Comment:
Well, trying to reproduce this from the provided details will likely take
much longer than an hour.
--
Ticket URL: <https://code.djangoproject.com/ticket/24525#comment:4>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: new
(models, ORM) |
Severity: Normal | Resolution:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* status: closed => new
* resolution: needsinfo =>
Comment:
Reporter provided the SQL from the queries that are being ANDed. Maybe
this is enough to be helpful to an ORM expert; reopening on that basis.
{{{
SELECT "soufle_app_stack"."id", "soufle_app_stack"."datetime_created",
"soufle_app_stack"."datetime_updated", "soufle_app_stack"."_name",
"soufle_app_stack"."owner_id", "soufle_app_stack"."is_public_reading",
"soufle_app_stack"."is_discussion_stack",
"soufle_app_stack"."datetime_canceled" FROM "soufle_app_stack" LEFT OUTER
JOIN "soufle_app_stack__readers" ON ( "soufle_app_stack"."id" =
"soufle_app_stack__readers"."stack_id" ) LEFT OUTER JOIN
"soufle_app_stack__writers" ON ( "soufle_app_stack"."id" =
"soufle_app_stack__writers"."stack_id" ) WHERE
("soufle_app_stack"."datetime_canceled" IS NULL AND
("soufle_app_stack"."is_public_reading" = True OR
"soufle_app_stack__readers"."user_id" = 147 OR
"soufle_app_stack__writers"."user_id" = 147 OR
"soufle_app_stack"."owner_id" = 147)) ORDER BY
"soufle_app_stack"."datetime_created" DESC
SELECT "soufle_app_stack"."id", "soufle_app_stack"."datetime_created",
"soufle_app_stack"."datetime_updated", "soufle_app_stack"."_name",
"soufle_app_stack"."owner_id", "soufle_app_stack"."is_public_reading",
"soufle_app_stack"."is_discussion_stack",
"soufle_app_stack"."datetime_canceled" FROM "soufle_app_stack" LEFT OUTER
JOIN "soufle_app_stack__readers" ON ( "soufle_app_stack"."id" =
"soufle_app_stack__readers"."stack_id" ) LEFT OUTER JOIN
"soufle_app_stack__writers" ON ( "soufle_app_stack"."id" =
"soufle_app_stack__writers"."stack_id" ) INNER JOIN "soufle_app_follow" ON
( "soufle_app_stack"."id" = "soufle_app_follow"."stack_id" ) WHERE
("soufle_app_stack"."datetime_canceled" IS NULL AND
("soufle_app_stack"."is_public_reading" = True OR
"soufle_app_stack__readers"."user_id" = 147 OR
"soufle_app_stack__writers"."user_id" = 147 OR
"soufle_app_stack"."owner_id" = 147) AND NOT ("soufle_app_stack"."id" IN
(SELECT U1."private_stack_id" AS "private_stack_id" FROM "soufle_app_user"
U1 WHERE (U1."id" = 147 AND U1."private_stack_id" IS NOT NULL))) AND NOT
("soufle_app_stack"."id" IN (SELECT U1."public_stack_id" AS
"public_stack_id" FROM "soufle_app_user" U1 WHERE (U1."id" = 147 AND
U1."public_stack_id" IS NOT NULL))) AND "soufle_app_follow"."user_id" =
147) ORDER BY "soufle_app_stack"."datetime_created" DESC
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/24525#comment:5>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: new
(models, ORM) |
Severity: Normal | Resolution:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by coolRR):
Can anyone possibly help with this? I'm stuck on this problem and I don't
know how to solve it.
--
Ticket URL: <https://code.djangoproject.com/ticket/24525#comment:6>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: new
(models, ORM) |
Severity: Normal | Resolution:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by jarshwah):
Can you try your queryset out with django 1.8 please? I've tried to
emulate the idea by combining two querysets, one that has distinct
applied, and that behaviour is not allowed (in 1.8):
{{{
In [9]: Company.objects.filter(name='HI').distinct() &
Company.objects.exclude(name='HI')
---------------------------------------------------------------------------
AssertionError Traceback (most recent call
last)
<ipython-input-9-b0c9de44f4df> in <module>()
----> 1 Company.objects.filter(name='HI').distinct() &
Company.objects.exclude(name='HI')
/Users/smeatonj/Development/django/django/db/models/query.py in
__and__(self, other)
305 combined = self._clone()
306 combined._merge_known_related_objects(other)
--> 307 combined.query.combine(other.query, sql.AND)
308 return combined
309
/Users/smeatonj/Development/django/django/db/models/sql/query.py in
combine(self, rhs, connector)
494 "Cannot combine queries once a slice has been taken."
495 assert self.distinct == rhs.distinct, \
--> 496 "Cannot combine a unique query with a non-unique
query."
497 assert self.distinct_fields == rhs.distinct_fields, \
498 "Cannot combine queries with different distinct
fields."
AssertionError: Cannot combine a unique query with a non-unique query.
}}}
So if you hit the same problem as I did in 1.8, then there's little
motivation to try to investigate the cause of your original problem in 1.7
since it's explicitly disallowed in 1.8. I would venture that the reason
it's disallowed in 1.8 (assuming it was "valid" in 1.7) is for similar
reasons that you're seeing now.
The SQL dump provided above doesn't appear to show any distinct clauses,
but your original problem description did. We don't have enough
information here to figure out the problem.
If you could at least provide the stacktrace when the assert is hit, that
might help narrow it down. But, honestly, if this is important enough to
your application then you should spend the time needed to create a
reproducible series of steps. We help where we can, but I think you'd be
hard pressed to find someone willing to try combination after combination
trying to arrive at a similar situation that you're in because it'd take
you "too much time".
--
Ticket URL: <https://code.djangoproject.com/ticket/24525#comment:7>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: new
(models, ORM) |
Severity: Normal | Resolution:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by coolRR):
Hi Josh,
Thanks for your help.
- Turns out I was wrong about the `.distinct` thing, it's to be applied
''after'' the failing code and not before it, so it's not relevant. That
was my mistake. In other words `.distinct` is definitely not used in the
failing query, so we can't expect to get the "Cannot combine a unique
query with a non-unique query" error.
- I tried to run my code on Django 1.8, and got the exact same error.
- Here is a traceback of the error: http://i.imgur.com/nM748ne.jpg The
frame just above the one visible is the frame in which I do `&` between
two querysets. (Didn't include these frames because they have client
code.) (This traceback was done on Django 1.8, if you want the same on
Django 1.7.7 I can do that too.) If you'll want me to expand any of the
"Local vars" sections, let me know.
Also, if you're interested enough and have the time to do a VNC or
TeamViewer session to my dev machine, I can show you the whole thing. Of
course, I'll totally understand if you're too busy. If so I'll try to do a
sample app. (Trying to avoid billing my client for the hours that this
will take.)
Thanks,
Ram.
--
Ticket URL: <https://code.djangoproject.com/ticket/24525#comment:8>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: new
(models, ORM) |
Severity: Normal | Resolution:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by jarshwah):
It's not likely that just seeing the problem is going to be enough to
reproduce on my own. I sympathise with your situation, but you're trying
to avoid billing your client by palming the work off for free to open
source contributors. We're not tier 3 support. You should (IMO) be billing
for bug fixes in situations like this.
I'm happy to meet you half way. If you can put a test case together then
we can work on fixing the issue.
Cheers
--
Ticket URL: <https://code.djangoproject.com/ticket/24525#comment:9>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: closed
Component: Database layer | Version: 1.7
(models, ORM) |
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* status: new => closed
* resolution: => needsinfo
--
Ticket URL: <https://code.djangoproject.com/ticket/24525#comment:10>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: closed
Component: Database layer | Version: 1.7
(models, ORM) |
Severity: Normal | Resolution: needsinfo
| Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by kevmitch):
* Attachment "and_assertion_error.zip" added.
minimal example project exhibiting the error
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: closed
Component: Database layer | Version: 1.7
(models, ORM) |
Severity: Normal | Resolution: needsinfo
| Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by kevmitch):
In Django 1.8, with the models
{{{
#!python
from django.db import models
class Year(models.Model):
date = models.DateField(auto_now_add=True)
class Student(models.Model):
year = models.ForeignKey('Year')
number = models.PositiveIntegerField()
courses = models.ManyToManyField('Course')
class Course(models.Model):
year = models.ForeignKey('Year')
department = models.CharField(max_length=4)
number = models.PositiveIntegerField()
}}}
the following
{{{
#!python
course_instance.student_set.all() &
year_instance.student_set.exclude(courses__in=[other_course_instance])
}}}
will raise
{{{
Traceback (most recent call last):
File
"/home/kevmitch/projects/apprentice/and_assertion_error/enrollment/tests.py",
line 24, in test_and
result = math101.student_set.all() &
year.student_set.exclude(courses__in=[anth100])
File
"/home/kevmitch/projects/apprentice/apprentice/env/local/lib/python2.7
/site-packages/django/db/models/query.py", line 211, in __and__
combined.query.combine(other.query, sql.AND)
File
"/home/kevmitch/projects/apprentice/apprentice/env/local/lib/python2.7
/site-packages/django/db/models/sql/query.py", line 556, in combine
w.relabel_aliases(change_map)
File
"/home/kevmitch/projects/apprentice/apprentice/env/local/lib/python2.7
/site-packages/django/db/models/sql/where.py", line 285, in
relabel_aliases
child.relabel_aliases(change_map)
File
"/home/kevmitch/projects/apprentice/apprentice/env/local/lib/python2.7
/site-packages/django/db/models/sql/where.py", line 287, in
relabel_aliases
self.children[pos] = child.relabeled_clone(change_map)
File
"/home/kevmitch/projects/apprentice/apprentice/env/local/lib/python2.7
/site-packages/django/db/models/lookups.py", line 185, in relabeled_clone
new.rhs = new.rhs.relabeled_clone(relabels)
File
"/home/kevmitch/projects/apprentice/apprentice/env/local/lib/python2.7
/site-packages/django/db/models/sql/query.py", line 329, in
relabeled_clone
clone.change_aliases(change_map)
File
"/home/kevmitch/projects/apprentice/apprentice/env/local/lib/python2.7
/site-packages/django/db/models/sql/query.py", line 787, in change_aliases
assert set(change_map.keys()).intersection(set(change_map.values()))
== set()
AssertionError
}}}
The tests of attached project demonstrates it in action.
Essential elements for reproduction:
* One of the querysets generated from related manager of the ManyToMany
target
* The other queryset is generated by exclude()ing on the ManyToManyField
from the related manager of a "versioning" ForeignKey model
--
Ticket URL: <https://code.djangoproject.com/ticket/24525#comment:11>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: new
(models, ORM) |
Severity: Normal | Resolution:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by kevmitch):
* status: closed => new
* resolution: needsinfo =>
--
Ticket URL: <https://code.djangoproject.com/ticket/24525#comment:12>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: new
(models, ORM) |
Severity: Normal | Resolution:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by kevmitch):
* cc: kevmitch (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/24525#comment:13>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: new
(models, ORM) |
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* severity: Normal => Release blocker
* stage: Unreviewed => Accepted
Comment:
Bisected to 01f2cf2aecc932d43b20b55fc19a8fa440457b5f
--
Ticket URL: <https://code.djangoproject.com/ticket/24525#comment:14>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: new
(models, ORM) |
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* has_patch: 0 => 1
Comment:
[https://github.com/django/django/pull/5163 Removing the assertion] seems
to work. Is there a better solution?
--
Ticket URL: <https://code.djangoproject.com/ticket/24525#comment:15>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: new
(models, ORM) |
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* needs_better_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/24525#comment:16>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: closed
Component: Database layer | Version: 1.7
(models, ORM) |
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Changes (by Tim Graham <timograham@…>):
* status: new => closed
* resolution: => fixed
Comment:
In [changeset:"2dc9ec5616a942de3a0886a707f93988f56dd594" 2dc9ec56]:
{{{
#!CommitTicketReference repository=""
revision="2dc9ec5616a942de3a0886a707f93988f56dd594"
Fixed #24525 -- Fixed AssertionError in some complex queries.
Thanks Anssi Kääriäinen for providing the solution.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/24525#comment:17>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: closed
Component: Database layer | Version: 1.7
(models, ORM) |
Severity: Release blocker | Resolution: fixed
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Comment (by Tim Graham <timograham@…>):
In [changeset:"62347208bb0cef6990699d7373fa06e2e12ead07" 62347208]:
{{{
#!CommitTicketReference repository=""
revision="62347208bb0cef6990699d7373fa06e2e12ead07"
[1.8.x] Fixed #24525 -- Fixed AssertionError in some complex queries.
Thanks Anssi Kääriäinen for providing the solution.
Backport of 2dc9ec5616a942de3a0886a707f93988f56dd594 from master
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/24525#comment:18>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: new
(models, ORM) |
Severity: Release blocker | Resolution:
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Changes (by Christian Karrié):
* status: closed => new
* resolution: fixed =>
Comment:
Hi all, unfortunately I have to reopen this issue.
After migrating from Django 1.10.6 to 1.11.9 the error reoccurs:
Code:
{{{
qs_out = qs.filter(*out_filters)
qs_in = qs.filter(*in_filters)
qs_merged = qs_in | qs_out
}}}
Vars:
`in_filters = [<Q: (NOT (AND: ('contract__customer_id', 1)))>, <Q: (NOT
(AND: ('contract__category__deletion', True)))>, <Q: (NOT (AND:
('contract__date_from__lte', datetime.date(2018, 1, 10)),
('contract__category__blocked', True), ('contract__date_until__gte',
datetime.date(2018, 1, 10))))>, <Q: (AND: ('id', 0))>]`
`out_filters = [<Q: (OR: (AND: (NOT (AND: ('contract__source_customer_id',
1))), ('contract__customer_id', 1)), (AND: (NOT (AND:
('contract__source_customer_id', 1))), (NOT (AND:
('contract__customer_id', 1)))))>, <Q: (NOT (AND:
('contract__category__incoming', True)))>]`
It seems to fail, if `qs_in` or `qs_out` is empty. But I'm not sure.
Printing `change_map` gives me:
{{{
{'T7': u'invoice_customer', u'invoice_customer': 'T7'}
}}}
Removing the assert works.
--
Ticket URL: <https://code.djangoproject.com/ticket/24525#comment:19>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: new
(models, ORM) |
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Changes (by Christian Karrié):
* version: 1.7 => 1.11
--
Ticket URL: <https://code.djangoproject.com/ticket/24525#comment:20>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: closed
Component: Database layer | Version: 1.11
(models, ORM) |
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Changes (by Tim Graham):
* status: new => closed
* resolution: => fixed
Comment:
As the fix for this ticket is released, please open a new ticket with
complete details to reproduce. Ideally, you could also
[https://docs.djangoproject.com/en/dev/internals/contributing/triaging-
tickets/#bisecting-a-regression bisect] to find the commit where the
behavior changed.
--
Ticket URL: <https://code.djangoproject.com/ticket/24525#comment:21>