[Django] #24705: Exception when negating Q object in annotate
Django
----------------------------------------------+--------------------
Reporter: karyon | Owner: nobody
Type: Bug | Status: new
Component: Database layer (models, ORM) | Version: 1.8
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------------------+--------------------
any of the following
{{{
UserProfile.objects.annotate(is_member=ExpressionWrapper(~Q(groups__name="Group
name"), output_field=BooleanField()))
UserProfile.objects.annotate(is_member=Case(When(~models.Q(groups__name='group
name'), then=Value(False)), default=Value(True),
output_field=models.BooleanField()))
UserProfile.objects.annotate(is_member=Case(When(~Q(groups__name="Group
name"), then=Value(False)), default=Value(True),
output_field=BooleanField()))
}}}
gives the following exception:
{{{
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/usr/local/lib/python3.4/dist-
packages/django/db/models/manager.py", line 127, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/usr/local/lib/python3.4/dist-packages/django/db/models/query.py",
line 794, in annotate
obj.query.add_annotation(annotation, alias, is_summary=False)
File "/usr/local/lib/python3.4/dist-
packages/django/db/models/sql/query.py", line 977, in add_annotation
summarize=is_summary)
File "/usr/local/lib/python3.4/dist-
packages/django/db/models/expressions.py", line 779, in resolve_expression
c.cases[pos] = case.resolve_expression(query, allow_joins, reuse,
summarize, for_save)
File "/usr/local/lib/python3.4/dist-
packages/django/db/models/expressions.py", line 713, in resolve_expression
c.condition = c.condition.resolve_expression(query, allow_joins,
reuse, summarize, False)
File "/usr/local/lib/python3.4/dist-
packages/django/db/models/query_utils.py", line 88, in resolve_expression
clause, _ = query._add_q(self, reuse, allow_joins=allow_joins)
File "/usr/local/lib/python3.4/dist-
packages/django/db/models/sql/query.py", line 1328, in _add_q
current_negated=current_negated, connector=connector,
allow_joins=allow_joins)
File "/usr/local/lib/python3.4/dist-
packages/django/db/models/sql/query.py", line 1177, in build_filter
can_reuse, e.names_with_path)
File "/usr/local/lib/python3.4/dist-
packages/django/db/models/sql/query.py", line 1570, in split_exclude
if alias in can_reuse:
TypeError: argument of type 'NoneType' is not iterable
}}}
removing the ~ "fixes" this.
see https://github.com/fsr-itse/EvaP for full code (a custom UserProfile
is in evaluation.models), but it looks like this also happens with the
default User model.
--
Ticket URL: <https://code.djangoproject.com/ticket/24705>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: karyon | Owner: nobody
Type: Bug | Status: new
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by jarshwah):
* needs_better_patch: => 0
* stage: Unreviewed => Accepted
* needs_tests: => 0
* needs_docs: => 0
Comment:
{{{
UserProfile.objects.annotate(is_member=ExpressionWrapper(~Q(groups__name="Group
name"), output_field=BooleanField()))
}}}
Isn't really supported. Q objects are for filters or case expressions
only. They aren't designed to be used as annotations, even though they
share similar APIs (for Case/When support).
{{{
UserProfile.objects.annotate(is_member=Case(When(~Q(groups__name="Group
name"), then=Value(False)), default=Value(True),
output_field=BooleanField()))
}}}
Probably should be supported. Negating a when clause seems like something
that should be supported, but I haven't yet investigated why this is
throwing errors. For the moment, as a workaround, you can stop negating
the clause and switch the "then" and "default" values to then=True,
default=False I think.
Accepting based on the fact that negated Q objects don't work with case
expressions though.
--
Ticket URL: <https://code.djangoproject.com/ticket/24705#comment:1>
Django
Reporter: karyon | Owner: nobody
Type: Bug | Status: new
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by karyon):
* cc: karyon (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/24705#comment:2>
Django
Reporter: karyon | Owner: nobody
Type: Bug | Status: new
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by MarkusH):
* cc: MarkusH (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/24705#comment:3>
Django
Reporter: karyon | Owner: nobody
Type: Bug | Status: new
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by martsberger):
I am seeing this issue as well and the suggested workaround:
> For the moment, as a workaround, you can stop negating the clause and
switch the "then" and "default" values to then=True, default=False I
think.
Doesn't work for more complex Q objects. For example, a negated Q & a non-
negated Q.
{{{#!python
queryset =
SomeModel.objects.annotate(value=Case(When(Q(some__field=some_value) &
~Q(another__field=another_value))))
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/24705#comment:4>
Django
Reporter: karyon | Owner: nobody
Type: Bug | Status: new
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by akaariai):
I know the reason for this, but can't work on a patch right now. The
reason is that _add_q() calls build_filter() with in_negated=True, and
build_filter() automatically pushes negated multijoin filters to
subqueries. This is correct when constructing a Where clause, but
incorrect for Case(When()).
To fix this in 1.8 we need to add a new kwarg to _add_q(), something like
disable_split_exclude, and add a similar kwarg to build_filter(). When Q()
objects are added to a query outside of filter() handling, we set this
flag to True (this is done in Q().resolve_expression()). This should
hopefully fix this issue.
For 1.9 this will be fixed by https://github.com/django/django/pull/4385.
We can of course add the 1.8 version to master, too, if pull 4385 hasn't
been committed by the time we fix this issue.
We need to also make sure Q() objects added for Case(When()) do not
regenerate aliases. They should use an existing alias, and multiple Q()
objects in the same annotation should use a single alias. So, for example:
{{{
.annotate(v=Case(When(friends__age=123), then=V(1)),
When(friends__age=321), then(V(2)))
}}}
should generate just one join to friends. If a join for friends pre-
exists, then that join should be reused.
I think I can try to work on this next week, but can't guarantee anything.
--
Ticket URL: <https://code.djangoproject.com/ticket/24705#comment:5>
Django
Reporter: karyon | Owner: nobody
Type: Bug | Status: new
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* has_patch: 0 => 1
Comment:
[https://github.com/django/django/pull/4677 PR]
--
Ticket URL: <https://code.djangoproject.com/ticket/24705#comment:6>
Django
Reporter: karyon | Owner: nobody
Type: Bug | Status: new
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* needs_better_patch: 0 => 1
Comment:
Patch looks okay to me, except that tests don't pass on Oracle.
--
Ticket URL: <https://code.djangoproject.com/ticket/24705#comment:7>
Django
Reporter: karyon | Owner: nobody
Type: Bug | Status: closed
Component: Database layer | Version: 1.8
(models, ORM) |
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Changes (by Tim Graham <timograham@…>):
* status: new => closed
* resolution: => fixed
Comment:
In [changeset:"bc87061a3c7c8d6b4d2469f35cc78683c6cff647" bc87061a]:
{{{
#!CommitTicketReference repository=""
revision="bc87061a3c7c8d6b4d2469f35cc78683c6cff647"
Fixed #24705 -- Fixed negated Q objects in expressions.
Avoided split_exclude() for Q when used as an expression.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/24705#comment:8>
Django
Reporter: karyon | Owner: nobody
Type: Bug | Status: closed
Component: Database layer | Version: 1.8
(models, ORM) |
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Comment (by Tim Graham <timograham@…>):
In [changeset:"db656609289b9eaf4661eb73f9318b7c66540301" db65660]:
{{{
#!CommitTicketReference repository=""
revision="db656609289b9eaf4661eb73f9318b7c66540301"
[1.8.x] Fixed #24705 -- Fixed negated Q objects in expressions.
Avoided split_exclude() for Q when used as an expression.
Backport of bc87061a3c7c8d6b4d2469f35cc78683c6cff647 from master
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/24705#comment:9>