[Django] #35147: Unexcepted behaviour for the filter() method of the QuerySet in version 5.0.1
Django
5.0.1
-------------------------------------+-------------------------------------
Reporter: Darbados | Owner: nobody
Type: Bug | Status: new
Component: Database | Version: 5.0
layer (models, ORM) | Keywords: annotate, bitand,
Severity: Normal | filter
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
Hi team,
I faced an unexpected behaviour of the bitand method of the F model when
used with a BigIntField value.
The change is that when I try to annotate and then filter on the promoted
field, the query set is always empty, when it shouldn't be.
Code is as follows:
{{{
records_qs =
Model.objects.annotate(flag_some_flag=F("flags").bitand(2**34)).filter(flag_some_flag=2**34)
}}}
The returned queryset is empty, but it shouldn't be.
Simple test with using only annotate and further filtration using python
list comprehension from the queryset using the annotated field, returns
the needed records.
The filtration works properly for flag values < {{{ 2**31 }}}.
I was looking for some notes in the Django 5 release notes, but I found
nothing, and this is the version where the problem is faced. With Django
4.2.8 this problem is not faced.
Regards,
Petar Netev
--
Ticket URL: <https://code.djangoproject.com/ticket/35147>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
5.0.1
-------------------------------------+-------------------------------------
Reporter: Darbados | Owner: nobody
Type: Bug | Status: new
(models, ORM) |
Severity: Normal | Resolution:
Keywords: annotate, bitand, | Triage Stage:
filter | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
A clew would be Simon Charette's answer:
I'm fairly certain this relates to a commit meant to address a SQLite
crash and a DDOS vector on Postgres only present in 5.0 so far[0].
I suspect we'll want to adapt `_connector_combinations` so when bitwise
operators are used between integers fields the output field is the wider
of both and not systematically IntegerField.
I suspect the problem manifests itself in other cases as well such as
IntegerField + IntegerField which can overflow into a BigIntegerField.
--
Ticket URL: <https://code.djangoproject.com/ticket/35147#comment:1>
Django
5.0.1
-------------------------------------+-------------------------------------
Reporter: Darbados | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: 5.0
(models, ORM) |
Severity: Normal | Resolution:
Keywords: annotate, bitand, | Triage Stage:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: Simon Charette (added)
* keywords: annotate, bitand, filter => annotate, bitand, filter,
integerfield, overflow
Comment:
Thank you for your report Petar.
The output field resolving logic or the ORM, particularly the combinator
part used by `bitand` and friends, is far from perfect and it is to blame
here as any `IntegerField` subclass
[https://github.com/django/django/blob/9c6d7b4a678b7bbc6a1a14420f686162ba9016f5/django/db/models/expressions.py#L585-L597
combined with another for bitwise connectors] will result in a
`CombinedExpression(output_field=IntegerField())`.
The problem does not only manifests itself in expression combinations
though, functions such as
`django.db.models.functions.Power("integer_field", "integer_field")` can
also overflow.
What makes combinator particularly problematic though is that they cannot
have the user provide an explicit `output_field` like it's the case with
other expressions such as functions. The only solution I can think of with
the current state of things is to use `ExpressionWrapper`
[https://docs.djangoproject.com/en/5.0/ref/models/expressions
/#expressionwrapper-expressions which is documented to be required in this
case]
> `ExpressionWrapper` is necessary when using arithmetic on `F()`
expressions with different types as described in Using `F()` with
annotations.
And from
[https://docs.djangoproject.com/en/5.0/ref/models/expressions/#using-f
-with-annotations the annotation documentation] about the usage of `F`
> If the fields that you’re combining are of different types you’ll need
to tell Django what kind of field will be returned. Most expressions
support output_field for this case, but since `F()` does not, you will
need to wrap the expression with `ExpressionWrapper`:
You haven't provided your model definition but I suspect wrapping
`F("flags").bitand(2**34))` in
`ExpressionWrapper(F("flags").bitand(2**34)), BigIntegerField())` will
address your problem?
If that's the case I'm not against adjusting the 5.0 release notes to
mention that non-wrapped arithmetic against integer fields might require
adjustments, as documented, and possibly accepting a ticket to try to make
the logic more ''generic'' wrt/ to integer field subclasses. Since this is
clearly documented though I'm not convinced this qualifies as a release
blocker that would warrant a code backport.
--
Ticket URL: <https://code.djangoproject.com/ticket/35147#comment:2>
Django
5.0.1
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: Database layer | Version: 5.0
(models, ORM) |
Severity: Normal | Resolution:
Keywords: annotate, bitand, | Triage Stage:
filter, integerfield, overflow | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Hi Simon,
Thanks for your notes.
Indeed using ExpressionWrapper produces the expected behaviour. In our
case we did took another approach usiing the Exact {{{
Exact(F("flags").bitand(flag_value), flag_value) }}}.
It is not a blocker, as you said.
--
Ticket URL: <https://code.djangoproject.com/ticket/35147#comment:3>
Django
5.0.1
-------------------------------------+-------------------------------------
Reporter: Petar Netev | Owner: nobody
Cleanup/optimization |
Component: Database layer | Version: 5.0
(models, ORM) |
Severity: Normal | Resolution:
filter, integerfield, overflow |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* type: Bug => Cleanup/optimization
* stage: Unreviewed => Accepted
Comment:
Given the information shared so far (thanks Simon and Petar!), I agree
this is not a release blocker and that documentation changes could help
future users. Accepting on such basis.
Petar, would you be able/like to prepare a patch for the docs
improvements?
--
Ticket URL: <https://code.djangoproject.com/ticket/35147#comment:4>