[Django] #29244: Passing dict to a Func subclass fails silently on hash step
Django
----------------------------------------+------------------------
Reporter: Rob Jauquet | Owner: nobody
Type: Bug | Status: new
Component: Core (Other) | Version: 2.0
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
----------------------------------------+------------------------
With this code:
{{{#!python
from django.contrib.postgres.search import SearchQuery, Value, Func
from django.core.paginator import Paginator
from django.db.models import F, TextField, DateTimeField, Models
from django.utils import timezone
class Entry(Model):
text = TextField()
published_on = DateTimeField(default=timezone.now())
class Headline(Func):
function = 'ts_headline'
output_field = TextField()
def __init__(self, text, query, options=None):
extra = {}
expressions = [text, query]
if options:
opt_str = ''
for key, value in options.items():
opt_str += f'{key}={value},'
expressions.append(Value(opt_str[:-1]))
super().__init__(*expressions, **extra)
entries = Entry.objects.annotate(
text_highlighted=Headline(
F('text'),
SearchQuery('house'),
options={'HighlightAll': False},
)
).order_by('-published_on')
paginator = Paginator(entries, 20)
# evaluates the queryset
paginator.page(1)
}}}
Passing options by {{{dict}}} to the {{{Func}}} subclass causes the count
step for pagination to fail silently in paginator.py line 85 [#link1 (1)]
which is in turn caused by a silenced TypeError during the hash in
expressions.py line 372 [#link2 (2)].
Then, because of the fallback to {{{len}}} on paginator.py line 90 [#link3
(3)] the entire queryset is evaluated instead of evaluating just the
specified page (and in this specific case, calling ts_headline on every
row instead of just the first page).
The correct way of passing options using {{{**options}}} instead of
{{{options=None}}} fixes this case, but the silent error and fallback to
calling {{{len}}} on a queryset causes an unexpected expensive query when
using pagination.
[=#link1 (1)]
https://github.com/django/django/blob/master/django/core/paginator.py#L85
[=#link2 (2)]
https://github.com/django/django/blob/master/django/db/models/expressions.py#L372
[=#link3 (3)]
https://github.com/django/django/blob/master/django/core/paginator.py#L90
--
Ticket URL: <https://code.djangoproject.com/ticket/29244>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: Rob Jauquet | Owner: nobody
Type: Bug | Status: new
Component: Core (Other) | Version: 2.0
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Tim Graham):
Do you have a proposal of how to address this? By the way, if you press
the "y" key when viewing a GitHub page, you can get a link that includes a
commit hash. Without that, the links can quickly go stale as line numbers
change.
--
Ticket URL: <https://code.djangoproject.com/ticket/29244#comment:1>
Django
Reporter: Rob Jauquet | Owner: nobody
Type: Bug | Status: new
Component: Core (Other) | Version: 2.0
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Description changed by Rob Jauquet:
Old description:
New description:
With this code:
paginator = Paginator(entries, 20)
[=#link1 (1)]
https://github.com/django/django/blob/281c0223b376d6fa1a11e0726d824ed35cfe7524/django/core/paginator.py#L85
[=#link2 (2)]
https://github.com/django/django/blob/281c0223b376d6fa1a11e0726d824ed35cfe7524/django/db/models/expressions.py#L372
[=#link3 (3)]
https://github.com/django/django/blob/281c0223b376d6fa1a11e0726d824ed35cfe7524/django/core/paginator.py#L90
--
--
Ticket URL: <https://code.djangoproject.com/ticket/29244#comment:2>
Django
Reporter: Rob Jauquet | Owner: nobody
Type: Bug | Status: new
Component: Core (Other) | Version: 2.0
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Rob Jauquet):
Thanks for the tip. I've updated the links.
For my use case, it would make sense to not call {{{len}}} on a queryset
in paginator.py and instead surface the exception. The {{{len}}} fallback
could be used only if the thing passed in is not a queryset.
--
Ticket URL: <https://code.djangoproject.com/ticket/29244#comment:3>
Django
--------------------------------------+------------------------------------
Reporter: Rob Jauquet | Owner: nobody
Component: Core (Other) | Version: 2.0
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham):
* type: Bug => Cleanup/optimization
* stage: Unreviewed => Accepted
Comment:
It looks like the best solution would be to remove the try/except in
`Paginator.count()`. `AttributeError` can be removed with
`hasattr(self.object_list, 'count')` and `TypeError` with some checking
that count is a method without any arguments. Perhaps a new function in
`django/utils/inspect.py` would help with that.
--
Ticket URL: <https://code.djangoproject.com/ticket/29244#comment:4>
Django
--------------------------------------+------------------------------------
Component: Core (Other) | Version: 2.0
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham):
* has_patch: 0 => 1
Comment:
[https://github.com/django/django/pull/10270 PR]
--
Ticket URL: <https://code.djangoproject.com/ticket/29244#comment:5>
Django
--------------------------------------+------------------------------------
Component: Core (Other) | Version: 2.0
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Tim Graham <timograham@…>):
In [changeset:"f1bf069ec1a3d4761ab03ccae00961bc3f2aea8a" f1bf069]:
{{{
#!CommitTicketReference repository=""
revision="f1bf069ec1a3d4761ab03ccae00961bc3f2aea8a"
Refs #29244 -- Fixed django.utils.inspect.method_has_no_args() for bound
methods.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/29244#comment:6>
Django
--------------------------------------+------------------------------------
Component: Core (Other) | Version: 2.0
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham <timograham@…>):
* status: new => closed
* resolution: => fixed
Comment:
In [changeset:"3767c7ff391d5f277e25bca38ef3730ddf9cea9c" 3767c7f]:
{{{
#!CommitTicketReference repository=""
revision="3767c7ff391d5f277e25bca38ef3730ddf9cea9c"
Fixed #29244 -- Prevented Paginator.count() from silencing TypeError and
AttributeError.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/29244#comment:7>