[Django] #28219: Paginator warning about unordered object should include stack trace
Django
------------------------------------------+------------------------
Reporter: Denise Mauldin | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 1.11
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
------------------------------------------+------------------------
I just updated from 1.10 to 1.11 and I get a ton of these
UnorderedObjectListWarnings when running my tests. However, I have no
idea where they're coming from, so I'm going through and searching for
`filter` in my codebase and adding an `order_by` on the end of it. This
is a quite annoying process and would be helped if the warning returned a
stack trace of where it was being called from.
--
Ticket URL: <https://code.djangoproject.com/ticket/28219>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: Denise Mauldin | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 1.11
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Simon Charette):
Hi Denise, thank you for your report.
The `stacklevel` of the `UnorderedObjectListWarning` warning was adjusted
to try to point at the right place in #28109 (shipped in 1.11.1) but it
only accounts for direct `Paginator` manipulation. If you use `ListView`
or `ModelAdmin` pagination features, which rely on `Paginator` internally,
it's possible the warning is not pointing at the right user code location.
I'm afraid this will this be hard to solve without either adjusting all
internal usages of `Paginator` to capture `UnorderedObjectListWarning`
warnings and ''rewarn'' with a `stacklevel` matching the first possible
user entry point but it sounds like a lost cause given the dynamic nature
of queryset definition and generation.
For example
{{{#!python
class FooListView(ListView):
queryset = Foo.objects.all()
paginate_by = 10
}}}
Maybe adjusting `ListView.paginate_queryset()` and `ModelAdmin.
get_paginator()` to rewarn with a message mentioning
`self.__class__.qualname__` would be enough?
--
Ticket URL: <https://code.djangoproject.com/ticket/28219#comment:1>
Django
Reporter: Denise Mauldin | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 1.11
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Denise Mauldin):
Hi Simon,
I'm not sure what to do. If I edit django/core/paginator.py to do a
traceback.print_stack() call before the warning is called, then I get
something really long that doesn't include any of my code other than the
API call:
{{{
File "manage.py", line 23, in <module>
execute_from_command_line(sys.argv)
File "/home/vagrant/.virtualenvs/ann/lib/python3.5/site-
packages/django/core/management/__init__.py", line 363, in
execute_from_command_line
utility.execute()
File "/home/vagrant/.virtualenvs/ann/lib/python3.5/site-
packages/django/core/management/__init__.py", line 355, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/vagrant/.virtualenvs/ann/lib/python3.5/site-
packages/django/core/management/commands/test.py", line 29, in
run_from_argv
super(Command, self).run_from_argv(argv)
File "/home/vagrant/.virtualenvs/ann/lib/python3.5/site-
packages/django/core/management/base.py", line 283, in run_from_argv
self.execute(*args, **cmd_options)
File "/home/vagrant/.virtualenvs/ann/lib/python3.5/site-
packages/django/core/management/base.py", line 330, in execute
output = self.handle(*args, **options)
File "/home/vagrant/.virtualenvs/ann/lib/python3.5/site-
packages/django/core/management/commands/test.py", line 62, in handle
failures = test_runner.run_tests(test_labels)
File "/home/vagrant/.virtualenvs/ann/lib/python3.5/site-
packages/django/test/runner.py", line 603, in run_tests
result = self.run_suite(suite)
File "/home/vagrant/.virtualenvs/ann/lib/python3.5/site-
packages/django/test/runner.py", line 567, in run_suite
return runner.run(suite)
File "/usr/local/python/3.5.1/lib/python3.5/unittest/runner.py", line
176, in run
test(result)
File "/usr/local/python/3.5.1/lib/python3.5/unittest/suite.py", line 84,
in __call__
return self.run(*args, **kwds)
File "/usr/local/python/3.5.1/lib/python3.5/unittest/suite.py", line
122, in run
test(result)
File "/home/vagrant/.virtualenvs/ann/lib/python3.5/site-
packages/django/test/testcases.py", line 213, in __call__
super(SimpleTestCase, self).__call__(result)
File "/usr/local/python/3.5.1/lib/python3.5/unittest/case.py", line 648,
in __call__
return self.run(*args, **kwds)
File "/usr/local/python/3.5.1/lib/python3.5/unittest/case.py", line 600,
in run
testMethod()
File "/var/www/ann/ann/orders/tests/test_api.py", line 58, in
test_order_list_denied
response = self.client.get('/api/orders/')
File "/home/vagrant/.virtualenvs/ann/lib/python3.5/site-
packages/rest_framework/test.py", line 282, in get
response = super(APIClient, self).get(path, data=data, **extra)
File "/home/vagrant/.virtualenvs/ann/lib/python3.5/site-
packages/rest_framework/test.py", line 208, in get
return self.generic('GET', path, **r)
File "/home/vagrant/.virtualenvs/ann/lib/python3.5/site-
packages/django/test/client.py", line 416, in generic
return self.request(**r)
File "/home/vagrant/.virtualenvs/ann/lib/python3.5/site-
packages/rest_framework/test.py", line 279, in request
return super(APIClient, self).request(**kwargs)
File "/home/vagrant/.virtualenvs/ann/lib/python3.5/site-
packages/rest_framework/test.py", line 231, in request
request = super(APIRequestFactory, self).request(**kwargs)
File "/home/vagrant/.virtualenvs/ann/lib/python3.5/site-
packages/django/test/client.py", line 483, in request
response = self.handler(environ)
File "/home/vagrant/.virtualenvs/ann/lib/python3.5/site-
packages/django/test/client.py", line 144, in __call__
response = self.get_response(request)
File "/home/vagrant/.virtualenvs/ann/lib/python3.5/site-
packages/rest_framework/test.py", line 251, in get_response
return super(ForceAuthClientHandler, self).get_response(request)
File "/home/vagrant/.virtualenvs/ann/lib/python3.5/site-
packages/django/core/handlers/base.py", line 124, in get_response
response = self._middleware_chain(request)
File "/home/vagrant/.virtualenvs/ann/lib/python3.5/site-
packages/django/core/handlers/exception.py", line 41, in inner
response = get_response(request)
File "/home/vagrant/.virtualenvs/ann/lib/python3.5/site-
packages/django/utils/deprecation.py", line 140, in __call__
response = self.get_response(request)
File "/home/vagrant/.virtualenvs/ann/lib/python3.5/site-
packages/django/core/handlers/exception.py", line 41, in inner
response = get_response(request)
File "/home/vagrant/.virtualenvs/ann/lib/python3.5/site-
packages/django/utils/deprecation.py", line 140, in __call__
response = self.get_response(request)
File "/home/vagrant/.virtualenvs/ann/lib/python3.5/site-
packages/django/core/handlers/exception.py", line 41, in inner
response = get_response(request)
File "/home/vagrant/.virtualenvs/ann/lib/python3.5/site-
packages/django/utils/deprecation.py", line 140, in __call__
response = self.get_response(request)
File "/home/vagrant/.virtualenvs/ann/lib/python3.5/site-
packages/django/core/handlers/exception.py", line 41, in inner
response = get_response(request)
File "/home/vagrant/.virtualenvs/ann/lib/python3.5/site-
packages/django/utils/deprecation.py", line 140, in __call__
response = self.get_response(request)
File "/home/vagrant/.virtualenvs/ann/lib/python3.5/site-
packages/django/core/handlers/exception.py", line 41, in inner
response = get_response(request)
File "/home/vagrant/.virtualenvs/ann/lib/python3.5/site-
packages/django/utils/deprecation.py", line 140, in __call__
response = self.get_response(request)
File "/home/vagrant/.virtualenvs/ann/lib/python3.5/site-
packages/django/core/handlers/exception.py", line 41, in inner
response = get_response(request)
File "/home/vagrant/.virtualenvs/ann/lib/python3.5/site-
packages/django/utils/deprecation.py", line 140, in __call__
response = self.get_response(request)
File "/home/vagrant/.virtualenvs/ann/lib/python3.5/site-
packages/django/core/handlers/exception.py", line 41, in inner
response = get_response(request)
File "/home/vagrant/.virtualenvs/ann/lib/python3.5/site-
packages/django/utils/deprecation.py", line 140, in __call__
response = self.get_response(request)
File "/home/vagrant/.virtualenvs/ann/lib/python3.5/site-
packages/django/core/handlers/exception.py", line 41, in inner
response = get_response(request)
File "/var/www/ann/ann/common/middleware.py", line 33, in __call__
return self.get_response(request)
File "/home/vagrant/.virtualenvs/ann/lib/python3.5/site-
packages/django/core/handlers/exception.py", line 41, in inner
response = get_response(request)
File "/home/vagrant/.virtualenvs/ann/lib/python3.5/site-
packages/django/utils/deprecation.py", line 140, in __call__
response = self.get_response(request)
File "/home/vagrant/.virtualenvs/ann/lib/python3.5/site-
packages/django/core/handlers/exception.py", line 41, in inner
response = get_response(request)
File "/home/vagrant/.virtualenvs/ann/lib/python3.5/site-
packages/django/core/handlers/base.py", line 185, in _get_response
response = wrapped_callback(request, *callback_args,
**callback_kwargs)
File "/usr/local/python/3.5.1/lib/python3.5/contextlib.py", line 30, in
inner
return func(*args, **kwds)
File "/home/vagrant/.virtualenvs/ann/lib/python3.5/site-
packages/django/views/decorators/csrf.py", line 58, in wrapped_view
return view_func(*args, **kwargs)
File "/home/vagrant/.virtualenvs/ann/lib/python3.5/site-
packages/rest_framework/viewsets.py", line 86, in view
return self.dispatch(request, *args, **kwargs)
File "/home/vagrant/.virtualenvs/ann/lib/python3.5/site-
packages/rest_framework/views.py", line 486, in dispatch
response = handler(request, *args, **kwargs)
File "/home/vagrant/.virtualenvs/ann/lib/python3.5/site-
packages/rest_framework/mixins.py", line 42, in list
page = self.paginate_queryset(queryset)
File "/home/vagrant/.virtualenvs/ann/lib/python3.5/site-
packages/rest_framework/generics.py", line 173, in paginate_queryset
return self.paginator.paginate_queryset(queryset, self.request,
view=self)
File "/home/vagrant/.virtualenvs/ann/lib/python3.5/site-
packages/rest_framework/pagination.py", line 208, in paginate_queryset
paginator = self.django_paginator_class(queryset, page_size)
File "/home/vagrant/.virtualenvs/ann/lib/python3.5/site-
packages/django/core/paginator.py", line 32, in __init__
self._check_object_list_is_ordered()
File "/home/vagrant/.virtualenvs/ann/lib/python3.5/site-
packages/django/core/paginator.py", line 110, in
_check_object_list_is_ordered
traceback.print_stack()
}}}
So I'm not sure why this warning is being triggered. I set Meta: ordering
defaults on all of my models and apiviews. What else am I missing?
Maybe I should report this on rest_framework's bug tracker?
--
Ticket URL: <https://code.djangoproject.com/ticket/28219#comment:2>
Django
Reporter: Denise Mauldin | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 1.11
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Simon Charette):
I'm pretty sure this is caused by your `Order` model's queryset not being
ordered but it's hard to tell without your model and DRF view definitions.
In all cases there's not Django can do to solve your specific DRF issue.
It should make sure the queryset passed to paginator is ordered and decide
what to do from there.
For the record, the `catch_warnings()` method I suggested in my previous
comment about ''rewarning'' won't work as it's not threadsafe.
--
Ticket URL: <https://code.djangoproject.com/ticket/28219#comment:3>
Django
Reporter: Denise Mauldin | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 1.11
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Denise Mauldin):
Replying to [comment:3 Simon Charette]:
> I'm pretty sure this is caused by your `Order` model's queryset not
being ordered but it's hard to tell without your model and DRF view
definitions. In all cases there's not much Django can do to solve your
specific DRF issue. It should make sure the queryset passed to paginator
is ordered and decide what to do from there.
>
> For the record, the `catch_warnings()` method I suggested in my previous
comment about ''rewarning'' won't work as it's not threadsafe.
Yeah, but when there's 50 `Order` model tests and 2 of them throw this
error, then it's hard to tell what's causing them when the stacktrace just
points to paginator. Someone on SO did suggest that I run the tests on
verbose mode (-v2), so that did help pinpoint which test was throwing the
error. However, it'd be nice if the UnorderedObject warning would have an
option to throw the full traceback so I could tell that it was line 58 in
my test_api.py. Maybe that's not possible?
--
Ticket URL: <https://code.djangoproject.com/ticket/28219#comment:4>
Django
--------------------------------------+------------------------------------
Reporter: Denise Mauldin | Owner: nobody
Component: Core (Other) | Version: 1.11
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham):
* type: Uncategorized => Cleanup/optimization
* component: Uncategorized => Core (Other)
* stage: Unreviewed => Accepted
Comment:
Tentatively accepting for further investigation, even though it's unclear
if a solution is feasible.
--
Ticket URL: <https://code.djangoproject.com/ticket/28219#comment:5>
Django
--------------------------------------+------------------------------------
Component: Core (Other) | Version: 1.11
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Simon Charette):
> However, it'd be nice if the UnorderedObject warning would have an
option to throw the full traceback so I could tell that it was line 58 in
my test_api.py. Maybe that's not possible?
Did you try running your test suite with `python -W error`? You could use
`warnings.simplefilter` to only error in the `UnorderedObjectListWarning `
case
{{{#!python
import warnings
from django.core.paginator import UnorderedObjectListWarning
warnings.simplefilter('error', UnorderedObjectListWarning)
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/28219#comment:6>
Django
--------------------------------------+------------------------------------
Component: Core (Other) | Version: 1.11
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Simon Charette):
FWIW the re-warning approach could eventually be used if
[https://www.python.org/dev/peps/pep-0550/ PEP 505] gets adopted as it
would make `warnings.catch_warnings()` safe to use in multi-threaded
environments.
--
Ticket URL: <https://code.djangoproject.com/ticket/28219#comment:7>