[Django] #23742: Option to run Django tests in reverse
Django
-----------------------------------+--------------------
Reporter: wrwrwr | Owner: nobody
Type: New feature | Status: new
Component: Testing framework | Version: master
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 1
Easy pickings: 0 | UI/UX: 0
-----------------------------------+--------------------
The idea is to add a `--reverse` option to be able to more thoroughly
check for test side-effects.
Django test suite currently has some tests that have side-effects causing
rare failures in tests usually run before them (or successes in tests
usually run after them?). Debugging such issues is somewhat cumbersome as
the exact test order depends on the environment.
One example is `assertNumQueries` dependence on cache priming (to be
handled in another ticket) causing `test_group_permission_performance` to
fail (for instance just when the full suite is run with a database on a
ramdisk and memcached enabled). With the
[https://github.com/wrwrwr/django/compare/feature/reverse-test-order
branch] you may try:
{{{#!python
./runtests.py admin_views.tests.GroupAdminTest --reverse
}}}
or
{{{#!python
./runtests.py --pair
admin_views.tests.GroupAdminTest.test_group_permission_performance
--reverse
}}}
to see a failure when run after `fixtures` tests.
--
Ticket URL: <https://code.djangoproject.com/ticket/23742>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: wrwrwr | Owner: nobody
Type: New feature | Status: new
Component: Testing framework | Version: master
Keywords: | Triage Stage: Unreviewed
Has patch: 1 | Needs documentation: 1
Needs tests: 1 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by wrwrwr):
* needs_better_patch: => 0
* needs_tests: => 1
* needs_docs: => 1
Comment:
Some tickets that are probably (not verified) experiencing the
`assertNumQueries` cache dependence: #10899, #17377, #20432, #23641.
--
Ticket URL: <https://code.djangoproject.com/ticket/23742#comment:1>
Django
Reporter: wrwrwr | Owner: nobody
Type: New feature | Status: new
Component: Testing framework | Version: master
Keywords: | Triage Stage: Unreviewed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by wrwrwr):
* needs_docs: 1 => 0
* needs_tests: 1 => 0
Comment:
Added some suite ordering tests and short docs for `--bisect`, `--pair`
and `--reverse`.
Attached the output of failing tests from running the whole suite in
reverse with the default settings. 94 in total, some could be probably
helped, but some may be hard to avoid. The other ticket with
`assertNumQueries` fix mentioned in the description is #23746.
--
Ticket URL: <https://code.djangoproject.com/ticket/23742#comment:2>
Django
Reporter: wrwrwr | Owner: nobody
Type: New feature | Status: new
Component: Testing framework | Version: master
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* needs_better_patch: 0 => 1
* stage: Unreviewed => Accepted
Comment:
This seams reasonable and helpful. It would be better if the documentation
for the existing `runtests.py` options was submitted separately. There is
also some missing documentation for the new argument to
`DiscoverTestRunner`.
--
Ticket URL: <https://code.djangoproject.com/ticket/23742#comment:3>
Django
Reporter: wrwrwr | Owner: nobody
Type: New feature | Status: new
Component: Testing framework | Version: master
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by wrwrwr):
* needs_better_patch: 1 => 0
Comment:
Rebased [https://github.com/wrwrwr/django/compare/ticket_23742 the branch]
on top of the merged existing options documentation (thanks for improving
it so much) and added documentation for the new `DiscoverTestRunner`
argument.
Alternatively, we could implement some shuffling, for example like the
"more random order"
[https://docs.python.org/devguide/runtests.html#running -r flag] of the
Python's suite.
--
Ticket URL: <https://code.djangoproject.com/ticket/23742#comment:4>
Django
Reporter: wrwrwr | Owner: nobody
Type: New feature | Status: new
Component: Testing framework | Version: master
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Comment (by wrwrwr):
Update: rebased [https://github.com/django/django/pull/3543 the pull],
split some line breaking fixes into another commit, moved the new test to
a better place and made the option reverse test cases within test classes
(as defined in
[https://docs.djangoproject.com/en/dev/topics/testing/overview/#order-in-
which-tests-are-executed testing docs]), but leave the classes in the
working order.
As
[https://github.com/wrwrwr/django/commit/00a5eebf2e798a4cc53b18e77aacefcef2efe482#commitcomment-8682804
noticed] by Preston, there is actually another level of reversing to
consider. First, reversing labels should always work. Secondly, Django
sorts tests into `django.TestCase`, `django.SimpleTestCase` and
`unittest.TestCase`, these groups shouldn't be probably reversed as the
order is guaranteed in documentation. Thirdly, cases within groups should
be independent. And lastly, test methods would better be independent too
(though there can be some reasonable exceptions).
--
Ticket URL: <https://code.djangoproject.com/ticket/23742#comment:5>
Django
Reporter: wrwrwr | Owner: nobody
Type: New feature | Status: new
Component: Testing framework | Version: master
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Comment (by wrwrwr):
Making the option respect the classes order decreased the number of
failures to 44 (when running the whole suite in reverse); attached an
updated list with some guesses as to what is the issue in each case -- it
seems all should be fairly easy to fix now.
--
Ticket URL: <https://code.djangoproject.com/ticket/23742#comment:6>
Django
Reporter: wrwrwr | Owner: nobody
Type: New feature | Status: new
Component: Testing framework | Version: master
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Comment (by prestontimmons):
This patch looks good to me.
--
Ticket URL: <https://code.djangoproject.com/ticket/23742#comment:7>
Django
Reporter: wrwrwr | Owner: nobody
Type: New feature | Status: new
Component: Testing framework | Version: master
Keywords: | Triage Stage: Ready for
Has patch: 1 | checkin
Needs tests: 0 | Needs documentation: 0
Easy pickings: 0 | Patch needs improvement: 0
| UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by prestontimmons):
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/23742#comment:8>
Django
Reporter: wrwrwr | Owner: nobody
Type: New feature | Status: closed
Component: Testing framework | Version: master
Keywords: | Triage Stage: Ready for
Has patch: 1 | checkin
Needs tests: 0 | Needs documentation: 0
Easy pickings: 0 | Patch needs improvement: 0
| UI/UX: 0
-------------------------------------+-------------------------------------
* status: new => closed
* resolution: => fixed
Comment:
In [changeset:"e22c64dfc0230134c0e9d9ae1d9a5c1589fc8b6e"]:
{{{
#!CommitTicketReference repository=""
revision="e22c64dfc0230134c0e9d9ae1d9a5c1589fc8b6e"
Fixed #23742 -- Added an option to reverse tests order.
This is useful for debugging side effects affecting tests that
are usually executed before a given test. Full suite and pair
tests sort cases more or less deterministically, thus some test
cross-dependencies are easier to reveal by reversing the order.
Thanks Preston Timmons for the review.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/23742#comment:9>