[Django] #32914: DiscoverRunner doesn't always group tests by class
Django
-------------------------------------+-------------------------------------
Reporter: Chris | Owner: Chris Jerdonek
Jerdonek |
Type: | Status: assigned
Cleanup/optimization |
Component: Testing | Version: dev
framework |
Severity: Normal | Keywords:
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
The following is an issue that has been present in the test runner since
before I started working on this area of the code this year (e.g. since
this commit:
https://github.com/django/django/tree/ec0ff406311de88f4e2a135d784363424fe602aa
), but I hadn't taken time to report it until now.
Namely, if tests are provided to the test runner on the command-line that
aren't already grouped by test-case class, then `DiscoverRunner` won't
group them by test-case class.
Here is an example (class A, class B, then class A again):
{{{
$ ./tests/runtests.py
test_runner.tests.DependencyOrderingTests.test_simple_dependencies
test_runner.tests.ManageCommandTests.test_custom_test_runner
test_runner.tests.DependencyOrderingTests.test_chained_dependencies -v3
Testing against Django installed in '/.../django/django'
Importing application test_runner
Found 3 test(s).
Skipping setup of unused database(s): default, other.
System check identified no issues (0 silenced).
test_simple_dependencies (test_runner.tests.DependencyOrderingTests) ...
ok
test_custom_test_runner (test_runner.tests.ManageCommandTests) ... ok
test_chained_dependencies (test_runner.tests.DependencyOrderingTests) ...
ok
----------------------------------------------------------------------
Ran 3 tests in 0.002s
OK
}}}
This can interfere with things like parallel tests, which assume the tests
are grouped by class, as well as the new shuffling code.
This can be fixed in
[https://github.com/django/django/blob/f42ccdd835e5b3f0914b5e6f87621c648136ea36/django/test/runner.py#L987-L1015
reorder_tests()], which already iterates over all tests, as it has logic
to group tests by test type and remove duplicates.
--
Ticket URL: <https://code.djangoproject.com/ticket/32914>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
-------------------------------------+-------------------------------------
Type: | Jerdonek
Cleanup/optimization | Status: assigned
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* stage: Unreviewed => Accepted
--
Ticket URL: <https://code.djangoproject.com/ticket/32914#comment:1>
Django
-------------------------------------+-------------------------------------
Type: | Jerdonek
Cleanup/optimization | Status: assigned
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* has_patch: 0 => 1
Comment:
PR: https://github.com/django/django/pull/14621
--
Ticket URL: <https://code.djangoproject.com/ticket/32914#comment:2>
Django
-------------------------------------+-------------------------------------
Type: | Jerdonek
Cleanup/optimization | Status: assigned
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Chris Jerdonek):
With the patch:
{{{
$ ./tests/runtests.py
test_runner.tests.DependencyOrderingTests.test_simple_dependencies
test_runner.tests.ManageCommandTests.test_custom_test_runner
test_runner.tests.DependencyOrderingTests.test_chained_dependencies -v3
Testing against Django installed in '/.../django/django'
Importing application test_runner
Found 3 test(s).
Skipping setup of unused database(s): default, other.
System check identified no issues (0 silenced).
test_simple_dependencies (test_runner.tests.DependencyOrderingTests) ...
ok
test_chained_dependencies (test_runner.tests.DependencyOrderingTests) ...
ok
test_custom_test_runner (test_runner.tests.ManageCommandTests) ... ok
----------------------------------------------------------------------
Ran 3 tests in 0.002s
OK
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/32914#comment:3>
Django
-------------------------------------+-------------------------------------
Type: | Jerdonek
Cleanup/optimization | Status: assigned
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/32914#comment:4>
Django
-------------------------------------+-------------------------------------
Type: | Jerdonek
Component: Testing framework | Version: dev
Keywords: | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"5848b3a1d70f09aaf598d794569ad758183f41db" 5848b3a1]:
{{{
#!CommitTicketReference repository=""
revision="5848b3a1d70f09aaf598d794569ad758183f41db"
Fixed #32914 -- Prevented test --shuffle from skipping test methods.
"test --shuffle" skipped test methods when test classes were mixed.
This changes runner.py's reorder_tests() to group by TestCase class.
Regression in 90ba716bf060ee7fef79dc230b0b20644839069f.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/32914#comment:5>