[Django] #32602: Clarify wording re: parallel testing and test case vs. test case class
Django
------------------------------------------------+------------------------
Reporter: Chris Jerdonek | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Testing framework | Version: 3.1
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
------------------------------------------------+------------------------
Currently, there are some places in the docs, code, and code comments that
aren't quite clear or right in the way they talk about parallel testing as
it relates to test cases versus test case classes.
For example, here in the docs:
https://docs.djangoproject.com/en/3.1/topics/testing/advanced/#django.test.runner.DiscoverRunner
it says:
> If there are fewer test cases than configured processes, Django will
reduce the number of processes accordingly.
But it should say, "If there are fewer test case ''classes'' than
configured processes."
Similarly, this code comment:
https://github.com/django/django/blob/41850eec99366a51f98123f7c51e5bc5a8b2798c/django/test/runner.py#L653-L654
should say:
{{{#!python
# Since tests are distributed across processes on a per-TestCase
# *class* basis, there's no need for more processes than TestCase
# *classes*.
}}}
And also, `partition_suite_by_case()`:
https://github.com/django/django/blob/41850eec99366a51f98123f7c51e5bc5a8b2798c/django/test/runner.py#L845
should really be called something like `partition_suite_by_class()` or
`partition_suite_by_test_class()` since it groups by `type`, and the
docstring updated accordingly.
I think this is subtle but important because, without being clear, it's
easy for people to mistakenly think that the parallel test runner is
parallelizing individual test cases, when it's really distributing out the
test cases grouped by class.
--
Ticket URL: <https://code.djangoproject.com/ticket/32602>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: Chris Jerdonek | Owner: nobody
Cleanup/optimization |
Component: Testing framework | Version: 3.1
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Description changed by Chris Jerdonek:
Old description:
New description:
Currently, there are some places in the docs, code, and code comments that
aren't quite clear or right in the way they talk about parallel testing as
it relates to test cases versus test case classes.
For example, here in the docs:
https://docs.djangoproject.com/en/3.1/topics/testing/advanced/#django.test.runner.DiscoverRunner
it says:
> If there are fewer test cases than configured processes, Django will
reduce the number of processes accordingly.
But it should say, "If there are fewer test case ''classes'' than
configured processes."
Here is similar language elsewhere in the docs:
https://docs.djangoproject.com/en/3.1/ref/django-admin/#envvar-
DJANGO_TEST_PROCESSES
Similarly, this code comment:
https://github.com/django/django/blob/41850eec99366a51f98123f7c51e5bc5a8b2798c/django/test/runner.py#L653-L654
should say:
{{{#!python
# Since tests are distributed across processes on a per-TestCase
# *class* basis, there's no need for more processes than TestCase
# *classes*.
}}}
And also, `partition_suite_by_case()`:
https://github.com/django/django/blob/41850eec99366a51f98123f7c51e5bc5a8b2798c/django/test/runner.py#L845
should really be called something like `partition_suite_by_class()` or
`partition_suite_by_test_class()` since it groups by `type`, and the
docstring updated accordingly.
I think this is subtle but important because, without being clear, it's
easy for people to mistakenly think that the parallel test runner is
parallelizing individual test cases, when it's really distributing out the
test cases grouped by class.
--
--
Ticket URL: <https://code.djangoproject.com/ticket/32602#comment:1>
Django
Reporter: Chris Jerdonek | Owner: nobody
Cleanup/optimization |
Component: Testing framework | Version: 3.1
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Tim Graham):
I disagree with your definitions. In my interpretation of
[https://docs.python.org/3/library/unittest.html unittest's docs], what
you call is "test case class" is a "test case" and what you call a "test
case" is a "test method."
--
Ticket URL: <https://code.djangoproject.com/ticket/32602#comment:2>
Django
Reporter: Chris Jerdonek | Owner: nobody
Cleanup/optimization |
Component: Testing framework | Version: 3.1
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Chris Jerdonek):
It says right at the top of the page you linked to:
> test case. A ''test case'' is the individual unit of testing. It checks
for a specific response to a particular set of inputs. `unittest` provides
a base class, `TestCase`, which may be used to create new test cases.
Its definition of test suite is consistent with that, too:
> test suite. A ''test suite'' is a collection of test cases, test suites,
or both. It is used to aggregate tests that should be executed together.
A test suite is made up of `TestCase` ''instances''. It's not a collection
of test classes.
--
Ticket URL: <https://code.djangoproject.com/ticket/32602#comment:3>
Django
Reporter: Chris Jerdonek | Owner: nobody
Cleanup/optimization |
Component: Testing framework | Version: 3.1
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Chris Jerdonek):
Also, from the "Organizing test code" section:
https://docs.python.org/3/library/unittest.html#organizing-test-code
> The basic building blocks of unit testing are ''test cases'' — single
scenarios that must be set up and checked for correctness. In `unittest`,
test cases are represented by `unittest.TestCase` instances. To make your
own test cases you must write subclasses of TestCase or use
FunctionTestCase.
> ... If the test fails, an exception will be raised with an explanatory
message, and `unittest` will identify the test case as a ''failure''.
--
Ticket URL: <https://code.djangoproject.com/ticket/32602#comment:4>
Django
Reporter: Chris Jerdonek | Owner: nobody
Cleanup/optimization |
Component: Testing framework | Version: 3.1
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Tim Graham):
Maybe I have a faulty understanding of what you're trying to say, but for
example:
{{{ #!python
class TestStringMethods(unittest.TestCase):
def test_upper(self):
self.assertEqual('foo'.upper(), 'FOO')
}}}
I would call `TestStringMethods` a test case and `test_upper` a test
method. My impression is that you would call `test_upper` a test case? (If
so, what's a test method?)
Under my definitions, I do think the unittest docs are confusing because a
test method seems more like an "individual unit of testing" than a test
case.
--
Ticket URL: <https://code.djangoproject.com/ticket/32602#comment:5>
Django
Reporter: Chris Jerdonek | Owner: nobody
Cleanup/optimization |
Component: Testing framework | Version: 3.1
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Chris Jerdonek):
In your example, `TestStringMethods` is a test case class. `unittest`
creates test cases from test methods, so test cases are in one-to-one
correspondence with test methods. (That's why the id of a test case
includes the method name.)
For example, if you read the docs for `loadTestsFromModule()`:
https://docs.python.org/3/library/unittest.html#unittest.TestLoader.loadTestsFromModule
it says:
> Return a suite of all ''test cases'' (my emphasis) contained in the
given module. This method searches ''module'' for classes derived from
`TestCase` and creates an instance of the class for each test method
defined for the class.
You can view test methods as the way to define the test cases that will
run.
--
Ticket URL: <https://code.djangoproject.com/ticket/32602#comment:6>
Django
Reporter: Chris Jerdonek | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Testing framework | Version: 3.1
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham):
* stage: Unreviewed => Accepted
Comment:
OK. I think there's confusion about this distinction across the ecosystem.
For example, [https://docs.python-guide.org/writing/tests/ The
Hitchhiker's Guide to Python] says, "Learn your tools and learn how to run
a single test or a test case" (where test case seems to mean all the tests
in a class).
In Django's comment, I think `per-TestCase` isn't confusing because the
casing implies a class. My guess is that most people read "test case" as
pertaining to classes, but it seems that being more precise won't hurt.
--
Ticket URL: <https://code.djangoproject.com/ticket/32602#comment:7>
Django
Reporter: Chris Jerdonek | Owner: Chris
Type: | Jerdonek
Cleanup/optimization | Status: assigned
Component: Testing framework | Version: 3.1
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Chris Jerdonek):
* owner: nobody => Chris Jerdonek
* status: new => assigned
--
Ticket URL: <https://code.djangoproject.com/ticket/32602#comment:8>
Django
Reporter: Chris Jerdonek | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Testing framework | Version: 3.1
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Mariusz Felisiak):
* cc: Natalia Bidart (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/32602#comment:8>
Django
Reporter: Chris Jerdonek | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Testing framework | Version: 3.1
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Natalia Bidart):
When I read the ticket description, my first reaction was the same as
Tim's: a test case, to me, is unequivocally the class inheriting from
`TestCase`.
But, after reading the follow up comments and links, I also agree with Tim
that being more explicit (which is better than implicit :-)) can benefit
future readers.
--
Ticket URL: <https://code.djangoproject.com/ticket/32602#comment:9>
Django
Reporter: Chris Jerdonek | Owner: nobody
Type: Cleanup/optimization | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
--------------------------------------+------------------------------------
Changes (by Natalia Bidart):
* component: Testing framework => Documentation
* easy: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/32602#comment:10>
Django
Reporter: Chris Jerdonek | Owner:
Cleanup/optimization | Status: assigned
Component: Documentation | Version: 3.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
-------------------------------------+-------------------------------------
Changes (by akbarahmadshah):
* owner: nobody => akbarahmadshah
* status: new => assigned
Comment:
Not new to django but new to open source, this seems to be a good issue to
get started with. Will start looking in developer docs from today.
--
Ticket URL: <https://code.djangoproject.com/ticket/32602#comment:11>
Django
Reporter: Chris Jerdonek | Owner: (none)
Type: Cleanup/optimization | Status: new
Component: Documentation | Version: 3.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
--------------------------------------+------------------------------------
* owner: Akbar Ahmad Shah => (none)
* status: assigned => new
--
Ticket URL: <https://code.djangoproject.com/ticket/32602#comment:12>
Django
Reporter: Chris Jerdonek | Owner: Mandadi
Type: | Mahendhar Reddy
Cleanup/optimization | Status: assigned
Component: Documentation | Version: 3.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
-------------------------------------+-------------------------------------
* owner: (none) => Mandadi Mahendhar Reddy
* status: new => assigned
--
Ticket URL: <https://code.djangoproject.com/ticket/32602#comment:13>
Django
Reporter: Chris Jerdonek | Owner: Mandadi
Type: | Mahendhar Reddy
Cleanup/optimization | Status: assigned
Component: Documentation | Version: 3.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
-------------------------------------+-------------------------------------
Comment (by MandadiMahendharReddy):
Dear Team, I have commenced working on this ticket and have made changes
to three files respectively.
1. django/_build/html/topics/advanced.html
2. django/_build/html/ref/django-admin.html
3. django/test/runner.py
I am able to view commit changes and commit third file in local i.e.
django/test/runner.py. However, I am unable to see files and changes in
git status for the first and second file in local.
I am unable to view the first and second file and their changes in git
status. I noticed that the **docs/_build/** folder is included in the
"**.gitignore**" file. Can you please suggest how to proceed with this?
--
Ticket URL: <https://code.djangoproject.com/ticket/32602#comment:14>
Django
Reporter: Chris Jerdonek | Owner: Mandadi
Type: | Mahendhar Reddy
Cleanup/optimization | Status: assigned
Component: Documentation | Version: 3.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
-------------------------------------+-------------------------------------
Comment (by Tim Graham):
You should edit the documentation in the `docs` folder, not in `_build.
The `_build` directory contains the html files generated from the source
txt files.
--
Ticket URL: <https://code.djangoproject.com/ticket/32602#comment:15>
Django
Reporter: Chris Jerdonek | Owner: Mandadi
Type: | Mahendhar Reddy
Cleanup/optimization | Status: assigned
Component: Documentation | Version: 3.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
-------------------------------------+-------------------------------------
Comment (by MandadiMahendharReddy):
Thank you Replying to [comment:15 Tim Graham]:
> You should edit the documentation in the `docs` folder, not in `_build.
The `_build` directory contains the html files generated from the source
txt files.
--
Ticket URL: <https://code.djangoproject.com/ticket/32602#comment:16>
Django
Reporter: Chris Jerdonek | Owner: Mandadi
Type: | Mahendhar Reddy
Cleanup/optimization | Status: assigned
Component: Documentation | Version: 3.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 1 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_better_patch: 0 => 1
* has_patch: 0 => 1
Comment:
[https://github.com/django/django/pull/17162 PR]
--
Ticket URL: <https://code.djangoproject.com/ticket/32602#comment:17>
Django
Reporter: Chris Jerdonek | Owner: Faishal
Type: | Manzar
Cleanup/optimization | Status: assigned
Component: Documentation | Version: 3.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 1 | UI/UX: 0
-------------------------------------+-------------------------------------
* owner: Mandadi Mahendhar Reddy => Faishal Manzar
--
Ticket URL: <https://code.djangoproject.com/ticket/32602#comment:18>
Django
Reporter: Chris Jerdonek | Owner: Faishal
Type: | Manzar
Cleanup/optimization | Status: assigned
Component: Documentation | Version: 3.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 1 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Mariusz Felisiak):
[https://github.com/django/django/pull/17319 New PR]
--
Ticket URL: <https://code.djangoproject.com/ticket/32602#comment:19>
Django
Reporter: Chris Jerdonek | Owner: Faishal
Type: | Manzar
Cleanup/optimization | Status: assigned
Component: Documentation | Version: 3.1
Severity: Normal | Resolution:
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
-------------------------------------+-------------------------------------
* needs_better_patch: 1 => 0
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/32602#comment:20>
Django
Reporter: Chris Jerdonek | Owner: Faishal
Type: | Manzar
Component: Documentation | Version: 3.1
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
-------------------------------------+-------------------------------------
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"f4e72e6523e6968d9628dfbff914ab57dbf19e6b" f4e72e6]:
{{{
#!CommitTicketReference repository=""
revision="f4e72e6523e6968d9628dfbff914ab57dbf19e6b"
Fixed #32602 -- Clarified wording of TestCase class.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/32602#comment:21>
Django
Reporter: Chris Jerdonek | Owner: Faishal
Type: | Manzar
Cleanup/optimization | Status: closed
Component: Documentation | Version: 3.1
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
-------------------------------------+-------------------------------------
Comment (by Mariusz Felisiak <felisiak.mariusz@…>):
In [changeset:"25a614639fe524aa1e9a97b1aee32c54a8fe310e" 25a61463]:
{{{
#!CommitTicketReference repository=""
revision="25a614639fe524aa1e9a97b1aee32c54a8fe310e"
[5.0.x] Fixed #32602 -- Clarified wording of TestCase class.
Backport of f4e72e6523e6968d9628dfbff914ab57dbf19e6b from main
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/32602#comment:22>