[Django] #30415: Using pytest as test runner
Django
---------------------------------------------+------------------------
Reporter: Daniel Hahler | Owner: nobody
Type: New feature | Status: new
Component: Testing framework | Version: master
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
---------------------------------------------+------------------------
I think it would be very nice if pytest could be used to run Django's
tests.
I've started working on this, by using a pytest plugin/conftest, which
reuses parts from `tests/runtests.py`, which I've therefore started to
make more modular (https://github.com/django/django/pull/11289).
It is not meant to replace the current test runner (or tests), but would
just allow to use pytest out of the box seamlessly, with all the
additional features and plugins it provides.
Is this something that is wanted in general?
--
Ticket URL: <https://code.djangoproject.com/ticket/30415>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: Daniel Hahler | Owner: nobody
Type: New feature | Status: new
Component: Testing framework | Version: master
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Carlton Gibson):
* stage: Unreviewed => Accepted
Comment:
Hi Daniel.
I'm very happy for some refactoring to help re-use here.
(What's counts as "Fixed" for this ticket?)
(Aside: would a test runner subclass letting us use plain `assert`s ever
get passed the line? Any interest in writing such? Fancy pinging the
DevelopersMailingList?)
--
Ticket URL: <https://code.djangoproject.com/ticket/30415#comment:1>
Django
-----------------------------------+------------------------------------
Reporter: Daniel Hahler | Owner: nobody
Type: New feature | Status: new
Component: Testing framework | Version: master
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
--
Ticket URL: <https://code.djangoproject.com/ticket/30415#comment:2>
Django
-----------------------------------+------------------------------------
Type: New feature | Status: new
Component: Testing framework | Version: master
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Daniel Hahler):
> What's counts as "Fixed" for this ticket?
When it is good enough for the pytest plugin/integration.
> would a test runner subclass letting us use plain asserts ever get
passed the line?
Not sure I follow, but the pytest wrapper would not be a subclass of the
Django test runner, would would rather use it, and parts from `tuntests`.
Any interest in writing such?
pytest then would allow to use plain asserts in tests - but that is out of
scope here.
--
Ticket URL: <https://code.djangoproject.com/ticket/30415#comment:3>
Django
-----------------------------------+------------------------------------
Type: New feature | Status: new
Component: Testing framework | Version: master
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by David Smith):
Hi all,
I've had a bit of a look at running the Django test suite with other test
runners (pytest).
One of the issues I've come accross is there are some classes which are
designed to **only** be inherited but start with the word 'Test'. Pytest
does not know this, tries to run them and fails. A name change for these
would be required. Here is an example:
I'm happy to investigate pytest further but can I confirm that I've
understood the ticket correctly (i.e. get the test suite running with
pytest) and would these type of changes be welcome. I'm sure there will be
other issues, but in addition to configuring a new test running it is
likely to also result in some of the existing tests need to be renamed /
refactored, (subtests ''could'' be an issue, for example).
--
Ticket URL: <https://code.djangoproject.com/ticket/30415#comment:4>
Django
-----------------------------------+------------------------------------
Type: New feature | Status: new
Component: Testing framework | Version: master
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Claude Paroz):
Maybe someone could summarize here the advantages of running Django's own
tests with a different test runner?
--
Ticket URL: <https://code.djangoproject.com/ticket/30415#comment:5>
Django
-----------------------------------+------------------------------------
Type: New feature | Status: new
Component: Testing framework | Version: master
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by David Smith):
All,
It will need someone much more experienced than myself to properly
articulate this, and to be credible on the topic. Carlton - I may be
reading too much into this but your comments sounded interested / positive
on this?
I think the benefits are subjective given current test runner works so any
benefits above that are about user experience. So may be hard to make a
good technical reason for the change. A few of my thoughts are:
Pros
#22449 - Pytest outputs have colour so would not need to fix this ticket
- Better output - get % progress, more information on failed tests.
- Integration with IDEs (e.g. can use pytest add-in for VSCode?)
- ability to use other pytest and pytest-addins functionality. e.g.
`pytest --durations=50` outputs the 50 slowest tests.
- Would pytest-xdist help speed up the tests?. (currently GSOC project on
this topic)
- ''IF'' project went full pytest then could use plain asserts.
Cons
- Some complications with implementation (e.g. subtests and some tests
may need to be re-written)
- If we went full pytest then it would be a dependency (Django tends to
avoid doing this)
- Current test runner may be better understood than getting pytest to work
(not sure how much expertise we have on the current test runner?)
Ideally I'd see this as a two step changes. 1) get it working with pytest
and the current test runner, 2) consider moving to pytest fully, this may
be after 1 has been working for a period of time.
Appreciate thoughts on next steps (is this a topic for the mailing list?)
--
Ticket URL: <https://code.djangoproject.com/ticket/30415#comment:6>
Django
Reporter: Daniel Hahler | Owner: nobody
Component: Testing framework | Version: master
Keywords: | Triage Stage:
| Someday/Maybe
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Carlton Gibson):
* status: new => closed
* resolution: => needsinfo
* stage: Accepted => Someday/Maybe
Comment:
> ...but your comments sounded interested / positive on this?
I think if a refactoring is available that would allow an option here,
then that would be worth investigating. `unittest` isn't going anywhere
but `pytest` is where all the progress is at the moment, and if there was
the option to take advantage of that, well... (end of previous sentence).
I had assumed Daniel had intended to work on the ticket... No issues that
not (as yet) but I think it's more a Someday/Maybe at this point.
(Updating: wouldn't object if somebody close it as needsinfo, i.e. what
the refactoring might look like, instead.)
> Appreciate thoughts on next steps ...
It needs a proof of concept. ''Here's what it would take...'' — Not sure
that's a good ROI on time spent: it's one thing for a pytest/pytest-django
maintainer to do it (they know the issues well) — it's another thing for
everyone else. Maybe as a learning exercise but...
So, having got thins far, I'm going to close it as needsinfo as a
Someday/Maybe:
1. Who's going to take it on? (Answer that, we can re-open, and assign.)
2. What does the patch look like? (Whether it's more than a Maybe...)
I hope that makes sense. (Do make any changes if you're not in agreement!)
Thanks for the thought and investigation David!
--
Ticket URL: <https://code.djangoproject.com/ticket/30415#comment:7>
Django
-------------------------------------+-------------------------------------
Component: Testing framework | Version: master
Keywords: | Triage Stage:
| Someday/Maybe
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Nick Pope):
Replying to [comment:6 David Smith]:
> - Some complications with implementation (e.g. subtests and some tests
may need to be re-written)
Note that there is the [https://github.com/pytest-dev/pytest-subtests
pytest-subtests] plugin which adds support for `unittest`'s `subTests()`.
--
Ticket URL: <https://code.djangoproject.com/ticket/30415#comment:8>