Replace the default django test runner to a more robust testing framework (like pytest)

[Diptesh Choudhuri]

Though the django default runner gets the job done, I think its high time to leverage the power of dedicated frameworks for testing django apps. I will take up the case of pytest here because other frameworks like nose are not widely used.

For everyone who has used pytest for testing in django, I think we will all agree that pytest is much better suited to this task. 

1. The passing tests look better.
2. Failing tests are much more verbose with where the error actually occurred.
3. Parameterizing the test cases using the default runner requires a third party library and looks difficult (I will be honest, I haven't used this package myself at all). 
4. Injecting runtime conditions into the tests is quite difficult, if not impossible, and requires a lot of hacky workarounds.
5. I am not sure about this, but the default runner also doesn't cache results from previously run tests, making retesting unchanged tests much slower.
6. This might sound non-trivial but typing pytest is much more easier than typing python manage.py test into the command line.
7. IDEs like VSCode can discover tests written for pytest but not the ones written using the default runner. This might not sound a big deal, but it is infinitely more easier to run tests using the GUI that these IDEs have. They provide real-time feedback and show exactly which tests have passed and what haven't in the code source itself. Here is a picture showing what I mean.
8. The directory structure created by python manage.py startapp my_app creates a tests.py file. This goes against testing best practices which say that all tests should be aggregated at one place (as opposed to inside individual apps).
9. A single tests.py file encourages new, inexperienced developers to write all their tests in a single file. This, yet again, goes against testing best practices which put a lot of emphasis on modularization of tests. Models should be tested in test_models.py, views in test_views.py etc. 
10. Though unittest's setUp and tearDown work fine, I personally find that pytest's fixture system provides a bit more freedom. I might want to put all my test methods in one class, but I might not want to run the setUp and tearDown for every test method. (Let me know if this point is a bit unclear.)

pytest and pytest-django are quire mature frameworks and don't need a lot of changes. We could make it easier to setup tests right out of the box. For example, django-admin startproject my_project could create a pytest.ini right out of the box in the project root. 

Most of my focus would be on rewriting and adding documentation. Even though widely used, pytest and pytest-django are pretty difficult to get started with for a newbie. I intend to fix this by adding a lot of examples to the docs (maybe even make a separate repository for this).

NOTE: I intend to work on this project for GSoC 2021. I am just scouting beforehand to see how the community receives it.

Please let me know what you think of this idea. Any critiques will be welcome.

Best,

Diptesh

[Jacob Rief]

There is already a ticket for this:

https://code.djangoproject.com/ticket/25707

I'm actually very much in favor of switching to pytest. In my newer projects I use pytest exclusively.

[Tobias Bengfort]

Hi Diptesh,

I think it is great that you want to work on this. I also think asking
for feedback now is the right way to go.

That said, I personall have not had the best experience with pytest. I
have mostly worked on smaller projects though, so my perspective might
be very limited.

My issue is mostly that pytest uses a lot of magic that is hard to
debug, e.g. the overwritten assert statement or implicit injection of
fixtures. Most pro arguments seem to depend on personal taste.

On 15/12/2020 04.37, Diptesh Choudhuri wrote:
> 3. Parameterizing the test cases using the default runner requires a

> third party library and looks difficult

> <http://witkowskibartosz.com/blog/parameterized-django-testcases-at-biking-endorphines.html> (I

> will be honest, I haven't used this package myself at all).

`subTest` has been in the standard library since python 3.4.

https://docs.python.org/3/library/unittest.html?highlight=unittest#distinguishing-test-iterations-using-subtests

tobias

[Tom Forbes]

I would be very much in favour of this. The developer experience of pytest is far better than the unittest module, the tests look more Pythonic and there is a lot more of an ecosystem we can tap into.

There was some recent work done about adding timings to the Djanfo test suite as part of the multi-process test runner. This was way more difficult than it should have been.

Unittest, along with the logging module, are very old and archaic stdlib modules that are ok-ish, but definitely hobbled by legacy interfaces and a “Java inspired” pre-pythonic code style.

I think it’s safe to say that most of the Python ecosystem has settled on pytest, and if we go along with that as well then we can reduce our maintenance burden whilst making our users happier.

Tom

On 15 Dec 2020, at 13:52, Diptesh Choudhuri <diptesh....@gmail.com> wrote:

Though the django default runner gets the job done, I think its high time to leverage the power of dedicated frameworks for testing django apps. I will take up the case of pytest here because other frameworks like nose are not widely used.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/8c5e0baa-5104-4876-a936-8792b3c8d5b8n%40googlegroups.com.

[René Fleschenberg]

Hi,

On 15.12.20 15:48, Tobias Bengfort wrote:
> My issue is mostly that pytest uses a lot of magic that is hard to
> debug, e.g. the overwritten assert statement or implicit injection of
> fixtures. Most pro arguments seem to depend on personal taste.

It should be noted that you can use pytest / pytest-django as a test
runner without having to use any other pytest features (it can run
unittest-style tests, you don't have to rewrite them). I do think it has
some features in the test runner. For example, the -k command line
option to quickly select tests, or --last-failed to rerun failed tests.

However, since pytest-django exists and has been working flawlessly for
me for several years, I am not sure how much benefit there would be in
having this in Django itself.

Regards,
René

[Ahmad A. Hussein]

Hi Diptesh,

This is a great idea! (And one that has had a fair bit of discussion). I don't think anyone holds an opinion against supporting pytest as a testing tool, but I believe the main hurdle is laying a convincing argument why this support should be in Django Core versus remaining in its current third-party state. I personally believe it should be in core because a lot of custom features we'd like to add to the test runner already exist in pytest's rich ecosystem. Just as Tom mentioned, adding timings was a lot harder than expected while such a feature already exists in pytest.

If we aren't going to add it to core, the question I'd like to raise (and preferably get an answer from experienced pytest-django users/developers) is what changes can we make in core to make it easier for the pytest-django package.

Lastly, if you intend to purely write documentation, I would recommend you apply for next year's Google Season of Docs instead of Google Summer of Code. It would be a better fit for the job description

--

[Diptesh Choudhuri]

Thank you everyone for your inputs! I don't know how you all are using the quote feature, so I will address you all by your names.

• Jacob: Thank you for your support! The ticket you linked to is very helpful. It is upside of 5 years old. pytest has evolved a lot over that time, as has django itself. The points raised in the ticket itself are still valid 5 years later and as django is a mature project now, some of its development prowess must go to specifically fixing its weaknesses (I believe the default runner is one). That being said, I also opened another ticket on the same website today about this. Should I close that and bump the one you linked instead?
• Tobias: I understand most of the issues raised are personal preferences. However these preferences have been cultivated by the django community itself, so it would be safe to assume that a lot of django users would have the same preferences. There is no harm in making these preferences official (by including them in the official docs). Once people have a common convention to follow, they will only appreciate it because it makes their lives easier. pytest indeed does a lot of magic behind the scenes but I intend to demystify this by writing extensive documentation in the Advanced Testing Topics page in the django docs. As for the subTest method, I admit to not having done enough research about it and it is completely new to me. However, since its not in common usage, we can safely assume that people don't like using it, don't you agree?
• Tom: Totally agree with everything you said! Especially being able to write more "pythonic" code with pytest. Even though our definitions of pythonic might be different, I would boldly suggest that functional testing, which pytest prefers, looks much more beautiful and readable than unittest's. class based ones.
• René: Though the option exists, I would humbly argue that its much easier to invoke pytest than it is to invoke python manage.py test. Most of my focus would be on improving the docs, along with a few changes to code-base itself to make pytest the default and generate a sane pytest.ini. On further thought, we can completely remove pytest.ini and instead do all configuration from settings.py itself. Though pytest-django exists, I feel that it can be very unforgiving to users who are completely new to testing (I was one not long ago). It doesn't work right out of the box. I sincerely believe this must and should be worked upon. 
  

Thank you everyone for your inputs and I would love to hear some more feedback. Also if anyone could teach me how to use the quote function, I would highly appreciate it.

Best

Diptesh

[Adam Johnson]

Hi

Whilst I appreciate the effort and love pytest, I don't think this it's feasible at the moment to even suggest to users that they use pytest, let alone make it a default or port Django's unittest-based framework.

For a first step, we have our backwards compatibility policy. Any move away from the unittest-based runner would break users' customizations such as subclasses

Secondly, pytest-django may seem mature, but it does not have feature parity with Django's test framework. Off the top of my head:

• It doesn't run system checks before tests.
  
• It doesn't support a setUpTestData-style shared database fixture with rollback between tests in the "functional" style of tests.
  
• It only has one test database fixture, so it either sets up *all* test databases or none. Django's TestCase supports a "databases" attribute to list the required databases needed in the tests, allowing tests that only use a subset of the databases to only set up those databases.
  
• It doesn't support the TestCase.available_apps attribute.
  
• I don't believe it rearranges TransactionTestCase tests to the end of the test run, allowing them to slow down the run.

There are other examples in the pytest-django issues list. Naturally it's the rare edge case, less-used features that few contributors want to work on that are missing - those users who need them find that pytest-django isn't for them and stick with the django testing framework. And there is a fairly limited bunch of people contributing to both pytest-django and django's test framework.

Third, pytest itself does not have complete feature parity with the basic run-tests options of Django's test framework. For example, Django has the --reverse flag to run tests in the reverse order. I suggested this be added to pytest core, but it was decided to make it into a plugin instead, pytest-reverse, which I now maintain.

I appreciate your willingness to contribute. I think any energy in this space is best focussed on improving pytest-django to improve its feature coverage, and that Django should not have any official support or suggestion to use pytest-django at this time. We could perhaps link to pytest as an alternative in one of the test topic guides, however we do tend to avoid such documentation links since they are prone to bitrot.

Thanks,

Adam

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/64886590-2d95-4c11-ad8d-24d0a746001dn%40googlegroups.com.

--

Adam

[Diptesh Choudhuri]

Ahmad, thanks for your views! Here's what I think.

> I personally believe it should be in core because a lot of custom features we'd like to add to the test runner already exist in pytest's rich ecosystem. Just as Tom mentioned, adding timings was a lot harder than expected while such a feature already exists in pytest.

Adding to this, since features like this already exist in pytest, I don't think we should reinvent the wheel and try to implement them from scratch.

I would say the main reason I would like integrate pytest into the django core is that since it is much loved among the community, it only makes sense to make it official. New users won't have to wrack their heads over unittest's basics (it took me a long time to understand setUp and tearDown and even longer to actually use it in a project). pytest-django also requires you to write much less code for the actual test (compare self.assertEquals(thing1, thing2) with assert thing1 == thing2).

I don't intend to write documentation only. I believe this idea has a lot of scope, and by integrating the features I have already suggested and the ones that will be suggested in the near future, I will end up writing a lot of code. So it will fit GSoC's job description. However I thank you for mentioning Google Summer of Docs, I will definitely look into it.

[Mariusz Felisiak]

Hi,

    I agree with René and Adam, pytest and pytest-django already exist, and I don't see why we should copy it to Django or change the default test runner.

Best,
Mariusz

[Arvind Nedumaran]

A case could be made along the same lines as was made for South.

Get Outlook for Android

---

From: django-d...@googlegroups.com <django-d...@googlegroups.com> on behalf of Mariusz Felisiak <felisiak...@gmail.com>
Sent: Tuesday, December 15, 2020 9:58:56 PM
To: Django developers (Contributions to Django itself) <django-d...@googlegroups.com>
Subject: Re: Replace the default django test runner to a more robust testing framework (like pytest)

 

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/60a4a090-8ff5-477e-9651-db1fbf5c397bn%40googlegroups.com.

[Diptesh Choudhuri]

> I agree with René and Adam, pytest and pytest-django already exist, and I don't see why we should copy it to Django or change the default test runner.

I am not talking about replicating any code-base whatsoever. What I am proposing instead is official django support for pytest. Kind of like how django supports argon2 for password hashing. We can make this a feature using pip install django[pytest] and make appropriate changes to the startproject command to create some sane test and configuration files.

Thanks for your input!

Best

Diptesh

[Diptesh Choudhuri]

> A case could be made along the same lines as was made for South.

Could you please elaborate a bit on this? Maybe provide the relevant tickets? I am quite new to this community.

Thanks!

Best

Diptesh

[Diptesh Choudhuri]

Adam,

Thanks for your detailed feedback.

I will be honest, I couldn't understand any of the points you mentioned (except the first one- this can be easily solved using a pytest hook). As you said, these are very rare cases indeed. As a django and pytest user for some time, I would have encountered atleast one of them when I wrote my tests. However I never have and hence I would say that these are very rare cases indeed. 

What I am trying to say is that if we include pytest in django core, the development of these features will only be accelerated. The default runner advantages you mentioned are not really features that are missing in pytest. Instead they are kind of addons that'd be nice to have. But the advantages that I mention of pytest over the default runner are significant and can tremendously improve developer experience while writing tests.

Please don't think I am disregarding your points. I respect and value your opinions.

Best

Diptesh

[Adam Johnson]

except the first one- this can be easily solved using a pytest hook

It's not easy at all. I tried 4 years ago and failed to find a satisfactory way of running the checks at the appropriate time since the pytest lifecycle (db setup, reporting, etc.) is so different from unittest: https://github.com/pytest-dev/pytest-django/pull/414

Instead they are kind of addons that'd be nice to have.

That's not how we treat stability in Django. These features are necessary, core features for the projects that use them, and we treat them as necessary. Please read the Deprecation Policy: https://docs.djangoproject.com/en/3.1/internals/release-process/#internal-release-deprecation-policy .

I appreciate your willingness to improve things and I agree generally pytest is a better test runner experience. But please channel some of your energy into understanding where Django is, where pytest is, how contributions work etc. You will get further than having a long back and forth where you keep asserting that it would be easy to implement.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/39dc54fb-43e9-424f-9ca3-10938741d733n%40googlegroups.com.

--

Adam

[Diptesh Choudhuri]

I understand what you mean. Here's a proposal.

Instead of removing the default test runner altogether, I can work on adding a TESTING_BACKEND variable in settings.py with possible values being 'django.testing.backends.pytest' (default) or 'django.testing.backends.DefaultRunner'. This way we need not compromise on the features that the default runner provides over pytest. Meanwhile I could work on some the issues pytest has by contributing to pytest-dev.

One small question- will a contribution to repositories such as pytest-dev or graphene-django count as contributions to the Django organization in GSoC?

Best

Diptesh

PS: I am also conducting a small survey about the same. I will post the results here once I get sufficient responses. It would be interesting to see how to common public reacts to such a change.

[James Bennett]

On Tue, Dec 15, 2020 at 6:07 PM Diptesh Choudhuri
<diptesh....@gmail.com> wrote:
> Instead of removing the default test runner altogether, I can work on adding a TESTING_BACKEND variable in settings.py with possible values being 'django.testing.backends.pytest' (default) or 'django.testing.backends.DefaultRunner'. This way we need not compromise on the features that the default runner provides over pytest. Meanwhile I could work on some the issues pytest has by contributing to pytest-dev.

I would be against both the original proposal to switch Django to
pytest, and this proposal.

And it's not just because I'm a little bit biased here -- though I am,
in that I personally dislike pytest and when I use it in personal
projects I use it solely as a runner, where the tests themselves all
use the unittest and mock modules in the Python standard library.

There are several independent reasons why I don't think either of
these proposals is a good fit for Django:

One is that even though it's easier at first to add dependencies to
Django now than it was 15 years ago, every dependency still should be
carefully considered. It wasn't too long ago that this proposal would
have been "switch the Django test runner to nose", and now nose is
unmaintained and deprecated (its last release was in 2015) and though
there's a maintained "nose2" fork, it's not compatible with the
original and doesn't have anywhere near the development momentum the
original had. You might observe that pytest shows no signs of having
that happen to it, but the fact that it *can* happen has to be
considered when we're talking about integrating it into one of
Django's core components; we can think it's likely, and be hopeful,
that pytest would still be around and actively maintained 5 years from
now, but we can be much more certain that the Python standard
library's unittest module will still be around and actively
maintained.

Another is that pytest would be effectively a test-only dependency,
and the history of *those* in Python projects is largely an unhappy
and complicated one. It would create a potential risk for unwary users
(forgetting to install the extra dependency, or to do "pip install
Django[pytest]" or similar), or force every installation of Django --
including production installations which often don't want a large
third-party framework that won't be used at all in production -- to
also install pytest.

Still another is the basic incompatibility between pytest and unittest
runners. While it's true that Django lets you configure a lot of
things, they all are handled through carefully normalized APIs, so
that you don't notice the difference. This is what lets you start out
development with a local SQLite database and switch to Postgres when
you deploy, for example, or swap out cache or session or
authentication or email-sending backends without needing to rewrite
your own code that uses those features, because Django provides
normalized APIs for accessing whichever backend it's been configured
to use. But pytest is not that simple: pytest can run tests written in
unittest style, but unittest cannot run tests written in pytest style.
Only pytest -- which understands its own conventions for setup,
fixture loading, and so on -- can do this. Which means that if I write
my Django app using pytest style, and you want to use it in your
project that's otherwise completely unittest style, you cannot
integrate with full testing support unless you also switch your whole
project's configuration to pytest. That's a *big* difference from,
say, switching to a new auth backend.

So I'm in agreement with others in this thread that if you want to
improve things for people who prefer to use pytest, contributing to
pytest, pytest-django, and related projects would be a fine way to do
so. But trying to switch Django itself to pytest, or to introduce a
"test backend" concept that would have a pytest runner, does not seem
to me like a good idea.

[Diptesh Choudhuri]

Alright I think the arguments made against this are pretty valid and this can't be continued. If you're interested, however, here is the result of the survey I mentioned earlier.

I will find some other way to make myself useful to this project. Thank you every one for your inputs!

[Diptesh Choudhuri]