[Django] #23778: Running tests outside the test runner needs clarification
Django
--------------------------------------+-----------------------------
Reporter: Starou | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Documentation | Version: 1.7
Severity: Normal | Keywords: tests run_tests
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
--------------------------------------+-----------------------------
Hi devs,
=== ''Some context first''
For one of my pluggable application I have two different test suites, one
of them being a Django dummy project.
In order to have a correct Coverage report, I run both of them in a
runtests.py script.
[https://docs.djangoproject.com/en/dev/topics/testing/advanced/#running-
tests-outside-the-test-runner The Documentation] is a bit confusing and I
think a working snippet could be useful here.
For example, it is not required to execute `setup_test_environment()` nor
`django.test.runner.DiscoverRunner.setup_databases()` if you execute
`DiscoverRunner.run_tests()` which is what you want in the end.
The following snippet could be useful to the newcomers don't you think ?
{{{
import os
import sys
import django
from django.test.runner import DiscoverRunner
sys.path.insert(0, '../..') # path to the development sources.
sys.path.insert(0, 'test_project') # path the the test project.
os.environ['DJANGO_SETTINGS_MODULE'] = 'test_project.settings'
os.environ['DJANGO_LIVE_TEST_SERVER_ADDRESS'] = '127.0.0.1:8082'
django.setup()
runner = DiscoverRunner()
runner.run_tests(['my_app'])
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/23778>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: Starou | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Documentation | Version: 1.7
Keywords: tests run_tests | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* needs_better_patch: => 0
* needs_docs: => 0
* needs_tests: => 0
* stage: Unreviewed => Accepted
Comment:
Yes, something like this seems like a good idea. We shouldn't suggest
using `sys.path` though, per #19941 (comment 9 and later). Would you like
to submit a patch? I'll be happy to review it.
--
Ticket URL: <https://code.djangoproject.com/ticket/23778#comment:1>
Django
Reporter: Starou | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Documentation | Version: 1.7
Keywords: tests run_tests | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Starou):
Replying to [comment:1 timgraham]:
> Yes, something like this seems like a good idea. We shouldn't suggest
using `sys.path` though, per #19941 (comment 9 and later).
In my case - AFAIR - using `PYTHONPATH` does not works because it came
after installed packages in the resolution order (because I want my test
project to run against the source located in `../..` and not against the
sources I may have system-wide installed.
This is only true for the first `sys.path.insert(0, '../../')`, the
test_project path can be put in `PYTHONPATH`.
In my case, I prefered not to mix both.
Anyway, the first insert is very specific and can be removed and the
second converted in a `PYTHONPATH` expression.
> Would you like to submit a patch? I'll be happy to review it.
With pleasure.
--
Ticket URL: <https://code.djangoproject.com/ticket/23778#comment:2>
Django
Reporter: Starou | Owner: Starou
Type: Cleanup/optimization | Status: assigned
Component: Documentation | Version: 1.7
Severity: Normal | Resolution:
Keywords: tests run_tests | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Starou):
* owner: nobody => Starou
* status: new => assigned
* has_patch: 0 => 1
Comment:
PR: https://github.com/django/django/pull/3485
--
Ticket URL: <https://code.djangoproject.com/ticket/23778#comment:3>
Django
Reporter: Starou | Owner: Starou
Type: Cleanup/optimization | Status: assigned
Severity: Normal | Resolution:
Keywords: tests run_tests | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by prestontimmons):
This example is okay. I have a couple thoughts, though:
I assume the most common case for this is creating a test runner for
reusable apps. In that case, messing with `PYTHONPATH` is often
unnecessary since you can either `pip install --editable` or `python
setup.py develop` to put your packages on the path. Is it worth mentioning
this?
The example hard-codes the app label in `runner.run_tests(['my_app'])`.
Even with reusable apps it's nice to have the option of passing a path to
a test case, like `./runtests.py myapp.tests.SomeCase.test`. Should the
example take that into consideration?
I usually do something like:
{{{
runner.runtests(sys.argv[1:] or ["myapp"])
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/23778#comment:4>
Django
Reporter: Starou | Owner: Starou
Type: Cleanup/optimization | Status: assigned
Severity: Normal | Resolution:
Keywords: tests run_tests | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Starou):
Replying to [comment:4 prestontimmons]:
> This example is okay. I have a couple thoughts, though:
>
> I assume the most common case for this is creating a test runner for
reusable apps. In that case, messing with `PYTHONPATH` is often
unnecessary since you can either `pip install --editable` or `python
setup.py develop` to put your packages on the path. Is it worth mentioning
this?
Hi Preston,
The PYTHONPATH in the PR snippet is about resolving
`DJANGO_SETTINGS_MODULE` by adding the path of the test project, not the
reusable app dev sources path (which is assumed). But maybe this is not
clear enough ?
>
> The example hard-codes the app label in `runner.run_tests(['my_app'])`.
Even with reusable apps it's nice to have the option of passing a path to
a test case, like `./runtests.py myapp.tests.SomeCase.test`. Should the
example take that into consideration?
>
> I usually do something like:
>
> {{{
> runner.runtests(sys.argv[1:] or ["myapp"])
> }}}
Good point.
I wanted the snippet as simple as possible (how to convert a `manage.py
test` shell invocation into Python code). In that regard, I would replace
`sys.argv[1:] or ['myapp']` by `sys.argv[1:]` and update the runtests.py
invocation example with some optional arguments ''à la Django''
(`[path.to.modulename|path.to.modulename.TestCase|path.to.modulename.TestCase.test_method]`).
What do you think ?
--
Ticket URL: <https://code.djangoproject.com/ticket/23778#comment:5>
Django
Reporter: Starou | Owner: Starou
Type: Cleanup/optimization | Status: assigned
Severity: Normal | Resolution:
Keywords: tests run_tests | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by prestontimmons):
I think there's merit to keeping your example minimal.
I just started imagining a best-practice type section of the docs for
running tests in reusable apps. It would cover things like:
* Using the test runner in a script
* Managing python path
* Adding test-only models
* Configuring settings manually when including an example project is
overkill
* etc.
This assumes there's a relative consensus of what "best practice" means.
I don't think that's in scope of this ticket, though.
--
Ticket URL: <https://code.djangoproject.com/ticket/23778#comment:6>
Django
Reporter: Starou | Owner: Starou
Type: Cleanup/optimization | Status: assigned
Severity: Normal | Resolution:
Keywords: tests run_tests | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Starou):
This came to my mind too: a more elaborate example in
[https://docs.djangoproject.com/en/1.7/intro/reusable-apps/ advanced
tutorial documentation] (''How to write reusable apps'').
The Django [https://github.com/django/django/blob/master/tests/runtests.py
runtests.py] script and the adopted structure of the test packages looks
like the obvious starting point to me.
But as you said, this is for another ticket!
--
Ticket URL: <https://code.djangoproject.com/ticket/23778#comment:7>
Django
Reporter: Starou | Owner: Starou
Type: Cleanup/optimization | Status: assigned
Severity: Normal | Resolution:
Keywords: tests run_tests | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Starou):
[https://github.com/django/django/pull/3485/files PR] updated with a more
elaborated example as suggested by @carljm too.
--
Ticket URL: <https://code.djangoproject.com/ticket/23778#comment:8>
Django
Reporter: Starou | Owner: Starou
Type: Cleanup/optimization | Status: assigned
Severity: Normal | Resolution:
Keywords: tests run_tests | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* needs_better_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/23778#comment:9>
Django
Reporter: Starou | Owner: Starou
Type: Cleanup/optimization | Status: assigned
Severity: Normal | Resolution:
Keywords: tests run_tests | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Comment (by Starou):
[https://github.com/django/django/pull/3485 PR] updated to get something
minimalistic. Thanks @carljm for putting me on that path!
--
Ticket URL: <https://code.djangoproject.com/ticket/23778#comment:10>
Django
Reporter: Starou | Owner: Starou
Component: Documentation | Version: 1.7
Severity: Normal | Resolution: fixed
Keywords: tests run_tests | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Changes (by Carl Meyer <carl@…>):
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"06726965c3e53e9a6b87e1532951a93d5f94f426"]:
{{{
#!CommitTicketReference repository=""
revision="06726965c3e53e9a6b87e1532951a93d5f94f426"
Fixed #23778 -- Added a doc section on using the Django runner for
reusable apps.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/23778#comment:11>
Django
Reporter: Starou | Owner: Starou
Type: Cleanup/optimization | Status: closed
Severity: Normal | Resolution: fixed
Keywords: tests run_tests | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Comment (by Carl Meyer <carl@…>):
In [changeset:"4ea39230fc2c77cc091ebbbdb3d445920b299746"]:
{{{
#!CommitTicketReference repository=""
revision="4ea39230fc2c77cc091ebbbdb3d445920b299746"
[1.7.x] Fixed #23778 -- Added a doc section on using the Django runner for
reusable apps.
Backport of 06726965c3e53e9a6b87e1532951a93d5f94f426 from master.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/23778#comment:12>