test discovery

[Carl Meyer]

Hi all,

Preston Timmons and I have been working the last several weeks to get
the test discovery branch (#17365) ready for merge, and I think we now
have a pull request ready for consideration:
https://github.com/django/django/pull/1050

Brief summary of the features, changes, and open questions:

* You can now place tests in any file matching the pattern 'test*.py',
anywhere in your codebase, rather than only in tests.py and models.py
modules of an INSTALLED_APP. The filename pattern is customizable via
the --pattern argument to "manage.py test".

* When you run "manage.py test" with no arguments, tests are discovered
and run in all subdirectories (recursively) of the current working
directory. (This means that django.contrib and other third-party app
tests are no longer run by default).

* When you pass arguments to "manage.py test", they are now full Python
dotted module paths. So if you have a "myproject.myapp" app, and its
tests.py contains "SomeTestCase", you would now run that single test
case via "manage.py myproject.myapp.tests.SomeTestCase" rather than
"manage.py myapp.SomeTestCase". This is longer, but allows more control
when an app's tests are split into multiple files, and allows for tests
to be placed anywhere you like.

* Doctests are no longer discovered by default; you will need to
explicitly integrate them with your test suite as recommended in the
Python docs: http://docs.python.org/2/library/doctest.html#unittest-api

* Tests in models.py and tests/__init__.py will no longer be discovered
(as those don't match the default filename pattern).

* The old test runner, and Django's extensions to the doctest module,
are deprecated and will be removed in Django 1.8; they could of course
be packaged separately if some people would like to continue using them.

Open question: how to handle the transition?

Jacob has suggested that back-compat breaks in test-running are not as
serious as in production code, and that we should just switch to the new
test runner by default in Django 1.6. This is what the pull request
currently does. This will mean that some people's test suites will
likely be broken when they upgrade to 1.6. They would have two options,
both documented in the release notes: they can update their test suite
to be discovery-compatible immediately, or they can just set TEST_RUNNER
to point to the old runner and get back the old behavior, which they can
keep using until Django 1.8 (or longer, if they package the old runner
externally).

The alternative would be to keep the old test runner as the default in
global_settings.py until it is removed in 1.8, and add the new runner as
the TEST_RUNNER in the startproject-generated settings.py. This would
provide a gentler transition for upgrading projects. On the other hand,
we just recently got the startproject settings.py all cleaned-up,
slimmed-down, and newbie-friendly, so it would make me sad to start
polluting it again with stuff that new projects generally shouldn't care
about, for solely legacy reasons.

Thoughts, questions, comments, code review and testing welcome! I'd like
to merge this on Friday (it's a bear to keep updated with trunk), so let
me know if you need longer.

Carl

[Donald Stufft]

On May 8, 2013, at 5:00 PM, Carl Meyer <ca...@oddbird.net> wrote:

> Jacob has suggested that back-compat breaks in test-running are not as
> serious as in production code, and that we should just switch to the new
> test runner by default in Django 1.6. This is what the pull request
> currently does. This will mean that some people's test suites will
> likely be broken when they upgrade to 1.6. They would have two options,
> both documented in the release notes: they can update their test suite
> to be discovery-compatible immediately, or they can just set TEST_RUNNER
> to point to the old runner and get back the old behavior, which they can
> keep using until Django 1.8 (or longer, if they package the old runner
> externally).

This sounds reasonable to me. Tests are not production code so I agree with Jacob.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[Aymeric Augustin]

On 8 mai 2013, at 23:00, Carl Meyer <ca...@oddbird.net> wrote:

> Jacob has suggested that back-compat breaks in test-running are not as
> serious as in production code, and that we should just switch to the new
> test runner by default in Django 1.6. This is what the pull request
> currently does.

That seems reasonable to me.

Thank you very much for working on this.

--
Aymeric.

[Anssi Kääriäinen]

It would be really nice to be able to use same syntax for running a
single Django's testcase at least for a while. I expect that there
will be problems if inspecting how a given test case behaves in older
versions compared to HEAD. Also, bisecting over the test case renaming
boundary will be really ugly. I don't believe end users will have too
much problems with this, but bug hunting and development in Django
itself will be a bit more tedious to do.

I don't have any bright ideas of how to do this nicely. Maybe just try
the new way, if no tests found, then try to convert appname.SomeClass
to projectname.appname.tests.SomeClass, and somehow do this only for
tests/runtests.py.

Otherwise the plan looks good to me. And the above issue isn't
anything severe, just something that would make developing Django a
bit easier in certain cases.

- Anssi

[Jacob Kaplan-Moss]

On Wed, May 8, 2013 at 2:00 PM, Carl Meyer <ca...@oddbird.net> wrote:
> Jacob has suggested that back-compat breaks in test-running are not as
> serious as in production code, and that we should just switch to the new
> test runner by default in Django 1.6.

I still think this. I don't see the need to maintain compatibility in
testing, especially when compatibility is just a TEST_RUNNER change
away.

Jacob

[Carl Meyer]

Hi Anssi,

On 05/08/2013 03:31 PM, Anssi K��ri�inen wrote:
> It would be really nice to be able to use same syntax for running a
> single Django's testcase at least for a while. I expect that there
> will be problems if inspecting how a given test case behaves in older
> versions compared to HEAD. Also, bisecting over the test case renaming
> boundary will be really ugly. I don't believe end users will have too
> much problems with this, but bug hunting and development in Django
> itself will be a bit more tedious to do.

I don't think this is enough of a problem to warrant more code to try to
work around it.

If you're bisecting, at worst this means having two commands in your
shell history instead of one and hopping up to the right one for each
bisect step. I think this is a minor irritation that will fade away soon
enough.

Most of our test apps are small/fast enough that you could just run the
whole app when bisecting instead of an individual TestCase, making this
a non-issue (since Django's test apps are top-level on sys.path, the
command to run a single app in Django's test suite is the same with the
new runner as with the old: "python runtests.py app_name").

Carl

[Tom Christie]

Hi Carl,

  Excellent, excellent stuff - many thanks to both yourself and Preston.

Couple of questions:

* Will it be possible to globally configure the default file/path pattern?

  Jannis's django-discover-runner includes support for a `TEST_DISCOVER_PATTERN` - is there anything similar, or just the command line `--patterns` option?  The use case I'm looking for is a single `tests` directory containing all the test modules, and I'm wondering if that'd be easy to support?

* Is this work broadly compatible with `django-discover-runner`?  I'd really like to make use of it in third party packages, but I'm wondering if I'd be able to do so in a way that lets me also provide compatibility with older versions of Django.

Thanks again,

  Tom

On Wednesday, 8 May 2013 23:04:29 UTC+1, Carl Meyer wrote:

Hi Anssi,

[Russell Keith-Magee]

Great work Carl and Preston! (And everyone else who had a hand in the patch - I know this has been kicking around for a while now)

 

Open question: how to handle the transition?

Jacob has suggested that back-compat breaks in test-running are not as
serious as in production code, and that we should just switch to the new
test runner by default in Django 1.6. This is what the pull request
currently does. This will mean that some people's test suites will
likely be broken when they upgrade to 1.6. They would have two options,
both documented in the release notes: they can update their test suite
to be discovery-compatible immediately, or they can just set TEST_RUNNER
to point to the old runner and get back the old behavior, which they can
keep using until Django 1.8 (or longer, if they package the old runner
externally).

The alternative would be to keep the old test runner as the default in
global_settings.py until it is removed in 1.8, and add the new runner as
the TEST_RUNNER in the startproject-generated settings.py. This would
provide a gentler transition for upgrading projects. On the other hand,
we just recently got the startproject settings.py all cleaned-up,
slimmed-down, and newbie-friendly, so it would make me sad to start
polluting it again with stuff that new projects generally shouldn't care
about, for solely legacy reasons.

I agree with Jacob's tests aren't production sensitive, so making a fast transition to the new test discovery has less impact.

However, we know from experience that it doesn't matter how much we flag this change in the release notes, they won't be read, and *someone* is going to get stung. In an ideal world, it would be good to be able to warn people who upgrade that their test suite may not be run as expected.

We've had a proposal kicking around for a while to add a management command that does an "upgrade check". I'm wondering if this might be an opportune time to dig into this some more.

To be clear, this isn't something I consider to be a merge blocker. I'd be happy to see 1.6 released with a fast transition to the new test runner. It would just be nice icing on the cake if we can get an upgrade validator included in the same release.

Yours,

Russ Magee %-)

[Andrew Godwin]

Just want to say that I'm happy with a "fast transition".

Is there a possibility we can detect the case where the tests might be broken (how might they be?) and print a helpful error message?

Andrew

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Carl Meyer]

Hi Tom,

On 05/09/2013 03:27 AM, Tom Christie wrote:
> * Will it be possible to globally configure the default file/path pattern?
> Jannis's django-discover-runner includes support for a
> `TEST_DISCOVER_PATTERN` - is there anything similar, or just the command
> line `--patterns` option? The use case I'm looking for is a single
> `tests` directory containing all the test modules, and I'm wondering if
> that'd be easy to support?

One of the goals here was to avoid adding more settings, so there isn't
a way to configure the pattern globally at this point. That may be
something we want to revisit, though; if you do need a different
pattern, you'll need it every time you run your tests, which basically
means you're either wrapping up "manage.py test" in a shell script or
overriding the test command with your own tweaked version. If we can
avoid the need for that by adding a setting, it might be worth it.

Your use case is very easy to support if you don't mind naming your test
files inside the tests/ directory all test_something.py. If you don't
like that, then you might need to use something like --pattern "*.py".

> * Is this work broadly compatible with `django-discover-runner`? I'd
> really like to make use of it in third party packages, but I'm wondering
> if I'd be able to do so in a way that lets me also
> provide compatibility with older versions of Django.

I think the capabilities of the command-line interface in this patch are
a superset of the capabilities of discover-runner. It's smarter about
figuring some pythonpath things out, and better about handling multiple
test labels, but anything that works with discover-runner should work
here too.

If necessary it wouldn't be at all hard to backport this code into
discover-runner so they are exactly the same; I haven't discussed this
with Jannis yet.

Carl

[Carl Meyer]

Hi Russ,

On 05/09/2013 04:51 AM, Russell Keith-Magee wrote:
> Great work Carl and Preston! (And everyone else who had a hand in the
> patch - I know this has been kicking around for a while now)

Thanks for the reminder; I should have mentioned that Mahdi Yusuf did
the initial draft patch, based on Jannis' django-discover-runner, which
was based on code I showed on a slide in my "Django and Testing" talk at
PyCon 2012; it all comes full circle :-)

> I agree with Jacob's tests aren't production sensitive, so making a fast
> transition to the new test discovery has less impact.
>
> However, we know from experience that it doesn't matter how much we flag
> this change in the release notes, they won't be read, and *someone* is
> going to get stung. In an ideal world, it would be good to be able to
> warn people who upgrade that their test suite may not be run as expected.
>
> We've had a proposal kicking around for a while to add a management
> command that does an "upgrade check". I'm wondering if this might be an
> opportune time to dig into this some more.
>
> To be clear, this isn't something I consider to be a merge blocker. I'd
> be happy to see 1.6 released with a fast transition to the new test
> runner. It would just be nice icing on the cake if we can get an upgrade
> validator included in the same release.

It wouldn't be too hard to write something that tried test discovery
using the old runner and using the new runner and did a simple
comparison of test counts to warn you if they differ.

I'm not personally likely to work on this "upgrade checks" framework,
but if it does happen I'd be happy to contribute this check to it.

Carl

[Carl Meyer]

On 05/09/2013 07:50 AM, Andrew Godwin wrote:
> Just want to say that I'm happy with a "fast transition".
>
> Is there a possibility we can detect the case where the tests might be
> broken (how might they be?) and print a helpful error message?

Two kinds of potential breakage:

- The most common will be missing tests; doctests, or tests in files
that don't match the test*.py pattern, which are currently run because
they are in models.py or manually imported into a tests/__init__.py,
won't be discovered by the new runner. Some people may also consider it
a "breakage" that tests in their pip-installed external apps are no
longer run by default.

- Less common will be tests that are in the codebase for some reason
(vendored third-party code, for instance?) that weren't found or run by
the previous runner, but are found and run (and perhaps fail) with the
new runner.

The first kind of breakage I don't think we can reasonably discover at
runtime; it would mean trying both discovery methods on every single
test run and comparing test counts. We could do this in a special
"upgrade check" like Russ suggested, but not every time you run tests.

The second issue I'm not too concerned about. If the
previously-undiscovered tests pass, then there's not really a problem.
If they fail, it shouldn't be too hard to figure out what's going on.

Carl

[Russell Keith-Magee]

I thought about that approach, but my immediate reaction was that it will end up producing more false positives than helpful feedback. Even the simple case -- myapp.TestCase -> myapp.tests.TestCase -- is going to require some name munging to work out if the old test and the new test are actually the same one. Since this is for a short-lived transitional tool, the effort involved in getting it right exceeds the value gained by having it, IMHO.

I'd prefer something much simpler:

 * On syncdb/validate, check for a marker somewhere in user space. 

 * If the marker isn't present, do a check for likely problems. In this case, look for the value of TEST_RUNNER; if it's set to the new default value, display a warning about the change in test discovery format. Add similar checks for other setting changes, or warnings about features that have been fully deprecated (e.g., the final transition of the {% url %} behavior)

 * Provide a management command to set marker. 

 * On the next syncdb/validate, the marker is present, so the install is verified as being "updated", and no warnings are displayed.

This leaves the question of where to put the marker. Some initial ideas:

 - a file on disk (e.g., an .updated file next to the settings file)

 - a new setting (VERIFIED_VERSION = "1.5")

 - a key-value table in a database used only for Django administrivia

Essentially, this would give us a place to put the "NO REALLY, READ THIS" warnings that we need in release notes, and make those messages slightly targeted.

Yours,

Russ Magee %-)

[Carl Meyer]

I wasn't anticipating trying to actually match up individual tests by name/path, just trying both runners and seeing whether the test counts match. On second thought, though, I realize that contrib / third-party apps will mean this check would alert "missing tests!" for pretty much everyone, so it would need to be smarter to be useful, and you're right that's likely not worth it.
 

I'd prefer something much simpler:

 * On syncdb/validate, check for a marker somewhere in user space. 

 * If the marker isn't present, do a check for likely problems. In this case, look for the value of TEST_RUNNER; if it's set to the new default value, display a warning about the change in test discovery format. Add similar checks for other setting changes, or warnings about features that have been fully deprecated (e.g., the final transition of the {% url %} behavior)

 * Provide a management command to set marker. 

 * On the next syncdb/validate, the marker is present, so the install is verified as being "updated", and no warnings are displayed.

This leaves the question of where to put the marker. Some initial ideas:

 - a file on disk (e.g., an .updated file next to the settings file)

 - a new setting (VERIFIED_VERSION = "1.5")

 - a key-value table in a database used only for Django administrivia

Essentially, this would give us a place to put the "NO REALLY, READ THIS" warnings that we need in release notes, and make those messages slightly targeted.

In theory this makes sense to me, but I think the question of where to put the "I already ran the upgrade check" marker is thorny. Database is ugly, since there's no requirement for a Django project to even use the ORM at all; and requiring projects using the ORM to have a magical extra not-related-to-a-voluntarily-installed-app table in their database sticks in my craw. Similar with adding a file to the filesystem; we'd essentially be requiring everyone to have this file in order to avoid the extra overhead (and possible noise) of the check on startup; forcing that type of cruft on everyone using Django is a non-starter in my book. Of those three ideas, I'd say a new setting has the least downside, but even that is extra cruft we're forcing into everyone's settings.py file.

I think perhaps better is simply an "upgradecheck" management command that does checks like this relative to the last version of Django. Of course, this has to be manually run, so it doesn't have the "catch everyone" benefit of something that happens automatically at validation, but if it's useful I think it's likely more people would run it than bother to read through the release notes.

Carl

[Carl Meyer]

I merged this patch tonight. Thanks to everyone who contributed! Now let's see how the CI servers feel about it...

Carl

[Yo-Yo Ma]

Wow, this is awesome! Thanks so much guys. Too long have I dreamt of the day when I could include tests for my "lib" and "util", etc. without having to couple them to one app or another. I'm so excited that this was added so quickly.

Thanks
Yo-yo

[Florian Apolloner]

Not good, at least our Jenkins runner which should generate xml output doesn't like it :/

[Florian Apolloner]

Hi Carl,

before I read all the tickets etc; why does runtests.py not use the TEST_RUNNER from settings.py (see https://github.com/django/django/commit/9012833af857e081b515ce760685b157638efcef#L60L149)? We'd need that for jenkins to produce xml files as output.

Thanks,
Florian

[Carl Meyer]

Hi Florian,

On May 11, 2013, at 7:51 AM, Florian Apolloner <f.apo...@gmail.com> wrote:

Hi Carl,

before I read all the tickets etc; why does runtests.py not use the TEST_RUNNER from settings.py (see https://github.com/django/django/commit/9012833af857e081b515ce760685b157638efcef#L60L149)? We'd need that for jenkins to produce xml files as output.

No good reason, just an oversight I think. If that's all that's needed to make the CI happy, feel free to change it, should be a simple fix.

Carl

[Florian Apolloner]

On Saturday, May 11, 2013 2:38:23 PM UTC+2, Carl Meyer wrote:

No good reason, just an oversight I think. If that's all that's needed to make the CI happy, feel free to change it, should be a simple fix.

Perfect, pushed a fix, let's see what jenkins says.

[Chris Wilson]

Hi all,

On Fri, 10 May 2013, Carl Meyer wrote:

> I merged this patch tonight. Thanks to everyone who contributed! Now
> let's see how the CI servers feel about it...

I think Travis is unhappy about something in this commit. Any ideas?

======================================================================
ERROR: test_file_path
(test_runner.test_discover_runner.DiscoverRunnerTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/home/travis/build/aptivate/django/tests/test_runner/test_discover_runner.py",
line 65, in test_file_path
["test_discovery_sample/"],
File "/home/travis/build/aptivate/django/django/test/runner.py", line
63, in build_suite
tests = self.test_loader.loadTestsFromName(label)
File "/usr/lib/python2.7/unittest/loader.py", line 91, in
loadTestsFromName
module = __import__('.'.join(parts_copy))
ImportError: Import by filename is not supported.

<https://next.travis-ci.org/aptivate/django/jobs/7275360>

Cheers, Chris.
--
Aptivate | http://www.aptivate.org | Phone: +44 1223 967 838
Future Business, Cam City FC, Milton Rd, Cambridge, CB4 1UY, UK

Aptivate is a not-for-profit company registered in England and Wales
with company number 04980791.

[Chris Wilson]

On Sat, 18 May 2013, Chris Wilson wrote:

> I think Travis is unhappy about something in this commit. Any ideas?
>
> ======================================================================
> ERROR: test_file_path (test_runner.test_discover_runner.DiscoverRunnerTest)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File
> "/home/travis/build/aptivate/django/tests/test_runner/test_discover_runner.py",
> line 65, in test_file_path
> ["test_discovery_sample/"],
> File "/home/travis/build/aptivate/django/django/test/runner.py", line 63,
> in build_suite
> tests = self.test_loader.loadTestsFromName(label)
> File "/usr/lib/python2.7/unittest/loader.py", line 91, in loadTestsFromName
> module = __import__('.'.join(parts_copy))
> ImportError: Import by filename is not supported.
>
> <https://next.travis-ci.org/aptivate/django/jobs/7275360>

It seems to be checking if a file exists, using a relative path, in
django/test/runner.py:65:

if not os.path.exists(label_as_path):
tests = self.test_loader.loadTestsFromName(label)

And that will behave differently depending if you run your tests from
inside the tests directory like this:

./runtests.py -v2 --settings=test_postgres_nogis test_runner

Or from the parent directory, as Travis currently does, like this:

python -Wall tests/runtests.py --selenium --verbosity 2 \
--settings=django_settings

I can work around it in Travis by changing working directory before
running the tests, but is it worth fixing this in the test runner, perhaps
using an absolute path based on __file__?

[Chris Wilson]

Hi all,

Another odd behaviour of the new test runner. This runs the tests, but
fails to add the test app to INSTALLED_APPS, so they all fail because
their tables are not created:

PYTHONPATH=.. python -Wall runtests.py --selenium --verbosity 2 \
--settings=tests.travis_configs.test_postgres_nogis \
tests.transactions.tests.AtomicInsideTransactionTests

Changing the spec to remove the initial "tests." results in the app being
loaded and the tests run:

PYTHONPATH=.. python -Wall runtests.py --selenium --verbosity 2 \
--settings=tests.travis_configs.test_postgres_nogis \
transactions.tests.AtomicInsideTransactionTests

Is this expected behaviour? It's rather counter-intuitive to me.

[Carl Meyer]

Hi Chris,

On May 18, 2013, at 9:42 AM, Chris Wilson <ch...@aptivate.org> wrote:
> Another odd behaviour of the new test runner. This runs the tests, but fails to add the test app to INSTALLED_APPS, so they all fail because their tables are not created:
>
> PYTHONPATH=.. python -Wall runtests.py --selenium --verbosity 2 \
> --settings=tests.travis_configs.test_postgres_nogis \
> tests.transactions.tests.AtomicInsideTransactionTests
>
> Changing the spec to remove the initial "tests." results in the app being loaded and the tests run:
>
> PYTHONPATH=.. python -Wall runtests.py --selenium --verbosity 2 \
> --settings=tests.travis_configs.test_postgres_nogis \
> transactions.tests.AtomicInsideTransactionTests
>
> Is this expected behaviour? It's rather counter-intuitive to me.

The initial "tests." prefix shouldn't be used, and wasn't intended to be supported. There is no __init__.py in the tests/ directory, and when running runtests.py the tests/ directory itself is placed on sys.path, so every module in there is a top-level module when running tests. I'm actually surprised unittest itself even runs the tests when specified that way, given the lack of __init__.py.

That said, if this turns out to to be something that trips a lot of people up, it would be trivial to add a special-case in runtests.py to look for and remove the initial "tests." in provided test labels.

Carl

[Carl Meyer]

Hi Chris,

On May 18, 2013, at 8:34 AM, Chris Wilson <ch...@aptivate.org> wrote:

> On Sat, 18 May 2013, Chris Wilson wrote:
>
>> I think Travis is unhappy about something in this commit. Any ideas?
>>
>> ======================================================================
>> ERROR: test_file_path (test_runner.test_discover_runner.DiscoverRunnerTest)
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>> File "/home/travis/build/aptivate/django/tests/test_runner/test_discover_runner.py", line 65, in test_file_path
>> ["test_discovery_sample/"],
>> File "/home/travis/build/aptivate/django/django/test/runner.py", line 63, in build_suite
>> tests = self.test_loader.loadTestsFromName(label)
>> File "/usr/lib/python2.7/unittest/loader.py", line 91, in loadTestsFromName
>> module = __import__('.'.join(parts_copy))
>> ImportError: Import by filename is not supported.
>>
>> <https://next.travis-ci.org/aptivate/django/jobs/7275360>
>
> It seems to be checking if a file exists, using a relative path, in django/test/runner.py:65:
>
> if not os.path.exists(label_as_path):
> tests = self.test_loader.loadTestsFromName(label)
>
> And that will behave differently depending if you run your tests from inside the tests directory like this:
>
> ./runtests.py -v2 --settings=test_postgres_nogis test_runner
>
> Or from the parent directory, as Travis currently does, like this:
>
> python -Wall tests/runtests.py --selenium --verbosity 2 \
> --settings=django_settings
>
> I can work around it in Travis by changing working directory before running the tests, but is it worth fixing this in the test runner, perhaps using an absolute path based on __file__?

I don't think this should be fixed in the test runner itself; in general, file-path test labels _should_ be interpreted as relative to wherever you are running the tests from.

But it should be fixed in the test_runner.test_discover_runner.DiscoverRunnerTest.test_file_path test - that test apparently needs to isolate itself better by setting the CWD for the duration of the test, or something similar. Mind filing a bug? I should be able to take a look soon.

Carl

[Chris Wilson]

Hi Carl,

On Sat, 18 May 2013, Carl Meyer wrote:

> I don't think this should be fixed in the test runner itself; in
> general, file-path test labels _should_ be interpreted as relative to
> wherever you are running the tests from.
>
> But it should be fixed in the
> test_runner.test_discover_runner.DiscoverRunnerTest.test_file_path test
> - that test apparently needs to isolate itself better by setting the CWD
> for the duration of the test, or something similar. Mind filing a bug? I
> should be able to take a look soon.

Done, thanks: https://code.djangoproject.com/ticket/20449