Adding an option to re-test only failed tests
Rob Madole
really like is the ability to run the tests again but filter only the
ones that caused a problem.
I'm thinking it would look something like this
./manage.py test --failed
Does this sound worthwhile to anybody?
Rob
Simon Willison
which tests failed somewhere? If so, where?
If we're talking about features from nose, the two I'd really like in
Django's test runner are --pdb and --pdb-failures:
--pdb = when an error occurs, drop straight in to the interactive
debugger
--pdb-failures = when a test assertion fails, drop in to the debugger
Cheers,
Simon
Alex Gaynor
suite as normal but as soon as it hits a failure it spits out the fail
and ends. Unladen Swallow added this to their test suite and it's a
huge help in terms of not running the full test suite to see what the
failure was.
Alex
--
"I disapprove of what you say, but I will defend to the death your
right to say it." -- Voltaire
"The people's good is the highest law." -- Cicero
"Code can always be simpler than you think, but never as simple as you
want" -- Me
Waylan Limberg
>> I've been using nose for our tests, and one of the features that I
>> really like is the ability to run the tests again but filter only the
>> ones that caused a problem.
>>
>> I'm thinking it would look something like this
>>
>> ./manage.py test --failed
>>
>> Does this sound worthwhile to anybody?
>
> I don't understand how this works - does it persist some indication of
> which tests failed somewhere? If so, where?
>
My recollection is yes, nose persists which tests passed/failed for
the previous run, but I don't recall where.
Personally, I've never used this (except to see how it works in nose)
because I'm always concerned that while a recent change may fix a
failing test, it could cause some other test to fail that previously
passed. So I'm going to run the whole test suite anyway - or at least
all the tests for that module, etc.
Alex suggestion of --failfast seems like a much more useful way to
shorten test runs.
--
----
\X/ /-\ `/ |_ /-\ |\|
Waylan Limberg
Rob Madole
The way this works in nose is through the testid plugin. Typically you
do this:
nosetests --with-id --failed
This will create a file called .noseids in the current working
directory.
You can make it use something else by saying:
nosetests --with-id --id-file=/somewhere/else/.noseids --failed
As far as storing the data of which test failed for Django, I'm not
sure what the *best* approach would be. Ned Batchelder's coverage
module does a similar thing. It keeps a .coverage file in the root I
think. Maybe just call ours .failedtests. Kinda gross, and not my
first choice, but it would work.
Or, perhaps use Python's tempfile module. But I'm not sure how to
grab a hold of the temp file again for the second pass through (maybe
tempfile.NamedTemporaryFile but this has problems on some platforms
according to the docs).
On one hand, I can see this argument: If you are adding 3 features
from nose, why not just use nose. But setting up nose and Django to
use it as the test runner isn't trivial the last time I checked.
We're using buildout to ease the pain.
Thanks for the input.
Rob
Rob Madole
ticket on that. So make that 4 features from nose...
Which would be great if the test is third or fourth in the stack. If
it's the last test in 50, it would loose it's effectiveness.
I know, I know. If you are running 50 tests you can reduce that down
to the module that is causing the problem.
Maybe time would be better spent making the use of nose really super
easy.
In settings.py:
TEST_RUNNER = 'django.contrib.test.nose.run_tests'
There might be some futzy bits to make that actually work, but I think
it'd doable.
Eh?
Rob
Rob Madole
It's certainly been done and doesn't require changes to Django.
Simon Willison
> TEST_RUNNER = 'django.contrib.test.nose.run_tests'
>
> There might be some futzy bits to make that actually work, but I think
> it'd doable.
implemented as an external project - but if it's as useful as it
sounds I'd personally be up for including it in core, if only to
promote the usage of nose amongst Django developers (and hence help
weaken the impression that Django doesn't integrate well enough with
the wider Python community).
Cheers,
Simon
Rob Madole
this. It looks solid to me.
Ticket and patches forthcoming...
Jeff Balogh
>
> I'll see if I can talk Jeff into adding what he's got as a start to
> this. It looks solid to me.
>
> Ticket and patches forthcoming...
The nose test-runner that I'm currently using is at
http://gist.github.com/197593. (The code I point to in the blog has
gotten more convoluted than I'd like; it turns out that can be a
problem if you're pointing at a vcs repo.)
For the most part, switching test runners is simple since Django
exposes all the setup and teardown functions. The hacky part is
passing extra arguments to nose: Django doesn't recognize nose
arguments so it complains and dies if you try something like
`./manage.py test --pdb-fail`. Since I just wanted to get something
working, I pass all arguments after a '--' on the command line to nose
and Django happily ignores them. If something like nose_runner is
shipped with Django it should run normally and be able to show the
same help screen that `nose --help` shows.
It was one of those "I'll come back and fix it later" sort of things. ;)
Cheers,
jeff
Russell Keith-Magee
I'm yet to be convinced that Nose should be the default test runner
for the simple reason that it doesn't come out of the box with Python.
However, I agree that using Nose with Django should be as painless as
possible. I included the TEST_RUNNER setting specifically for this
reason.
I haven't seen any reports in Trac that the capabilities of
TEST_RUNNER aren't suitable for the task, and I've seen several
snippets and blog posts indicating that Nose can be used with Django.
If I'm incorrect on this, please let me know (or better still, open a
ticket in Trac).
Then there is the issue of whether Django should be responsible for
shipping a Nose-compatible test runner, or whether that is Nose's
responsibility. Historically, I've been of the opinion that it should
be Nose's responsibility to ship a test runner, but I'm open to other
opinions on this.
I'm slightly hesitant to start building extra features like --failfast
into Django's testing tools, especially when tools like Nose already
cover this territory. Django isn't a test framework tool, and I'm
hesitant to support proposals that try to turn it into one.
Russ %-)
Jeff Balogh
I've updated the script at
https://gist.github.com/197593/9763da971f67c2c3bd25b2a7e4754e8b5d625087
to expose nose options as test_runner.options. Mixing options for the
two packages is kind of hairy, but I don't think any of that is
exposed to developers running the tests.
Integrating outside options will require a patch to
core/management/commands/test.py to import a test runner and ask it if
it has extra options. Importing the test runner before creating the
Command class should be backwards compatible unless people are writing
test runners that import test.py (there'd be a circular import issue).
It looks like this: http://gist.github.com/197797.
I'll get a proper bug filed if this looks acceptable/interesting.
Simon Willison
wrote:
> for the simple reason that it doesn't come out of the box with Python.
> However, I agree that using Nose with Django should be as painless as
> possible. I included the TEST_RUNNER setting specifically for this
> reason.
people less, not more likely to run them.
> Then there is the issue of whether Django should be responsible for
> shipping a Nose-compatible test runner, or whether that is Nose's
> responsibility. Historically, I've been of the opinion that it should
> be Nose's responsibility to ship a test runner, but I'm open to other
> opinions on this.
work fine. I think this is fits in to the conversation at DjangoCon
about how we should go about encouraging Django users to explore the
wider Python ecosystem. The important thing is that we can have some
official (or semi-official) documentation somewhere saying "to access
enhanced test running commands, install nose and drop this string in
to your TEST_RUNNER setting." The string itself could point at
dango.something or nose.something, the important thing from my point
of view is that they don't have to grab a test runner script from
another third party, then figure out where to put it within their
project. If nose don't want to ship the test runner, I'd be fine with
putting it in django.contrib somewhere.
Cheers,
Simon
Rob Madole
> work fine. I think this is fits in to the conversation at DjangoCon
> about how we should go about encouraging Django users to explore the
> wider Python ecosystem. The important thing is that we can have some
> official (or semi-official) documentation somewhere saying "to access
> enhanced test running commands, install nose and drop this string in
> to your TEST_RUNNER setting." The string itself could point at
> dango.something or nose.something, the important thing from my point
> of view is that they don't have to grab a test runner script from
> another third party, then figure out where to put it within their
> project. If nose don't want to ship the test runner, I'd be fine with
> putting it in django.contrib somewhere.
document their work. I wouldn't care if it's in Nose or Django that
has the test runner, just as long as doing an "easy_install nose" and
changing the TEST_RUNNER is all a user has to do to make the switch.
I just can't imagine the Nose guys including a Django test runner in
their base install. Maybe I'm wrong.
Simon's comment about documentation makes me think there needs to be
some detailed docs about how you would use this (beyond the "change
your TEST_RUNNER"). The Nose docs are good, but they don't have the
same flavor that the Django docs do. The audience is different I
think. If this is really going to be easy, I'd like to see docs about
how to use --pdb and --failed and --coverage built right into the
Django testing docs.
This may sunset or compete with two other tickets that are slotted for
1.2 though:
http://code.djangoproject.com/ticket/4501 (via Nose's coverage
support)
http://code.djangoproject.com/ticket/11613 (you can use --pdb and --
pdb-failures to get a similar behavior)
Rob
Ned Batchelder
From the point of view of encouraging the usage of nose, either would work fine. I think this is fits in to the conversation at DjangoCon about how we should go about encouraging Django users to explore the wider Python ecosystem. The important thing is that we can have some official (or semi-official) documentation somewhere saying "to access enhanced test running commands, install nose and drop this string in to your TEST_RUNNER setting." The string itself could point at dango.something or nose.something, the important thing from my point of view is that they don't have to grab a test runner script from another third party, then figure out where to put it within their project. If nose don't want to ship the test runner, I'd be fine with putting it in django.contrib somewhere.It's hard to get people to write tests. It's hard to get them to document their work. I wouldn't care if it's in Nose or Django that has the test runner, just as long as doing an "easy_install nose" and changing the TEST_RUNNER is all a user has to do to make the switch. I just can't imagine the Nose guys including a Django test runner in their base install. Maybe I'm wrong.
I would think the same logic applies to Django. Nose needs to work with lots of different projects, so they can't own the Django details, since by that logic they'd also own the TurboGears logic, the Pylons logic, the Twisted logic, etc... But the Nose plugin API isn't changing much now, so Django could include a nose plugin which would only have to change when Django details change. And we'd get all sorts of nose features without having to implement them in the Django code.
--Ned.
http://nedbatchelder.com
Simon Willison
> I would think the same logic applies to Django. Nose needs to work with
> lots of different projects, so they can't own the Django details, since
> by that logic they'd also own the TurboGears logic, the Pylons logic,
> the Twisted logic, etc... But the Nose plugin API isn't changing much
> now, so Django could include a nose plugin which would only have to
> change when Django details change. And we'd get all sorts of nose
> features without having to implement them in the Django code.
shipping plugins that get Django to work well with other parts of the
Python ecosystem sends a powerful message that Django wants to work
well with other parts of the Python ecosystem.
Cheers,
Simon
Harro
another one?
Rob Madole
> Sounds like a bad plan, what if by fixing the failed test you break
> another one?
fast, easy to use, and doesn't punish you.
My goal where I work was to make testing so simple that the other devs
on the team gravitate towards it because it's quicker to write a test
than to: write a view, change urls.py, create a template, write some
boilerplate view code, write some glue, ./manage.py runserver, open
Firefox on localhost, and see the results.
So from a purest point of view, yes, you can break another test pretty
easily by fixing another. If you don't have a continuous build /
integration server this is more dangerous. But it's also been my
experience that the users that check-in code that breaks a test will
never do a ./manage test -h either to find the other options, I doubt
they ever see it. I believe pretty strongly in CI, so I think that's
my official defense :D
Javier Guerra
>
> Sounds like a bad plan, what if by fixing the failed test you break
> another one?
while testing, when i found some not-obvious test failure and i have
to run the test repeatedly, i try to run just this one until it
passes. then i rerun the whole set.
it would be very nice to have a flat to run just the ones that failed
last time. after that, rerun the whole thing.
in the (very rare) case that fixing a test breaks another, it would be
caught by the full run at the end, maybe there could be an option so
that:
- if there's no 'failure record' run all
- if there's some record, first test those that have failed the last time
- if they still fail, stop there
- if there's no further failures, rerun the whole set
that seems the easiest use, while still not running unrelated tests
again and again.
--
Javier
Rob Madole
> - if there's some record, first test those that have failed the last time
> - if they still fail, stop there
> - if there's no further failures, rerun the whole set
before but it makes total sense. I think it's the way everyone does
it anyway, it would just take the manual command line changes out of
the process. I wonder what you would call such an option. ./bin/test
--smartfail?
Rob
Dougal Matthews
- if there's no 'failure record' run all
- if there's some record, first test those that have failed the last time
- if they still fail, stop there
- if there's no further failures, rerun the whole set