Re-run only tests that failed last time

[Gergely Polonkai]

Hello,

I have a pretty extended test suite for my application, which can run for about 15 minutes. Now when I introduce an error, I will be notified of it pretty late, and a one-liner fix needs another 15 minutes to check. Of course if only one of my tests fail, I can add the name of the test case as a parameter to manage.py test, but in case of many failures it’s pretty hard to do the same.

Is there a way to run only the tests that failed last time, even if I have to install a separate app/module for that? A quick Google search gave me no usable results.

Best,

Gergely

[Andreas Kuhne]

If you have created packages for your tests, you can run all tests in a package. If the tests aren't in the same package, it won't help you however.

Regards,

Andréas

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CACczBU%2B-SR3uASD_eQc_c%2BKFp%3DvCgaWga6pfUgA%2BJHr5HgkCHQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Erik Cederstrand]

> Den 24/03/2015 kl. 11.27 skrev Gergely Polonkai <ger...@polonkai.eu>:
>
> I have a pretty extended test suite for my application, which can run for about 15 minutes. Now when I introduce an error, I will be notified of it pretty late, and a one-liner fix needs another 15 minutes to check. Of course if only one of my tests fail, I can add the name of the test case as a parameter to manage.py test, but in case of many failures it’s pretty hard to do the same.
>
> Is there a way to run only the tests that failed last time, even if I have to install a separate app/module for that? A quick Google search gave me no usable results.

Only re-running failed tests partly defeats the purpose of testing. Your fix for failing test case A may introduce a failure in test case B.

Instead, I would look into either tweaking you very slow test so they run faster, or splitting up your test cases into fast and slow tests, so you can run your slow tests less frequently depending on some setting.

Erik

[Ramiro Morales]

Mine found:

https://pypi.python.org/pypi/django-juno-testrunner
http://nose.readthedocs.org/en/latest/plugins/testid.html#looping-over-failed-tests
plus https://github.com/django-nose/django-nose

--
Ramiro Morales
@ramiromorales

[Gergely Polonkai]

@Erik: On the CI side, certainly, I will run all the tests upon each commit. I want to utilize this thing only on my developer machine, so I can do a quick run after I think I fixed a problem. Naturally, if those failing tests are green, I will run the whole suite again.

Separating/tweaking my current tests is not really an option; the whole project is pretty big, with some large apps (yes, those must be cut in smaller pieces, but that’s not today’s work yet). It will happen, though, but thanks for pointing it out!

@Ramiro Morales: This juno test runner seems promising, thank you for the link!

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAO7PdF_-Km0%2BHC9jcJ%3DfvFJEJZqQoA_Xp_uSL6r0cTCSXjhN%2Bg%40mail.gmail.com.

[Russell Keith-Magee]

Hi Gergely,

One option is to use a test suite GUI tool, like cricket:

http://pybee.org/cricket

Cricket lets you view your entire test suite as a tree, select subsets of the suite to run, see test results as the suite executes, and re-run the failures from the previous execution.

Yours,

Russ Magee %-)

--

You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CACczBU%2B-SR3uASD_eQc_c%2BKFp%3DvCgaWga6pfUgA%2BJHr5HgkCHQ%40mail.gmail.com.

[Gergely Polonkai]

Hello Russ,

I've already installed the juno test runner, but a graphical tool cannot hurt locally 😃 I'll check it out tomorrow, thanks for the link!

Best,
Gergely

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAJxq84-hZn2VOe_dHRS63BXTmFJ8h3pbdsuKgQE1uFJo6csBZw%40mail.gmail.com.

[Carl Meyer]

Hi Gergely,

I use py.test [1], another test runner which has plugins that offer this
feature. The pytest-django [2] plugin provides the necessary integration
to make py.test able to run a normal Django test suite. Then the
pytest-cache [3] plugin offers an ``--lf`` (last-failed) option to run
only the tests that failed in the last run, and the pytest-xdist [4]
plugin offers the even-more-advanced ``--looponfail`` option, which "run
your tests repeatedly in a subprocess. After each run py.test waits
until a file in your project changes and then re-runs the previously
failing tests. This is repeated until all tests pass after which again a
full run is performed."

If you're interested in advanced Python testing, I think py.test is near
the top of the list of tools you should at least take for a spin.

Carl

[1] http://pytest.org/latest/
[2] https://pypi.python.org/pypi/pytest-django/
[3] https://pypi.python.org/pypi/pytest-cache
[4] https://bitbucket.org/pytest-dev/pytest-xdist

[Gergely Polonkai]

Hello Carl,

thanks, I will take a look at this, too!

Best,

Gergely

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/5511E627.8090409%40oddbird.net.

[aRkadeFR]

Great tool, thanks :)

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAJxq84-hZn2VOe_dHRS63BXTmFJ8h3pbdsuKgQE1uFJo6csBZw%40mail.gmail.com.