A setting that skips tests for thirdparty applications

[lakin]

http://code.djangoproject.com/ticket/13873

As part of a normal testing routine, I find it annoying that django
tests ALL of the applications listed in INSTALLED_APPS. While I do
want it to test most of them, I rarely want it to test third party
applications. This ticket has a patch that adds a TEST_SKIP_APP_TESTS
setting that allows for a developer to explicitly request that certain
apps are not tested when one runs ./manage.py test

The tests for applications listed in TEST_SKIP_APP_TESTS doesn't stop
one from asking for that application to be tested by passing it's app
label to ./manage.py test as a command line argument.

In any case, I was hoping to get a review/feedback on the patch. I'm
more than willing to improve it in order to get it included. The
original idea came from za...@nox.cx

Lakin Wecker

[Luke Plant]

Thanks for the ticket and patch. However, I for one am struggling to
understand why it is so painful to do:

./manage.py test app1 app2 app3 app4

...especially if you create an alias or a shell script wrapper e.g.:

alias foo_project_default_tests=./manage.py test app1 app2 app3 app4

(For myself I've just learnt to use the history features of bash and
that's just as fast [1]).

I also wonder whether this would be an appropriate use of a setting - it
feels more appropriate as command line option for manage.py than a
setting, since settings are things that 'belong' to a project, or to a
project/machine combination, but this is more of a 'developer
preference'. Once you make it a command line option, then I imagine you
are back in the realm of wanting to make an alias/shell script wrapper
so that you don't have to remember it, in which case it's of dubious
usefulness IMO.

Just my 2¢

Luke

[1] http://mike-austin.com/blog/2007/03/history-search-backward.html

--
"The truth will set your teeth free." (Calvin and Hobbes)

Luke Plant || http://lukeplant.me.uk/

[John-Scott Atlakson]

It's also not too terribly difficult to write your own test runner that can skip tests based on your particular criteria.

I worked on a project where they altered the db to make User.email unique (using email instead of username for login). This broke many of django's internal tests, but we weren't interested in seeing these results every time buildbot ran the full test suite. So we just wrote a test runner that skipped any app in installed apps that started with 'django.'.

As long as you're willing to trust that Django and the third party apps only release code that passes their own tests , you may want to do this to speed up testing by only testing your own custom application code.

I haven't attempted this with the new class based test runners in 1.2, but from a cursory glance at the docs[1], it looks even simpler than it once was.

Cheers,
John-Scott

[1] http://docs.djangoproject.com/en/1.2/topics/testing/#defining-a-test-runner

[lakin]

> Thanks for the ticket and patch.  However, I for one am struggling to
> understand why it is so painful to do:
>
>   ./manage.py test app1 app2 app3 app4

As someone who works on quite a few projects, it's painful to have to
remember the exact app names for all 10 projects that I frequently
work on.
./manage.py test package.app1 package.app2 package.app3 package.app4
./manage.py test package2.app5 package2.app6 package2.app7
package2.app8
./manage.py test package3.app9 package3.app10 package3.app11
package3.app12
etc.

I disagree that it's solely a developer preference. Depending on the
team, the project it may very well be the project policy that certain
tests shouldn't be run (such as the John-Scott's use-case).

Thanks for the feedback.

Lakin