This is an interesting proposal. I'm not completely convinced, though,
for a couple of reasons.
1) runserver is not especially robust to certain types of errors, such
as SyntaxError. I will admit that I haven't tried the snippet that you
referenced, but it looks like it will be prone to the same problem.
I'd like to fix this problem, but while this problem exists, adding a
continuous integration server that crashes every time the developer
makes a Syntax Error doesn't sound like an especially good idea.
2) I'm not necessarily convinced that a CI services should be running
on a developers box during development. Django's test suite takes a
long time to run - I've heard reports of test durations measured in
*hours* for some setups. Test durations like this aren't really
compatible with a "retest on every file change" CI approach. Given the
current state of Django's test framework, I'm more convinced that a CI
server based on checkins is more appropriate.
3) Most of my hesitation on (2) is driven by the fact that
setup/teardown is such an expensive operation. Most of the time spend
running the test suite isn't spent running tests - it's spent setting
up and tearing down tests. We've made some big improvements by using
transactions to speed up this process, but it's still slow. If you
want to convince me that per-file-modification CI is worthwhile,
you'll need to work out a way to dramatically reduce the
setup/teardown time for tests.
4) An alternate approach is to come up with a smarter way to run tests
so that the tests that are important to the file being modified
(whatever that means) are run more often, rather than just blanket
running the entire suite. This might mean reducing the CI process to a
Monte Carlo style sampling that eventually results in 100% coverage if
the file base remains stable for long enough.
Yours,
Russ Magee %-)
That's great - but if you leave it on github, it's going to get forgotten.
This issue is being tracked on ticket #9589. If you want to ensure
this contribution isn't forgotten, turn your github changeset into a
patch, and upload that patch to the ticket. You can also reference the
github changeset in the ticket comments, but not all the Django core
uses git.
> But, just like the test command, the runtester command takes a list of
> apps, or even a TestCase class within an app. That way you are simply
> running a subset of the tests on the app actively being worked on.
> For example:
>
> ./manage.py runtester some_app.SubTest
>
> With the above command, only the SubTest tests are run, which will
> drastically reduce the time to run the tests. Granted time to setup
> and teardown of the test environment is still there.
Well, sure, except for the fact that if you do this, you no longer
have continuous integration :-) The point of a test suite is that you
run *all* the tests, so that you can catch *all* the possible
regressions. Sure - if you're just doing a quick check of something,
you might just run a single test, but you need to run the full suite
to be sure.
That's why I suggest that CI is better suited as a checkin trigger. A
checkin is an indication that you think a body of work is complete.
Running the full test suite is then an appropriate validation step. A
single file save doesn't give you that sort of certainty that a body
of work is complete.
When it takes a while for a test suite to run (which will be true in
Django even if you're only running a small subset of the overall
suite), it could be easy for a test suite to start running on a
partially saved set of updates, which could result in confusion over
exactly which version of a which files were used to run a test.
kGiven the fact that it takes a long time to setup and teardown
Django's test suite, I feel that it is better to leave test execution
as a manually invoked process. That way the developer can be certain
of the exact file versions that will be used for a test run.
Yours,
Russ Magee %-)
Have you looked at http://github.com/lacostej/nosyd yet? This seems might be
of interest based on what you are looking for. As an added bonus it allready
supports Django :)
-Thomas Adamcik
Is it possible to autorun only the failing tests? It would be very
useful (at least for me) if the following was supported