should manage.py test run system checks?

[Tim Graham]

A ticket [1] and pull request [2] note that `manage.py test` ran the system checks in Django 1.7 (as a side effect of call_command('migrate')), but this is no longer the case in Django 1.8 since call_command() doesn't trigger the system checks anymore.

me: I'm of the opinion that running the system checks as part of the manage.py test command should be opt-in (for example, by writing a test that asserts the call_command('check') output is empty. For example, when debugging a single test, it doesn't seem necessary to have the overhead of running check. As more options are added to check (e.g. #25500), a default implementation as currently proposed could become increasing inflexible. I'd like to document the change in the 1.8 release notes and suggest the alternative.

Adam: I don't think they should be optional, or if they are, they should be opt-out. The checks are a brilliant guard against error, but not running them as part of test invites them not being run at all in a TDD workflow, as often code can be developed with nothing but running the tests. It is also surprising that *only* test doesn't run them, since every other manage command does.

At YPlan we couldn't do without them as part of tests. Our aforementioned 'installed packages' check saves a lot of time that would otherwise be wasted understanding confusing error messages about imports not working, and our other custom checks do verification similar to Django's, for issues that without resolution it does not make sense to even attempt do any tests. Also we don't notice any real overhead, we can still get a single test to run in 1 second (with --keepdb :) ) despite all our extra messing around with pip freeze etc.

Other opinions?

[1] https://code.djangoproject.com/ticket/25415
[2] https://github.com/django/django/pull/5293

[David Filipovic]

How about making them opt-out instead of opt-in (for instance with a --skip-checks flag)? That way anybody who is aware of the fact that checks are being run every time tests are run and is seeking an increase in performance can choose to opt-out, whereas anybody not aware (or at least not fully aware) of it could probably benefit from having them run anyways.

[David Filipovic]

Sorry for the double post, but I think allowing two levels of control, by possibly even adding a setting: TEST_SKIP_CHECKS (in addition to the --skip-checks management command flag) which defaults to False would be ideal. That way any developer worried about test performance can just add this flag to their settings file and still use their old workflows.

[Aymeric Augustin]

2015-10-20 2:48 GMT+02:00 Tim Graham <timog...@gmail.com>:

me: I'm of the opinion that running the system checks as part of the manage.py test command should be opt-in (for example, by writing a test that asserts the call_command('check') output is empty. For example, when debugging a single test, it doesn't seem necessary to have the overhead of running check.

I share this opinion. Single-test run time is a very important use case, not only for TDD practitioners, but also for everyone who tries to fix a stubborn failing test.

--

Aymeric.

[Shai Berger]

Just brainstorming : by all of the above, it would seem that the best default is to run checks unless specific tests are selected. Of course, this could be perceived as inconsistent, and that would be a problem.

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[Marcin Nowak]

As I said some time ago - there should be a possibility to disable system checks (generally), and run them only when needed (i.e. by adding --system-check argument to mgmt command(s)).

Kind Regards,

Marcin

[Žan Anderle]

Adam: I don't think they should be optional, or if they are, they should be opt-out. The checks are a brilliant guard against error, but not running them as part of test invites them not being run at all in a TDD workflow, as often code can be developed with nothing but running the tests. It is also surprising that *only* test doesn't run them, since every other manage command does.

I agree with this. I don't see why an opt-out option is not valid? 

Dne torek, 20. oktober 2015 02.48.29 UTC+2 je oseba Tim Graham napisala:

[Adam Johnson]

Y'all know my position (original author). How long are the checks taking for people? We have a very large project with >100 models, ~30 apps, and it still takes less than a second.

[Žan Anderle]

Same for us. It takes slightly longer because of some custom checks, but it's still under a second.

Dne nedelja, 31. januar 2016 10.23.30 UTC+1 je oseba Adam Johnson napisala:

[Tim Graham]

It looks like about 250 milliseconds for Django's test suite. It's quicker in the case of running an individual test since the checks will be restricted to the application that contains that test. I guess it's fine with me if we proceed with this change. After all, no one complained about in Django 1.7.