Remove TestRunner from 4.0?

[Ingo Schommer]

In order to reduce complexity in core, I'd like to suggest removing the TestRunner functionality to run PHPUnit tests via the "sake" binary in 4.0.

This is a bit of a historical artefact from early 2.x releases when we didn't support the "phpunit" binary directly.

Nowadays we bundle "phpunit" through composer, and it doesn't need all this scaffolding.

In fact, the "phpunit" binary is a lot more powerful (e.g. filtering by method name), see https://phpunit.de/manual/3.7/en/textui.html#textui.clioptions

Particularly in the light of the recent 3.1.13 release, I'm keen to reduce our "security surface" on things like development tasks.

What will change:

- No more test execution via web browser. That's probably the most contentious change, but my take is that devs caring about unit testing should be comfortable with basic CLI tooling, or have an IDE which can provide a UI like PHPStorm. Can somebody think of a good reason that you'd want to run unit tests on a webroot where you don't have CLI access?

- dev/tests/coverage: Built-in to PHPUnit, already described on http://docs.silverstripe.org/en/3.0/howto/phpunit-configuration/

- dev/tests/browse: Listing available tests 

- dev/tests/module: Simply point the "phpunit" binary at a folder

- dev/tests/suite: phpunit --testsuite

- dev/tests/all: Simply let phpunit auto-discover tests on the webroot

- dev/tests/SomeTest,SomeOtherTest: No replacement, can be emulated with suites in phpunit.xml

- SkipTests=1: Can be emulated with --filter

What's required:

- Remove TestRunner.php and SapphireTestSuite.php

- Move TestRunner::$coverage_filter_dirs to phpunit.xml.dist

- Move TestRunner::use_test_manifest() to SapphireTest

- Move dev/tests/cleanupdb to a new BuildTask

- Remove "sake" chapter on http://docs.silverstripe.org/en/3.0/topics/testing/

I'd also like to remove JSTestRunner.php for similar reasons: It doesn't provide any benefit over a Gruntfile, tooling has moved on since it was written, and it hasn't been used much in practice.

Any objections?

[Cam Findlay]

Maybe raise as an RFC? If you whip this up today it can be discussed at the next core committers hangout (which is a private one tomorrow).

C

[Ingo Schommer]

RFC at https://github.com/silverstripe/silverstripe-framework/issues/4254