Running cakephp 3.0 testsuite with phpunit

[José Lorenzo]

Hey guys,

Now that https://github.com/cakephp/cakephp/pull/1355 is already merged I would like to share some tips and start a discussion on the future of the test shell.

First the tips:

You might have noticed that the new way of running the php core testsuite looks something like this:

cd lib/Cake;

db_class='Cake\Database\Driver\Sqlite' db_dsn='sqlite::memory:' phpunit --stderr

That seems a bit verbose. The reason is that it is a lot easier to pass variables from the environment than creating config files on the fly when you do automated testing. To avoid this problem just add a you own phpunit.xml file to lib/Cake and tell git to ignore it. I do it in my global .gitignore file.

An example of this file is in this link:

http://bin.cakephp.org/saved/103585

Now the discussion:

Should we remove the test (and web test interface) from the framework and only add support for running tests with the phpunit command?

Why I think it is time to use the axe on those:

* Installing and running phpunit executable is way easier than a couple years ago.

* Our test harness does not actually offer anything useful on top of phpunit

* Creates confusion for newcomers that are already acquainted to phpunit

* Using the console with php does not suck anymore

* Newbies rarely do tests, more advanced developers are used to the console

Thoughts?

[Simon East]

I (for one) love the web-interface, and don't tend to use the command line very often.  We also incorporate it with our server monitoring service (NodePing) that requests various test cases via HTTP remotely and sends emails/SMS notifications if the tests fail.

I guess if the web runner is too hard to update and/or maintain, it could be moved to a plugin, or perhaps provide some other way to trigger tests over HTTP.

Simon East  |  Technical Lead  |  surfacedigital

Level 1, 60 Hardware Lane, Melbourne VIC 3000 Australia
Ph (03) 9670 7062   Skype simoneast_oz   Tw @surfacedigital   www.surfacedigital.com.au

--
You received this message because you are subscribed to the Google Groups "cakephp-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cakephp-core...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Jose Diaz-Gonzalez]

If a plugin were provided that were kept up to date, then that would be stellar. I can imagine something like getting the test output as json would be cool.

I personally think running via continuous integration or otherwise will be fine with cli, but for the novice tester, the web ui is a god send, and it's actually quite nice to see visual feedback and have that warm fuzzy feeling of all green :)

[Graham Weldon]

Agreed. I think maintaining the web UI would be ideal, elas a plugin if necessary, but it's quite an invaluable tool. For example, I write tests and run via the UI, but others on my team don't write tests so much, so they use the UI to check that their code changes don't break existing tests. 

[mark_story]

I don't know if a plugin would work for the webrunner. Currently the webrunner uses some non-standard extensions of PHPUnit to make it all work. If the webrunner is important (it sounds like it is) it should probably stay. With that said if we adopted `phpunit` in the CLI as the only way to run tests, would needing to have/maintain a phpunit.xml file be a pain point for webrunner users?

-Mark

[Graham Weldon]

Sounds like we could do some phpunit XML generating thingis to assist web users. Moar work

[mark_story]

If we went with phpunit as the only way to run tests in CLI it would be ideal to generate phpunit.xml files as part of new applications/plugins. Webrunner issues aside I think using `phpunit` has a few benefits/drawbacks worth discussing.

Good:

* Get rid of more custom shims that make cake different from every other framework/library. I see this as adopting more 'PHP Conventions'.

* Simpler for people to learn as it is just phpunit. No additional wrappers required.

* We can more easily leverage other phpunit extensions.

* Core tests are more stable even when applications have modified configuration. Currently re-configuring an app can cause core tests to fail which is annoying.

Bad:

* Change, people don't like change.

* phpunit.xml file becomes required.

Keeping the webrunner working is important for new/existing users so any changes we make to the test runners should keep the webrunner in a functional state.

-Mark

[Ravage84]

* Installing and running phpunit executable is way easier than a couple years ago.
* Our test harness does not actually offer anything useful on top of phpunit
* Creates confusion for newcomers that are already acquainted to phpunit
* Using the console with php does not suck anymore
* Newbies rarely do tests, more advanced developers are used to the console

I agree with everything.

Some more thoughts from a "young" (6 months now) CakePHP user:

* The web UI is certainly nice for newbies and for devs wanting to show their bosses something "visual".
* I personally never really got the point of this web UI thingy beyond the mentioned "visual" part.
* If Cake's test environment provides other ways for CI and monitoring tools then it should be fine
* If we provide ready to cook recipes for executing the tests in the console for newbies then they could get along without the web UI (and learn to use it, too).
* If the web UI test runner was a single fat script that was maintained externally and could be inserted into the project for those who need it, it would take away some burden off the core project.
* Is there an alternative solution to Cake's web UI? Does anybody know if there is something similar to it?
* People really don't like changes but CakePHP 3.0 comes with changes already. Why not one thing more if it's for a good cause?

Marc
* If devs coming from other PHP frameworks/standard applications they don't have to learn something new if we use phpunit standard stuff, which would help transitions to Cake easier.

[mark_story]

To my knowledge, CakePHP is the only project that provides a web UI to PHPUnit based test runners. At least I've not seen a web based runner for phpunit in other popular frameworks.

-Mark

[euromark]

And I think its a good thing. It helps to ease first steps.

It also helps to test session based stuff without all the overhead required, otherwise.

The output for quick debugging is more readable (larger screen, sql queries at the bottom, ...)

Why removing this when its already there? Useful tool for some situations and therefore a good thing we should not drop.