Better support for testing out of the box

116 views
Skip to first unread message

Martijn

unread,
May 12, 2016, 3:19:33 AM5/12/16
to SilverStripe Core Development
After I have seen how easy it is to write tests in a clean Laravel install, I really think Silvertripe lacks in this compared to Laravel.

See also this ticket :

https://github.com/silverstripe/silverstripe-installer/issues/116

After a composer install you want to invite the developer by running tests from mysite by providing a copy/paster like :

vendor/bin/phpunit mysite

Without the need to search for documentation on how to setup and run tests. This works for Laravel, so it should be possible with Silverstripe as well.

I think this is an important enhancement that promotes TDD and will lead to better modules and maybe even a better adoption of Silverstripe as a framework in the corporate world.

As far as I can oversee for now, we need the following enhancements:

  • adding a global $_FILE_TO_URL_MAPPING; by default
    • Something like $_FILE_TO_URL_MAPPING[__DIR__] = 'http://localhost';
    • Maybe even provide a _ss_environment.php file by default? so localhost can be changed
  • find out a way that (sake/ php framework/cli-script.php) dev/build on the command line also refresh the cache for the browser.
    • This is probably caused by my osx having different php versions. but forcing sake to use my mamp php versions did not help.
    • This might be operator error ;)
  • adding a test folder and a SampleTest.php in mysite

And this would be really nice :

  • allow installing on the command line.
    • A clean install with composer create-project requires you to move to the browser and enter the database and login details.
    • This could be done with symfony's console component. (see my sample implementation/sandbox https://github.com/axyr/silverstripe-console)


Now I would really love to put some time in this, but before I do that I want to know if there is even a demand for this and if this fits in the philosophy of the Silverstirpe core team.


And of course I might be missing some key aspects in achieving this, or there might be better ways of implementing this.


What do you think?

off...@netwerkstatt.at

unread,
May 12, 2016, 3:28:58 AM5/12/16
to silverst...@googlegroups.com

+1,

 

though testing with many fixtures is dead slow on mysql. There are some workarounds to use sqlite for testing which speeds up a lot, e.g. http://www.silverstrip.es/blog/how-to-speed-up-unit-tests-using-sqlite/

 

Maybe we could bundle some stuff as a dev-requirement in a seperate module?

 

Cheers,

 

Werner

--
You received this message because you are subscribed to the Google Groups "SilverStripe Core Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to silverstripe-d...@googlegroups.com.
To post to this group, send email to silverst...@googlegroups.com.
Visit this group at https://groups.google.com/group/silverstripe-dev.
For more options, visit https://groups.google.com/d/optout.

Martijn

unread,
May 12, 2016, 3:38:55 AM5/12/16
to SilverStripe Core Development

Ah yeah, I forgot that one ;) :

  • Having a single test database configuration that can be a sqlite database.


 I now added this :

public function tearDown()
    {
        parent::tearDown();
        parent::kill_temp_db();
    }


to remove the test datbases, cause on a normal test-driven-marathon-coding-session hundreds test databases are created but not removed...

off...@netwerkstatt.at

unread,
May 12, 2016, 4:21:48 AM5/12/16
to silverst...@googlegroups.com

Well, sometimes you want to check the DB after a test has failed.

 

Some sql statements to delete all test DBs might be fine.

 

Von: marv...@gmail.com [mailto:silverst...@googlegroups.com] Im Auftrag von Martijn


Gesendet: Donnerstag, 12. Mai 2016 09:39
An: SilverStripe Core Development

--

Martijn

unread,
May 12, 2016, 4:48:11 AM5/12/16
to SilverStripe Core Development

With one test database that would still be possible. one test database per install that will be cleared before the test runs is fine. not 100 ss_tmpdb12334's

Nicolaas Thiemen Francken - Sunny Side Up

unread,
May 12, 2016, 6:46:41 AM5/12/16
to silverstripe-dev
  • find out a way that (sake/ php framework/cli-script.php) dev/build on the command line also refresh the cache for the browser.
+1 

Martijn

unread,
May 14, 2016, 5:51:31 PM5/14/16
to SilverStripe Core Development
So I start digging and I came pretty far :

https://github.com/axyr/silverstripe-cli-installer

After :

$ composer global require axyr/silverstripe-cli-installer

You can create a fresh installation anywhere with :

$ silverstripe new projectname

Much nicer then :

$ composer create-project silverstripe/installer projectname

if you ask me...



It needs more environment requirements checking, the questions need validation and I would like more sensible defaults to shorten the process, but it currently does :

  1. install installer/cms/framework
  2. write _ss_environment.php and mysite/_config.php
  3. build database and cache
  4. writes a SampleTest.php and phpunit.xml
  5. invites to run the test that checks the homepage works

If a _ss_environment.php file is present in the directory where from the command is executed, it takes that values as default for the config questions.


I tested it on OSX and Ubuntu, and it works pretty good in the direction I had in mind.


Once all the packages are in cache, you get a new project in seconds WITH a working UnitTest setup.


Please let me know what you think of it!




Patrick Nelson

unread,
May 15, 2016, 4:23:03 PM5/15/16
to silverst...@googlegroups.com
Here here. I actually ended up with several hundred temp databases and I too assumed they would have been cleaned up and not piling up like a garbage dump. I did t realize the "cleandb" script already existed so u ended up writing my own bash script clear it all out. 

I agree very strongly that the overhead involved with getting up and running with unit tests is quite high and must be reduced. The biggest impediment is the dependency upon having silverstripe itself installed first and thus the easier you can make that, the better. Also that should (by default) make phpunit work straight out of the box with little or no overhead. 

I've always wondered ain't this but why not use composer to autoload classes? I know the manifest is needed right now for the lack of namespaces so maybe there's just a lot of work to be done there. Anyway his way you don't need he class manifest in order to get new classes working as classes are discovered very quickly and automatically via simple algorithm.  Even if you did need a 'manifest' composer already can generate a class map for you 🤓


- Patrick Nelson
Sent from my phone.

On May 12, 2016, at 1:48 AM, Martijn <marv...@gmail.com> wrote:


With one test database that would still be possible. one test database per install that will be cleared before the test runs is fine. not 100 ss_tmpdb12334's

--

Christopher Pitt

unread,
May 15, 2016, 4:35:05 PM5/15/16
to SilverStripe Core Development
On the subject of better testing...I've been working on this for a while: https://www.youtube.com/watch?v=f8SO_mL6-e4

Planning to trial it in a new, small SS project. Will finish the docs thereafter and make an alpha release!

Sam Minnée

unread,
May 16, 2016, 7:32:36 PM5/16/16
to SilverStripe Core Development
I've always wondered ain't this but why not use composer to autoload classes? I know the manifest is needed right now for the lack of namespaces so maybe there's just a lot of work to be done there. Anyway his way you don't need he class manifest in order to get new classes working as classes are discovered very quickly and automatically via simple algorithm.  Even if you did need a 'manifest' composer already can generate a class map for you 🤓

It's something we've discussed, but I don't think it would help testing.

In SS4 I'd like to see us move to a PSR-4 autoloader. The composer classmap would just be a duplication of our own manifest, with fewer features, so I don't really see the point in that.

The manifest will still be necessary for running ClassInfo::subclassesFor() and ClassInfo::implementorsOf() calls. However, if we're not using the manifest for class loading, we should be able to push it out of the bootstrap and into more of an on-demand utility.

The biggest place where ClassInfo::subclassesFor() is used is in the ORM. However, if we could shard the manifest data into smaller pieces, we may be able to work with much smaller manifest datasets (e.g. just the DataObject tree, rather than the all classes).

Martijn

unread,
May 17, 2016, 8:21:14 AM5/17/16
to SilverStripe Core Development
The problem is that right now, we have to dev/build on the command line and in the browser when switching.

so :

1. php framework/cli-script.php dev/build
2. vendor/bin/phpunit mysite
3. visit my.website.dev

will result in memory exhausted error.

4. visit my.website.dev/dev/build

browser works

5. vendor/bin/phpunit mysite

will result in memory exhausted error

repeat from 1.

this is kinda anoying when developing...


Sam Minnée

unread,
May 17, 2016, 2:31:34 PM5/17/16
to SilverStripe Core Development
Ah, yes. So, there are a few issues here:

1. The manifest needs to perform class-by-class parsing that is kind of memory intensive. It's possible that there are memory leaks in ClassManifest or TokenisedRegularExpression that could be resolved, to help with this. This does more than Composer's class scanner does, which is why it's more complex (it's not a complete parser, but it needs to reliably identify implements and extends references while respecting namespace semantics). Sometimes, when this topic comes up, people complain that it seems harder than it should be. Very few projects rely on automatic traversal of the class-tree like we do, but it does prevent a lot of double-handling. We could move to a system based on nikic/php-parser, but that would be a lot more code to do the same job, so it's not a clear win.

2. The manifest is necessary to visit the homepage. The ORM relies on the manifest to know what the subclasses of SiteTree are, in order to construct the queries. We could potentially reduce its dependence to dev/build, but only if there's some subsequent cache, and it's hard to know where to reliably cache that, because...

3. The manifest cache is unique-per-user. So unless you run your cli scripts with the same user as your webserver (which is something that we do with a lot of automated post-deploy scripts and the like), there's a separate copy for your WWW instances and your CLI instances. We do this because we don't want to make the files world-writeable, and it's difficult to know a set of permissions that would reliably cover both the needs of both users, in all systems. Before we made the cache unique per user developers would often get weird permission errors. One solution here would be use something other than a filesystem to cache this information, but it would need to be fast. A regular database wouldn't cut it; memcache might. This is a good idea but adds set-up burden, which given that the initial problem was "it's too hard to get started with unit testing", I don't think it's of much help.

4. The manifest cache is kind of hard to configure, since one of the things it is used for is the caching of compiled configurations! Since _ss_environment.php can be used to configure these top-level things, it's not a showstopper. However, it would probably be good to look at making the cache more granular that a big, all-encompassing *manifest*.

Hopefully that provides some insight!

If we want to improve things, I'd probably start with #1.

The other place I would look into things is #3: It should be possible to tell the system details of a group that both www-data and your CLI user belong to and to create group-writeable cachefiles. If you look at the Symfony docs on this — http://symfony.com/doc/2.0/book/installation.html#configuration-and-setup — it's not trivial, but it would be nicer if it were at least possible. We'd also need to consider windows usage.

I've raised some tickets about the relevant items:

--
You received this message because you are subscribed to the Google Groups "SilverStripe Core Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to silverstripe-d...@googlegroups.com.
To post to this group, send email to silverst...@googlegroups.com.
Visit this group at https://groups.google.com/group/silverstripe-dev.
For more options, visit https://groups.google.com/d/optout.
--
Sam Minnée
CEO
SilverStripe Limited

Sam Minnée

unread,
May 24, 2016, 6:42:40 AM5/24/16
to SilverStripe Core Development
1. php framework/cli-script.php dev/build
2. vendor/bin/phpunit mysite
3. visit my.website.dev

will result in memory exhausted error.


Martijn, can you give a bit more detail about this? What version of PHP and of SilverStripe are you using? I'm having trouble replicating this issue. 

Simon Erkelens

unread,
May 24, 2016, 5:30:55 PM5/24/16
to SilverStripe Core Development
Is your CLI on the same php-config as your www-data user? Because it sounds like it isn't.
Reply all
Reply to author
Forward
0 new messages