Running Tests

[Christopher Pitt]

One of the first things I like to do, when contributing to a new project, is running the tests. I was surprised to learn that the framework is tested through the process of creating a fresh [application] install and running the tests [there] with sake. That's a little different from previous projects I've worked on, and if it works then that's cool.

Would anybody be interested in the idea of making the tests runnable directly from the framework repository? They're based on PHPUnit and Behat, so they could be runnable without a fresh installation [or a separate repository]. There are a few benefits to this:

• Fewer steps to test
  
• Easier for newcomers
  
• Simpler Travis configuration
  
• Easier to document

Anybody interested/have thoughts about this?

[Ingo Schommer]

Hey Chris, great suggestion! I never liked the crazy dance we have to do on our CI (Travis) to get tests running (https://github.com/silverstripe-labs/silverstripe-travis-support). But I think this will be a difficult task to achieve in isolation, particularly around Behat where you need a fully functional webroot being served. This usually requires the creation of an assets/ folder as well as YML config for database access. The config usually lives in a mysite/ folder, but doesn't have to. 

How would you approach this without a separate repo (the "installer")? For example, Symfony also has a "standard edition" repo to set up new projects, although of course for CI tests they neither need a DB nor webroot due to separation of components.

While we could modify the test bootstrap to create a project "around" the module, that's just going to be confusing on other levels, right?

  

On a related note, there's been a few proposals over the years to restructure the folder layout (most recent one from 2012). I think we should pick this up again. Basically investigate the use of Laravel packages or Symfony bundles to structure current modules consistently in a vendor/ folder, with a separate webroot. But even then you'd still have the same testing troubles around creating a project for a checked out module.

--
You received this message because you are subscribed to the Google Groups "SilverStripe Core Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to silverstripe-d...@googlegroups.com.
To post to this group, send email to silverst...@googlegroups.com.
Visit this group at http://groups.google.com/group/silverstripe-dev.
For more options, visit https://groups.google.com/d/optout.

--

Ingo Schommer | Solutions Architect
SilverStripe (http://silverstripe.com)
Mobile: 0221601782
Skype: chillu23

[Matthew Bonner]

I think the majority of problems are surrounding the CMS, the framework itself should be unit testable, and if I'm not mistaken it already is?

[Patrick Nelson]

Just wanted to throw this gripe out there, may be totally unfounded, but may relate: Things are more difficult to test properly when so much global state is being maintained. I could be pretty far off on this :)

On Wed, Feb 25, 2015 at 10:14 AM, Matthew Bonner <matthew...@gmail.com> wrote:

I think the majority of problems are surrounding the CMS, the framework itself should be unit testable, and if I'm not mistaken it already is?

--

[Christopher Pitt]

Ingo:

> fully functional webroot being served

A headless browser + PHP development server might be enough. 

> structure current modules consistently in a vendor/ folder

I made this a while ago: https://github.com/typedphp/composer-path-plugin. It's explained in https://leanpub.com/typedphp.

The basic idea is to require a plugin which will look for paths [in "extras"] and place the package in those paths:

Themes could be "themes/foo"

Extensions could be "extensions/bar"

Alternatively we could stat using PSR-4 autoloading more [and the vendor folder as intended]. Low impact for extensions. High impact for the framework components...

[Ingo Schommer]

Hey Chris, we already have some custom SilverStripe composer package types (bundled in the default composer installer), namely "silverstripe-module" (installs on root folder) and "silverstripe-theme" (installs in /themes). I'm not sure I follow the purpose of the composer-path-plugin. So its turning around the path declarations, making the consuming package responsible for paths rather than the included package itself. But would it make the silverstripe/installer obsolete?

--

You received this message because you are subscribed to the Google Groups "SilverStripe Core Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to silverstripe-d...@googlegroups.com.
To post to this group, send email to silverst...@googlegroups.com.
Visit this group at http://groups.google.com/group/silverstripe-dev.
For more options, visit https://groups.google.com/d/optout.

[Matthew Bonner]

The problem with states can be overcome by mocking objects, .NET does this massively for testing the .NET framework, and I think others do too. SilverStripe already uses mocking to test where state is applicable.

We also need to consider the testing like so:

• Backend testing
• Frontend testing

The backend testing can ensure things like the API work, but we need to ensure the frontend works too, which again already happens, as Selenium is used to automate frontend testing.

Personally, I don't think the travis configuration is complicated, so I can't see how making it simpler is of any benefit.

Maybe I'm not understanding what you are suggesting, as it is coming across to me that some people are duplicating effort when the functionality already exists to carry out automated testing.

We don't set up a new website every time we want to test what has been worked on, PHPUnit and Behat are already used along with Selenium.

The documentation is old, so you'll have to poke around with the code to see what's already been done, but as I mentioned earlier, pretty much everything that is required to unit test the framework, the CMS, and a project based on using both is already in place. Maybe updating the docs might be a good starting point?

[Christopher Pitt]

It would obsolete the composer-installers plugin. It supports the path being set by the consumed package and falls back to path configuration in the consumer. So silverstripe-[cms|framework] could set default paths for modules and themes, and (in rare cases, for good reasons) the themes/modules could override these.

It would probably allow us to do away with the installer (as a starting point). In that situation, the installer would still be required, but rather as a ss module/git submodule, for initial DB setup.