Proposal and POC: silverstripe-recipe module type

[Damian Mooyman]

As we are nearing the date of 4.0.0 alpha2, and are getting ready to prepare for 4.0.0 stable, there has been a lot of discussion regarding the release process of the SilverStripe core, as well as the way that we version and manage dependencies.

The problem

Up until now, we have relied on a single core installer project, which declares dependencies on any "core" module via a "self.version" constraint. See https://github.com/silverstripe/silverstripe-installer/blob/98ad9085321522227c548784ac5ca4823cccaca5/composer.json for reference. An issue with this is that self.version normally requires manual replacement immediately after installation, and also locks us into using a single one-version-for-all-modules approach to dependencies.

In 4.0.0 we have introduced a new module (so far), https://github.com/silverstripe/silverstripe-asset-admin/ which is currently not versioned as `self.version`, as in fact the current version is 1.0.0-alpha1. As we continue to split the silverstripe core into independent components, we are looking to tie less and less into a single versioning system, thus the demand for a new versioning and release process has become apparent.

Yet another issue we have struggled with is inter-dependency between modules; Releasing a new version of one module requires a release of all other core modules to simply satisfy the self.version dependency. For modules such as reports or siteconfig, which rarely change, this is highly unnecessary.

We also run into issues where the installer project doesn't quite match the desired configuration; We currently release both with-cms, and framework only distributions of installer, normally involving manual changes in order to generate the framework-only version. What we are looking to potentially do in future releases is to include groups of complementary modules as "recipes", which are typically 1-6 modules with a set of default configuration.

The proposal

My proposal is to do away with the silverstripe/installer, as it's far too limited and rigid, and to instead introduce a new module type; `silverstripe-recipe`. A recipe is a library which may EITHER be installed directly via `composer create-project`, but also can be composed into other recipes which may extend on them.

I've built a proof of concept of how a recipe system could work; By generating a tractorcow/recipe-core module (representing a framework-only recipe), and a tractorcow/recipe-cms module (representing a standard recipe with cms and asset admin).

The repos for these are:

 - https://github.com/tractorcow/recipe-core

 - https://github.com/tractorcow/recipe-cms

This system is supported by a custom composer plugin built to manage project-assets. https://github.com/tractorcow/recipe-plugin. Any recipe may declare a "project-files" option, which is a list of files (or file wildcards) which represent files which should be copied to the project root, and should be modified directly by the developer. These files are only copied once, when the module is added for the first time. This means that assets between recipes are not duplicated, and allows recursive inheritance of other recipe assets.

E.g. the recipe-cms module declares `Page.php` as a cms-specific project file https://github.com/tractorcow/recipe-cms/blob/master/composer.json, whereas the recipe-core module includes a number of necessary core files required to bootstrap a silverstripe environment. https://github.com/tractorcow/recipe-core/blob/master/composer.json

Instead of declaring 'self.version' as the dependency module, recipes will be locked to specific versions. E.g. `4.0.0`. The intent is that we would control installed version via branch-alias tags in source control (e.g. alias dev-master as 4.0.0), as well as the `minimum-stability` in recipe composer.json. We would update our release process to automate the increment of these versions so that "everything works" while doing rapid releases.

In the future

Another feature that I had discussed with Sam earlier was the need to shift dependency management up into the top level composer.json. The intent is that once a recipe is declared, they should be able to be installed "inline", which means that rather than a single require being added to your root composer.json when pulling in a new recipe, the recipe will actually copy its own dependencies up into the top level. Similar to how top-level project files are installed in this proof of concept, this should give developers much greater fine-level control over which parts of any recipe make its way into a project. Not every part of a recipe is always necessary, and the ability to remove sub-components is just as important as the ability to add them.

This will probably be achieved via a custom composer command. https://getcomposer.org/doc/articles/plugins.md#command-provider

E.g. a theoretical command `composer add-recipe silverstripe/recipe-blog` would:

 - copy a default `blog.yml` into mysite/_config/

 - install a set of dependency modules (blog, lumberjack, comments)

 - Add installed dependencies to the root composer.json require

We could also include `composer upgrade-recipe` to perform incremental upgrades.

Testing it out

You can test out the proof of concepts for core and cms with the following commands:

  composer create-project tractorcow/recipe-core ./recipe-test-coreonly dev-master

Or

  composer create-project tractorcow/recipe-cms ./recipe-test-cms dev-master

Please let me know what you think. :D

Kind regards,

Damian

[Ingo Schommer]

That's genius, I like the "recipe composability" this approach allows :) It paves the way for having bundles of well-maintained SilverStripe modules incl. their supporting code. For example, a "blog+comments" recipe could come preconfigured with blog templates including comment ability.

I also agree that independent versioning of core modules is important, so they can move at their own speed: We might want to create API-breaking major releases of the "reports" module without waiting for a major "framework" module release. It'll be a bit weird for a while, e.g. you might get a recipe with rep...@5.0.0 and fram...@4.0.0 - but the recipes make this a lot easier to handle for devs.

One aspect that hasn't been mentioned, and initially confused me here: Eventually we'll want all SilverStripe modules to run in vendor/, in which case we'll need to extend the existing 'silverstripe-module' composer plugin to copy public assets into the webroot. Same approach, different topic from recipes.

--
You received this message because you are subscribed to the Google Groups "SilverStripe Core Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to silverstripe-dev+unsubscribe@googlegroups.com.
To post to this group, send email to silverstripe-dev@googlegroups.com.
Visit this group at https://groups.google.com/group/silverstripe-dev.
For more options, visit https://groups.google.com/d/optout.

--

Ingo Schommer | Solutions Architect
SilverStripe (http://silverstripe.com)
DDI: 04978 7330 ext 4422

Mobile: 0221601782
Skype: chillu23

[Damian Mooyman]

> One aspect that hasn't been mentioned, and initially confused me here: Eventually we'll want all SilverStripe modules to run in vendor/, in which case we'll need to extend the existing 'silverstripe-module' composer plugin to copy public assets into the webroot. Same approach, different topic from recipes.

We could also implement intelligent route handling. E.g. a request for `public/silverstripe/blog/css/Blog.css` is routed to `vendor/silverstripe/blog/public/css/Blog.css`. Given a general enough convention, the rules could be handled via pure apache rewrite syntax.

To unsubscribe from this group and stop receiving emails from it, send an email to silverstripe-d...@googlegroups.com.
To post to this group, send email to silverst...@googlegroups.com.

Visit this group at https://groups.google.com/group/silverstripe-dev.
For more options, visit https://groups.google.com/d/optout.

[Daniel Hensby]

I tested this today and it's working pretty sweetly for a proof of concept. I couldn't get a coreonly version to then upgrade using the CMS recipe, but I'll talk to Damian about that separately. 

[Damian Mooyman]

Oh, what problem did you have?

I tried this

composer create-project tractorcow/recipe-core ./test-core-3 dev-master

cd ./test-core-3/

composer require tractorcow/recipe-cms dev-master

The result gave me all the necessary modules and project files.

[Damian Mooyman]

Another thing I'll need to think about is how we integrate recipes with travis configuration. At the moment the travis support module does a lot of configuration manually. It would be nice to move the logic out of this, and into something more configurable at the .travis.yml level.

[Damian Mooyman]

I've posted this to the github issues list as an RFC at https://github.com/silverstripe/silverstripe-framework/issues/5992

[Chris Joe]

I liked the idea when you first mentioned it!

This could reduce the barriers for upgrading a module quite a bit and allows for newcomers to be less in the situation of "you're using an outdated version" if they're using the dev-master/alpha/nightly builds.

Moving modules to run in vendor/ has many solutions available, but this is out of scope for this discussion :)