Module path clean-up!

148 views
Skip to first unread message

Christopher Pitt

unread,
Apr 14, 2015, 3:59:07 AM4/14/15
to silverst...@googlegroups.com
Hello you fine folks!

How would you all feel about someone [read: me] extending the module loader code to allow it to look for modules within a ./modules folder, as well as the project root? Perhaps I'm overlooking something but it seems like it could be a relatively straightforward change, which would greatly reduce the noise in the project root...

Yay or Nay, ye scallywags!

Nicolaas Thiemen Francken - Sunny Side Up

unread,
Apr 14, 2015, 4:11:38 AM4/14/15
to silverstripe-dev
sounds great, but ...

I wonder - apart from mysite, themes, .htaccess, composer.json, and composer.lock what else is there in the document root?   In moving all the modules to ./modules are you not just shifting the problem? 



--
You received this message because you are subscribed to the Google Groups "SilverStripe Core Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to silverstripe-d...@googlegroups.com.
To post to this group, send email to silverst...@googlegroups.com.
Visit this group at http://groups.google.com/group/silverstripe-dev.
For more options, visit https://groups.google.com/d/optout.



--
Nicolaas Thiemen Francken
  www.sunnysideup.co.nz
  phone: +64221697577

Christopher Pitt

unread,
Apr 14, 2015, 4:16:57 AM4/14/15
to silverst...@googlegroups.com, n...@sunnysideup.co.nz
Currently, every extra module lands in the project root. It's horrible in particularly large applications.

Marijn Kampf

unread,
Apr 14, 2015, 4:21:20 AM4/14/15
to silverst...@googlegroups.com
Would be a Yay from me.

Nicolaas, don't forget assets, cms, framework and vendor.

I would love to be able to tidy up all the modules in a separate folder, but wonder how many modules will actually break if they are in a subfolder?

Marijn

Loz Calver

unread,
Apr 14, 2015, 4:28:57 AM4/14/15
to silverst...@googlegroups.com
Sounds great! Andrew Short did some work on something similar I think, though I can’t seem to find the posts on here about it

Thoughts:
  • How do we get Composer to install the modules there instead of the document root? We’d probably need a new “type” value instead of “silverstripe-module” for backwards compatibility, unless we can modify the installer to check framework version before installing each module? Doubtful, as it looks like the installer can only access one package at a time :(
  • If we’re putting them in a modules/ directory and adding autoloader detection for that, could we do the same for vendor/ too? That way, modules can exclude the “type” param in their composer.json and will be installed directly into vendor/ automatically. Dunno how we’d detect which modules have already been autoloaded by composer’s autoload.php, so I’ve probably just shot my own idea down!
Loz

Corey Sewell

unread,
Apr 14, 2015, 4:34:22 AM4/14/15
to silverst...@googlegroups.com

This was a bit of a pain for me when starting with Silverstipe. Especially with maintaining the .gitignore. The reason why I created https://github.com/guru-digital/SSAutoGitIgnore

It would be nice to have modules in their own sub directory but breaking modules may be an issue.

I know in some of my modules I assume the path will always be in the root when using requirements to include js/CSS.

In later modules I started to use dirname(__FILE__) to set a constant for the modules base directory.

Might not be such an issue if requirements is also modified to search the "modules" folder.

But who knows what else devs may do assuming the module will always be in the root.

lu...@lerni.ch

unread,
Apr 14, 2015, 4:49:36 AM4/14/15
to silverst...@googlegroups.com
I would like to see this happening. I've never had go with it myself but there is:
https://github.com/axyr/silverstripe-modulefolder

Not really the same issue but it would help to make things even more clear if there would be a recommendation for module-naming (silverstripe prefix or not) since it's ugly to search for two possible names when you search a module in your project- / module -folder. See discussion https://groups.google.com/forum/#!topic/silverstripe-dev/MMZkS76s19k

Am 14.04.15 um 10:34 schrieb Corey Sewell:

Ingo Schommer

unread,
Apr 14, 2015, 5:22:26 AM4/14/15
to silverst...@googlegroups.com
Sounds like this will require any modules with hardcoded path references into the module folder to change their logic (e.g. Requirements calls). I'd say that's the majority of all SilverStripe modules, so hundreds. While I understand the urge to clean up the module root folder, given the impact of this change we should do it once, and properly. We've wanted to move out the webroot from the document root for a while now, which in turn would allow us to simply keep all modules in the vendor/ namespace along non-SilverStripe dependencies. There's been numerous discussions: https://groups.google.com/forum/#!searchin/silverstripe-dev/module$20directory$20structure

webroot/
  assets/
  silverstripe/
    blog/ <- might be a symlink
      js/
      css/
vendor/
  silverstripe/
    blog/
      code/
      tests/
    framework/
      ...
    cms/
      ...
  unclecheese/
    betterbuttons/
      ...
composer.json

  
I think as a first step, we need to agree on a module definition format (to get a path dynamically, identify registered/active modules, etc.). Which means comparing Andrew's existing work (thread) with other emerging solutions like Symfony Bundles or Laravel Packages. Overall, I think this is an issue worth going through the RFC process.

Given this change will likely require many module authors to adjust their code (remove hardcoded paths), it should be a priority to figure out well before a 4.0 release.

Ingo Schommer | Solutions Architect
SilverStripe (http://silverstripe.com)
Mobile: 0221601782
Skype: chillu23

Christopher Pitt

unread,
Apr 14, 2015, 6:17:50 AM4/14/15
to silverst...@googlegroups.com
I confess I haven't been through as many modules as you have. The only paths I tend to see are the ones each module defines relative to their own root folder. That's not something that this change is likely to affect.

I agree that a recommended folder structure is an excellent thing to shoot for. Does it block a change to module loader code that allows both approaches?

Ingo Schommer

unread,
Apr 14, 2015, 6:34:01 AM4/14/15
to silverst...@googlegroups.com
File references are rarely relative to the module root folder. Just check BlogEntry.php as a quick example (search for "blog/"). Due to the way some SilverStripe APIs are designed, we don't even allow for a proper relative path at the moment (e.g. in SiteTree::$icon). So it sounds like this would be an "opt-in" change for module authors, rather than something we can just change in the SilverStripe composer installer. And for those kind of large scale changes, I'd rather ask hundreds of module authors once to do this properly than have the format potentially change again in a year's time.

On 14 April 2015 at 22:17, Christopher Pitt <cgp...@gmail.com> wrote:
I confess I haven't been through as many modules as you have. The only paths I tend to see are the ones each module defines relative to their own root folder. That's not something that this change is likely to affect.

I agree that a recommended folder structure is an excellent thing to shoot for. Does it block a change to module loader code that allows both approaches?

--
You received this message because you are subscribed to the Google Groups "SilverStripe Core Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to silverstripe-d...@googlegroups.com.
To post to this group, send email to silverst...@googlegroups.com.
Visit this group at http://groups.google.com/group/silverstripe-dev.
For more options, visit https://groups.google.com/d/optout.

Patrick Nelson

unread,
Apr 14, 2015, 1:44:53 PM4/14/15
to silverst...@googlegroups.com
Re: Christopher

Yay from me as well. Also: I like the cut of your jib ;) This is something I nagged about in a recent thread here: https://groups.google.com/forum/#!msg/silverstripe-dev/MMZkS76s19k/zDjooWeeTh4J

You can see how relevant this is just from a general organizational standpoint. I'm glad this is being received well because, if you ask me, it's a long time coming. I don't think it should be mutually exclusive from also scanning the root directory, but I do think it should be configurable (i.e. via composer.json and/or also in PHP) on which directory that I as the developer would like to group all of my modules! That way my site directory (sort of a module of its own) can be in the root, but all installable modules are setup in a nicely organized ./modules directory as discussed.


Re: Concerns on moving (in general)

Breaking modules unfortunately may be a necessary evil and could be a positive part of a progressive community effort to upgrade their modules, including changing the mindset of the practice of just putting everything in the root directory (which also happens to be public). I think it's much better practice to allow grouping these together somehow possibly in a public directory (for simplicity) unless we go the Laravel route where then you can have [what should be] a non-public directory (like ./vendor or ./workbench) where you can build/prototype your module code but easily publish public code via a copy task, which looks pretty simple, but that's obviously not really relevant at this moment.

At the moment I'd suggest a simpler progressive enhancement to maintain current functionality, but globally defined a desired MODULE_DIR (constant) and configuration in composer.json (or some other config accessible to the SilverStripe module installer) so that modules get installed in that location. It might also make sense to define (via config) the per-module exclusion list to facilitate this transition, for those modules which are still in the dark ages and not yet compatible :) 

As far as I'm concerned, you (the module author) shouldn't be hard-coding the name of your module folder into any of your code. Your code shouldn't care what directory it's under. Getting the correct full path to the base directory of  your module can be as simple as using the __DIR__ magic constant (available circa PHP 5.3) or via __FILE__ as Corey mentions. If you need to require assets using root relative references, reference something in the framework to get that information. The difficult part may be in having the modules easily generate a correct site root-relative path to files. With a properly configured site ($_FILE_TO_URL_MAPPING being defined) we could setup a helper in the framework to translate an absolute path from __FILE__ (e.g. /home/user/silverstripe/modules/foobar/file.php)  to a root relative reference like /modules/foobar/file.php, if that doesn't already exist.

- Patrick


Patrick Nelson

unread,
Apr 14, 2015, 1:59:53 PM4/14/15
to silverst...@googlegroups.com
Also, it may make sense to enhance the Requirements::css() method to allow absolute paths, not just root relative paths, to help simplify the process. 

So, instead of:

Requirements::css("modulename/js/cms.js");
You could do this:

Requirements::css(__DIR__ . "static/js/cms.js");

Then, you wouldn't be so tightly bound to the module name as highlighted above.

Christopher Pitt

unread,
Apr 14, 2015, 3:20:39 PM4/14/15
to silverst...@googlegroups.com
Ok, so how do we go about deciding/asking for this change? I don't mind doing setup work to make it happen. Let's get the ball rolling! :)

Gerald Villacarlos

unread,
Apr 15, 2015, 3:53:11 AM4/15/15
to silverst...@googlegroups.com
nice! much much easier to understand than the current structure.

Florian Thoma

unread,
Apr 15, 2015, 6:07:25 PM4/15/15
to silverst...@googlegroups.com
I do agree to some degree that moving the modules to a module folder could clean up thing.

But if the goal is to make the structure easier to understand I wouldn't add the "vendor" names like "silverstripe" or "unclecheese" as folders in there. If you are looking for a module you would have to know who wrote it (which I don't necessarily). If you use a lot of modules they will probably be from a lot of different people and adding these as folders only bloats the structure unnecessarily.

Also, I would keep external sources (currently 'vendors') separate from ss modules. IMO keeping the existing vendors directory and moving the modules to 'modules' would keep things cleaner.

And on a side note, when moving modules to their own folder the "silverstripe-" prefix can be left out as well to keep things simpler.

Since technically only 'framework' is not a module, the structure would look like this:

webroot/
  assets/
  framework/
  modules/
    betterbuttons/
    blog/

    cms/
    ...
  vendor/
    composer/
  composer.json

Moving the modules to a module folder only moves the problem. Only 'assets' and 'framework'  are effectively separated from the rest.

Of course the vendor and modules folder could also be moved out of the webroot if that's the preferred way, but it doesn't change the main issue of having many folders at one place.

Christopher Pitt

unread,
Apr 15, 2015, 6:43:16 PM4/15/15
to silverst...@googlegroups.com
I think it might also be good for mysite and cms to be in the same level as framework and assets...

Ingo Schommer

unread,
Apr 18, 2015, 3:25:50 AM4/18/15
to silverst...@googlegroups.com
Hey Chris, regarding on how to get this effort started, here's my suggested approach:

1. Get familiar with Andrew's existing work (overview, thread 1, thread 2git, blog).
Unfortunately Andrew has since taken down his blog with progress updates and more details, so we have to stick to discussions and code.
2. Agree on a minimal module interface which allows module authors to avoid hardcoding paths (see Andrew's ModuleInterface).
3. Figure out how a module is registered and code gets access to configuration like the module public assets path. Make sure its backwards compatible.
4. Identify all SilverStripe APIs which need to change to allow dynamic paths (e.g. SiteTree::$icon)
5. Write up an RFC so community can decide without re-reading long threads from 2012 :)
6. Send a pull request to SilverStripe core once agreement has been reached
7. Implement change in all SilverStripe maintained modules, educate module authors, send pull requests to high profile modules
8. Now that we have the basics in place: Figure out how to restructure directories and webroot through another RFC (lots of opinions here)

Sorry if that sounds complicated, but its a change at the centre of the SilverStripe architecture, so I don't think there's any shortcuts.

Uncle Cheese

unread,
Apr 18, 2015, 5:07:52 PM4/18/15
to silverst...@googlegroups.com
We don't have Andrew's blog anymore, but we do still have the interview he gave us on our podcast. This is well worth a listen: http://silverscoop.org/home/episode/5

Uncle Cheese

unread,
Apr 18, 2015, 5:17:21 PM4/18/15
to silverst...@googlegroups.com
The meat of that topic doesn't start until around 13:00, so you probably want to jump ahead unless you want to learn about the fundamentals of some new tool called "Composer."

Christopher Pitt

unread,
Apr 18, 2015, 9:05:16 PM4/18/15
to silverst...@googlegroups.com
Thanks gents! I'll have a read/listen...

Conrad Dobbs

unread,
Apr 18, 2015, 9:12:29 PM4/18/15
to silverst...@googlegroups.com
Wasn't there also talk of moving the the code out of the public directory? (Or am I dreaming)

If so it would seem related as it would need to cover off how assets are handled.

Aaron Carlino

unread,
Apr 19, 2015, 5:11:10 PM4/19/15
to silverst...@googlegroups.com
Yup, that's all part of it. Basically a module exposes the assets that need to be web accessible and they get symlinked by a composer script. 


On Sunday, 19 April 2015, Conrad Dobbs <con...@webtorque.co.nz> wrote:
Wasn't there also talk of moving the the code out of the public directory? (Or am I dreaming)

If so it would seem related as it would need to cover off how assets are handled.

--
You received this message because you are subscribed to a topic in the Google Groups "SilverStripe Core Development" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/silverstripe-dev/wKVrc0vQ3Zc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to silverstripe-d...@googlegroups.com.

Conrad Dobbs

unread,
Apr 20, 2015, 5:49:03 PM4/20/15
to silverst...@googlegroups.com
A good example to look at might be Laravel 5 which uses psr-4 autoloading for everything.


On Monday, April 20, 2015 at 9:11:10 AM UTC+12, Uncle Cheese wrote:
Yup, that's all part of it. Basically a module exposes the assets that need to be web accessible and they get symlinked by a composer script. 

On Sunday, 19 April 2015, Conrad Dobbs <con...@webtorque.co.nz> wrote:
Wasn't there also talk of moving the the code out of the public directory? (Or am I dreaming)

If so it would seem related as it would need to cover off how assets are handled.

--
You received this message because you are subscribed to a topic in the Google Groups "SilverStripe Core Development" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/silverstripe-dev/wKVrc0vQ3Zc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to silverstripe-dev+unsubscribe@googlegroups.com.
To post to this group, send email to silverstripe-dev@googlegroups.com.

Marcus Nyeholt

unread,
Apr 20, 2015, 6:02:22 PM4/20/15
to silverst...@googlegroups.com
As Ingo's mentioned, Andrew addressed most (perhaps all?) of the points mentioned above. While there's been some improvements in the wider landscape around some things that could be useful (such as autoloading), the following repo should be cloned to get a very very solid starting point


to dig into what's needed. 

Cheers,

Marcus


On Tue, Apr 21, 2015 at 7:49 AM, Conrad Dobbs <con...@webtorque.co.nz> wrote:
A good example to look at might be Laravel 5 which uses psr-4 autoloading for everything.

On Monday, April 20, 2015 at 9:11:10 AM UTC+12, Uncle Cheese wrote:
Yup, that's all part of it. Basically a module exposes the assets that need to be web accessible and they get symlinked by a composer script. 

On Sunday, 19 April 2015, Conrad Dobbs <con...@webtorque.co.nz> wrote:
Wasn't there also talk of moving the the code out of the public directory? (Or am I dreaming)

If so it would seem related as it would need to cover off how assets are handled.

--
You received this message because you are subscribed to a topic in the Google Groups "SilverStripe Core Development" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/silverstripe-dev/wKVrc0vQ3Zc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to silverstripe-d...@googlegroups.com.
To post to this group, send email to silverst...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "SilverStripe Core Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to silverstripe-d...@googlegroups.com.
To post to this group, send email to silverst...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages