Module System Refactoring, PHP 5.3 + GSOC

[Andrew Short]

Hey All,

For those who don't know me my name's Andrew Short. I'm an engineering and computer science student at the University of Wollongong, and am also employed by SilverStripe Australia. I've been involved with SilverStripe for quite a while now, and have worked on major features in the past. I'm currently looking for a project to work on for GSOC, and the improved module listing site caught my eye and got me thinking about the whole module system, which could do with a a lot of work.

I was thinking a possible project would be to refactor the way that modules are handled in SilverStripe - currently there's no way to define dependencies, organise modules into subfolders or easily submit them to a central repository. With the recent surge in popularity of the composer/packagist pair, I think it would be great to take advantage of this really useful system rather than inventing our own. I would propose the following course of action:

• Update the module system to one based on composer/packagist. This would allow module authors to define all the metadata for a module inside a JSON file (tags, requirements etc etc), and all the versioning would be inferred from source control. This means submitting a module would be as easy as submitting the URL to your repository.
  
• Allow modules to be organised into application/modules/framework sub-folders.
• Implement proper application/module/framework priorities for code and themes.
  
• Update the requirements system to allow requirements to be included by specifying their name rather than file path - Require::javascript('jquery').
• Create a better module listing repository based on composer/packagist. This could perhaps be done by one of the other GSOC students.
  

Part of this would mean dropping support for PHP 5.2 as composer only supports PHP 5.3. I think is a good excuse to drop 5.2 support - there's lots of new features that would be very useful. However, it's probably a bit late in the 3.0 release cycle to do this, so I would propose marking 5.2 as deprecated in the initial release, and perhaps merging this work back for the 3.1 release, and dropping 5.2 support. We could also alter evaluate if we want to take advantage of namespaces and other features.

There's also a number of other issues to think about - such as whether it is actually possible to develop the module listing site in parallel with the module system, and how much effort would be involved. The packagist site is fully open source and forms a very solid base. Other issues that would need to be evaluated include if anything needs to changed/deprecated for the 3.0 release, how to best manage backwards compatibility and several other items.

It'd be great if people could comment on this - there's probably heaps of alternative approaches which have their own merits. It would also be good if people could chime in on dropping 5.2 support, since this will be a pretty big change.

Cheers,
Andrew Short.

[Frank Mullenger]

Hey Andrew,

Personally I think you have the right idea to leverage composer/packagist for this task. I like the concept of just submitting a URL for the repository of a module and having the meta data extracted automatically. 

Dropping 5.2 support is not important for me, I'm happy with the move to 5.3.

It sounds like a really cool system, I'm looking forward to it already.

Cheers,

Frank

--
You received this message because you are subscribed to the Google Groups "SilverStripe Core Development" group.
To post to this group, send email to silverst...@googlegroups.com.
To unsubscribe from this group, send email to silverstripe-d...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/silverstripe-dev?hl=en.

[xeraa]

+1 for the general approach and dropping PHP 5.2 support.

Hope we can find a student for this *hint*!

Am Donnerstag, 22. März 2012 19:42:28 UTC+1 schrieb Frank Mullenger:

Hey Andrew,

Personally I think you have the right idea to leverage composer/packagist for this task. I like the concept of just submitting a URL for the repository of a module and having the meta data extracted automatically. 

Dropping 5.2 support is not important for me, I'm happy with the move to 5.3.

It sounds like a really cool system, I'm looking forward to it already.

Cheers,

Frank

On Thu, Mar 22, 2012 at 11:58 PM, Andrew Short  wrote:

Hey All,

For those who don't know me my name's Andrew Short. I'm an engineering and computer science student at the University of Wollongong, and am also employed by SilverStripe Australia. I've been involved with SilverStripe for quite a while now, and have worked on major features in the past. I'm currently looking for a project to work on for GSOC, and the improved module listing site caught my eye and got me thinking about the whole module system, which could do with a a lot of work.

I was thinking a possible project would be to refactor the way that modules are handled in SilverStripe - currently there's no way to define dependencies, organise modules into subfolders or easily submit them to a central repository. With the recent surge in popularity of the composer/packagist pair, I think it would be great to take advantage of this really useful system rather than inventing our own. I would propose the following course of action:

• Update the module system to one based on composer/packagist. This would allow module authors to define all the metadata for a module inside a JSON file (tags, requirements etc etc), and all the versioning would be inferred from source control. This means submitting a module would be as easy as submitting the URL to your repository.
  
• Allow modules to be organised into application/modules/framework sub-folders.
• Implement proper application/module/framework priorities for code and themes.
  
• Update the requirements system to allow requirements to be included by specifying their name rather than file path - Require::javascript('jquery').
• Create a better module listing repository based on composer/packagist. This could perhaps be done by one of the other GSOC students.
  

Part of this would mean dropping support for PHP 5.2 as composer only supports PHP 5.3. I think is a good excuse to drop 5.2 support - there's lots of new features that would be very useful. However, it's probably a bit late in the 3.0 release cycle to do this, so I would propose marking 5.2 as deprecated in the initial release, and perhaps merging this work back for the 3.1 release, and dropping 5.2 support. We could also alter evaluate if we want to take advantage of namespaces and other features.

There's also a number of other issues to think about - such as whether it is actually possible to develop the module listing site in parallel with the module system, and how much effort would be involved. The packagist site is fully open source and forms a very solid base. Other issues that would need to be evaluated include if anything needs to changed/deprecated for the 3.0 release, how to best manage backwards compatibility and several other items.

It'd be great if people could comment on this - there's probably heaps of alternative approaches which have their own merits. It would also be good if people could chime in on dropping 5.2 support, since this will be a pretty big change.

Cheers,
Andrew Short.

--
You received this message because you are subscribed to the Google Groups "SilverStripe Core Development" group.

To post to this group, send email to silverstripe-dev@googlegroups.​com.
To unsubscribe from this group, send email to silverstripe-dev+unsubscribe@​googlegroups.com.

[Marcus Nyeholt]

I've not really looked much into Composer's format yet, but does it have metadata for 'this module is commonly used with this other module' ? Not dependencies, but a way to find other modules that work well together. 

Cheers,

Marcus

To view this discussion on the web visit https://groups.google.com/d/msg/silverstripe-dev/-/AEZD0qltIIoJ.

To post to this group, send email to silverst...@googlegroups.com.
To unsubscribe from this group, send email to silverstripe-d...@googlegroups.com.

[Andrew Short]

Not at the moment - it'd be useful to have the ability to add "optional dependencies", which are listed when the user installs the module (e.g. the event manage module optionally depends on the payment module). This would be a nice to have though. There's nothing stopping us from adding extra data to the file.

Andrew Short.

[vikas srivastava]

Hi Andrew
I am trying to research on this idea from past few days and got
similar idea from Drupal . They are using .info file in root of module
directory for getting info about module (I am not sure if they are
extracting same file for getting information on drupal.org module
download page). But this basic idea can work in our case. The only
thing we need to check that storing all kind of the meta data (Those
who will help to improve the module page) with Composer's format is
possible or not ?

[Andrew Short]

You can store extra/custom data in the composer file, so the person can add support for custom fields if they want.

Andrew Short.

--
You received this message because you are subscribed to the Google Groups "SilverStripe Core Development" group.

[vikas srivastava]

ok then we should try to implement composer in this project .

--
Warm Regards
Vikas Srivastava  :)

[Ingo Schommer]

Thanks so much for writing this up Andrew. I think of all the GSOC project ideas,

this one would be the most central to SilverStripe's ongoing health as an open source project.

Here's some pro/con points I see for using packagist vs. reusing the ss.org/modules codebase:

- Pro: Less code to maintain, well documented base (Symfony)

- Pro: Getting future enhancements for free by backing a larger project

- Pro: Already has tags and search built in

- Pro: Less coupling to Andrew's module archirecture GSOC work (in terms of parallel development)

- Con: Less knowledge about Symfony in the community => harder to maintain?

- Con: Need to implement community rating in Symfony code. I'd just use Disqus for comments, so technology agnostic.

- Con: Needs an opensearch layer to expose to "global search" on ss.org

- Con: Harder to display modules on ss.org profiles (but I'd be happy to scrap that until somebody does an OAuth implementation)

- Con: No (easy) single-sign-on with ss.org

- Con: Requires reimplementing ss.org theme (or at least branding)

- Con: No easy way to upload non-versioned files (could theoretically be handled via a packages.json file) - but no big loss to me

- Con: Packagist codebase not very well unit tested

- Neutral: Would have to run under subdomain rather than www.ss.org, not as integrated, but also means easier to scale maintenance in community

Looks like more cons than pros, but there's also some *big* pros in there ;)

We'll have to reach a conclusion for/against packagist very soon, as students need to know what they're applying for.

@Andrew: You suggested that the Packagist aspect could be separate from your GSOC project.

I actually think this could work in our favour, as a "replace ss.org/modules" project could focus on more or less

generic improvements which can be tested without requiring SilverStripe to support composer.

But: This additional project would be more of a Symfony than SilverStripe project, meaning

both the mentor and student should be proficient in it.

Ingo

[Andrew Short]

If you look at the Packagist source, you'll notice it's relatively simple and quite modular:

https://github.com/composer/packagist/tree/master/src/Packagist/WebBundle
https://github.com/composer/packagist/blob/master/src/Packagist/WebBundle/Entity/Package.php (most of the Composer interaction lives here)

It all seems nicely decoupled from the actual Composer API - Packagist itself isn't doing anything magic. I think this wouldn't be too hard to implement in SilverStripe, especially as we can re-use some of the classes and patterns. I think this would be the best approach though since it means we're using proven code and won't be unnecessarily reinventing the wheel.

Andrew Short.

[Sam Minnée]

I'd be much more comfortable if we were using SilverStripe to run this site.  It seems a shame for people to have to work with different frameworks and templating languages to help work on SilverStripe's module site.  I also think that the goals of packagist will wind up diverging from ours to the point that we need to hack on the codebase a lot, which would mean hacking on a Symfony code base.  As an example, I'd assume that this site is going to need to deal with themes and possibly widgets.

It's one thing to use a 3rd party system for an internal library or a tool only used by a subset of developers, but this modules site will be an important and very visible part of the SilverStripe ecosystem, and to base it on something other than SilverStripe just because there is a similar-ish codebase already available doesn't seem like the right tradeoff to make.

This site is essentially "putting a face on Composer", and I'd like to see such faces built on the SilverStripe framework.

If Andrew thinks that's a reasonable approach then I'd vote for that.

[Dan Rye]

This site is essentially "putting a face on Composer", and I'd like to see such faces built on the SilverStripe framework.

I thought this was what we were talking about when I through my vote in for Composer.

[Ingo Schommer]

After looking at the Packagist codebase a bit more with Andrew,
I agree that a lot can be reused "under the hood" because
Packagist is a fairly light layer on top of Composer.
So another vote for SilverStripe'ing the face! :)

[Sam Minnée]

On 27/03/2012, at 11:21 AM, Dan Rye wrote:

This site is essentially "putting a face on Composer", and I'd like to see such faces built on the SilverStripe framework.

I thought this was what we were talking about when I through my vote in for Composer.

Well, SilverStripe modules could certainly be uploaded to packagist.org if they were built as Composer modules.  And we could probably structure things with framework and cms such that the whole framework could be pulled from packagist.org.  In this sense, packagist.org acts as something analogous to PEAR or RubyGems - a centralised repository that many different unrelated projects use.  This seems like a good thing.  It would also lead into things like pulling our Zend dependencies from packages.

There is a second question: what do we do about addons.silverstripe.org or silverstripe.org/modules or whatever?  Ingo seemed to be suggesting we could use the Packagist.org codebase as a starting point, and I don't think that's the right idea.  Sure, packagist.org as-is has some features that are better than silverstripe.org/modules, but it's still quite a long way from what we want.

[Ingo Schommer]

Andrew has put up a website where he updates his status,

and blogs about the solution: http://ajshort.github.com/gsoc/

Its a github page, so either watch the repo (https://github.com/ajshort/gsoc)

or subscribe to its RSS feed (http://github.com/ajshort/gsoc/commits/gh-pages.atom)