Module Manager

[Andreas Piening]

Hi everybody,

in the context of designing the new SilverStripe 3.0 I would like to kick of a discussion about a Module Manger and share some thoughts about how I would picture it. In the fashion of the http://open.silverstripe.org/wiki/development/NewDataMapper I created a page in the wiki and encourage people comment on it here in the dev list. Here it is [drum roll]:

http://open.silverstripe.org/wiki/development/ModuleManager

Cheers

Andy

[Sigurd Magnusson]

Andreas, great work and thanks for putting up some thoughts.

How did you see this working with the notion of a website bridging various environments, such as development, testing, and production?

The ideas put forth appear to expect either;

1) Module installation/etc can act on the production server directly. Many people may consider this to be useful for smaller/simple websites, or where the installations being performed are minor (addition of a widget, rather than a full, complex, module). I expect many people would deem the ability to add, upgrade, remove modules on production servers to be risky on high-profile or complex websites.

2) Module Manager operates on an environment other than production, and therefore some mechanism deploys code changes to production once the test site is considered stable.

Questions;

1. Is the goal of MM to allow both types of use (acting on production and pre-production)

2. Do we see merit in being able to restrict the use of MM, rendering the GUI in a read-only mode that shows installed modules and whether they are out of date, but doesn't allow installation, upgrading, or removal? This would likely be set on a production environment. For security, we may also want to allow a developer to disable command line operation of MM as an option.

3. Where MM is used on pre-production and not production, it would be worth agreeing whether the goal of the Module Manager is to assist with the deployment of modules, or if existing techniques (use of SVN, Git, FTP, etc.) would be required.ness requirements.

4. Does it make sense to classify modules in terms of risk of installing/upgrading so that "low risk" updates can be performed (e.g. widgets marked as stable and meet certain unit-test coverage criteria, etc) whilst preventing a user from installing "higher risk" code (e.g. full modules, or anything marked as pre-stable, etc.)

I assume the questions 1-4 will be asked by others too, so it would make sense for the answers to eventually bubble up to the wiki page you've produced.

Cheers,

Sigurd.

--
You received this message because you are subscribed to the Google Groups "SilverStripe Core Development" group.
To post to this group, send email to silverst...@googlegroups.com.
To unsubscribe from this group, send email to silverstripe-d...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/silverstripe-dev?hl=en.

[Ingo Schommer]

On Wednesday, January 26, 2011 12:57:35 PM UTC+13, Sigurd Magnusson wrote:

Andreas, great work and thanks for putting up some thoughts.

How did you see this working with the notion of a website bridging various environments, such as development, testing, and production?

In my opinion, staging and environment awareness into the module manager is out of scope - there's proven solutions for managing codebases and their dependencies throughout the deployment process.

High-profile sites simply wouldn't use a module manager to install modules, unless they really like to live on the edge ;)

In terms of priorities, I'm really interested in describing module metadata and its dependencies, first and foremost.

Every dev wants to know if blog 0.2 doesn't work with 2.4.3 or lower, regardless of his deployment process.

Does anybody know good PHP dependency management frameworks? I'd rather avoid writing that ourselves...

I assume we can't get something similiar to Rubygems+Bundler for PHP easily, right?

We've got pear in the PHP world, but it sucks ... example: http://sebastian-bergmann.de/archives/899-PHPUnit-3.5-Upgrading-Woes.html

Automated module install (and a central repo+API to support this) are nice, but not crucial IMHO.

Worth discussing of course, but I think in terms of near-term implementation plans, we have

to start to prioritize.

[Dan Rye]

Andreas, I like were you are headed with this.  And I agree with Ingo.  Two separate discussions could be forked from this:

1. Proven deployment strategies (with SS & modules, on GIT?) across various deployment code bases (DEV, QA, UAT, PROD).
2. PHP Frameworks for module/plugin/extension dependencies.

The first my not belong on this list, but I'd be interested in having a section of the SS documentation reflect this sort of thing, so maybe it is a topic to be initiated on the documentation list.

As for #2, coming from a java background myself I can suggest Maven for PHP (http://www.php-maven.org/) - It is way more than just a dependency management framework, but much better than pear.

-Dan

[Barry Swaisland]

Andreas,

I agree with your document and since I also suggested it on the 3.0
discussion I wish to show support for this again now.

The current situation where there are many great modules listed as
unsupported or worse still only on github and linked to every now and
again on IRC - if module install was automatic then these could be
accessed in a much easier way because the list of modules could be
shown that are registered as working with your current install of
silverstripe - again the community can indicate whether or not a
module works on a particular build.

Regarding the very extensive list you have provided - I'd like to be
able to filter the modules available and also add my own not-connected-
with-silverstripe modules to the list.

Oh and I'd also wish the same for widgets as with modules :)

[matt clegg]

Just my 2cents but (from a users point of view) I really like the way Wordpress allows you to install modules from inside the admin area.

(from a dev point of view) maybe there could be a way of adding something like a Silverstripe repository list so that you can choose which group of modules ie trunk/branches etc ?

Matt

--

[Hamish Campbell]

Hey all,

This is something I've been thinking about a lot recently, given that
the move to git has thoroughly complicated our deployment strategy
(yes, I'll let it go now).

Firstly - absolute agree that the first task is to define a common
method of identified & describing module meta-data. Can I make the
following suggestions/requests:

1. The unique identifier is a publicly accessible URI.

2. The document at the URI contains the module definition.

3. The module definition is machine readable and [also] sits in the
module root.

4. Checksums should be optional, but (for example) not including it
would preclude it appearing in 'trusted' source lists, or from GUI
installers.

This works very cleanly with apache subversion handlers, git, GitHub
(point it at the raw source), or you can just post a static file.

An example of an XML definition would be:

<module>
<name>AwesomeSauceModule</name>
<uri>http://www.example.com/modules/AwesomeSource/trunk/
definition.xml</uri>
<version>1.3.3.7</version>
<checksum type='sha-1'>ABC123</checksum>
<source type='git'>http://www.example.com/modules/AwesomeSource.git</
source>
...

Why?

So that module lists can be aggregated without the need for a central
repository and dependencies can be resolved without referring back to
a central location. This is also easily (and optionally) extended with
other features like author information, comments, ratings, etc. You
should also be able to define a complete deployment from an XML
definition containing the dependant modules.. svn:externals in another
guise :P.

Secondly - I personally would like to see this as the primary method
for building a SilverStripe deployment.

If you have even a medium priority site you'll do this on your test
box before syncing to prod, but I personally think that there is room
for a targeted SilverStripe specific deployment solution. There are as
many build tools as their are languages, operating systems and IDEs.
The idea of introducing more dependencies into the already
considerable SilverStripe learning curve (even for intermediate devs)
doesn't appeal.

And after all - if Module Manager is not meant to "manage your
modules", what is it for?

Finally - I don't see an in-CMS installer as a priority at all.
SilverStripe is reasonably developer-centric at the moment, and the
modules that are out there don't really lend themselves to that sort
of install process at the moment. I'm sure that will come with a solid
module management solution, but at the moment there are some other
steps that need to be taken.

Regards

Hamish

[Nicolaas Thiemen Francken - Sunny Side Up]

one thing I am wondering about is form fields.  That is, it would be really awesome to have a central repository of form fields that you can select for your project.

I often find myself creating form fields for "custom" modules (e.g. date pickers, colour pickers, drop-and-drag sorters, etc... etc...) - right now, they just become part of the "custom" module.  However - it would be better if I could make them part of a general collection of form fields.

Then, in Sapphire, we could reduce the number of "core" form fields to a bare minimum - with the option for people to add more form fields as required.... It could even be as simple as (pseudo pseudo code)

...

$field = new FormfieldMaker("SpecialformField");

...

function FormFieldMaker($className) {

 if(! class_exists($className)) {

   //download formfield into code base here

  }

}

etc...

In no time we would have 200+ formfield that would help everyone in building faster and better modules.

- Nicolaas

[Jean-Fabien Barrois]

Hi all,
I am really happy to see that this module manager thread initiates so
many interesting discussions.

A recurring theme in the discussion above is who are we developing
this module manager for?
That's a very good question!
Web developers? Well they don't really need a module manager do they?
They usually know how to use source code management softwares, but
they wonder if that version of that particular module will fit well in
that SS installation, or play along well with these other modules. So
they clone the code, test thoroughly, and deploy when happy. That's in
an ideal world of course. Still the dependency manager described
earlier would be priceless in that respect. And using a cli for
installing/upgrading/removing modules could be interesting for
developers.

But many web developers work for clients who asked for a cms for a
good reason usually (as opposed to static html like 10 years ago) :
getting as much control over their website a possible. I don't know if
you are like me, but 90% of my clients are small to very small
businesses, hosting their website somewhere on a shared server. They
are quite keen on installing a module, widget or a theme, but yeah cli
is not their cup of tea (they don't even have access to it anyway) but
they are now comfortable with SS backend.

I would love to see that module manager makes everybody happy
actually.
So a web developer would probably like some sort of package manager
that would resolve depedencies if any, and offer piston support if
possible since git is used from now on.
And a website owner would like a cms based interface for installing
modules (still resolving dependencies), but still be able to roll back
if things go haywire.

Making everybody happy is one of the most challenging things in web
development, but that's the exciting bit of it isn't it?

Cheers

On Jan 27, 5:49 pm, Hamish Campbell <hn.campb...@gmail.com> wrote:
> Hey all,.

[Jason Stratford]

Hi all,

Another opinion to chip in...

Web developers? Well they don't really need a module manager do they?
They usually know how to use source code management softwares, but
they wonder if that version of that particular module will fit well in
that SS installation, or play along well with these other modules. So
they clone the code, test thoroughly, and deploy when happy. That's in
an ideal world of course. Still the dependency manager described
earlier would be priceless in that respect. And using a cli for
installing/upgrading/removing modules could be interesting for
developers.

I would agree with the above mostly. Proper dependency checking and easier module upgrade would be good. Although with SVN externals upgrade is pretty easy (have to look for an alternative in the post GIT world). From a developers point of view CMS GUI management of modules is not required in my opinion. Separation of configuration and content has always been a strong plus for Silverstripe.

 

But many web developers work for clients who asked for a cms for a
good reason usually (as opposed to static html like 10 years ago) :
getting as much control over their website a possible. I don't know if
you are like me, but 90% of my clients are small to very small
businesses, hosting their website somewhere on a shared server. They
are quite keen on installing a module, widget or a theme, but yeah cli
is not their cup of tea (they don't even have access to it anyway) but
they are now comfortable with SS backend.

Here however is where I have to disagree (not because your wrong - just a different perspective), from our experience business owners come to us because they don't want to manage these sorts of things, they barely want to manage their content in most cases. Silverstripe is perfect for us as we can lock it down to just the content management functions that the client needs without over complicating or confusing their view with features they do not require. As the customers requirements and experience grows Silverstripe grows with them and we are able to modify and integrate new modules with ease, all without risk for the customers live environment.

As with most things, if its not globally required and destined for Core than whatever solution is put in place must be optional.

Some interesting ideas as always though.

Cheers

Jason

[Hamish Friedlander]

The target audience for the module manager has been a bit of a subject of thought for me too. I'd personally love to see a developer aimed module manager, and don't necessarily see a lot of call for a end-user aimed module manager, but I'd rather see a limited scope done right than a half-attempt at pleasing everybody. 

The biggest problem with trying to write a module manager for developers is that it'd need to support all the different development & source code management strategies - git, svn, bazaar? piston, svn externals, or just copying in the code?

In the long term I'd like to see sake expanded to be more of a general task manager, similar to rake for the rails world. This'd let people specify how to perform particular actions for their environment, which a developer aimed module manager could leverage to get the job done.

But in the mean time focusing on getting the metadata right is a good start, and a production-site module installer is a good way to make sure we've got most of the information we need available.

I generally agree with Other Hamish's basic spec (although xml? eew. yaml is already used for test fixtures in SS and seems a more natural language). There is a bit of a problem with describing possible version controlled sources, but nothing unsurmountable. There are plenty of other systems that solve similar problems (linux package managers like debian debs, other language module managers like python's eggs and ruby's gems), so I'd suggest borrowing good ideas from them.

Further in the future, getting ss.org/modules to understand those specs would be awesome. Alternatively, maybe a separate metadata spec for lists of modules (including references to other lists), so that particular dev houses could nominate their preferred sources for particular modules or add in-house modules to the list of modules available?

Hamish Friedlander

[Marcus Nyeholt]

One thing that I think a module manager needs to provide is information about updates that are available to the codebases, and what exactly is available in the update. As a developer, I want to be able to run a sake task or look at an admin tab in the CMS and see that there's updates available for one or more of my modules, and what exactly has changed between my current version and the new version. This implies that the module spec file needs to at least have information about versions of the module that are available, and the module tool knows how to create a diff between those versions, whether it's from SVN/Git/tarball. 

Marcus

[Matt Bower]

I agree about the target audience for the Module Manager. One of the beauties of SS is that all the configuration is done by the developer at the code level and adding a way for a site maintainer to upgrade their modules steps away from that paradigm. Not to mention that site functionality would be break on an upgrade if one of the source files in the currently installed version of a module had a bug that was manually patched or the original site developer hacked core files.

My vote is to leave the Module Management to the developers and leave managing content to the site maintainers.

An alternate idea would be functionality in SS that listed and linked to the modules available on ss.org for developers to have quick access to and maybe a download builder on ss.org so you could select all the modules you wanted and then download them in 1 package file. Then, for the site maintainers, it could use the proposed module metadata to call home and check if there were updates to the core or modules and inform the maintainer if they're critical or not. Then they would contact their developer to do the necessary upgrades. It would provide more of an impetus for repeat work for us developers ;-)

In my experience, most of the people maintaining these websites barely know how to use a computer, let alone upgrade software. Upgrading web software by a layman is risky business to say the least.

Matt

[Hamish Campbell]

On Jan 28, 9:23 am, Hamish Friedlander <ham...@silverstripe.com>
wrote:

> I generally agree with Other Hamish's basic spec (although xml? eew. yaml is
> already used for test fixtures in SS and seems a more natural language).

Add yaml support to RestfulService and I'm all for it. Unless this
already exists, in which case, 8-o.

[Andreas Piening]

hi everybody,

i've been following the discussion closely over last days. thanks for the many good ideas. i'm on leave until early march and i will then catch up on the thread. since there has been so much positive response about the mm i am confident that we will pull it off.

thanks

andy

[dalesaurus]

Just throwing in my $0.02, with regards to the Module Manager. Maybe
working in baby steps would be good:

1. Work out a simple dependency system (the YML in a module root seems
to be a good start)
2. Build API out to simply move things around the filesystem via CLI
(build on ye olde SS-REPL code?)
3. Add in compressed files and wget/curl wrappers
4. Figure out the GUI situation once all the hard work is done

Subnotes:
-I see that pretty much every framework/language/project has ended up
building their dependency system. It'd be great to build on something
else, but I'd expect to end up with a custom system.
-Building out the CLI first would make every developer happy and skirt
most/any initial permission issues above.

In my eyes a basic dependency for SS would be:
-Namespace top levels (core-*,cms-*,thirdparty-*,module-*,widget-*),
require unique namespaces for all subs
-Standardize versioning (Three numbers X.y.z seems fairly universal)
-Enforcing logic (require, greater than, less than)

This would be a modest start!

[Mark Stephens]

Good point, although I think it might need to be able to show changes at a higher level than just diffs. For example, if a module release has a changed environment/third party dependency you'd need to see that too. Essentially you need to understand:

- what are the benefits of using the updated module - is it of value to me?

- what are implications of that module - are there any gotchas that could cause me headaches?

- what actions are required to apply the update other than an automatic update. - ok, how do I do it?

I think developers are generally the audience - too many modules require code configuration etc, or set up of third party products. There may be a subset of modules that don't require this, but I'm not sure its worth giving non-devs an interface to easily manage this subset.

Mark

-------

Mark Stephens | Development Manager

SilverStripe

http://www.silverstripe.com

Phone: +64 4 978 7330 xtn #63

Skype: mr.morphic

[Jean-Fabien Barrois]

Has anyone got any experience using apache ivy (dependency manager)?
It does what maven does but without all the PM stuff that we wouldn't
need.
It is designed for java apps initially, but can be used for other
things according to the doc.
Still I haven't found any examples on the web of people using it with
php projects.

Jean-Fabien

On Jan 28, 5:38 pm, Mark Stephens <m...@silverstripe.com> wrote:
> Good point, although I think it might need to be able to show changes at a higher level than just diffs. For example, if a module release has a changed environment/third party dependency you'd need to see that too. Essentially you need to understand:
> - what are the benefits of using the updated module - is it of value to me?
> - what are implications of that module - are there any gotchas that could cause me headaches?
> - what actions are required to apply the update other than an automatic update. - ok, how do I do it?
>
> I think developers are generally the audience - too many modules require code configuration etc, or set up of third party products. There may be a subset of modules that don't require this, but I'm not sure its worth giving non-devs an interface to easily manage this subset.
>
> Mark
>
> On Jan 28, 2011, at 11:50 AM, Marcus Nyeholt wrote:
>
>
>
>
>
>
>
>
>
> > One thing that I think a module manager needs to provide is information about updates that are available to the codebases, and what exactly is available in the update. As a developer, I want to be able to run a sake task or look at an admin tab in the CMS and see that there's updates available for one or more of my modules, and what exactly has changed between my current version and the new version. This implies that the module spec file needs to at least have information about versions of the module that are available, and the module tool knows how to create a diff between those versions, whether it's from SVN/Git/tarball.
>
> > Marcus
>
> > On Fri, Jan 28, 2011 at 7:23 AM, Hamish Friedlander <ham...@silverstripe.com> wrote:
>
> > The target audience for the module manager has been a bit of a subject of thought for me too. I'd personally love to see a developer aimed module manager, and don't necessarily see a lot of call for a end-user aimed module manager, but I'd rather see a limited scope done right than a half-attempt at pleasing everybody.
>
> > The biggest problem with trying to write a module manager for developers is that it'd need to support all the different development & source code management strategies - git, svn, bazaar? piston, svn externals, or just copying in the code?
>
> > In the long term I'd like to see sake expanded to be more of a general task manager, similar to rake for the rails world. This'd let people specify how to perform particular actions for their environment, which a developer aimed module manager could leverage to get the job done.
>
> > But in the mean time focusing on getting the metadata right is a good start, and a production-site module installer is a good way to make sure we've got most of the information we need available.
>
> > I generally agree with Other Hamish's basic spec (although xml? eew. yaml is already used for test fixtures in SS and seems a more natural language). There is a bit of a problem with describing possible version controlled sources, but nothing unsurmountable. There are plenty of other systems that solve similar problems (linux package managers like debian debs, other language module managers like python's eggs and ruby's gems), so I'd suggest borrowing good ideas from them.
>
> > Further in the future, getting ss.org/modules to understand those specs would be awesome. Alternatively, maybe a separate metadata spec for lists of modules (including references to other lists), so that particular dev houses could nominate their preferred sources for particular modules or add in-house modules to the list of modules available?
>
> > Hamish Friedlander
>
> > --
> > You received this message because you are subscribed to the Google Groups "SilverStripe Core Development" group.
> > To post to this group, send email to silverst...@googlegroups.com.
> > To unsubscribe from this group, send email to silverstripe-d...@googlegroups.com.

> > For more options, visit this group athttp://groups.google.com/group/silverstripe-dev?hl=en.

>
> > --
> > You received this message because you are subscribed to the Google Groups "SilverStripe Core Development" group.
> > To post to this group, send email to silverst...@googlegroups.com.
> > To unsubscribe from this group, send email to silverstripe-d...@googlegroups.com.

> > For more options, visit this group athttp://groups.google.com/group/silverstripe-dev?hl=en.

>
> -------
> Mark Stephens | Development Manager

> SilverStripehttp://www.silverstripe.com

[Nicolaas Thiemen Francken - Sunny Side Up]

I would like to make a topic / tag / "area of interest" based list of all silverstripe modules ever made - like this:

DATAOBJECT EDITING 

 - module A [click to see more info], [svn / git location goes here], author

 - module B [click to see more info], [svn / git location goes here], author

EMAIL + FORUM

 - member messages module [click to see more info], [svn / git location goes here], author

SOCIAL MEDIA

 - facebook module [click to see more info], [svn / git location goes here], author

MAPPING

 - GoogleMap Module [click to see more info], [svn / git location goes here], author

DATABASE

- db plumber  Module [click to see more info], [svn / git location goes here], author

SEARCH

- sphinx Module [click to see more info], [svn / git location goes here], author

ECOMMERCE

- Ecommerce Module [click to see more info], [svn / git location goes here], author

ECOMMERCE

- Ecommerce Module [click to see more info], [svn / git location goes here], author

My questions are:

a. does this exist already?

OR

b. can you please provide any lists? (Silverstripe People, maybe you can provide an export from your website????)

I can provide this list as XML / JSON / WHATEVER so that people can retrieve it to their own websites, etc...

Please add any modules to: http://tinyurl.com/silverstripe-modules (as a starting point). 

To add a module with ease: http://tinyurl.com/add-silverstripe-module.

Thank you 

Nicolaas

PS reasons are:

* we develop a lot that has probably already been done

* there seems to be a lot of overlap in modules - I am not really against this, but it would be good to combine forces at times

* make it easier to find a module 

[Will Rossiter]

a. does this exist already?

OR

b. can you please provide any lists? (Silverstripe People, maybe you can provide an export from your website????)

I can provide this list as XML / JSON / WHATEVER so that people can retrieve it to their own websites, etc...

It makes complete sense to open up a RestFul API for silverstripe.org/modules.. In fact, I would imagine it would be required for the core part of the module manager system to work properly.

--
Will Rossiter | Developer
SilverStripe
http://www.silverstripe.com

Mobile: +64 27 389 3454
Office: +64 4 978 7330 ext 58
Skype: will.rossi

Level 5, 97-99 Courtenay Place
Wellington, New Zealand

[Nicolaas Thiemen Francken - Sunny Side Up]

Hi Will

Thanks for your message.  Are you saying that there is a RESTFul service operating or that it can be turned on?

Cheers

Nicolaas

[Will Rossiter]

Thanks for your message.  Are you saying that there is a RESTFul service operating or that it can be turned on?

There will be one in the future. Me and Andreas had a discussion about the specifics of it (possible, other concerns etc) a couple weeks ago but no progress on opening it yet. Will keep this thread up to date on any progress.

[Nicolaas Thiemen Francken - Sunny Side Up]

is there any chance to get a csv export for the moment?

[Uncle Cheese]

My only request is that the API be open enough that developers could
create their own frontends for the module manager. Surely there will
be a lot of different ideas about what to include and how the UI
should function. So I think the primary role of SS NZ in this project
is the development, deployment, and documentation of that API.

Also, it would be great to start doing some analytics on the modules.
If you could track the number of downloads and add tags to modules,
that would open up a whole new world to a frontend module manager that
could work with some level of intelligence and dynamism, rather than
getting stuck in the search/browse pattern.


On Feb 5, 8:49 pm, Nicolaas Thiemen Francken - Sunny Side Up

[Sam Minnée]

> My only request is that the API be open enough that developers could
> create their own frontends for the module manager. Surely there will
> be a lot of different ideas about what to include and how the UI
> should function. So I think the primary role of SS NZ in this project
> is the development, deployment, and documentation of that API.

Yeah, I think the initial focus will be to create a PHP/sake based API, on top of which UIs can be built.

The function of the silverstripe.org website in any module manager will be to deliver some kind of json/XML/csv/whatever index.

It's not clear at this stage what silverstripe.org/modules will be come once the application itself has its own module manager. It's feature-set *may* be reduced a little but I would expect there to still be value in having a standalone module browser, for newcomers to the platform.

> Also, it would be great to start doing some analytics on the modules.
> If you could track the number of downloads and add tags to modules,
> that would open up a whole new world to a frontend module manager that
> could work with some level of intelligence and dynamism, rather than
> getting stuck in the search/browse pattern.

Yeah I agree that as the number of modules grows, we need better tools to sort through them.

[Nicolaas Thiemen Francken - Sunny Side Up]

Hi

I am not sure if I understand this module manager in all its details. However, here are some comments about the concept. 

For me, if I want to find out what has been done in the area of, for example

* share icons

* maps

* database management

then I have to look at:

* several third-party websites

* SS widgets (hard to browse quickly or to make a selection)

* SS modules  (hard to browse quickly or to make a selection)

* trac (at times you can find real gems there)

* the fora (some people just post their work there)

* etc...

I would be delighted BIG TIME if there was one place that I could find all the widgets, half-baked code, full modules, best modules, examples, etc... for one topic.  To me this is worth 110x more than having a module installer.  I am pretty sceptical that an automated module installer will work any time soon. It will only work on some hosts and it will only work for fully functional modules such a blog. Furthermore, installing a new module takes around 10 minutes for a developer (add to svn externals / git repo, run dev/build, test it is working), but customising takes much longer (e.g. reading documentation, _config settings, css, html, making it work with your existing templates, etc...).  Having said that, imagine your client decides to install a module (if this becomes an option).  It is 10pm and her side goes down .... then what?

In summary, my recommendation would be to abandon the "module manager" as discussed here (if I understand it right) and focus this energy on:

a. making it easier to find modules

b. creating clearer documentation for modules (dependencies, works with what versions, etc...)

c. making it easier to configure modules (allowing more settings to be adjusted through the site config for example). 

To follow is a bit of background on how I setup _config files.  I am just adding this to give some background on my mode of thinking.  Read on if you are curious. 

I try to write my module _config files like this (JUST AN EXAMPLE - hopefully it is not too wide to fit on nice lines)

// ====================== START googlemapbasic MODULE ======================

//MUST SET

//Object::add_extension('SiteTree', 'GoogleMapBasic');

//Object::add_extension('ContentController', 'GoogleMapBasic_Controller');

//GoogleMapBasic::set_key("abcdef"); //see http://code.google.com/apis/maps/signup.html

//MAY SET

//GoogleMapBasic::set_exclude_from_classes(array("HomePage")); // add relevant page classes

//GoogleMapBasic::set_include_in_classes(array("ContactUsPage")); // add relevant page classes

//ADVANCED CUSTOMISATION

//GoogleMapBasic::set_js_location("mysite/javascript/MyMap.js");//customise the JS

// ====================== END googlemapbasic MODULE ======================

the developer can then copy these settings to their mysite/_config.php file and set as required.

In this way you can:

a. add the module to a site without "turning it on" (i.e. all settings are commented out by default).  in the example above, I could add the basic google map to a site without the CMS editor owner being aware of it - then turning it on when they ask for a google map by simply adjusting the mysite/_config file.

b. update the module to a later version without loosing the settings (all settings are in mysite/_config)

c. find out what settings are available for a module, customising it without having to read through all the code.

d. quickly review the settings for the entire site in your mysite/_config.php on a module by module basis.

Hope that makes sense. 

Nicolaas 

[Sam Minnée]

> In summary, my recommendation would be to abandon the "module manager"

The module manager is going to be increasingly important for managing things like:

- dependencies between modules
- knowing which modules need upgrade
- knowing which versions of which module work with which versions of Sapphire (and SilverStripe CMS, and forum, and...)

Most O/Ses and development platforms have some kind of package manager, because they're a good idea.

On sites that I develop, I'm unlikely to enable any kind of module installation user interface on production sites. It's more of a development / configuration tool. But some people using SilverStripe in more of a self-service manner might find the module UI useful.

That said, the whole 'I installed a module and now the module manager is broken' risk is something we're going to need to address, most likely by some kind of rollback support in the manifest.

[Will Rossiter]

In summary, my recommendation would be to abandon the "module manager" as discussed here (if I understand it right) and focus this energy on:

As the number of modules increase and we build abstracted modules which tightly integrate together, there needs to be a way to manage dependencies better so is what I am really looking forward to.

Take for instance a simple blog with comments and spam protection. If SilverStripe 3.0 didn't have the module manager you would have to download modules blog, comments, spam protection and a spam protector all of which have different versions, tags, branches and some may be in silverstripe, some in silverstripe-labs but with the module management backend I imagine you'll do something like

sake modules install blog comments spamprotection recaptcha

And rely on the module manager to make decisions on version support. That sounds a whole lot easier than requiring developers to "know" or have to "read" documentation on versions.

[Nicolaas Thiemen Francken - Sunny Side Up]

Ok, I hear what you are saying - it is a bit like introducing a version management system on a module level (something we could not live without on a file level). That is, SVN / GIT can tell you if you are using the right file, but not what modules go together.  

Perhaps we could make a central web-page where you can enter all your externals to compute any conflicts / problems. This page could then recommend any changes and you could also enter any problems you encountered (i.e. the system learns as more people use it).

How it works

a. you enter a list of svn / git external addresses of the modules you are using.

b. it works out any problems and list recommendations (e.g. if you use ecommerce 0.5, you must also use payment 0.8 - you have not included this)

c. you run your website and go back to the page at a later date to note any problems (or lack thereof) you encountered (e.g. change recommendation from payment 0.8 to payment 0.9).

d. each combination of externals could have a unique ID (sub-page) and we could leave comments and see how many people (and even who) is using the same combination - so that ideas can be shared on each combination. 

e. you dont end up with managing the version of the module manager to manage the versions of the modules :-)

f. at some stage, we could introduce a small piece of code in sapphire that directly interacts with this external checking page. 

Were you thinking about something like this or something completely different?

Cheers

Nicolaas

[Will Rossiter]

you enter a list of svn / git external addresses of the modules you are using.

The idea would be even that modules have shortcodes i.e forum, dataobjectmanager and you would install by name (and the manager would know the url based off the module API). This would mean that modules will need to be uniquely named. I think the only conflict at the moment is ImageGallery and that's easy enough to fix. 

b. it works out any problems and list recommendations (e.g. if you use ecommerce 0.5, you must also use payment 0.8 - you have not included this)

Yes as mentioned, dealing with dependencies is one of the big selling points for this. Not just between modules but with sapphire core.

If you are looking for more specific information there is some documentation on the wiki.

http://open.silverstripe.org/wiki/development/ModuleManager

[Marcus Nyeholt]

Something to consider, and I'm not sure if there's enough of a performance benefit for...

Has anyone investigated whether combining modules together provides any form of speed increase at all? I've got a couple of projects with over 20 required modules, and I'd be interested to know if that has a noticeable performance impact or not. 

Aside from that, as a module developer, it'd be really nice to have a tool that easily allows me to create a 'release' version of a module that includes any relevant dependencies; whether that's 'combined' into the module by bringing all files and merging _config files, or keeping as separate folders. 

How do other people manage this other than having exit("Module X requires Module Y") if a class from Module Y doesn't exist? 

Marcus

[Sam Minnée]

On 10/02/2011, at 5:45 PM, Marcus Nyeholt wrote:

> Something to consider, and I'm not sure if there's enough of a performance benefit for...
>
> Has anyone investigated whether combining modules together provides any form of speed increase at all? I've got a couple of projects with over 20 required modules, and I'd be interested to know if that has a noticeable performance impact or not.

The only significant impact would come from the number of decorators.

> Aside from that, as a module developer, it'd be really nice to have a tool that easily allows me to create a 'release' version of a module that includes any relevant dependencies

Why? If the module manager will install the dependency for you I don't see the benefit. What if someone installs 2 modules which both have the same dependency? If it's just a great idea, why doesn't apt-get do it?

[Marcus Nyeholt]

Oh definitely, a module manager would solve that problem - but what if the user can't use the manager? Will modules still be downloadable from the web as a package file? The analogy with apt would mean downloading the relevant .debs manually, and as an ubuntu user it can be bloody annoying having to go on a mad chase after dependent .debs; it'd be nice to have them packaged alongside the one I actually downloaded. 

You raise an interesting point though - what happens when two modules have the same dependency, but require different versions of that dependent module? 

Marcus

[Al Twohill]

> You raise an interesting point though - what happens when two modules have
> the same dependency, but require different versions of that dependent
> module?

The easiest way would be to try not to have maximum allowed versions,
ie Foo requires Bar2.3 or higher.

In the case Foo only works with Bar2.3-2.9, you'd be prevented from
installing/updating to Bar3.0 or anything that depends on it, until
Foo is removed or updated to work with Bar3.0...

--
The project has begun...

[Sam Minnée]

It's a nice aim but won't be feasible 100% of the time.

[Sam Minnée]

Well, I've found the Debian package manager to close enough to perfect to never have much reason to complain about. That said, I'd suggest a simple tool that keeps the modules broken out but bundles several module directories into a single tar.gz

That way, developers can more easily resolve module duplication.

In the case of managing decency conflicts - this is what makes constructing such an algorithm nontrivial. Some kind of heuristic search will probably be the best idea. Time to crack open my AI textbooks!