News on extensions.silverstripe.org

164 views
Skip to first unread message

Ingo Schommer

unread,
Feb 25, 2013, 6:20:22 AM2/25/13
to silverst...@googlegroups.com
Hello everybody!

The module/widgets/themes listing on silverstripe.org direly needs a rethink,
since its current design and process approach wasn't suitable any longer for hosting all the shiny
modules you guys create (thanks by the way!).

There's been many previous discussions on this topic. One major factor influencing
our way forward was the recent surge of using Composer for package management.
There's nearly 250 SilverStripe modules on packagist.org already!

Our GSOC student Vikas Srivastava has started this work late last year (git repo), 
and has provided a lot of valueable direction on how to handle the changed requirements,
as well as helping various module authors make the switch to Composer - thank you Vikas!
Slightly in parallel, Andrew Short has worked on his own version of a modules listing.

Andrew's work can be previewed at http://extensions.andrewshort.name/

We've decided to base the new modules listing on Andrew's work, and merge in Vikas's
contributions as required. This decision was mainly driven by the fact that we changed
our mind since GSOC 2012 on how we manage modules: Packagist turns out to be a really useful
and stable platform, we didn't want to replicate this effort. Andrew's project uses
Packagist for its module information, so you'll need to register modules there,
and we'll just read out the JSON data.

So on the new module listing, all modules and themes are required to contain a composer.json
file, and be registered on packagist.org. I've updated the installation instructions
to reflect this fact, and added an FAQ around common Composer questions:

Another important decision was to make the module listing a separate website from silverstripe.org,
to allow for a greater extent of collaboration in the community. It will eventually be hosted on extensions.silverstripe.org.
and I've added a bunch of enhancements there already. Contributions welcome!

This effort is mainly driven by volunteers and their spare time, so we can't promise
a launch date - but the more you help (with code, testing, feedback), the faster we get it done :)

Please let us know what you think! Does the new approach serve your needs as module authors and module users?

Regards
Ingo

Martimiz

unread,
Feb 25, 2013, 7:17:16 AM2/25/13
to silverst...@googlegroups.com
Looking great!

Would it be an idea - or even possible - to dynamically read and include the extended documentation from github, if a docs directory exists?

Martine

Anselm Christophersen

unread,
Feb 25, 2013, 7:37:40 AM2/25/13
to silverst...@googlegroups.com
Probably worth mentioning that there's already a bunch of todos ready to take on here:

I guess it's just about picking one, and letting the admins know through the comments that you're working on it?

--
You received this message because you are subscribed to the Google Groups "SilverStripe Core Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to silverstripe-d...@googlegroups.com.
To post to this group, send email to silverst...@googlegroups.com.
Visit this group at http://groups.google.com/group/silverstripe-dev?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

Ingo Schommer

unread,
Feb 25, 2013, 7:48:21 AM2/25/13
to silverst...@googlegroups.com
Absolutely. We obviously prefer issues being filed there where the author
is actually willing to contribute work hours himself, but generally any reasonable
idea can be placed there. Reasonable meaning: "Its likely that somebody in the community
will take this on in the next 3-6 months". If you expect it to require some discussion and consensus
in the community first, its better placed on this thread for now though.

Sam Minnée

unread,
Feb 25, 2013, 3:00:56 PM2/25/13
to silverst...@googlegroups.com
It would be great to have documentation available for all available SilverStripe modules.

- I'm not sure of the processing / disk requirements of this, because it would mean that each module would need to be checked out onto the server. I presume that's okay but it would need to be monitored. We've gotten in trouble in the past with these auto-fetch-from-git systems -- bugs in them are a common cause of silverstripe.org outages. It can be done, it just needs to be built well - probably with some kind of message/worker pattern, rather than a simple cronjob.

- I'm not sure whether it would make more sense to run that on extensions.silverstripe.org or doc.silverstripe.org, since some of the modules will have their docs on doc.silverstripe.org regardless. The distinction might become arbitrary to visitors. In either case, we'd be using the docsviewer module.

My initial thought is that putting the documentation onto doc.silverstripe.org seems weird, but upon further reflection it's not weirder than putting it on another vhost on the server (extensions.silverstripe.org). We would, of course, need to ensure that the IA supported the inclusion of a few hundred modules.

Sam Minnée

unread,
Feb 25, 2013, 3:05:06 PM2/25/13
to silverst...@googlegroups.com
Hi everyone,

This system looks for everything on packagist where type is "silverstripe-module" or "silverstripe-theme".  So, the tagging of modules as "silverstripe" is optional, although that still makes it easier to identify SilverStripe stuff directly on the Packagist.org site.

One thing it *does* mean is that using the "silverstripe/" vendor prefix is definitely unnecessary now, and one of the feature suggestions I've made for the new site is to have a "browse by vendor" view, in addition to the "browse by maintainer".

Ingo Schommer

unread,
Feb 25, 2013, 4:27:24 PM2/25/13
to silverst...@googlegroups.com
The extensions codebase uses Composer PHP classes to download the repos already, that's how the README is fetched as well.
(pointing to the old repo, Andrew has reset the github repo which seemed to have made some classes disappear)

At the moment the checkout/download and parsing happens synchronously for all changed modules,
I've typed up a ticket reasoning for a queue implementation: https://github.com/silverstripe/extensions.silverstripe.org/issues/27

I would be inclined to keep module docs on extensions.ss.org, simply because it leaves all module info in one place,
and already have the IA to support a large number of modules. It also keeps doc.ss.org more focused on a simpler IA,
the existing content structure for core docs is already hard enough to understand ;)

Martimiz

unread,
Feb 27, 2013, 5:37:55 AM2/27/13
to silverst...@googlegroups.com

If the modules are already downloaded and kept up-to-date, the docsviewer could still be part in this. But only if  it could register the requested module/version on the fly,  as registering all 250+ modules in advance on every pageview isn't an option :-)  Then each extention page can generate its link to the docs.

Martine

Martimiz

unread,
Feb 27, 2013, 10:38:34 AM2/27/13
to silverst...@googlegroups.com

But only if  it could register the requested module/version on the fly...

Just tried, and registering modules on the fly is really easy to accomplish :-) There are two things though:

1. Probably best rebuild Lucene indexes in batches, might need to adapt the task - sync to queue mentioned before?  Is search necessary anyway on individual module docs?

2. Each documentation would be approached individually, based on the repo's current branch, so versions are not accessible? In that case the header with module/version links can be optionally stripped.

Martine

Ingo Schommer

unread,
Feb 27, 2013, 12:17:43 PM2/27/13
to silverst...@googlegroups.com
I don't think we'll need any Lucene indexing for docsviewer to be honest.
We're looking into replacing the main ss.org search with a Google Custom Search Engine,
and doc.ss.org plus extensions.ss.org would be easy to add there. The specialized extensions.ss.org
search including tags, dependencies etc will need to stay on ElasticSearch, but I don't
see the need to invest time into the docsviewer Lucene search at the moment. @Will, what do you think?
In terms of resource usage, my preference is starting with only exposing docs for the respective module master.

@Martine: Do you have exts.ss.org running locally? I think the COmposer PHP classes just
check out into a temporary folder somewhere, not sure how they go about cache cleaning policies.

Nicolaas Thiemen Francken - Sunny Side Up

unread,
Feb 27, 2013, 4:30:04 PM2/27/13
to silverst...@googlegroups.com
On 28 February 2013 06:17, Ingo Schommer <in...@silverstripe.com> wrote:
I don't think we'll need any Lucene indexing for docsviewer to be honest.
We're looking into replacing the main ss.org search with a Google Custom Search Engine,



We have done a bit of research into the GCSG stuff and we are implementing this for a client at the moment. 

Martimiz

unread,
Mar 1, 2013, 10:57:40 AM3/1/13
to silverst...@googlegroups.com


@Martine: Do you have exts.ss.org running locally? I think the COmposer PHP classes just
check out into a temporary folder somewhere, not sure how they go about cache cleaning policies.


No I haven't... I may give it a go this weekend. But, brainstorming, if downloads are indead only temporarily cached, what if we would save the markdown into db 'as is' on download, and then have an adapted docsviewer version read those instead of fysical pages?

Martimiz

unread,
Mar 20, 2013, 6:07:41 AM3/20/13
to silverst...@googlegroups.com
Are there any updates on this?

Looks like https://github.com/silverstripe/extensions.silverstripe.org/ no longer exists, and I can't really find anything else on GitHub - but that may be me, of course :)

Martine

Ingo Schommer

unread,
Mar 20, 2013, 6:13:24 AM3/20/13
to silverst...@googlegroups.com
Hey Martine, it has simply been renamed to https://github.com/silverstripe/addons.silverstripe.org.
We felt that "addons" is a less ambiguous way of describing what the site contains,
since it doesn't cause any confusions with the "Extension" PHP class in SilverStripe,
and makes it clearer that the site will also contain themes.

Martimiz

unread,
Mar 20, 2013, 6:59:22 AM3/20/13
to silverst...@googlegroups.com
Hi Ingo, thanks. As to if (and if so, how) the docs for individual addons are stored anywhere on the server for use with DocsViewer, I haven't been able to sort that out. It would be really great (and so much quicker) if Andrew could just say yes or no :)

Martine


Op woensdag 20 maart 2013 11:13:24 UTC+1 schreef Ingo Schommer het volgende:

Andrew Short

unread,
Mar 20, 2013, 7:00:33 AM3/20/13
to SilverStripe Development Mailing List
At the moment modules are downloaded, but only the README is extracted. Docs viewer integration would be very nice, and is definitely doable, but I'm not working on it at the moment,

Andrew Short.

Martimiz

unread,
Mar 20, 2013, 7:52:32 AM3/20/13
to silverst...@googlegroups.com

Hi Andrew,

At the moment modules are downloaded, but only the README is extracted. Docs viewer integration would be very nice, and is definitely doable, but I'm not working on it at the moment,



You mean you're not working on the addons platform? Is anyone else? Or is it just temporary on hold?

If the docs directories are fysically present on the server, implementing the DocsViewer would mean nothing more then installing it and generating a link on each module's main page.

The only upgrade the Docsviewer would need is to be able to register modules on the fly instead of registering every module on each pageview. But I already tried, and it's a simple addition really, I could do that. I also sent a pull request enabling the DocsViewer to include README pages.

One extra thing would be enabling search: if you'd want to use it, the task of rebuilding the indexes would need a slight alteration as well - but that would probably not be very hard - and most modules wouldn' really need it...

Martine

Andrew Short

unread,
Mar 20, 2013, 8:23:25 AM3/20/13
to SilverStripe Development Mailing List
I just meant I'm not working on that specific feature. I don't think it would be quite as trivial as you make it sound :) At the moment I'm only focusing on essential features.

Andrew Short.


--
You received this message because you are subscribed to the Google Groups "SilverStripe Core Development" group.

Martimiz

unread,
Mar 20, 2013, 8:55:27 AM3/20/13
to silverst...@googlegroups.com

>> I don't think it would be quite as trivial as you make it sound :)

Why would you think that :) ? At the moment I'm sort of immersed in the DocsViewer - and I really think for you it wouldn't mean much more then

- dynamically adding a link like http://extensions.andrewshort.name/docs/<modulename>
- install the DocsViewer (supporting dynamic registration, which I already tried and it's easy to add
- point it to the fysical location of the modules - if there is one...
- think about search indexes as a next step...

If you think I can install your addons module with any chance of success, I'd be happy to set up a trial. I'm just scared of having to download all addons and crashing my system. Would there be a way to limit that to, say ten or so?


Reply all
Reply to author
Forward
0 new messages