core directory structure

[Ingo Schommer]

i'm just trying to describe the update procedures of silverstripe core,
and am failing to get a concise description of which core folders to
delete/overwrite, and which ones to leave alone. as toplevel folders
will be added/removed from future releases, i can't just list the
usual suspects (cms,sapphire,jsparty,auth_openid). especially as we're
planning a more modularized structure
(= more toplevel folders), i think this points to a broader problem in
the directory structure, as well as our definition of a "module".

currently we have these types of "modules" all in the toplevel:
- core (cms, sapphire)
- thirdparty (jsparty)
- project (mysite, tutorial, mycoolproject, ...)
- modules (auth_openid, blog)
- widgets
- themes (with subfolders)
- custom directories (e.g. PEAR)

some requirements that a directory structure would have to support:
- easy packaging, release and replacement of core
- easy identification of application specific folders
- distinction between "core modules" (delivered with release,
required) and "user modules" (added by the user)
- symlinking of one core to multiple projects
- definition through svn:externals
- no dependencies of PHP-configuration to find application folders
(like global $project)
- (optional) usage of core directories outside of the webroot
(symlinking)
- (long term) determining of autoload priorities based on folder: user
modules override core modules

i've looked at other library projects for common patterns, coming to
this proposed directory structure:

assets
app (replaces "mysite")
doc (new folder with packaged api docs)
modules
blog
forum
system
cms
sapphire
modules
auth_openid
thirdparty
jquery
prototype
tinymce
themes
blackcandy
othertheme
widgets
language_chooser

- all path references use the constant BASE_PATH defined in main.php
define('BASE_PATH',
dirname(dirname(dirname($_SERVER['SCRIPT_FILENAME'])));
- in addition, we have MODULES_PATH, THIRDPARTY_PATH, THEMES_PATH,
SYSTEM_PATH, SYSTEM_MODULES_PATH
- depending on how much we want to promote sapphire at this current
stage
as a cms-independent framework, this folder could move to the toplevel.
- all toplevel directories would be created by default, some of them
empty.
the distinction/naming of /modules (user defined) and /system/modules
(core stuff)
could be clearer -any ideas?
- it would also mean the end of the strangely named "jsparty" folder -
although
we could also keep that for nostalgic reasons haha
- non-PHP files can't rely on a fixed hierarchy, e.g. css-rules like
url("../../sapphire/images/logo.gif")
relative referencing WITHIN the module directory is required. if
e.g. a module references images from CMS, best practice would now be
to copy those into the module for relative referencing (decoupling)
- currently _ss_environment is simply looked for in "../", "../../",
"../../../" - its up for discussion how feasible this is
- do we need a "system themes" directory? how about "system widgets"?
- do we need a "system thirdparty" directory separate from "user
thirdparty"?
- should each module know about its path so we don't have to define
CMS_PATH, SAPPHIRE_PATH etc?
- i would suspect that lots of non-native speakers have trouble
understanding the meaning of the "assets" folder, how about "uploads"?
"userfiles"?
- lots of documentation will need to change
- modules referencing mostly jsparty will need to adjust
- (long term) should we have a /config directory within modules and
system folder with multiple files instead of a single _config.php per
module

i've shuffled files from the existing trunk around in that fashion,
fixed a lot of path references, introduced some constants.
basically have the folder setup from above running
(admin loading, tests passing), minus some images not loading.
manifestbuilder is currently just patched from traversing/assuming one
sublevel
to doing two sublevels, so thats a temporary hack.

i've made a commit to trunk (http://trac.silverstripe.com/silverstripe/changeset/63154
)
changing 90% of the hardcoded path infos
(mostly Requirements calls) to using constants, which untangles a lot
of the assumptions
and makes a switch to new structures easier. for example, even if we
decide in the future to change to a class-based environment config, we
can simply replace the used constants with something more dynamic.

so, most of the legwork is done, its proven to (mostly) work,
question now is if we decide that the 2.3 preparations are a good
point in time to make that switch. because we're adding to the confusion
by spinning out more core functionality into modules, and as we need
to do a lot of testing anyway,
i tend to a "yes", but like to hear other thoughts.

Ingo

[Sam Minnee]

I agree that we need to improve the current status quo, but I feel
like your proposal is too complex.

Requirements I dispute:

"distinction between "core modules" (delivered with release, required)
and "user modules" (added by the user)"

--> The only required module is sapphire. At least, that's the
architecture we're moving towards

"easy identification of application specific folders"

--> Well, for most people it's true that there will be a binary
distinction between "application specific" and "modules", but for much
of what we produce there's more of a continuum.
* sapphire is required by everything
* cms is required by most
* ecommerce is required by some
* nzctbase is required by a couple
* mysite is required by one
The implication is that we must treat any "application specific"
special behavior as something convenient for the common case than a
hard-and-fast rule

"no dependencies of PHP-configuration to find application folders
(like global $project) "

--> Well, you're either going to need a global or a define, and it's
going to need to be project-specific, because "app" might be "mysite"
on a legacy project. When it comes to producing a loosely coupled
architecture, defines are just globals with a little lipstick. That
said, we should move towards alteration of these globals from being
the exception rather than the rule. Which I think that we have done
lately; most projects use "mysite" as their app folder now.

"(long term) determining of autoload priorities based on folder: user
modules override core modules "

--> This is a nice-to-have, not a requirement, which I assume you
agree with given that you've said, "long term".

Design decisions I disagree with:

* Widgets & Modules aren't sufficiently different to warrant separate
folders like this. In particular, modules will likely contain
widgets, and what could be considered a "widget" in the classification
you've given is really just a module containing one widget. My point
is that there's a fluid continuum between modules and widgets, and
that in the longer view of the framework, widgets are really just one
kind of "nugget of functionality" that we might ask developers to
provide. Should we have a separate folder for "authentication
plugins?"
* Having separate "modules" and "system/modules" directories doesn't
make any sense. If you're concerned about where auth_openid should
live, I would suggest that we update the installer to provide module
installation in 2.4, and we have auth_openid be demoted to just one of
many modules that users might choose to incorporate into their site
out of the box.
* If we're going to release sapphire as a framework on its own,
without the cms, it's arguable whether cms should live in the
"modules" or the "system" directory. Which gets us to the point where
system contains nothing but sapphire, so why don't we just put
sapphire at the top level?
* The monolithic "third party" directory is a problem. Currently,
most users of silverstripe trunk download an enormous amount of stuff
that they never use. If we're going to go to the trouble of cleaning
this up (which, to be honest, might not really be something we want to
try and squeeze into 2.3) then we should deal with this better. It we
have a dependency manager for modules then it all becomes much easier
because we can just say something like "cms depends on jquery, jquery-
forms, and tinymce" and have the module manager take care of it.
* Based on the above, we might not want to have a flat structure for
modules. In fact, we might want to keep something very similar to our
current system and just allow slashes in module names. I discovered
that you can put a line like "test/blog http://svn.silverstripe.com/open/modules/blog/trunk"
into an svn:externals entry; it will automatically create the test
folder for you. This is great news!

We might just say that our module list for a particular project is
something like this:

ecommerce
auth/openid
blog/main
blog/importer
thirdparty/jquery
thirdparty/tinymce

If we were going to take this approach - let's call it the "module
names have slashes" approach - then we *could* do the things that I
argued against previously - put widgets and system modules into
separate directories - because we wouldn't be treating any directories
as special. They would all just be "part of the module hierarchy".

The simplicity that I'm interested in is not a reduction in the number
of folders; rather, it's a reduction in the number of special cases.

[Matt Peel]

> - distinction between "core modules" (delivered with release,
> required) and "user modules" (added by the user)
>

As I think Sam mentioned, sapphire should be the only "core module" -
everything else is (or should soon be) optional.

> doc (new folder with packaged api docs)
>

I like the idea of packaged api docs, but always disliked it when these
were included in the root of the site. If you're going to put them in
the .tar.gz, I would instead propose that at the root of the .tar.gz is
a 'site' or 'upload' directory, and docs is a sibling of that instead.
The install notes should be modified to say 'upload the 'site' directory
to the root of your website (or where-ever you want SS to be
installed)'. Alternatively, deliver the docs separately. I've never seen
a need for docs if there is a website (e.g. api.silverstripe.com) - so
long as api.silverstripe.com has a way to download the API for those who
have internet like New Zealand's or want local docs.
> thirdparty
> jquery
> prototype
> tinymce
>
thirdparty is an interesting idea, but I think will get into problems -
what if you have a module that uses third-party code? Like... well I
can't think of any good examples, but where would it live? Split between
thirdparty/ and modules/? All in modules/? Also, you've only got JS in
here - is that just because all of the third party SS uses so far is JS?
There's random other bits and pieces - e.g. the bbcode parser
(where-ever it lives now) is (or was) a bunch of PEAR classes.
> themes
> blackcandy
> othertheme
>
Does that mean that you would have /themes/blackcandy/blog/ and
/themes/blackcandy/forum/ or /themes/blackcandy_blog/ and
/themes/blackcandy_forum/ as it is now. I prefer the former approach -
it keeps the themes directory tidy, and when its as easy as it is in SS
to swap from one theme to another, I'd like to install a whole bunch and
then pick which one I liked the best.
> widgets
> language_chooser
>
One thing to note is that a number of modules have widgets in them as
well (e.g. the Youtube module also has a widget I think?). Would you
separate them out in this structure, or would widgets that require
features only present in a module be lumped in with that module?

> - all path references use the constant BASE_PATH defined in main.php
> define('BASE_PATH',
> dirname(dirname(dirname($_SERVER['SCRIPT_FILENAME'])));
> - in addition, we have MODULES_PATH, THIRDPARTY_PATH, THEMES_PATH,
> SYSTEM_PATH, SYSTEM_MODULES_PATH
>

I'd like to see THEME_PATH added to that list (or some equivilent) which
is a link to the current theme's base folder. This would only work if
you had the first directory structure I proposed above - e.g.
/themes/blackcandy/blog/ etc, because otherwise it could be confusing as
to what exactly THEME_PATH should point to if you are on a blog page.

> the distinction/naming of /modules (user defined) and /system/modules
> (core stuff)
> could be clearer -any ideas?
>

I'm in favour of just dropping it together, having sapphire/ at the root
(and possibly everything else inside modules/ - I dunno about that).

> - currently _ss_environment is simply looked for in "../", "../../",
> "../../../" - its up for discussion how feasible this is
>

I'd like the ability to define where this file is - like
/etc/silverstripe or something. But that's probably going way overboard
- it is nice to have cfg in one place :)

> - should each module know about its path so we don't have to define
> CMS_PATH, SAPPHIRE_PATH etc?
>

Automatically defining something for that would be cool - define a
PATH_* for each module name that is automatically created by sapphire
during a manifest build and stored in the manifest file. So long as the
module knows what it's called, it can make use of PATH_SAPPHIRE or
similar elsewhere.

Need to think about namespacing here - SSPATH_SAPPHIRE maybe... I think
having a common prefix makes more sense than a common suffix in any
case, but that might just be a matter of style.

> - i would suspect that lots of non-native speakers have trouble
> understanding the meaning of the "assets" folder, how about "uploads"?
> "userfiles"?
>

How hard do you think it would be to make this customizable? Then people
could change it to what they feel comfortable with, and my job of
creating a document management system for someone would be a lot easier,
because I could leaverage the upload capabilities of the existing
/admin/assets/ admin page, but have files uploaded to a directory
outside of the web root, then have an ACL to define who gets access to
what file.

> - lots of documentation will need to change
>

Have fun with that Ingo :D

> so, most of the legwork is done, its proven to (mostly) work,
> question now is if we decide that the 2.3 preparations are a good
> point in time to make that switch. because we're adding to the confusion
> by spinning out more core functionality into modules, and as we need
> to do a lot of testing anyway,
> i tend to a "yes", but like to hear other thoughts.
>

I'd think if you're already making 2.3 a backwards-incompatible release
(I assume it is, but not totally sure), then you might as well do as
much as you can now.

Matt.

[Ingo Schommer]

On 28.09.2008, at 07:22, Sam Minnee wrote:

>
> "distinction between "core modules" (delivered with release, required)
> and "user modules" (added by the user)"
> --> The only required module is sapphire. At least, that's the
> architecture we're moving towards

But thats not where we are, or will be anytime soon.
And its an unrealistic use case for 95% of our projects.
With "system", I mean "whats packaged with our cms release".
And if you really want to, you can distribute a "system"
directory just containing "sapphire". The path configuration
should also allow you to configure "sapphire" and "cms" as toplevel
directories, but I don't think its appropriate for our main product,
Silverstripe CMS.

>
> "easy identification of application specific folders"
> --> Well, for most people it's true that there will be a binary
> distinction between "application specific" and "modules", but for much
> of what we produce there's more of a continuum.

True, modules have interdependencies. In the core release,
all dependencies should be contained within the /system/modules folder.
If somebody now installs forum, and its relying on the (imaginary)
textile module not delivered with the core, he has to install both
modules in his "user modules" folder.

Using modules directories, it'd be much clearer if you
have multiple application specific directories, like "nzct" and
"nzctbase"
on the toplevel.

As a developer, I want to look at a project I didn't write,
and clearly see what has been altered from the default installation.
That shouldn't involve digging through releases to see what
modules were delivered at this point in time, just to find
out which are custom. This distinction might be obvious to us core-
developers,
and we have a low number of system modules (currently!),
but its hard to explain to newcomers. Instead of "delete all folders
system folders and modules before upgrading",
I'll have to write in documentation: "delete the folders cms,
sapphire, jsparty, auth_openid,
and by the way, if you have 2.3 that'll include userform and comments
and whatnot as well" ;)

>
> "no dependencies of PHP-configuration to find application folders
> (like global $project) "
> --> Well, you're either going to need a global or a define, and it's
> going to need to be project-specific, because "app" might be "mysite"
> on a legacy project. When it comes to producing a loosely coupled
> architecture, defines are just globals with a little lipstick.

You're right, "mysite" is broadly used now - as a term, it doesn't
really
describe the full breadth of whats built with silverstripe, but i guess
the same is true of "app". Constant or global, it shouldn't be defined
by users in most cases, and hence not appear in mysite/_config.php
generated by the installer.

>
>
> * Widgets & Modules aren't sufficiently different to warrant separate
> folders like this. In particular, modules will likely contain
> widgets, and what could be considered a "widget" in the classification
> you've given is really just a module containing one widget. My point
> is that there's a fluid continuum between modules and widgets, and
> that in the longer view of the framework, widgets are really just one
> kind of "nugget of functionality" that we might ask developers to
> provide.

Fair enough.

> Should we have a separate folder for "authentication
> plugins?"

That should be the folder for "system plugins" - why single out
"authentication" here?

>
> * Having separate "modules" and "system/modules" directories doesn't
> make any sense. If you're concerned about where auth_openid should
> live, I would suggest that we update the installer to provide module
> installation in 2.4, and we have auth_openid be demoted to just one of
> many modules that users might choose to incorporate into their site
> out of the box.

So you're saying the distinction of system- and user modules would
just be given
in some metadata by a module installer that we'd still have to write?
I think having them in
separate folders is a bit more straightforward here.
>

> * The monolithic "third party" directory is a problem. Currently,
> most users of silverstripe trunk download an enormous amount of stuff
> that they never use. If we're going to go to the trouble of cleaning
> this up (which, to be honest, might not really be something we want to
> try and squeeze into 2.3) then we should deal with this better. It we
> have a dependency manager for modules then it all becomes much easier
> because we can just say something like "cms depends on jquery, jquery-
> forms, and tinymce" and have the module manager take care of it.

Good point, so we would get rid of "jsparty" as an organising structure
alltogether? In the end, its mostly modules which have thirdparty
dependencies,
and thirdparty code isn't really more than a module itself.

>
> * Based on the above, we might not want to have a flat structure for
> modules. In fact, we might want to keep something very similar to our
> current system and just allow slashes in module names. I discovered
> that you can put a line like "test/blog http://svn.silverstripe.com/open/modules/blog/trunk
> "
> into an svn:externals entry; it will automatically create the test
> folder for you. This is great news!

Yeah recursive folder generation from svn:externals could come in handy.

> We might just say that our module list for a particular project is
> something like this:
>
> ecommerce
> auth/openid
> blog/main
> blog/importer
> thirdparty/jquery
> thirdparty/tinymce
>
> If we were going to take this approach - let's call it the "module
> names have slashes" approach - then we *could* do the things that I
> argued against previously - put widgets and system modules into
> separate directories - because we wouldn't be treating any directories
> as special. They would all just be "part of the module hierarchy".

So this would be a de facto namespace for a module? Could work,
but has to be clearly documented which namespaces you can use
for your modules. Otherwise somebody makes a blog add-on
called blog/mycoolfeature, the user downloads the official blog module,
and wonders why /mycoolfeature is not included.

[Sam Minnee]

I think that this can't really be resolved without a decent module
manager - the module manager affects the design of it too much.
Rather than change it in 2.3, and change it again in 2.4, let's just
leave this whole issue until 2.4.

I've opened a couple of tickets about this

http://open.silverstripe.com/ticket/2864
http://open.silverstripe.com/ticket/2865

[Simon J Welsh]

Breaks the installer. I think having a working installer may be
important?

On 28/09/2008, at 5:07, Ingo Schommer wrote:

> i've made a commit to trunk (http://trac.silverstripe.com/silverstripe/changeset/63154
> )
> changing 90% of the hardcoded path infos
> (mostly Requirements calls) to using constants, which untangles a lot
> of the assumptions
> and makes a switch to new structures easier. for example, even if we
> decide in the future to change to a class-based environment config, we
> can simply replace the used constants with something more dynamic.

---
Simon Welsh
Admin of http://simon.geek.nz/

Who said Microsoft never created a bug-free program? The blue screen
never, ever crashes!

http://www.thinkgeek.com/brain/gimme.cgi?wid=81d520e5e

[Ingo Schommer]

On 29.09.2008, at 00:27, Simon J Welsh wrote:

>
> Breaks the installer. I think having a working installer may be
> important?

Doh, thanks for noticing. Fixed in r63204.