Reorganising -extras
Michael C. Harris
overwhelmingly passed[1]. The manner of reorganising hasn't been
finalised however, though that thread discusses the options. I believe
this is urgent, as the FormUI refactoring will break every plugin that
uses it, and we need a way to manage their update.
The two options are to split the entire tree or to split each
plugin/theme. These are the goals, as I see them:
1. Continue to encourage contributions from lots of people, especially
as a gateway to getting involved in Habari
2. Allow stable releases
3. Support old releases, as per
http://wiki.habariproject.org/en/Release_Policy
4. Allow automated production of zips for dist
5. Allow branching of individual plugins/themes
6. Restrict who can tag something as stable
7. Relatively easily managed permissions
As far as I can tell (and Owen, who suggested it, please correct me if
I've read your post on that thread incorrectly) the main argument for
splitting at the root is ease of maintenance, in that it would be
easier to restrict and grant privileges on half the repo than
individual mini-repos. I don't know enough about svn permissions to
know if that's a reasonable argument. I'm the only one who ever
commits to the only repos I manage :)
I'm not sure how branching and old releases would work if we split at
the root though.
I think splitting each plugin/theme at it's base into mini-repos
covers the first 5 goals well. It also has the advantage of being an
understood system, as that's what we use for the core and what most
other projects use. 6 and 7 are inter-related, and again, I don't have
enough experience to judge how hard they would be to meet. If we can
meet those I'm leaning towards this option.
Opinions? Technical know-how? Let's get this happening.
--
Michael C. Harris, School of CS&IT, RMIT University
http://twofishcreative.com/michael/blog
Scott Merrill
<michael...@gmail.com> wrote:
> The two options are to split the entire tree or to split each
> plugin/theme. These are the goals, as I see them:
>
> 1. Continue to encourage contributions from lots of people, especially
> as a gateway to getting involved in Habari
> 2. Allow stable releases
> 3. Support old releases, as per
> http://wiki.habariproject.org/en/Release_Policy
> 4. Allow automated production of zips for dist
> 5. Allow branching of individual plugins/themes
> 6. Restrict who can tag something as stable
> 7. Relatively easily managed permissions
>
> As far as I can tell (and Owen, who suggested it, please correct me if
> I've read your post on that thread incorrectly) the main argument for
> splitting at the root is ease of maintenance, in that it would be
> easier to restrict and grant privileges on half the repo than
> individual mini-repos. I don't know enough about svn permissions to
> know if that's a reasonable argument. I'm the only one who ever
> commits to the only repos I manage :)
http://svnbook.red-bean.com/en/1.4/svn.serverconfig.pathbasedauthz.html
However we do it, there will be a fair amount of manual intervention
to add users to the authz file, and granularly assign access to
directories within the repository.
> I'm not sure how branching and old releases would work if we split at
> the root though.
If I understand correctly, each project (plugin, theme, what-have-you)
would be its own repository. These independent repositories would be
isolated little kingdoms with discrete user permissions and their own
revision numbers.
> I think splitting each plugin/theme at it's base into mini-repos
> covers the first 5 goals well. It also has the advantage of being an
> understood system, as that's what we use for the core and what most
> other projects use. 6 and 7 are inter-related, and again, I don't have
> enough experience to judge how hard they would be to meet. If we can
> meet those I'm leaning towards this option.
Creating /trunk, /tags/ and /branches inside each project does not
create a new repository. Instead, each commit would increment the
global revision number. So a brand new project might have a gigantic
revision number for its first commit. That's not a big deal, but some
people get worked up over it.
http://svnbook.red-bean.com/en/1.4/svn.reposadmin.planning.html#svn.reposadmin.projects.chooselayout
The only real drawback I see to having /trunks, /tags and /branches
inside each project directory is that someone could more easily
pollute the entire repository with a bad commit. Maybe I don't know
enough about how Subversion works and this actually isn't a big deal.
I think a single repository having /trunk, /tags and /branches inside
each project directory is the best way to go. The question then
becomes: what naming convention do we want to enforce for tagged
stable versions so that whatever infrastructure we create can find
those and package them automatically?
Cheers,
Scott
Michael C. Harris
> On Wed, May 28, 2008 at 3:15 AM, Michael C. Harris wrote:
[snip]
... and a disclaimer, it's late, and I haven't read your links yet
> > I'm not sure how branching and old releases would work if we split at
> > the root though.
>
> If I understand correctly, each project (plugin, theme, what-have-you)
> would be its own repository. These independent repositories would be
> isolated little kingdoms with discrete user permissions and their own
> revision numbers.
That's not what I thought Owen was saying. I believe he was saying we
maintain one repository and split it at the root into a stable tree
and an unstable tree.
> > I think splitting each plugin/theme at it's base into mini-repos
> > covers the first 5 goals well. It also has the advantage of being an
> > understood system, as that's what we use for the core and what most
> > other projects use. 6 and 7 are inter-related, and again, I don't have
> > enough experience to judge how hard they would be to meet. If we can
> > meet those I'm leaning towards this option.
>
> Creating /trunk, /tags/ and /branches inside each project does not
> create a new repository. Instead, each commit would increment the
> global revision number. So a brand new project might have a gigantic
> revision number for its first commit. That's not a big deal, but some
> people get worked up over it.
> http://svnbook.red-bean.com/en/1.4/svn.reposadmin.planning.html#svn.reposadmin.projects.chooselayout
People who care about revision numbers care about it :) I wasn't
counting that as an issue, but if people here do care they should
speak up about that too.
> The only real drawback I see to having /trunks, /tags and /branches
> inside each project directory is that someone could more easily
> pollute the entire repository with a bad commit. Maybe I don't know
> enough about how Subversion works and this actually isn't a big deal.
I'm not sure I follow. If we restrict access to writes to /tags, it's
as resilient as it is now.
> I think a single repository having /trunk, /tags and /branches inside
> each project directory is the best way to go. The question then
> becomes: what naming convention do we want to enforce for tagged
> stable versions so that whatever infrastructure we create can find
> those and package them automatically?
I'd suggest that certain people be given permission to write to tag
directories, commits to tag directories get put wherever we're
distributing zips based on the name of the directory created under the
tag directory, hopefully with the commit message displayed somewhere.
So we have version 1.2 of the diigo silo committed as
/plugins/diigo/tags/1.2 and the commit message "Stable against Habari
0.7" and that would get pushed as a zip to the diigo post (of content
type plugin) as the latest stable download, marked with the commit
message. 1.1 and 1.0 would still be available for download, as well as
a zip from trunk marked unstable available.
Scott Merrill
>> would be its own repository. These independent repositories would be
>> isolated little kingdoms with discrete user permissions and their own
>> revision numbers.
>
> That's not what I thought Owen was saying. I believe he was saying we
> maintain one repository and split it at the root into a stable tree
> and an unstable tree.
Oh, right, so the choices are these:
/habari-extras/plugins/foo/trunk
/habari-extras/plugins/foo/tags/1.0
/habari-extras/trunk/plugins/foo
/habari-extras/tags/1.0/plugins/foo
I greatly prefer the former.
>> The only real drawback I see to having /trunks, /tags and /branches
>> inside each project directory is that someone could more easily
>> pollute the entire repository with a bad commit. Maybe I don't know
>> enough about how Subversion works and this actually isn't a big deal.
>
> I'm not sure I follow. If we restrict access to writes to /tags, it's
> as resilient as it is now.
I assume we're going to open up this repository to more people, but
simultaneously restrict the number of people who can commit to which
directories. So every author who wants to use our system can keep
their plugin or theme in our repository under their control, but they
would not be able to commit to other people's directories (unless they
specifically asked -- perhaps we need a ticket tracking system to
record who requests what access -- if it's just an email, it might get
lost in the noise).
Even still, someone could accidentally copy a bunch of stuff into
their own project directory. I guess it's not that big of a deal
right now.
Cheers,
Scott
Owen Winkler
>
>>> I'm not sure how branching and old releases would work if we split at
>>> the root though.
>> If I understand correctly, each project (plugin, theme, what-have-you)
>> would be its own repository. These independent repositories would be
>> isolated little kingdoms with discrete user permissions and their own
>> revision numbers.
>
> That's not what I thought Owen was saying. I believe he was saying we
> maintain one repository and split it at the root into a stable tree
> and an unstable tree.
Yes, I was suggesting that we split the tree in half, leaving branches
and trunk on one side, and tags on the other. This would allow us to
appoint release managers for -extras that could publish plugins to tags
when they're ready for release.
In contrast, I think that creating separate individual repos for each
plugin is not a good idea, because it would be too difficult to maintain.
It may yet be possible not to split the repo at the root, but to put the
tags/branches/trunk directories into each plugin and theme. An
automated tool to manage permissions would be much more appreciated in
this case.
Also keep in mind the technical barrier for use. It may be easy for us
to conceive of these repo properties, but lay users will need to operate
them. For example, a web interface that allows theme developers to
upload a zip file of their latest revisions would be very useful. An
interface that allows a release manager to package a new release into
tags would be useful. The GUI component almost has to exist, and we
could lay the burden of permission maintenance on that tool to retain a
similar structure to the one we currently have, as an alternative to
breaking the root as suggested above. Obviously, there are drawbacks to
both of these options.
>> I think a single repository having /trunk, /tags and /branches inside
>> each project directory is the best way to go. The question then
>> becomes: what naming convention do we want to enforce for tagged
>> stable versions so that whatever infrastructure we create can find
>> those and package them automatically?
>
> I'd suggest that certain people be given permission to write to tag
> directories, commits to tag directories get put wherever we're
> distributing zips based on the name of the directory created under the
> tag directory, hopefully with the commit message displayed somewhere.
>
> So we have version 1.2 of the diigo silo committed as
> /plugins/diigo/tags/1.2 and the commit message "Stable against Habari
> 0.7" and that would get pushed as a zip to the diigo post (of content
> type plugin) as the latest stable download, marked with the commit
> message. 1.1 and 1.0 would still be available for download, as well as
> a zip from trunk marked unstable available.
Might I suggest doing it similarly to how Drupal does it?
Branch releases are written as "1.0-0.3". This means "Compatible with
Habari version 1.0, this plugin version is 0.3". That allows there to
be two branches, "1.0-0.3" and "2.0-0.3" that maintain compatibility
with separate versions of Habari.
In this situation it might be worthwhile to do all development in
branches, rather than trunk, and then produce releases in tags that are
similar to the branch names, ala "1.0-0.2". This way, you'd know the
latest release by number and the version of Habari it is compatible with
without having to read the commit logs.
All of this works much better if there is a web-based directory to do
the heavy-lifting.
Owen
Michael C. Harris
>
> >> If I understand correctly, each project (plugin, theme, what-have-you)
> >> would be its own repository. These independent repositories would be
> >> isolated little kingdoms with discrete user permissions and their own
> >> revision numbers.
> >
> > That's not what I thought Owen was saying. I believe he was saying we
> > maintain one repository and split it at the root into a stable tree
> > and an unstable tree.
>
> Oh, right, so the choices are these:
>
> /habari-extras/plugins/foo/trunk
> /habari-extras/plugins/foo/tags/1.0
>
> /habari-extras/trunk/plugins/foo
> /habari-extras/tags/1.0/plugins/foo
>
> I greatly prefer the former.
I fear that I'm misrepresenting Owen's original plan, so I'll stop
saying what I think it was.
> >> The only real drawback I see to having /trunks, /tags and /branches
> >> inside each project directory is that someone could more easily
> >> pollute the entire repository with a bad commit. Maybe I don't know
> >> enough about how Subversion works and this actually isn't a big deal.
> >
> > I'm not sure I follow. If we restrict access to writes to /tags, it's
> > as resilient as it is now.
>
> I assume we're going to open up this repository to more people, but
> simultaneously restrict the number of people who can commit to which
> directories. So every author who wants to use our system can keep
> their plugin or theme in our repository under their control, but they
> would not be able to commit to other people's directories (unless they
> specifically asked -- perhaps we need a ticket tracking system to
> record who requests what access -- if it's just an email, it might get
> lost in the noise).
Yes, I think it's important that people have broad commit access. I
wouldn't want to see people needing to ask for access to individual
repos in -extras.
> Even still, someone could accidentally copy a bunch of stuff into
> their own project directory. I guess it's not that big of a deal
> right now.
Everything can be reverted :)
Michael C. Harris
>
> Michael C. Harris wrote:
> >
> >>> I'm not sure how branching and old releases would work if we split at
> >>> the root though.
> >> If I understand correctly, each project (plugin, theme, what-have-you)
> >> would be its own repository. These independent repositories would be
> >> isolated little kingdoms with discrete user permissions and their own
> >> revision numbers.
> >
> > That's not what I thought Owen was saying. I believe he was saying we
> > maintain one repository and split it at the root into a stable tree
> > and an unstable tree.
>
> Yes, I was suggesting that we split the tree in half, leaving branches
> and trunk on one side, and tags on the other. This would allow us to
> appoint release managers for -extras that could publish plugins to tags
> when they're ready for release.
>
> In contrast, I think that creating separate individual repos for each
> plugin is not a good idea, because it would be too difficult to maintain.
Just to be clear, I wasn't suggesting individual repos ...
> It may yet be possible not to split the repo at the root, but to put the
> tags/branches/trunk directories into each plugin and theme. An
> automated tool to manage permissions would be much more appreciated in
> this case.
... I was suggesting this.
> Also keep in mind the technical barrier for use. It may be easy for us
> to conceive of these repo properties, but lay users will need to operate
> them. For example, a web interface that allows theme developers to
> upload a zip file of their latest revisions would be very useful. An
> interface that allows a release manager to package a new release into
> tags would be useful. The GUI component almost has to exist, and we
> could lay the burden of permission maintenance on that tool to retain a
> similar structure to the one we currently have, as an alternative to
> breaking the root as suggested above. Obviously, there are drawbacks to
> both of these options.
Perhaps starting with the end user requirements and working backwards
would be a better way to work out what should be done. If we do
approach things that way, however, I don't think we can restructure
-extras in time for the submission of the FormUI code, and we
certainly don't want to do this twice. Maybe we should just break all
the plugins with the new FormUI code for now?
That all sounds reasonable.
Chris Meller
Having a pretty advanced GUI available through the plugin directory is obviously going to be necessary for wide adoption, but the first implementation wouldn't have to be much more advanced than, say, Trac's system. Maintain a list of users, groups, and plugins/themes and generate a simple text file. Particularly easy if you just want to throw together a stand-alone app to get things rolling. Integrating with hp.o's ACL system's user database later on would be kick ass, but shouldn't be considered a major goal for the initial release.
As far as the actual permissions given to users, I would definitely recommend that each plugin/theme have its own access groups. Allowing everyone to commit to the entire -extras repo and change any extra is just begging for trouble. The lay-people example is a good one... sooner or later someone who doesn't know how to use SVN *will* F things up, it's just a matter of time. Limiting the scope of the F'ing would be a very good thing.
As for Owen's suggestion of using the Drupal tags structure, I don't see how that's going to achieve much. Does Foo's 1.0-0.3 release support up to 1.0, or 1.0 and above? What if it supports 1.0 to 1.5, but not 0.5 or 2.0? Obviously if there's a 2.0-0.4 release also available it's pretty clear it should be preferred for a 2.0 release over the 1.0 version, but in its absense there's not much indication as to what's meant by the version number.
I would suggest adding min and max versions to the info() method, but I despise the Firefox method of extension compatibility. If nothing has broken with my plugin, I shouldn't have to update it for every new Firefox release just to change a max version number. I would say that, unless we can find a better solution, we should keep version compatibility as meta-information only displayed within the plugin directory, if we keep it at all. I still think having a 'stable' tag in each extra's mini-repo would be the simplest method for providing a reliable yet low-comprehension-cost indicator of what should be grabbed. A more complex system could be adopted later on without breaking that pattern (ie: the stable tag could point to the same contents as the 1.0-0.3 tag).
Breaking existing things with FormUI changes should be its own topic, as it doesn't really have much to do with the repo and more to do with how we want to handle backwards compatibility in general (which is a much larger discussion).
Owen Winkler
>
> As for Owen's suggestion of using the Drupal tags structure, I don't see
> how that's going to achieve much. Does Foo's 1.0-0.3 release support up
> to 1.0, or 1.0 and above? What if it supports 1.0 to 1.5, but not 0.5 or
> 2.0? Obviously if there's a 2.0-0.4 release also available it's pretty
> clear it should be preferred for a 2.0 release over the 1.0 version, but
> in its absense there's not much indication as to what's meant by the
> version number.
Since Habari API changes only happen on certain version number
increments, plugins should be able to rely on the major.minor version
combination for a working plugin for the indicated Habari version.
Keep in mind that as the project grows, it may be important to maintain
older versions of plugins for people who are not yet ready to upgrade.
As such, a "stable" tag alone might not be helpful for that purpose.
Also, a "stable" tag would likely go out of date the instant that we
released a new version of Habari, since the tag wouldn't point at the
version that works with the current release any more unless there was
some automated method of updating it (for which we'd need something like
the branch naming, anyway), or the developer/release manager was on top
of the new release.
That's why I think having explicit branches for compatible versions is
helpful, to explain my suggestion. It would give users a clear place to
look for versions of plugins that were compatible with specific versions
of Habari, and delineate a pattern of release that allows us to choose
the correct version to supply during automation.
Owen
Chris Meller
Also, a "stable" tag would likely go out of date the instant that we
released a new version of Habari, since the tag wouldn't point at the
version that works with the current release any more unless there was
some automated method of updating it (for which we'd need something like
the branch naming, anyway), or the developer/release manager was on top
of the new release.
That's a fair point.
That's why I think having explicit branches for compatible versions is
helpful, to explain my suggestion. It would give users a clear place to
look for versions of plugins that were compatible with specific versions
of Habari, and delineate a pattern of release that allows us to choose
the correct version to supply during automation.
I think we may be making this project larger than it needs to be at this point. How about we re-organize -extras to create a branches, tags, and trunk for every extra, and we can continue to discuss the variety of ways in which we could enhance this structure to allow for things like this?
Chris Meller
Can I get a yay / nay on creating a "mini-repo" branches, tags, trunk structure for each -extra, so that people can branch their extras?
Further discussion on tagging and integration with a plugin directory is welcome, but shouldn't hold up this step any longer.
Scott Merrill
> To keep things moving, since this conversation has died twice already, I'd
> like to limit the scope and put it up to a vote.
>
> Can I get a yay / nay on creating a "mini-repo" branches, tags, trunk
> structure for each -extra, so that people can branch their extras?
+1 for having each -extras project directory contain a /trunk,
/branches, /tags directory structure.
rick c
Rick Cockrum
Chris J. Davis
Rich Bowen
Ali B.
To keep things moving, since this conversation has died twice already, I'd like to limit the scope and put it up to a vote.
Can I get a yay / nay on creating a "mini-repo" branches, tags, trunk structure for each -extra, so that people can branch their extras?
Further discussion on tagging and integration with a plugin directory is welcome, but shouldn't hold up this step any longer.
Ali B / dmondark
http://www.awhitebox.com
Michael C. Harris
> To keep things moving, since this conversation has died twice already,
> I'd like to limit the scope and put it up to a vote.
> Can I get a yay / nay on creating a "mini-repo" branches, tags, trunk
> structure for each -extra, so that people can branch their extras?
> Further discussion on tagging and integration with a plugin directory
> is welcome, but shouldn't hold up this step any longer.
+1.
Raman Ng (tinyau)
To keep things moving, since this conversation has died twice already, I'd like to limit the scope and put it up to a vote.
Can I get a yay / nay on creating a "mini-repo" branches, tags, trunk structure for each -extra, so that people can branch their extras?
Further discussion on tagging and integration with a plugin directory is welcome, but shouldn't hold up this step any longer.
Ali B.
To keep things moving, since this conversation has died twice already, I'd like to limit the scope and put it up to a vote.
Can I get a yay / nay on creating a "mini-repo" branches, tags, trunk structure for each -extra, so that people can branch their extras?
Further discussion on tagging and integration with a plugin directory is welcome, but shouldn't hold up this step any longer.
Michael C. Harris
directly with svn they can no longer be updated with svn up. If you
do, you'll get the new branches, tags, trunks structure instead of the
plugin itself.
To change to trunk, change to the directory of the plugin and run
this (using defensio as an example):
svn switch http://svn.habariproject.org/habari-extras/plugins/defensio/trunk .