Implementation Strategies - To plugin or not to plugin....

[Danayel]

As mentioned by Anthony and Nick here :

http://groups.google.com/group/joomla-3pd-extension-standards/browse_thread/thread/618404c01f3f92e9

(it would be great if you two could copy your comments to here)

We should come up with reliable methods of implementation for each of
the standards.

- It encourages other developers to participate if things are just
plug(in) and play
- it reduces the chances of conflicts between versions (lifestream vs
NB example from Nick)
- it will hopefully make it easier to create extensions implementing
the standards, than to create them not implementing it (providing huge
incentives to do it)

It will be impossible to come up with a universal implementation
method as we are dealing with differing technologies - a plugin is
useless for implementing a standard on HTML and CSS classes, but
essential to implementing Javascript loading.

Anthony mentioned that he is concerned about having to get his users
to install another plugin, this is a valid concern.

Nick mentioned he is concerned that will end up with conflicting
versions of the specified plugins on sites. Also a valid concern.

I think both of these could have a technological, or social answer
though.

e.g. Signing up to a spec requires you to ship the latest version at
all times, and we hold off new versions of plugins until 80% of
signees are ready.

One thing that should be noted is that going down this path is almost
definitely going to require all of us to make some changes in how we
build and distribute our software. All of us are doing our own thing
right now, so it will be difficult, if not impossible, to bring that
"behavior soup" together without having to make changes.

The aim however is to make sure the rewards greatly outweigh the cost
of those changes. I feel that this is achievable.

To reply to a few of Nicks, points:

> * *Future-proof*. The exposed API should be immutable, or provide
> backwards compatibility whenever it's updated. We should do it
> like Microsoft did in the transition from Windows 3.1 to Win32s to
> Windows 95 to Windows XP to Windows Vista. By providing a
> backwards compatible API, applications written for Windows 3.1
> were easily ported to Windows Vista with minimal changes (only
> about the parts which had to go away). Diametrically opposite to
> this approach is Joomla! 1.0 to 1.5 to 1.6. Each of those version
> changes completely screwed up our code and we had to make
> extensive changes. IMHO, only the 1.0 to 1.5 upgrade was justified
> for breaking the site. The 1.5 to 1.6 was a blunder. APIs changed
> for no reason and caused a lot of grief. Let's not do that.

Agreed, there should never be changes to the existing API without good
reason. If a feature must be reworked, then we should probably create
an entirely new API rather than rework an old one. That woudl allow
fro progressive upgrades without breaking sites.

Well thought out specs should help alleviate this problem however.

> * *Dependency tracking*. We need a method to track down
> dependencies. For instance, my extension should indicate that it
> depends on jQuery and jQuery UI. If they are not available they
> should be installed, transparently. This is similar to a Linux
> distribution's package management system. I have proposed a
> possible structure for such a component as a Nooku Incubator proposal.

This the third time I have said this to you :-D We should hook up and
flesh out my appcasting RFC, dependency tracking and history tracking
(to facilitate rollbacks) are part of that.

Using appcasted plugins would also help solve some of our version
conflict dillemas, as all extensions could automagically download the
latest version of the required plugins at install time.

> 1. *The Library*. The core of what we are discussing.
> 2. *The Installer*. Because Joomla!'s own installer can't handle
> dependency tracking and roll-back.
> 3. *Adoption*. Just because we build it, it doesn't mean they will
> come. We will need big projects and clubs migrating to our
> solution and eventually having Joomla! itself adopt this solution.
> The former is difficult, but doable. The latter is less likely to
> happen than a snow storm in Sahara, pigs learning to fly or hell
> freezing over.

Regarding point 3, I agree getting 3pds on board is more likely than
the core, unless we can get one of the PLT to join us, but we can
always offer the tech back as a branch or patch.

Alternatively, some of the distros around would be likely to adopt
them.

You are right that just building it doesn't mean people will come,
however if we don't build it, then we are guaranteed that no one will
come. Besides which we have enough people here already to make a good
dent in the market. around 100 extensions and a few hundred templates
between us.

[Danayel]

I'd like to get this worked out as soon as possible as it will be the
tech upon which we build a lot of our standards.

e.g. a universal JQuery version cannot be implemented (except as a
temporary measure) until we work out the best way to reliably install
universal plugins.

Given that a we want the following:

1. It needs to not be down-gradable - i.e. if I install an old version
of an extension that carries an old set of plugins, it shouldn't
install them over the new set.

2. We should be able to force a rollback if necessary (i.e. a fatal
bug is found). note: rollback != downgrade.

I think that we should devise a remote installation procedure (with a
direct download option if remote fails) for any plugins.

Alternately, an option for JS is that we get a CDN account together,
and upload the chosen version as whatever.js, with no version
information so we can replace when we as we decide on a new version.

This CDN option does have a downside that it may break some scripts,
and relies on external sites being up.

The remote installation option does have an issue that if the host is
down or unreachable it becomes a manual process. But we could get
around this by having the files stored in the extension, but when
installed it automatically (or optionally?) checks if there is an
update. An added bonus would be to make sure that it doesn't install,
but still version checks, if an already installed version is higher.

This final option would provide us the best balance of keeping things
up to date, and having reliable installs.

There is a lot more we can do with this, but I think an install script
that takes care of this should be one of the first things we develop
and distribute, then we can add to it later.

If we have a script that takes care of it, then all we need to do is
include that script in our install.php and it should take care of the
rest - Installing the plugin, checking the version, and updating or
suggesting an update if needed (and possible).

Thoughts?

[Nicholas K. Dionysopoulos]

Danyel,

Your concerns seem to be addressed by my proposal for an extension manager. Since it is modelled after the way Linux package managers work, it shouldn't be a problem. For example, if Extension A tries to install Plugin X 1.0 and the site has already installed Plugin X 1.1 then the plugin won't be installed. Also, I have described backing up extension files and tables before applying an upgrade/installation, as well as support for rollbacks if things go sour.

The biggest challenges with this approach are:

• Developers should be convinced to use that system. Much like APT can't know about software you compile n your own, an extensions manager can't know about extensions not monitored by it / installed through it.
• We'd need a centralized registry of extensions, much like Ubuntu's universe/multiverse repositories. This means that we have to devise our own repository manifest format so that users can easily add third party repositories. The upside is that even if the remote repository server is off-line, the package information still exists in the local database.

The biggest downside of an extension manager approach is that on cheap servers which block server-to-server HTTP communications it wouldn't work. This would require developers supporting direct installation without the extension manager and, in that case, all bets are off :(

Best regards,

Nicholas K. Dionysopoulos
Lead Developer, AkeebaBackup.com
Web: www.AkeebaBackup.com
Blog: www.Dionysopoulos.me

[Richie Mortimer]

Might be missing the point here, but is it not just simple to host this in an SVN somewhere / GitHub. and simply package it up along with the extension?

Then we do two things in our install scripts, 

1. check that before installing the plugin the current version installed is not higher than the one we are trying to install

2. if the current version is lower, take a backup of it and store for rollback purposes

I may be over simplifying here, but permitting that we followed the rules that any new version would not knowingly break existing functionality then we would be okay. 

there are tools like phing that can make this packaging up quite simple. 

also i think changing the way the user interacts with Joomla! causes problems, as Anthony mentioned he would be concerned his users have to install another plugin, getting them to install via some other extension manager, is just added complication.

IMO the user shouldn't care or need to know that it exists only that the extension works, the more we can do without user interaction the better. plug and play

of course getting it into Joomla! would be the best option, but its not likely, especially in this concept stage. build it and propose it, who knows...

Richie

[Daniel Chapman]

My preferred suggestion was a mix of both packaged and remote.

The reason why I am not keen on relying on a purely packaged solution is because packages get old if the extension is not updated regularly.