Plugin API
Maxence Bernard
Alexander and I have recently continued discussing the idea of a
plugin framework.
The discussion has been summarized on a Wiki page:
http://trac.mucommander.com/wiki/PluginAPI
Two separate approaches, dubbed 'Open bar' and 'Guided', are outlined
in the document.
At this point, the 'Guided' approach seems like a sensible choice. But
before moving forward, we'd like to have your input on this.
Could you guys please take a moment and let us know what you think?
Also, feel free to add to the wiki page if you have additional ideas.
Thanks !
Maxence
Johann Schmitz
I have two suggestions:
- Plugin install directory
I would prefer a "system wide" place for the plugins (in the application folder), since i often copy the whole
app dir to another box, most of the time without my profile in ~/.mucommander/. Maybe the two places can be
combined - first scan the applications plugins dir, then the users profile plugin dir.
- Configuration
Whats about a GUI for the configuration of plugins? A new tab in the preferences dialog would be the easiest
way, since all configuration related method (save/load) are located in the base class.
But i would like to sperarate the plugin configuration from the app configuration.
Mybe a new entry in the main menu bar (e.g. "Plugins") could contain the configuration dialogs. Plugins should
provide a configuration dialog extending the configuration base dialog.
Since not all plugins need to be configured, this class name should be optional in the MANIFEST.MF.
--
Johann Schmitz
http://www.j-schmitz.net
Nicolas Rinaudo
I just updated the Wiki page with a quick description of why I believe
the "guided" approach is the wrong way to go: http://trac.mucommander.com/wiki/PluginAPI#Acasefortheopenbarapproach
Any thoughts?
Nicolas
A.Yerenkow
> Hey,
>
> I just updated the Wiki page with a quick description of why I believe
> the "guided" approach is the wrong way to go: http://trac.mucommander.com/wiki/PluginAPI#Acasefortheopenbarapproach
>
> Any thoughts?
>
I read what you wrote.
I really think that our internal code changes shouldn't touch plugins.
Interoperability with plugins should be by protocol, and protocol must
be carefully developed and finalized.
Let's say that Plugins specification 1.0 not provide all possibility to
plugins, but some (like file handling), and should be finalized and
supported no matter how muCommander will grow/evolve.
For example, we have a construction site (building site = mucommander)
and some guy came (plugin), and said: "Hey guys, I can work with pipes
and electricity" (zip/unzip archives).
And we could use him later, because we know what he can do.
Let's say we've got fired some manager, and change some office programs
(internal rework), is we really need after this "forget" what this guy
can, and demand from him to fill some weird new form? (please, use other
method for register self) I think that's not normal. Whatever we do
internally, any new guy could came and say "Hey, I can do that work", in
the same way, not changed from beginning.
When we'll have some working simple plugins, we could elaborate and make
new specification, like 2.0, for example supports for panels/toolbars,
additional actions on extension/mime types etc.
Something like this.
Nicolas Rinaudo
I see your point, but let's take a concrete example.
Imagine that we have to change the contract of the ProtocolProvider
interface: it used to live in com.mucommander.file, but there's a
compelling reason for us to move it to com.mucommander.protocol.
No amount of plumbing will insulate us from the fact that people who
created file protocol plugins will need to adapt and recompile them.
Or we'd need to have thought about that beforehand and created a
wrapper interface for the ProtocolProvider interface, which seems like
a lot of work to duplicate features that are essentially already there.
This seems like a shame, since the file API was designed with plugins
in mind: register a provider and you're done.
Not having this duplicated API doesn't mean that we can't be backward
compatible. If we have to change a method's contract, we can always
keep the old method, deprecate it and make it call the new method with
acceptable defaults.
Let's take the desktop API for example. It's possible to dynamically
register new desktops, but let's imagine that we need to add an
integer parameter to the register(DesktopAdapter) method (priority of
the desktop, for example).
We'd have:
@deprecated
public void register(DesktopAdapter adapter) {
register(adapter, DEFAULT_PRIORITY);
}
public void register(DesktopAdapter adapter, int priority) {
// ...
}
Old plugins would still work, new plugins could take advantage of the
more fine grained options.
Another point is that we intend to split muCommander in various sub-
projects - file API, configuration API... The goal being for other
developers to be able to use our APIs for their code. Doing that
forces us to use the "open" approach on a per-subproject basis anyway.
Moreover, insulating plugin developers from the underlying APIs means
that they won't become familiar with these APIs, or that people that
have used our APIs for other projects need to learn another layer of
complexity before being able to roll out plugins.
The "guided" approach is a good one in principle, but there will come
a point where we need to change some core components which will either
break plugins ("open" approach) or the plugin API ("guided" approach).
Regardless of the case, old plugins will need to be adapted. Why
create more work for us if the end result is the same?
Nicolas
A.Yerenkow
Plugin API would be something like 5-10, max 20 interfaces. I don't
think that such additional layer of tiny interfaces is really
complexity, and that this is hard to maintain...
> The "guided" approach is a good one in principle, but there will come
> a point where we need to change some core components which will either
> break plugins ("open" approach) or the plugin API ("guided" approach).
> Regardless of the case, old plugins will need to be adapted. Why
> create more work for us if the end result is the same?
>
There is no "ideal" way, here we had to choose least evil :)
BTW, you say that there is coming global refactoring (
com.mucommander.file -> com.mucommander.protocol )
We can make file-only PluginApi after that.
Nicolas Rinaudo
Currently there are 1000+ java files in /source.Plugin API would be something like 5-10, max 20 interfaces. I don't
think that such additional layer of tiny interfaces is really
complexity, and that this is hard to maintain...
BTW, you say that there is coming global refactoring (
com.mucommander.file -> com.mucommander.protocol )
We can make file-only PluginApi after that.
Maxence Bernard
I have followed the discussion and given this more thought, and I have
to say Nicolas raised some very valid concerns about the 'Guided
approach'.
The two following requirements render backwards compatibility
impossible to guarantee in practice, no matter what approach we choose:
A) the Plugin API has manipulate some of the core classes
(ProtocolProvider, AbstractFile, MuAction, ...) to harness the
existing APIs (file API, action API, ...).
B) we can't afford to freeze core API as they are not fully mature ;
we want to keep improving them over time, and muCommander to benefit
from those improvements.
=> that means whenever core APIs change (class renamed, package moved,
method deleted), plugins will fail to load.
I think there is no way around it, unless we have proxy classes and
methods for every single core class that we want to expose in the
Plugin API ... which would be a nightmare to develop and maintain, and
wouldn't completely shield us from changes in the underlying core APIs.
If backwards compatibility can't be guaranteed, then the main argument
in favor of the 'Guided approach' is gone. In that case, the 'simpler
is better' principle applies and we should go for the 'Open bar'
approach as it is easier for us to roll out.
With that being said, we should do our best to offer backwards
compatibility ('best effort' mode) :
- limit changes in the core APIs as much as possible
- favor class and method deprecation rather than plain removal
- consider sticking to the same core API versions between major
revisions. For instance, muCommander 0.9.1, 0.9.2, ... 0.9.x would all
use the same file / action / ... API versions. Note that we will have
to complete the extraction of core APIs first, see http://trac.mucommander.com/wiki/ApiSplitting
- Give proper guidelines to plugin developers, and list the APIs
that are safe to use (core APIs)
I think we're making good progress here. Feel free if you have more to
add, the discussion is still open.
Maxence
Arik
Hi All,
As the work on 0.9.1 release is coming to an end, and we plan to defined the next release as a "community release" with plugin APIs as its major feature I would like to continue the discussion about plugin APIs. It's been a long time since this discussion took place and I hope the guys that participated in the discussion are still interesting in this feature. I've read the discussion on the mailing list and the wiki page that was created, and I think that it was a good discussion that can be a basis for further discussion.
I think that first we should define what is the goal of using plugin APIs, what do we expect to achieve by using plugin APIs. it seems to me that the discussion around this feature was mainly focused on backward compatibility, how to ensure that previously developed plugins will still work as we continue to develop muCommander. IMO, backward compatibility is something that we should think about, as we don't want to update the plugins on every release of course, but IMO there are more important things that plugins should provide to muCommander:
1. plugins should make it technically easier to extend the application
1.1 when someone wants to develop an extension he won't have to be exposed to all muCommander's code. he would need to know only the parts in the code that are required for implementing the extension.
1.2 it should be easier to set the development environment for plugins development as not all the settings and external dependencies are required
2. plugins should increase the level of participartion in the project - one of the major advantages of muCommander IMO is that it is written in Java which is a very popular language today, and there are many libraries which are developed in Java, so the potential for extensions by the comumnity is high (and the next release should address this in different areas as well - by using github/gerrit, having tutorial videos on youtube channel/on our website, etc)
2.1 plugins will provide a way to implement extensions without maintainers of muCommander being involved in the process, as the code doesn't need to be reviewed by maintainer since it is not going to be merged to muCommader's source control
2.2 as opposed to code that is integrated within mucommander's source control, plugins don't have to be open sourced
3. plugins will provide a way to add/replace implementations without recompile
3.1 new extensions can be downloaded and loaded into the application without recompile
3.2 built-in components can be replaced with other components
3.2.1 can be used to replace component with different implementation, for example users will be able to replace the rar files extension with other implementation that use different library than junrar
3.2.2 can be used to update libraries. today users need to wait for us to upgrade the external libraries we use, with plugins users will be able to replace plugin with an updated version as long as it is compatible with mucommander's extension protocol. users can sometimes even just update the jar of the external library if it is backward compatible.
4. the API splitting was a great design change in terms of modulatiry and reusability (I've encountered a project that use our commons-file package, I can't remember its name..). enhance the current design with plugins architecture will make our code more modular and if the extensions protocols will be independent enough, the plugins could serve other projects as well.
5. the size of mucommader's jar file increase as we add more external libraries and dependencies to the project. by using a plugins architecture, we can extract some components from mucommander's core and provide them as an optional (as opposed to integrated) plugins. for example, in 0.9.1 we are adding support for vSphere protocol which will probably not be used by the majority of our users, so we can implement it as a plugin that is not integrated in our release of muCommander, but could be downloaded separately.
Few more word regarding the backward compatibility issue: I think that Nicolas had a point with his arguments, and Maxence summerized it well - there won't be ideal solution that will ensure backward compatibility always. but I don't think that it's one of the important goals of plugin design though. it's important to have versions so that when we break the protocol/API between the core part and the plugins, plugins that don't conform the API won't be loaded. if we break such protocol/API, it's ok to expect plugin developers will update the plugins in order to conform the new API. plugin APIs should be relatively stable of course, and we should try to minimized the number of changes in them. if we add functionality to the API, we'll need to decide whether to support plugins that are not updated with default implementation for the added functionality, or reject plugins that do not provide the new funcionality.
Not everything should be developed as a plugin though. I think we should define things which are extensible and can be defined with a clear API for extensions, for example:
1. file systems - like ftp, smb, etc. we already got few contribution in that area, one of them is the vSphere support that was lately integrated - the plugins should provide a panel (for the server dialog) and support for the file operations (ls, copy, remove, etc)
2. compression protocols - like zip, rar, 7zip, etc. such plugins should provide support for packing, unpacking (in the future we can add passwords and so on)
3. desktop types
4. file viewers/editors - like text, image, pdf, etc (viewer/editor).
5. more general plugins that get the paths of the selected files and can operate on them, such plugins should be able to open a new dialog, add menu-item etc
6. TBD ..
This post is already long so I'll cut it here, and I'll wait for your feedback about what has been written so far.
You are more than welcome to share you thoughts and insights.
Next I'll describe what I had on mind regarding the design of the plugin APIs and about OSGI which I checked not very long ago to serve as a basis for our plugin architecture.
Regards,
Arik
Arik
Let's take a step forward by presenting what I have in mind regarding the Plugins API and how it address the issues that were previously raised in this thread and in the wiki (http://trac.mucommander.com/wiki/PluginAPI):
Two approaches were discussed, 'Open bar' and 'Guided'. IMHO the 'Guided' approach is the better one for us. In my previous post I noted the benefits I see in using plugins in mucommander, the guided approach seems to meet them better: the APIs for extensions are better defined which makes it easier to develop extensions, and the better encapsulation will reduce coupling and will produce more reusable components. By comparing the pros and cons of the two approaches in the wiki page, we can also see that the guided approach seems like the better option.
Another argument for choosing the guided approach is OSGI infrastructure. after researching this area lately, OSGI seems to me like a very well-fit tool for this purpose. OSGI has become very popular for creating modular and extensible application. Eclipse are using OSGI for their plugins based architecture in a similar way to what we need in order to achieve our goals from using plugins in mucommander. By using custom class-loaders OSGI let one bundle to expose only a portion that is required for other bundles that want to extend it without changing the visibility of classes & methods, and to one bundle to import only a portion that is required for its operation, with a nice way to handle versioning (for backward and forward compatibility) and for adding and removing bundles at runtime. OSGI encourage you to expose only the portion that is needed to be exposed as in the guided approach, and its infrastructure eliminates some of the drawbacks mentioned regarding the guided approach such as complexity of maintaining an extra layer and backward compatibility issues.
For a detailed paper on how OSGI address backward compatibility issues see: http://www.osgi.org/wiki/uploads/Links/SemanticVersioning.pdf
I think that the API should be split according to the different kind of plugins we'll support, each one will have its own interface(s) and versions. It will make it easier to implement the extensions and for us to load and register them. Let's take for example 'archiver' plugins:
We already have an extensible design as there's the interface 'ArchiveFormatProvider' that each archive implementation should implement and we register them at 'FileFactory'. Now imagine that from commons-file 'bundle' (which is the OSGI term for its components, each plugin and its consumer(s) is a bundle) we export (make it accessible for other bundles) the 'ArchiveFormatProvider' interface with the other classes that 'archiver' plugins will need such as 'AbstractROArchiveFile' and so on. Each plugin as its own implementation of 'ArchiveFormatProvider' which it register as for other bundles as a 'ArchiveFormatProvider' service (another OSGI term). At 'FileFactory' we'll have OSGI's 'ServiceTracker' which will track 'ArchiveFormatProvider' services, and register/unregister them internally as it does today.I think that A. Yerenkow meant to something similar to this by saying that the plugins consumer don't have to know the plugin implementation in his previous post.
'Filesystem' plugins will, in a similar way, register a service that is based on the 'ProtocolProvider' interface we currently have, and (optionally) provide a Panel that will be presented in the 'Connect' dialog with a callback that gets back the input.
We would need to think more about the details like, for example, whether to have the classes in commons-file project that are being exported to the plugins bundles as a different bundle or in the same bundle as the rest of the commons-file classes. But as we can see from those examples, using OSGI we won't need to add intermediate abstract layers, and we could quite easily make things that currently have extensible design (other example, in the mucommander project it could be desktops (DesktopAdapter + DesktopManager), actions (MuAction + ActionManager) and such) to be exposed as APIs for plugins (and the current implementations will need to be changed to be as 'integrated' plugins, i.e plugins that come with the official release).
Regarding the other ideas mentioned in the wiki page, I'll describe how they will be address by this OSGI-based design (in short):
- Plugin files should be jar files. I don't see a reason for using a customized format, and jar file are the standard way in Java
- Plugin install directory should be placed under the application installation folder. We can save per-user data related to the plugins in a separate file under the preferences directory if needed
- Meta-information/Versioning should be placed in a MANIFEST.MF file inside the jar in the standrad OSGI way
- Configuration of the plugins should be handled by the plugin itself. Plugins with configuration can expose a panel with a callback that will be presented in mucommander, and each such plugin will have its own configuration file
- I'll need to think some more about localization..
Feel free to respond and sound your opinion on this,
Arik