how to do extensions?
Jeff Hammel
I have historically made many plugin-architecture software [*] using
setuptools entry points. I don't fundamentally believe that the
packaging system (i.e. setuptools) is the appropriate place for such
things. However its there and its convenient. I've asked in #python
on freenode a few times and besides "don't use setuptools" and "what
problem are you trying to solve?" I haven't gotten much response.
The problem I am trying to solve is this:
1. I tend to write software with either swappable components (which
backend model do you want; SQL, couch, flat files, etc) or software
that you can hang an ordered list of extensions on an entry point.
These shouldn't be hardcoded and in fact must have a string key for
driving via a CLI or a a configuration file
2. There should be the ability to add functionality (change these
plugins) -- easily -- without having to modify the core software.
To give an example, see
https://github.com/k0s/toolbox/blob/master/toolbox/dispatcher.py#L21 .
There are two model classes keyed on a string. For completely
self-contained software, as it currently is, a dict is fine: string
keys -> plugins. However, for a large class of problems I want to be
able to add extensions without having to modify the upstream software
(including monkey-patching). I hope that's clear. Whether or not its
necessary for this particular example is debatable, though I do have
other cases in the same module (e.g. handlers) that should be
pluggable if the software is going to truly be useful and extensible
software instead of a particular implementation.
I see our own Tarek has done http://pypi.python.org/pypi/extensions/ .
Is this the recommended course of action?
I'd like to find recipes/patterns/standards that can be (easily) used
by the Mozilla community. In other words, this isn't a "how do I solve
this problem?" question, as I can think of a few ways of doing that,
but rather "how do I move towards a plugin software architecture
generally useful to e.g. Mozilla?".
Jeff
[*] I don't necessarily believe that plugin architectures are a
particularly great architecture even at their best. I prefer more a
contract-based approach, such as zope.interface. But such approaches
are often difficult to actualize, and zope.interface isn't wonderful
(or maybe I just haven't mastered it). And plugin architectures are
easy to write, understand, and maintain. So I use them.
Tarek Ziadé
The problem of Plugins is two-fold
1/ how to declare a plugin for an application
2/ how to configure the application so that is uses the plugins
Setuptools' principle, where you declare extensions points (1), is
fine per se, but the way it's used out there (2) is often wrong:
applications tend to loop over entry points and use the ones that
correspond to a group *automatically*
IOW if I create a Python project that exposes an entry point, it will
be implicitly activated in an application that uses these entry
points.
and there will be potential issues:
- what if I want to install application A but not activate one of its
entry point ?
- what if two entries points are conflicting with each others ? which
one overrides the other ?
- ...
The important thing imo is to let the end-user explicitly manage
plugins configuration for an application via a config file, like
Mercurial does.
To resume: Implicitly activated plugins are not a good idea, But
discoverability is still a nice feature to have.
So, here's what I am planning for Packaging:
- Metadata in general will be extensible via X-Fields in setup.cfg, so
one project can define a new field. Let's say "X-Entry-Point".
- The pkgutil module in the stdlib will project discoverabilty for
metadata, thus for entry points
- Applications should use a configuration file to explicitly activate
plugins - we might provide helpers atop pkgutil
But we're not there yet. The goal of the "extensions" project I've
started is to provide a backport of the pkgutil APIs that will also be
backward compatible with today's Setuptools entry points, so people
can use it w/o a setuptools dependency.
In the meantime, what I would highly suggest is to go the Mercurial way:
1/ add in the application a config file with a section for plugins
2/ every plugin installed by a third party project, can be explicit
activated via its fully qualified name
for example:
[extensions]
foo = here.is.the.plugin
where plugin is a class or a module or whatever the app takes, and
here.is.the, the package/module where it lives.
>
>
> I'd like to find recipes/patterns/standards that can be (easily) used
> by the Mozilla community. In other words, this isn't a "how do I solve
> this problem?" question, as I can think of a few ways of doing that,
> but rather "how do I move towards a plugin software architecture
> generally useful to e.g. Mozilla?".
>
> Jeff
>
> [*] I don't necessarily believe that plugin architectures are a
> particularly great architecture even at their best. I prefer more a
> contract-based approach, such as zope.interface. But such approaches
> are often difficult to actualize, and zope.interface isn't wonderful
> (or maybe I just haven't mastered it). And plugin architectures are
> easy to write, understand, and maintain. So I use them.
Yeah agreed. what I did is defined interfaces but without
zope.interface. I use ABCs for this. Some people find this feature
overkill, but I find it quite nice to have to validate plugins.
That's what I do for now in Services, I have a plugin registry that
validates and registers plugins that are explicitly declared. See:
https://hg.mozilla.org/services/server-core/file/ce11a1c69122/services/pluginreg.py
And I control plugins via config files, and define them in my packages. see:
- the interface definition:
https://hg.mozilla.org/services/server-core/file/ce11a1c69122/services/auth/__init__.py#l47
- a concrete plugin :
https://hg.mozilla.org/services/server-core/file/ce11a1c69122/services/auth/sql.py#l79
Notice that like the ZCA, the plugin does not inherit from a base
plugin class, which avoid any class mixin hell. It just have to
implement some methods defined in the interface, and the plugin
registry controls this.
Cheers
Tarek
--
Tarek Ziadé | http://ziade.org
Erik Rose
> And I control plugins via config files, and define them in my packages.
+1 on config files. If your plugins need the concept of ordering, it's basically the only way to do it.
I personally prefer Python as a config file syntax, because why not? You don't end up reinventing the wheel to do factoring or conditionals (like ZCML did), humans don't need to learn yet another format, and you save a language boundary. Non-Python is a good choice if you need non-Python tools (like aptitude) to read the files or if you need to read untrusted config files (like in Packaging).
Erik
bear
I used to prefer Python as the config file of choice but lately i've
been moving towards JSON. It allows the same complexity (if needed)
and the config data is consumable by non-python environments.
--
Bear
be...@xmpp.org (email)
bea...@gmail.com (xmpp, email)
be...@code-bear.com (xmpp, email)
http://code-bear.com/bearlog (weblog)
PGP Fingerprint = 9996 719F 973D B11B E111 D770 9331 E822 40B3 CD29
Tarek Ziadé
> On Apr 25, 2011, at 1:26 PM, Tarek Ziadé wrote:
>
>> And I control plugins via config files, and define them in my packages.
>
> +1 on config files. If your plugins need the concept of ordering, it's basically the only way to do it.
>
On my side, I prefer ini-like files for the sake of simplicity. If
you don't have a strong policy, Python-based configuration end up
complicated and easy to break. And if you start adding conditionals in
your config, something's wrong. IOW the complexity can be in the
application side, where the configuration is read at initialization,
and the configuration file itself can stay a simple, flat ini-file.
Semi-replated: forcing developers to think hard about very simple
options to configure the app is a good practice imho, as it becomes
almost self-documented.
A side benefit is that Ops can tweak ini-like files without risking to
break your app code, as long as the boundaries for every value is
documented and/or checked at initialization. For the same reason, I
also dislike json or yaml: a missing parenthesis can break the file,
whereas a ini-like file is pretty robust on typos etc.
bear
that's exactly why I prefer JSON - if the config is wrong it *should* break IMO.
INI files just have too much "wiggle room". But +1 on the reasons to
avoid Python.
>
> Cheers
> Tarek
>
> --
> Tarek Ziadé | http://ziade.org
>
--
Jeff Hammel
DSL-esque in allowing and promoting what you can do. While of course
you can do just about anything in python, it is a bit magical and
isn't very illustrative. I tend to gravitate towards .ini too,
because its easy and ConfigParser is available on every damn system on
the planet, but more tend to find the right format for the job (.ini
is good if your configuration looks like a nested set of
dicts....which more often than I would think, it does....less so for
deeper hierarchies or other data formats. And then there is the
question...what of bools? What of lists?)
$0.02
Jeff
Jeff Hammel
bear
> Hah! I was just thinking JSON had too much wiggle room :P
hehe - it's wiggle is on the parsing/structure side - not the value
side I guess :)
Erik Rose
> you don't have a strong policy, Python-based configuration end up
> complicated and easy to break. And if you start adding conditionals in
> your config, something's wrong.
To round out the pro/con list, I've seen case after case where attempting to enforce layer separation or layer simplicity with language boundaries backfires. I've seen people embed business logic in Clearsilver templates, and we all know the overengineered mess that peak.config turned into, subsuming (but spelling differently) a whole raft of Python syntax. A good Python config file should look like this:
foo = 'value'
bar = 'other value'
baz = u'ünicøde'
things = [1, 4, 5] # No need to invent a new syntax for a simple structure
Django's settings.py is a good example.
Erik
Tarek Ziadé
> On Mon, Apr 25, 2011 at 17:08, Tarek Ziadé <ziade...@gmail.com> wrote:
>> On Mon, Apr 25, 2011 at 10:52 PM, Erik Rose <er...@mozilla.com> wrote:
>>> On Apr 25, 2011, at 1:26 PM, Tarek Ziadé wrote:
>>>
>>>> And I control plugins via config files, and define them in my packages.
>>>
>>> +1 on config files. If your plugins need the concept of ordering, it's basically the only way to do it.
>>>
>>> I personally prefer Python as a config file syntax, because why not? You don't nd up reinventing the wheel to do factoring or conditionals (like ZCML did), humans don't need to learn yet another format, and you save a language boundary. Non-Python is a good choice if you need non-Python tools (like aptitude) to read the files or if you need to read untrusted config files (like in Packaging).
>>
>> On my side, I prefer ini-like files for the sake of simplicity. If
>> you don't have a strong policy, Python-based configuration end up
>> complicated and easy to break. And if you start adding conditionals in
>> your config, something's wrong. IOW the complexity can be in the
>> application side, where the configuration is read at initialization,
>> and the configuration file itself can stay a simple, flat ini-file.
>> Semi-replated: forcing developers to think hard about very simple
>> options to configure the app is a good practice imho, as it becomes
>> almost self-documented.
>>
>> A side benefit is that Ops can tweak ini-like files without risking to
>> break your app code, as long as the boundaries for every value is
>> documented and/or checked at initialization. For the same reason, I
>> also dislike json or yaml: a missing parenthesis can break the file,
>> whereas a ini-like file is pretty robust on typos etc.
>
> that's exactly why I prefer JSON - if the config is wrong it *should* break IMO.
My statement was from a real production case where we forgot the
ending "}" and it took people some time to find it, because all
variables values looked perfectly fine.
What I meant to say was that JSON has extra syntax-thingies compared
to ini-like file to express the same thing, and the nesting JSON
offers is often not necessary for the configuration.
But that's a bit of bikesheding I guess ;)
Jeff Hammel
YAML! XML! ;)
Tarek Ziadé
> I prefer non-python for config as well, as (IMHO) config should be
> DSL-esque in allowing and promoting what you can do. While of course
> you can do just about anything in python, it is a bit magical and
> isn't very illustrative. I tend to gravitate towards .ini too,
> because its easy and ConfigParser is available on every damn system on
> the planet, but more tend to find the right format for the job (.ini
> is good if your configuration looks like a nested set of
> dicts....which more often than I would think, it does....less so for
> deeper hierarchies or other data formats. And then there is the
> question...what of bools? What of lists?)
>
Here's my attempt on this:
https://wiki.mozilla.org/Services/Sync/Server/GlobalConfFile
implementation:
https://hg.mozilla.org/services/server-core/file/ce11a1c69122/services/config.py
my .2 euros
Cheers
Ian Bicking
* What extensions do you want to enable?
Luckily everyone here agrees on explicitly enabled extensions.
* How do you specify where to find the extension?
I also have become annoyed with entry points, mostly due to the many levels of indirection.
I think it's useful to allow a dotted format, which is imported, but also a file format, where the file is execfile'd. Often extensions are about site-specific customizations or other things that don't warrant a whole package with imports etc. I personally prefer module:object (as opposed to module.object), and path/to/filename.py:object. But syntax is a little bike-sheddy ;) It should go in whatever config file you are using, which means a string is the least common denominator. You could actually import or exec something in settings.py for instance, but even Django people don't actually do that.
* How does the extension activate itself?
There's two basic models: application specific hooks, and monkeypatching.
Application specific hooks would mean you have a specific kind of plugin, and at the right time you call all the plugins of that type.
Monkeypatching means you load the plugin and rely on the plugin to have side effects that do the work. You can make this more pretty by structuring the side effects, e.g., have the plugin call register() functions or something.
There is a kind of hybrid I guess. Django applications are an example -- there are some structured parts of applications, but there are some parts where applications just poke away to add functionality. This is most noticeable to me when a Django application doesn't *do* anything, but just offers a new helper or filter or template tag or something -- clearly it's not an application, it's just relying on Django's eager importing of applications.
Monkeypatching means you can just have a list of plugins. If you have typed plugins you can either specify the type, or you rely on the plugin having metadata attached to it to indicate type. Trac uses attached metadata as an example. This makes really lightweight plugins a little harder, as a simple function is seldom enough. I find the metadata often oppressive (e.g., zope.interface stuff). But there's no reason the metadata couldn't be as simple as an attribute convention, where the attribute uses a string to describe what kind of plugin it is. OTOH, sometimes you want one conceptual plugin to affect many hooks. Maybe just special case lists?
* How is an extension configured?
Often extensions require configuration. Consider authentication -- one auth provider might need a db connection, another might need a filename where the usernames and passwords are stored. Almost any interesting extension requires configuration.
I see two basic models of this: (a) plugins are actually plugin factories that are instantiated with configuration (e.g., like Paste Deploy), and (b) plugins look in some global place for configuration (like Django). Django certainly is the simpler model, where anything can import settings and either look at what is already configured (serendipitously being able to reuse configuration) or require new configuration parameters. Name collision is possible but realistically doesn't happen often.
The two remind me some of the monkeypatch vs hook model. Instantiating with configuration makes the configuration a lot more complicated in a structural sense. INI isn't that great as a result. YAML does a lot better. Python-based configuration in this style starts looking a lot like real Python code.
Both explicit hooks and instantiating with explicit configuration are much easier to test.
----
So I guess that's my taxonomy of options without strong suggestions ;) I go back and forth in my own stuff, so I haven't formed a consistent opinion myself. Maybe just use an industry accepted standard: Spring! http://springpython.webfactional.com/