Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

plugin / intra process communication system

12 views
Skip to first unread message

Florian Ludwig

unread,
Feb 13, 2010, 4:50:43 AM2/13/10
to pytho...@python.org
Hi,

I'm looking for a module/plugin/intra-process-communication/hook system
for python. Maybe someone here could point me to some project I missed
or might have some good ideas if I end up implementing it myself.

Most systems I have found are "one to many" communications but I would
like "many to many", to clarify what I mean an *example* use-case:

Lets say you program a wiki and like to allow different kind of
authentications. So you create two plugins (for example one for
OpenID and one for Shibboleth).

Some time later you feel like reinventing the wheel again and
you program a blog. Again you like to allow different ways of
authentication and you already wrote plugins for exactly the
same for your wiki, right?

With most systems now some trouble begins - the blog software would need
to have the interface/extention-points/however-you-name-it that the wiki
software defined.

The plugins you wrote probably import them from the wiki module directly
which means your blog would depend on the wiki. (?!) So where to put the
interface/[...] definition that is imported by those plugins? Create a
package/module just containing the interface? This really get
troublesome if different people wrote the wiki, the blog and another
third one a plugin.

Also it will probably break if you try to create a program that includes
(imports and uses) the wiki and the blog at the same time.

While looking for existing solutions I came across several different
implementations and approaches with different advantages and
disadvantages. To not list them all, here one representative
implementation: trac's component architecture [0] as its well but still
short documented, shares several concepts with other implementations and
I failed to see how to solve the problem described above with it.

So, what's your solution?

Thanks for reading my long mail!

Really hoping for an constructive discussion,
Florian

[0] http://trac.edgewall.org/wiki/TracDev/ComponentArchitecture

signature.asc

Paul Kölle

unread,
Feb 14, 2010, 4:16:18 AM2/14/10
to pytho...@python.org
Am 13.02.2010 10:50, schrieb Florian Ludwig:
> Hi,
>
> I'm looking for a module/plugin/intra-process-communication/hook system
> for python. Maybe someone here could point me to some project I missed
> or might have some good ideas if I end up implementing it myself.
>
> Most systems I have found are "one to many" communications but I would
> like "many to many", to clarify what I mean an *example* use-case:
>
> Lets say you program a wiki and like to allow different kind of
> authentications. So you create two plugins (for example one for
> OpenID and one for Shibboleth).
>
> Some time later you feel like reinventing the wheel again and
> you program a blog. Again you like to allow different ways of
> authentication and you already wrote plugins for exactly the
> same for your wiki, right?
>
> With most systems now some trouble begins - the blog software would need
> to have the interface/extention-points/however-you-name-it that the wiki
> software defined.
>
> The plugins you wrote probably import them from the wiki module directly
> which means your blog would depend on the wiki. (?!) So where to put the
> interface/[...] definition that is imported by those plugins? Create a
> package/module just containing the interface? This really get
> troublesome if different people wrote the wiki, the blog and another
> third one a plugin.
If you are talking about code sharing you can move the common code out
of your applications in a seperate namespace. If you follow the model
trac is using you would install a module/package/egg with the basic
functionality of the pluginsystem (i.e. what's in core.py and env.py +
logging and whatever you think is necessary).
All shared code like your auth-plugins would go in a common plugin
directory to which you can refer via PYTHONPATH.

Another common technique is chaining of WSGI middleware..., check out
pythonpaste.org.

Then there is SOA where functionality is shared via RPC/webservices and
wsdl/uddi. But my feeling is this is mostly used in "Enterprise"
applications and is best used in Java/.NET where you already have
libraries doing all the XML stuff.

hth
Paul


Florian Ludwig

unread,
Feb 14, 2010, 7:05:06 AM2/14/10
to Paul Kölle, pytho...@python.org
On Sun, 2010-02-14 at 10:16 +0100, Paul Kölle wrote:
> Am 13.02.2010 10:50, schrieb Florian Ludwig:
> > Hi,
> >
> > I'm looking for a module/plugin/intra-process-communication/hook system
> > for python. Maybe someone here could point me to some project I missed
> > or might have some good ideas if I end up implementing it myself.
> >
> > [...]

> >
> > The plugins you wrote probably import them from the wiki module directly
> > which means your blog would depend on the wiki. (?!) So where to put the
> > interface/[...] definition that is imported by those plugins? Create a
> > package/module just containing the interface? This really get
> > troublesome if different people wrote the wiki, the blog and another
> > third one a plugin.

> If you are talking about code sharing you can move the common code out
> of your applications in a seperate namespace. If you follow the model
> trac is using you would install a module/package/egg with the basic
> functionality of the pluginsystem (i.e. what's in core.py and env.py +
> logging and whatever you think is necessary).

> All shared code like your auth-plugins would go in a common plugin
> directory to which you can refer via PYTHONPATH.

You're right, its about code sharing/reusing - should have said it more
clearly. What I am looking for is the pluginsystem that makes this easy.
Here there problem with the trac (and other plugin systems I've seen)
approach:

You need to define something like:
|
| class IAuthPlugin(Interface): [...]
|
in your blog software.

Now within the open_id_login_plugin:
| from blog import IAuthPlugin
|
| class OpenIdPlugin(object):
| implements(IAuthPlugin)
|

So the OpenIdPlugin is specific for the blog and hardly sharable.
Actually it is but it depends on the blog as it gets imported.

This might be thought of as an implementation detail but its seems
pretty common.


> Another common technique is chaining of WSGI middleware..., check out
> pythonpaste.org.

WSGI middleware just doesn't do it in some situations. Also it only
works out for web-based applications (my example was one but my question
was more general).


Thanks for your answer,
Florian

--
Florian Ludwig <di...@phidev.org>

signature.asc

Diez B. Roggisch

unread,
Feb 14, 2010, 12:47:30 PM2/14/10
to
Am 14.02.10 13:05, schrieb Florian Ludwig:

Why? Any reason you can't define it in a separate package the
blog-software depends on, as well as your wiki?


And then of course, this is not really needed. In Python, behavior
counts, not type-information. So you can get away without any explicit
declared interface. You might chose to not do that, for aestetic
reasons, or better documentation. But you aren't forced.

diez

Florian Ludwig

unread,
Feb 15, 2010, 5:12:23 PM2/15/10
to Diez B. Roggisch, pytho...@python.org
On Sun, 2010-02-14 at 18:47 +0100, Diez B. Roggisch wrote:
> > Here there problem with the trac (and other plugin systems I've
> seen)
> > approach:
> >
> > You need to define something like:
> > |
> > | class IAuthPlugin(Interface): [...]
> > |
> > in your blog software.
>
> Why? Any reason you can't define it in a separate package the
> blog-software depends on, as well as your wiki?

That's actually my point - most plugin systems I've seen, like the one
trac uses, are not encouraging you to do so. Having a module that just
defines an Interface is kind of weird - and in the real world no one is
doing it.

> And then of course, this is not really needed. In Python, behavior
> counts, not type-information. So you can get away without any explicit
> declared interface. You might chose to not do that, for aestetic
> reasons, or better documentation. But you aren't forced.

Actually some plugin-systems in python do force you and they check if
your "implementation" comply with the "interface".


Here is one solution I came up with. Based on the earlier example:


> > Lets say you program a wiki and like to allow different kind of
> > authentications. So you create two plugins (for example one for
> > OpenID and one for Shibboleth).
> >
> > Some time later you feel like reinventing the wheel again and
> > you program a blog. Again you like to allow different ways of
> > authentication and you already wrote plugins for exactly the
> > same for your wiki, right?

auth_openid.py - providing the authentication service
http://dpaste.com/hold/159619/

wiki.py - providing the wiki
http://dpaste.com/hold/159634/

Now putting both together:

> import pbus
>
> import wiki
> import auth_openid
> # or: import auth_shibboleth
>
> pbus.get("wiki").run()

No interface definitions.


What do you think? Any obvious pitfalls (besides reinventing something)?

Please keep in mind that syntax/api is not "done" or anything its just
an concept presentation.

Thanks,

signature.asc

Paul Kölle

unread,
Feb 16, 2010, 10:14:22 AM2/16/10
to pytho...@python.org
Am 15.02.2010 23:12, schrieb Florian Ludwig:
> On Sun, 2010-02-14 at 18:47 +0100, Diez B. Roggisch wrote:
[...]

>> And then of course, this is not really needed. In Python, behavior
>> counts, not type-information. So you can get away without any explicit
>> declared interface. You might chose to not do that, for aestetic
>> reasons, or better documentation. But you aren't forced.
>
> Actually some plugin-systems in python do force you and they check if
> your "implementation" comply with the "interface".
Which is a good thing IMO ;)

> Here is one solution I came up with.

[...]
Can you post working code? It's not clear what "pbus" is and how it works.

cheers
Paul

Diez B. Roggisch

unread,
Feb 16, 2010, 1:20:03 PM2/16/10
to
Am 15.02.10 23:12, schrieb Florian Ludwig:

> On Sun, 2010-02-14 at 18:47 +0100, Diez B. Roggisch wrote:
>>> Here there problem with the trac (and other plugin systems I've
>> seen)
>>> approach:
>>>
>>> You need to define something like:
>>> |
>>> | class IAuthPlugin(Interface): [...]
>>> |
>>> in your blog software.
>>
>> Why? Any reason you can't define it in a separate package the
>> blog-software depends on, as well as your wiki?
>
> That's actually my point - most plugin systems I've seen, like the one
> trac uses, are not encouraging you to do so. Having a module that just
> defines an Interface is kind of weird - and in the real world no one is
> doing it.

Just because nobody doesn't do it doesn't mean it's not desirable. IMHO
the ones using interfaces usually immediatly implement them for some
cases - so you actually get the interface for you authentication
framework, and also implement an OpenID plugin for it. And this forms
one package. Then you can add new packages that implement other things.

repoze.who and what are examples for such design.


>
>> And then of course, this is not really needed. In Python, behavior
>> counts, not type-information. So you can get away without any explicit
>> declared interface. You might chose to not do that, for aestetic
>> reasons, or better documentation. But you aren't forced.
>
> Actually some plugin-systems in python do force you and they check if
> your "implementation" comply with the "interface".

I didn't say nobody does it, I just said it isn't needed to get a
working plugin system. If you don't want to have a separate package
*and* don't want to pull in any unused code, this is pretty much your
only option.

> Here is one solution I came up with. Based on the earlier example:
>>> Lets say you program a wiki and like to allow different kind of
>>> authentications. So you create two plugins (for example one for
>>> OpenID and one for Shibboleth).
>>>
>>> Some time later you feel like reinventing the wheel again and
>>> you program a blog. Again you like to allow different ways of
>>> authentication and you already wrote plugins for exactly the
>>> same for your wiki, right?
>
> auth_openid.py - providing the authentication service
> http://dpaste.com/hold/159619/
>
> wiki.py - providing the wiki
> http://dpaste.com/hold/159634/
>
> Now putting both together:
>
>> import pbus
>>
>> import wiki
>> import auth_openid
>> # or: import auth_shibboleth
>>
>> pbus.get("wiki").run()
>
> No interface definitions.
>
>
> What do you think? Any obvious pitfalls (besides reinventing something)?

I don't know what pbus is supposed to do. Nor how it's laid out on a
python package level.

Diez

Florian Ludwig

unread,
Feb 16, 2010, 2:50:39 PM2/16/10
to Paul Kölle, pytho...@python.org
On Tue, 2010-02-16 at 16:14 +0100, Paul Kölle wrote:

> Am 15.02.2010 23:12, schrieb Florian Ludwig:
> > On Sun, 2010-02-14 at 18:47 +0100, Diez B. Roggisch wrote:
> >> [...]

> >> And then of course, this is not really needed. In Python, behavior
> >> counts, not type-information. So you can get away without any
> >> explicit
> >> declared interface. You might chose to not do that, for aestetic
> >> reasons, or better documentation. But you aren't forced.
> >
> > Actually some plugin-systems in python do force you and they check
> > if your "implementation" comply with the "interface".

> Which is a good thing IMO ;)

Type checking might be good but not pythonic - imo. The problem with
having interfaces needed to be declared (and therefore the possibility
to check) is as I outlined in my first mail:
Where to put those interface descriptions if you want both sides
pluggable? You probably end up depending on other modules/projects just
to import the interfaces.

> > Here is one solution I came up with.

> [...]
> Can you post working code? It's not clear what "pbus" is and how it
> works.

Oh, sorry. The idea is similar to "d-bus" - hence the name. Its "the
communcation system". The code is pretty simple and quick. Please
remember its just to show of the concept not to use it in any production
software or anything.

Instead of using classes as connecting-piece I thought using strings
(maybe path "/plugin/wiki" or "org.example.wiki") might solve my
problem. It might be possible to add optional interface validation
though.


pbus.py - just a simple concept implementation of a framework
http://dpaste.com/hold/159980/

auth_openid.py - providing the authentication service
http://dpaste.com/hold/159619/

wiki.py - providing the wiki
http://dpaste.com/hold/159634/

Now putting both together:

> import pbus
>
> import wiki
> import auth_openid
> # or: import auth_shibboleth
>
> pbus.get("wiki").run()

--
Florian Ludwig <di...@phidev.org>

signature.asc

Florian Ludwig

unread,
Feb 17, 2010, 4:29:37 AM2/17/10
to python-list
On Tue, 2010-02-16 at 19:20 +0100, Diez B. Roggisch wrote:
> Am 15.02.10 23:12, schrieb Florian Ludwig:
> > On Sun, 2010-02-14 at 18:47 +0100, Diez B. Roggisch wrote:
> >>> Here there problem with the trac (and other plugin systems I've
> >> seen)
> >>> approach:
> >>>
> >>> You need to define something like:
> >>> |
> >>> | class IAuthPlugin(Interface): [...]
> >>>
> >>> in your blog software.
> >>
> >> Why? Any reason you can't define it in a separate package the
> >> blog-software depends on, as well as your wiki?
> >
> > That's actually my point - most plugin systems I've seen, like the one
> > trac uses, are not encouraging you to do so. Having a module that just
> > defines an Interface is kind of weird - and in the real world no one is
> > doing it.
>
> Just because nobody doesn't do it doesn't mean it's not desirable.

Good point.

> IMHO the ones using interfaces usually immediatly implement them for some
> cases - so you actually get the interface for you authentication
> framework, and also implement an OpenID plugin for it. And this forms
> one package. Then you can add new packages that implement other things.

So if I want to implement another authentication mechanism but doesn't
use any of the auth-frameworks code you mentioned you still would depend
on it.

> repoze.who and what are examples for such design.

repoze.who is new for me, it might not actually solve my problem here
but it looks interesting for other tasks :) Thanks.


> >> And then of course, this is not really needed. In Python, behavior
> >> counts, not type-information. So you can get away without any explicit
> >> declared interface. You might chose to not do that, for aestetic
> >> reasons, or better documentation. But you aren't forced.
> >
> > Actually some plugin-systems in python do force you and they check if
> > your "implementation" comply with the "interface".
>
> I didn't say nobody does it, I just said it isn't needed to get a
> working plugin system. If you don't want to have a separate package
> *and* don't want to pull in any unused code, this is pretty much your
> only option.

Do you know somebody/some project/system/... who does it?

Pulling unused code I definitely want to avoid so I probably will stay
with this way.


> > [...]


> >
> > What do you think? Any obvious pitfalls (besides reinventing something)?
>
> I don't know what pbus is supposed to do. Nor how it's laid out on a
> python package level.

Its "the plugin system", connecting "service providers" (plugins) and
"service users". In my example the "users" also are providers at the
same time, the service "auth", provides the service "wiki".

Hope that clarifies my intentions a bit.

Thanks,
Florian


PS I removed you (diez) from the TO list as that bounced. Probably
related to the (not existing?) domain nospam.web.de


--
Florian Ludwig <di...@phidev.org>

signature.asc

Diez B. Roggisch

unread,
Feb 17, 2010, 2:46:00 PM2/17/10
to
>> IMHO the ones using interfaces usually immediatly implement them for some
>> cases - so you actually get the interface for you authentication
>> framework, and also implement an OpenID plugin for it. And this forms
>> one package. Then you can add new packages that implement other things.
>
> So if I want to implement another authentication mechanism but doesn't
> use any of the auth-frameworks code you mentioned you still would depend
> on it.

This depends on your definition of depending... (pun intended).

Sure, the code floats around. But it isn't *used* by your application.
Why do you bother so much?

>> I didn't say nobody does it, I just said it isn't needed to get a
>> working plugin system. If you don't want to have a separate package
>> *and* don't want to pull in any unused code, this is pretty much your
>> only option.
>
> Do you know somebody/some project/system/... who does it?
>
> Pulling unused code I definitely want to avoid so I probably will stay
> with this way.

Well, it's simply duck-typing, so you find it all over the place. Out of
my head I only know about e.g. TurboMail, that allows you to specify
plugins via pkg_resources entry points & simply uses them.

> Its "the plugin system", connecting "service providers" (plugins) and
> "service users". In my example the "users" also are providers at the
> same time, the service "auth", provides the service "wiki".

So it's hypothetical?

To summarize my point again: if you want explicit type-checking or
zope-style interface implementation, but don't want any code that
implements this (at least not imported) - you need an explicit package.
If you don't want this, use duck-typing.

>
> PS I removed you (diez) from the TO list as that bounced. Probably
> related to the (not existing?) domain nospam.web.de

I post through NNTP, and the email is obfuscated for the obvious reasons.

0 new messages