Pylons plugins proposal

[Rufus Pollock]

Hi,

At the Open Knowledge Foundation (http://www.okfn.org) we have used
Pylons for several of our projects (http://www.okfn.org/projects/). We
think it is a great framework.

However, there are some standard components which it would be nice to
be able to "plugin" in a standard way -- e.g. users + auth, admin,
settings, comments (more examples plus details below). In many other
frameworks/platforms (including e.g. Django) you do indeed see some
sort of plugin frameworks in which this sort of functionality does
indeed become a plugin of some kind.

Now, I'm well aware of Pylons different and more flexible (and IMO
better) approach compared to e.g. django. And that this architecture
may mitigate against providing any standard "plugin" (e.g. how do we
know which template framework -- mako, genshi, jinja, etc -- a given
pylons user is using ...).

However, I think it is still possible to do something useful along
these lines -- even if a formal plugin framework isn't
possible/required just "packaging" up best-practice(s) *pattern* for
how to do standard stuff it would be very valuable (especially if a
pattern kept up to date!).

Below I've given a list of some examples of the sorts of things that
could be "plugins". What else should be on that list? Is there
interest in having plugins?

Regards,

Rufus Pollock
Open Knowledge Foundation - http://www.okfn.org/


## Pylons plugin proposals

### 1. Users and Authentication - basic plugin to do users and
authentication out of the box

* Openid + Form (email confirmation etc)
* Build on existing plugins (repoze.who, authkit, whatever)
* Tie in to existing model (sqlalchemy) object (or provide demo code
to create it)
* Even if just a demo that *works* it would be useful
* We've now done this ~ 6 times. There are commonalities and even if
this were just a demo it would be useful

### 2. Admin - lightweight admin system

1. Model CRUD - provided an existing solution nicely wrapped up or
just document how to do it and ensure it really works -- we've used
FormAlchemy plenty (and other options) but ensuring something "just
worked" would be useful

2. Settings module - think wordpress settings/options system
* DB Table + web user interface
* Config in db rather than ini file

### 3. Mini-CMS

* Simple content management - post/page table plus editor
* Slave CMS - pull content from an RSS feed from e.g. wordpress or other CMS
* We implemented this ourselves in http://openshakespeare.org

### 4. Commenting

* Commenting is a common functionality across many applications
* A basic plugin that providing commenting functionality either
directly or pased on an existing pluggable service (e.g. disqus) would
be extremely useful

--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To post to this group, send email to pylons-...@googlegroups.com.
To unsubscribe from this group, send email to pylons-discus...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en.

[Mike Orr]

Pylons is intentionally small, so that things that would be plugins in
other frameworks can usually be implemented on top of Pylons. So for
instance, Repoze.who/what is a plugin system for auth^2, and there are
Pylons application templates on PyPI that preconfigure a
Pylons/who/what application for you. Eventually the best of these
templates might get official status and be featured in the docs.
Including them in the Pylons distribution would be more difficult --
we'd be committing to supporting them forever, and the state of the
art in auth, CRUD, etc changes so frequently.

There has been some work to base Pylons 2 on zope.component, which
would allow Pylons components themselves (routing, action calling) to
be pluggable and replaceable. An application could use the same
component system to configure its auth plugins, etc. Another goal of
Pylons 2 is to make applications more nestable, so that they could be
used as inner apps similar to Django applications. So this may help
for plugins, but first we have to get Pylons 1 finished.

If you look on PyPI you'll see the existing Pylons applicaition
templates. You might get together with those authors and set up a
central list of these, and find some way to distingush the best ones
and get rid of the worst ones so that users can see, "This is the best
recommended template for doing X."

--
Mike Orr <slugg...@gmail.com>

[Tim Black]

At least partial examples of 1 & 2 are provided by by TurboGears, though
not as plugins.

Though this is not what you're looking for, I'll mention that I'm
beginning to create a very simple app-specific plugin framework here
https://blueprints.launchpad.net/reformedchurcheslocator/+spec/feeds-plugin-framework.
I've outlined a basic strategy for implementing the framework, but hope
I might be able to glean some insight from other plugin frameworks as I go.

Tim

[Wyatt Baldwin]

I think at the very least a documented method for "blending" Pylons
apps would be useful. If I have an app and I want to add this or that
functionality from some other Pylons app, what's the best way to do
that? How do I pull in the static files and the templates?

In my case, at work, I'm creating a base app from which other apps
extend. There's a bunch of common stuff that I don't want to repeat
per app: base config from INI file, environment setup, shared static
files, shared templates, common error handling, etc.

I've got this pretty much worked out, and I'm feeling pretty good
about it, but I wonder if there's a "best practice" for doing this
sort of thing? It's possible that everyone will need to do custom
setup anyway, but maybe some recipe type documentation would be handy.
I'd be happy to share my base package if anyone's interested.

> Open Knowledge Foundation -http://www.okfn.org/

>
> ## Pylons plugin proposals
>
> ### 1. Users and Authentication - basic plugin to do users and
> authentication out of the box
>
>   * Openid + Form (email confirmation etc)
>   * Build on existing plugins (repoze.who, authkit, whatever)
>   * Tie in to existing model (sqlalchemy) object (or provide demo code
> to create it)
>   * Even if just a demo that *works* it would be useful
>   * We've now done this ~ 6 times. There are commonalities and even if
> this were just a demo it would be useful
>
> ### 2. Admin - lightweight admin system
>
>   1. Model CRUD - provided an existing solution nicely wrapped up or
> just document how to do it and ensure it really works -- we've used
> FormAlchemy plenty (and other options) but ensuring something "just
> worked" would be useful
>
>   2. Settings module - think wordpress settings/options system
>     * DB Table + web user interface
>     * Config in db rather than ini file
>
> ### 3. Mini-CMS
>
>   * Simple content management - post/page table plus editor
>   * Slave CMS - pull content from an RSS feed from e.g. wordpress or other CMS

>     * We implemented this ourselves inhttp://openshakespeare.org

[Ben Bangert]

On May 22, 2010, at 2:41 PM, Wyatt Baldwin wrote:

> I think at the very least a documented method for "blending" Pylons
> apps would be useful. If I have an app and I want to add this or that
> functionality from some other Pylons app, what's the best way to do
> that? How do I pull in the static files and the templates?
>
> In my case, at work, I'm creating a base app from which other apps
> extend. There's a bunch of common stuff that I don't want to repeat
> per app: base config from INI file, environment setup, shared static
> files, shared templates, common error handling, etc.
>
> I've got this pretty much worked out, and I'm feeling pretty good
> about it, but I wonder if there's a "best practice" for doing this
> sort of thing? It's possible that everyone will need to do custom
> setup anyway, but maybe some recipe type documentation would be handy.
> I'd be happy to share my base package if anyone's interested.

Right now, its kind of messy. It's totally possible, but it involves importing bits from here and there in your main app, etc. For example, you could make a package that had some models in it, then just import them in your models/__init__.py, and call the init_model for the other package with your configured engine.

And you could add in additional template paths, by using Mako's multiple template search paths, but yea, no best practices or examples of this. It's also not trivial to get the location of mako templates that are inside another package installed somewhere on the system.

Pylons does support dotted notation now for controllers, so you can hook up a controller like so:

map.connect('/home', controller='homepackage.controllers.home:HomeController', action='index')

One of the first things I want to add for Pylons 1.1, is better capabilities for extension points. While sub-classing seems ok for extending things, it quickly becomes a nightmare in larger apps. So I was planning on utilizing Marco so that we'd have an event system, and a registry where plugins could be registered, and handlers for the plugins could indicate they want to be triggered on an event.

Most of the common things Pylons does when loading an app will trigger events, which should provide lots of extension points.

Cheers,
Ben

[Mike Orr]

On Sat, May 22, 2010 at 3:19 PM, Ben Bangert <b...@groovie.org> wrote:
> One of the first things I want to add for Pylons 1.1, is better capabilities for extension points. While sub-classing seems ok for extending things, it quickly becomes a nightmare in larger apps. So I was planning on utilizing Marco so that we'd have an event system, and a registry where plugins could be registered, and handlers for the plugins could indicate they want to be triggered on an event.

That can be done in 1.1? I thought it would take till Pylons 2.

--
Mike Orr <slugg...@gmail.com>

[Ben Bangert]

On May 22, 2010, at 6:44 PM, Mike Orr wrote:

> That can be done in 1.1? I thought it would take till Pylons 2.

Mmm, well, at this point I'm not sure it'll be done using Marco. But I absolutely want events in for Pylons 1.1 so that its easier to make Pylons extensions. At this point its more a matter of whether the Blinker package might be a good solution for this, or a custom event system that handles inheritance as well (not sure if the blinker system lets you subscribe to everything emitted by a class AND its parent classes).

- Ben

[Rufus Pollock]

On 22 May 2010 21:15, Mike Orr <slugg...@gmail.com> wrote:
> On Sat, May 22, 2010 at 12:12 PM, Rufus Pollock <rufus....@okfn.org> wrote:

[...]

>> Below I've given a list of some examples of the sorts of things that
>> could be "plugins". What else should be on that list? Is there
>> interest in having plugins?
>>
>> Regards,
>>
>> Rufus Pollock
>
> Pylons is intentionally small, so that things that would be plugins in
> other frameworks can usually be implemented on top of Pylons. So for
> instance, Repoze.who/what is a plugin system for auth^2, and there are

Right, we have used repoze.who extensively :)

While this is pretty pluggable, if you want, for example,
user/password/email confirmation you have to plug this together with
the db, controllers etc yourself (sure repoze.who gives you the form
but you heed to plug it together with db, emailing etc).[^1]

Now I'm well aware that there will *never* be a one-size fits all
solution. However, I think there is enough commonality across enough
apps that a small plugin/example solution could be valuable to the
community -- and yes I know there are examples on the wiki (which were
very useful) but they aren't the same as some running code bundled up
as a small package (for one thing it is much easier to keep such
material up to date over time)

[^1]: Looking around for this sort of thing already being done there
is (AFACIT): http://blog.sadphaeton.com/node/39,
http://pypi.python.org/pypi/BlastOff (but does much more) and we
ourselves implemented this as part of http://www.isitopendata.org/
(though not factored out as yet)

> Pylons application templates on PyPI that preconfigure a
> Pylons/who/what application for you. Eventually the best of these
> templates might get official status and be featured in the docs.

Right, that would be great. It seems like templates would be getting
pretty close to plugins if they included, say, sqlalchemy model code.

> Including them in the Pylons distribution would be more difficult --
> we'd be committing to supporting them forever, and the state of the
> art in auth, CRUD, etc changes so frequently.

Sorry if this was not clear in my email: IMO *plugins* should *never*
be in pylons core (as that should be kept lean and mean). Plugins are
extensions of specific functionality However, you might want to have a
set of "semi-official" plugins that had good support from the core
community.

> There has been some work to base Pylons 2 on zope.component, which
> would allow Pylons components themselves (routing, action calling) to
> be pluggable and replaceable.  An application could use the same
> component system to configure its auth plugins, etc. Another goal of
> Pylons 2 is to make applications more nestable, so that they could be
> used as inner apps similar to Django applications. So this may help
> for plugins, but first we have to get Pylons 1 finished.

This sounds a little more orthogonal as its about making "components"
of pylons pluggable rather than about plugins extending pylons core.

[...]

>> ### 4. Commenting
>>
>>  * Commenting is a common functionality across many applications
>>  * A basic plugin that providing commenting functionality either
>> directly or pased on an existing pluggable service (e.g. disqus) would
>> be extremely useful
>
> If you look on PyPI you'll see the existing Pylons applicaition
> templates.  You might get together with those authors and set up a
> central list of these, and find some way to distingush the best ones
> and get rid of the worst ones so that users can see, "This is the best
> recommended template for doing X."

Ok this sounds promising. First step I guess is to start a wiki page
listing wants and existing work and then go from there.

Regards,

Rufus

[Rufus Pollock]

On 22 May 2010 21:25, Tim Black <timb...@gmail.com> wrote:
> At least partial examples of 1 & 2 are provided by by TurboGears, though
> not as plugins.

As I understand it (I had a looked a *bit* before) TG admin is based
on dbsprockets + toscawidgets:

<http://turbogears.org/2.0/docs/main/Extensions/Admin/index.html>

Is there any way for that tgext.admin to be re-usable in pylons?

> Though this is not what you're looking for, I'll mention that I'm
> beginning to create a very simple app-specific plugin framework here
> https://blueprints.launchpad.net/reformedchurcheslocator/+spec/feeds-plugin-framework.
> I've outlined a basic strategy for implementing the framework, but hope
> I might be able to glean some insight from other plugin frameworks as I go.

Interesting. From a quick look at the docs I assume this is to do with
caching (atom/rss?) feeds? If so that at least one other example of a
standard plugin use-case and I we ourselves have implement this kind
of feed caching in http://www.openshakespeare.org/ as a way of pulling
content from a wordpress CMS into the main site (that's how word of
the day is generated for example:
<http://www.openshakespeare.org/word/read/bilbo>)

It also reminds me that another useful plugin would be (AMQP)
messaging/celery integration. Again this is something we have
experience of and could share our current approach.

Regards,

Rufus

--
Open Knowledge Foundation
Promoting Open Knowledge in a Digital Age
http://www.okfn.org/ - http://blog.okfn.org/

[Tim Black]

On 05/23/2010 03:51 AM, Rufus Pollock wrote:
> On 22 May 2010 21:25, Tim Black <timb...@gmail.com> wrote:
>
>> At least partial examples of 1 & 2 are provided by by TurboGears, though
>> not as plugins.
>>
> As I understand it (I had a looked a *bit* before) TG admin is based
> on dbsprockets + toscawidgets:
>
> <http://turbogears.org/2.0/docs/main/Extensions/Admin/index.html>
>

It looks to me like it depends on SQLAlchemy, too.

> Is there any way for that tgext.admin to be re-usable in pylons?
>

I haven't tried, but since TurboGears is built on Pylons, tgext.admin
might just work the same way it works in TurboGears.

>> Though this is not what you're looking for, I'll mention that I'm
>> beginning to create a very simple app-specific plugin framework here
>> https://blueprints.launchpad.net/reformedchurcheslocator/+spec/feeds-plugin-framework.
>> I've outlined a basic strategy for implementing the framework, but hope
>> I might be able to glean some insight from other plugin frameworks as I go.
>>
> Interesting. From a quick look at the docs I assume this is to do with
> caching (atom/rss?) feeds?

Yes, it will cache atom/rss feeds, as well as "feeds" that are just
existing HTML pages that we'll screen-scrape for content.

> If so that at least one other example of a
> standard plugin use-case and I we ourselves have implement this kind
> of feed caching in http://www.openshakespeare.org/ as a way of pulling
> content from a wordpress CMS into the main site (that's how word of
> the day is generated for example:
> <http://www.openshakespeare.org/word/read/bilbo>)
>

Are you using VDM for the feed caching? I note this demo:
http://knowledgeforge.net/okfn/vdm/file/cb5116104817/vdm/sqlalchemy/demo_simple.py.
CKAN (http://knowledgeforge.net/ckan/trac/) has features similar to what
I'm trying to implement (versioning, tagging, groups, recent changes).
I note http://wiki.okfn.org/ckan/doc/schema/--is it possible to make the
following 4 kinds of packages (each with its own distinct database
schema): congregations, people, offices, and roles, then make a
one-to-many relationship where one congregation has many people, and
each person has one or more officers or roles? I need to store data for
church congregations and groups of congregations--data like the church
name, address, latitude & longitude, phone number, website URL, email
address, & names & contact info for its officers (pastor, elders,
deacons) and other roles (clerk of session, treasurer, Sunday School
superintendent).

I'm also finding the ideas in the CKAN documentation useful as I'm
figuring out how to manage our federated church directory data. Where
is the code or the docs on how you "exchange...metadata changesets
between different CKAN instances"? That's going to be a key feature of
the Reformed Churches Locator (RCL) software--one RCL instance will
regularly read in feeds of other RCL instances' data. I'm planning to
use UUIDs to avoid duplicate data, but am finding it tricky to prevent
duplicate data from non-RCL feeds that don't provide a UUID. See
https://blueprints.launchpad.net/reformedchurcheslocator/+spec/versioning for
my initial methods of avoiding duplicates. I think your
duplicate-avoiding code must be in here:
http://knowledgeforge.net/ckan/hg/file/f6c0d4e0950a/ckan/getdata/ ; is
that correct?

> It also reminds me that another useful plugin would be (AMQP)
> messaging/celery integration. Again this is something we have
> experience of and could share our current approach.
>

Thank you. I read through AMQP & celery's docs and can't figure out how
I would use them in RCL yet, but will keep them in mind for when I
finally realize I need them in the future. : )

Tim

[Rufus Pollock]

@Tim: apologies for huge delay in responding on this. Missed this when
first came through and only noticed this now when reading back threads
...

On 24 May 2010 10:05, Tim Black <timb...@gmail.com> wrote:
> On 05/23/2010 03:51 AM, Rufus Pollock wrote:
>> On 22 May 2010 21:25, Tim Black <timb...@gmail.com> wrote:

[...]

>> If so that at least one other example of a
>> standard plugin use-case and I we ourselves have implement this kind
>> of feed caching in http://www.openshakespeare.org/ as a way of pulling
>> content from a wordpress CMS into the main site (that's how word of
>> the day is generated for example:
>> <http://www.openshakespeare.org/word/read/bilbo>)
>>
> Are you using VDM for the feed caching?  I note this demo:
> http://knowledgeforge.net/okfn/vdm/file/cb5116104817/vdm/sqlalchemy/demo_simple.py.

Not particularly, we use VDM for 'revisioning' all our objects so we
can have version-control/wiki-like features (diffs, rollbacks,
changelogs etc)

> CKAN (http://knowledgeforge.net/ckan/trac/) has features similar to what
> I'm trying to implement (versioning, tagging, groups, recent changes).
> I note http://wiki.okfn.org/ckan/doc/schema/--is it possible to make the
> following 4 kinds of packages (each with its own distinct database
> schema):  congregations, people, offices, and roles, then make a
> one-to-many relationship where one congregation has many people, and
> each person has one or more officers or roles?  I need to store data for

Yes, you could easily support that schema and versioning and
recent-changes are supported out of the box by VDM (in essence vdm is
a plugin to sqlalchemy that allows you to revision/version objects and
relations.

[...]

> I'm also finding the ideas in the CKAN documentation useful as I'm
> figuring out how to manage our federated church directory data.  Where
> is the code or the docs on how you "exchange...metadata changesets
> between different CKAN instances"?  That's going to be a key feature of

There's no documentation in the core docs yet but the code itself as a
decent docstring and is reasonably readable:

<http://knowledgeforge.net/ckan/hg/file/7f472e98f5a6/ckan/model/changeset.py>

> the Reformed Churches Locator (RCL) software--one RCL instance will
> regularly read in feeds of other RCL instances' data.  I'm planning to
> use UUIDs to avoid duplicate data, but am finding it tricky to prevent
> duplicate data from non-RCL feeds that don't provide a UUID.  See
> https://blueprints.launchpad.net/reformedchurcheslocator/+spec/versioning for
> my initial methods of avoiding duplicates.  I think your
> duplicate-avoiding code must be in here:
> http://knowledgeforge.net/ckan/hg/file/f6c0d4e0950a/ckan/getdata/ ; is
> that correct?

No it's in that changeset file.

[...]

Rufus