Demo of plugin system

[billf]

I have created a new branch on launchpad under ~billferrett/web2py/
plugins. The purpose is to demo an approach to using plugins to
provide functionality to the web2py core.

I needed an example area to apply the idea to so I have taken sql.py
and moved all the db-engine specific code (and some general SQL code)
replacing it with calls to the selected plugin class.

The plugins are found in plugins/db. There is one "abstract" class
that acts as a template for "db" plugins and contains common
functions. There is one plugin file/class for each db-engine.

On startup, sql.py looks for plugins in the plugins/db folder,
ignoring any containing IS_ABSTRACT=True and asking each plugin to
check that required drivers are present. This information is
formatted so that widget.start() displays it in place of the normal
"Database drivers found".

When a SQLDB is created, the appropriate plugin is selected depending
on the uri. This can be overridden if required. The modified sql.py
then calls to the plugin to perform the necessary functions (get
dialect values, connect, create table, insert, update, delete, etc).

I have moved as much of the actual SQL to the plugins as was
practicable for the demo. All the plugin code is complete but only
the basic functionality for sqlite and mysql has been tested (create
table, insert, update, delete).

As I say, this is intended as a demo of an approach that I think could
be useful. I hope that some of you will have the time to look at it
and I look forward to your comments.

[mdipierro]

Thank you Bill. I will take a look.
This is very interesting and probably we should have something like
this but to clarify, you use the term plugin in a very different
context than I do.
I may be wrong but what you call a plugin is what I would call a
driver. This may be the cause of our apparent different opinion on
some issues.

Massimo

[billf]

I can see where you are coming from - in this case the "plugin" is one
step removed from the python/db driver, e.g. pyodbc, level. I think
of sql.py as providing a level of abstraction and the plugins
implementing it appropriately. As an aside, the db plugin interface of
the demo should ideally be able to integrate the gql.py and memdb.py
or any storage engine as just another type of db.

I think a similar approach could be applied to other areas, for
example, views/templates and authorisation.

With this type of "plugin" the key aspect is that the web2py core
defines a set of hooks (or points where it will call out - or API?)
that a compliant plugin must satisfy. For the demo I have made, each
plugin is a single file but there is no reason why the same approach
couldn't be applied to more complex plugins that resemble the "plugin
app" you have referred to elsewhere where the plugin comprises a
model, controllers, etc.

Obviously, to cater for this latter case a different method of
organising resources is needed but I think it still prefereable to
have the "plugins" separate from the applications folder. I envisage
there could be a lot of debate on that issue.

Anyway, I look forward to your comments.

[mdipierro]

Bill.

I like your approach here (despite the fact that this is nothing to do
with the talk on plugins as I define them).
I am trying to understand how extensive is the work you have done
here.

Do you feel this is good enough that it can replace the current
sql.py?
If not, what is missing?
If yes, it is not easy to replace sql.py for what you have created
without lots and lots of tests.
Do you have a mechanism in place for doing these tests?

I can see this or something like this becoming part of web2py in one
of the next releases.

Massimo


On Jan 26, 2:53 pm, billf <billferr...@blueyonder.co.uk> wrote:

[billf]

My intention is that all the functionality in the sql.py on which the
code was based (only 3 or 4 days ago) is in the demo version (new
sql.py + plugins). Unfortunately, I can't say hand-on-heart that it
could go into trunk without further testing. Hopefully, if the new
sql.py works for one plugin it works for them all.

Re. testing: there are 2 issues

1) having a comprehensive set of tests

2) having the full range of db-engines available (i'm sure other
people can help with this)

Re tests: a small addition would be to test the finding and allocation
of plugins. Other than that, as the functionality is the same as the
current sql.py, any existing tests would be the same. There are a lot
of tests at the end of sql.py - how complete are they?

Assuming we need to create more tests, there are a couple of
observations.
1) I need to get up-to-speed with creating and running tests under
python. I don't know enough yet.

2) In an ideal world, someone other than the coder writes the
tests :-) TBH I don't see that happening

3) In general the same tests should apply to each and every plugin.
We need to identify where, if at all, a different outcome is
acceptable.

4) I could do with help defining what else needs to be tested and
organising tests that so it is easy to find/modify a particular test.
Once that is done it would be easier for others to contribute missing
tests.

Plan of action:
I get up to speed on creating/running tests.
Someone tell me what is missing from existing tests (and I will try to
identify gaps myself).
I structure the tests (it may not be necessary to do anything) and
update branch on launchpad.
People run tests on available db-engines and notify of bugs and
missing tests and provide additional tests.
Iterate over last 2 steps until everyone happy :-)

Hopefully, at the end we have a full set of tests for sql.py.

Bill

[billf]

p.s. in an ideal world, any new test suite (minus the plugin
availability checks) should be able to run against the existing
sql.py.