Site Components - Requirements

1 view
Skip to first unread message

Patrick Lewis

unread,
May 27, 2008, 4:53:24 AM5/27/08
to TurboGears Trunk
Hello,

In starting work on ticket 1655 [1] (which is about a component
architecture for TG apps) , I've drafted up a requirements document
for discussion [2]. If you have some time and interest, I'd appreciate
any feedback or input that you might have. Thanks.

[1] http://trac.turbogears.org/ticket/1655
[2] http://docs.turbogears.org/2.0/RoughDocs/SiteComponentRequirements

Dean Landolt

unread,
May 28, 2008, 7:02:31 AM5/28/08
to TurboGears Trunk
This is something I've been hoping for for quite a while. Most of this
seems pretty well suited for setuptools, particularly upgrades and
dependencies, and customizations would be as easy as extracting and
modding the installed egg, but obviously that would make rolling in
the latest component changes untenable. It would pretty much force the
solution Mark suggested and just use mercurial to let the user
untangle the mess.

Of course, model changes would still require some sort of migration,
so no matter what there would have to be some sort of update script
provided by the component author anyway -- perhaps this is a decent
place to hook in for updates...

So I guess my question is, should these components be _customized_ or
_extended_? Breaking changes (like an ALTER on the model) will have to
be dealt with no matter how you slice it, but in most cases (template
customization) it would be trivial to override the default. I guess
this would be considered monkeypatching, but it doesn't seem like it
to me.

percious

unread,
May 29, 2008, 6:56:34 PM5/29/08
to TurboGears Trunk
I have in fact been developing "components" for TG for some time now.

DBMechanic could be seen as the first of said components, and probably
one of the more complex systems. I am now working on
tg.ext.silverplate, which is another component, intended to handle
registration and user administration.

Now, my components do not modify the database schema in anyway, and
this is not an easy problem to solve really. The only thing i could
suggest is that components should not be allowed to modify the schema
in any way, but can utilize the general TG2 database session.

The reason modifying the database schema is a hard problem is that you
end up with a mis-match between your model code and your database
schema, which may be ok in some cases. Personally I don't like that
kind of mismatch, which leads us to the idea of generating your model
code, which seems like a great solution. You may find that developers
will be reluctant to release control of their model/__init__.py in
favor of automatically created schema (I am one of those). This is
why I have not pushed hard to make dbmech create and delete columns/
tables/etc. I'm still trying to figure out the best way to handle
this. Personally, I am not a big fan of generated code (with the
exception of true assemblers). Also, I tend to break my model out
into separate files, and the idea of finding which file things are
defined in and modifying them is ludicrous. In fact, the database
schema may be defined in a completely different package which your
component does not have write access to.

All of this boils down to the simplest approach: components requiring
their own database tables/rows/etc should create their own separate
database schema and database. This also frees the developer of said
components to use whatever ORM or other database techniques they
choose. This schema should be configurable, so should the developer
choose to share the database with the general TG database he can do
so.

Here are the kinds of things I have been trying to figure out how to
do, and they are slowly coming out with the wash as silverplate is
being developed.

1) Identity schema information must be shared across the pylons config
object. This gives the extension (im calling them extensions not
components) some ability to use alternative database schema to access
the identity information. One example of this is using makeForm and
makeTable primitives form dbsprockets to generate registration forms
and user tables for management.

2) In order for extensions to maintain the look-and-feel of the
overall application, we have to figure out a way of mapping the
templates to the extensions and making them swappable. I am not sure
on how to do this, but I think I will figure it out eventually. If
anyone has ideas on this subject, please let me know.

Hope some of this helps.

cheers.
-chris

Mark Ramm

unread,
May 30, 2008, 9:37:06 AM5/30/08
to turbogea...@googlegroups.com
> All of this boils down to the simplest approach: components requiring
> their own database tables/rows/etc should create their own separate
> database schema and database. This also frees the developer of said
> components to use whatever ORM or other database techniques they
> choose. This schema should be configurable, so should the developer
> choose to share the database with the general TG database he can do
> so.

This is fine when you want super simple things. But take a look at
django contrib for a whole list of things that would be absolutely
impossible if "apps" couldn't change the model. (A simple example, a
comments app, which allows you to add comments to various content.)

--
Mark Ramm-Christensen
email: mark at compoundthinking dot com
blog: www.compoundthinking.com/blog

percious

unread,
May 30, 2008, 10:04:21 AM5/30/08
to TurboGears Trunk
Perhaps it would work if we made the restriction that extensions
cannot modify existing tables, but may add tables, defining them in a
separate model file?

cheers.
-chris

Jonathan LaCour

unread,
May 30, 2008, 10:58:29 AM5/30/08
to turbogea...@googlegroups.com
Mark Ramm wrote:

> This is fine when you want super simple things. But take a
> look at django contrib for a whole list of things that would be
> absolutely impossible if "apps" couldn't change the model. (A
> simple example, a comments app, which allows you to add comments
> to various content.)

I actually agree with both of you here. Components must be able to
add tables / model objects to your database. That being said, I
don't think that the only way to do this is to hardcode those model
definitions into the components themselves, for several reasons:

1. Its difficult to figure out what the component is creating
without either digging through the source or the documentation.
This creates a disconnect between how you interact with your
model typically, by simply looking at or editing your model
source.

2. Its difficult to change the data definition or plug in existing
model objects that provide the needed behavior without doing
weird things in some configuration file or monkey patching.

I'm tempted to suggest code generation as a solution for this, but I
think that also provides issues when you want to upgrade a component
for example.

Perhaps there could be a balance here, where if you just want the
default behavior of the component, and don't care about making
any tweaks to its behavior, you don't have to have much of the
component's code injected into your project, but once you start to
override things, you can have the code for that particular aspect of
the component copied into your project tree, where you can override
the behavior directly?

--
Jonathan LaCour
http://cleverdevil.org

Patrick Lewis

unread,
May 30, 2008, 12:38:11 PM5/30/08
to TurboGears Trunk
On May 30, 10:58 am, Jonathan LaCour <jonathan-li...@cleverdevil.org>
wrote:
>
> I'm tempted to suggest code generation as a solution for this, but I
> think that also provides issues when you want to upgrade a component
> for example.
>
> Perhaps there could be a balance here, where if you just want the
> default behavior of the component, and don't care about making
> any tweaks to its behavior, you don't have to have much of the
> component's code injected into your project, but once you start to
> override things, you can have the code for that particular aspect of
> the component copied into your project tree, where you can override
> the behavior directly?
>

Other than upgrading, are there other issues with injection /
templating? Is reuse in other projects / apps a concern?
Are there other issues? From the standpoint of seeing what's going on
and making changes though, injection is attractive (IMO).

How would a hybrid approach work? I'm guessing the end developer would
issue a command to spit the component into the project, but what
then? Is it totally in the end-developer's hands? From a component
writer's perspective, it seems like there would be more overhead, but
maybe you could make tools to help out.

Jonathan LaCour

unread,
May 30, 2008, 1:23:06 PM5/30/08
to turbogea...@googlegroups.com
Patrick Lewis wrote:

> Other than upgrading, are there other issues with injection /
> templating? Is reuse in other projects / apps a concern? Are
> there other issues? From the standpoint of seeing what's going on
> and making changes though, injection is attractive (IMO).

Not really. The big issue is that it adds a lot of code to your
project that you didn't write, and makes it difficult to perform
upgrades to the component itself.

> How would a hybrid approach work? I'm guessing the end developer
> would issue a command to spit the component into the project, but
> what then? Is it totally in the end-developer's hands? From a
> component writer's perspective, it seems like there would be more
> overhead, but maybe you could make tools to help out.

Well, it seems to me that you'd issue a command the very first
time that you wanted to use a component, and you could tell it
whether you wanted the entire component's source spit into your
project, or if you'd prefer to use the defaults. Either way,
I think code should be added to your project, but the default
would be placeholders that import from the component, aliasing the
classes/functions from the component itself into your project's
namespace.

If you wanted to later tweak things, you could issue another
command, which could selectively start replacing those placeholders
(model, templates, API, etc) with the actual generated
code/templates, which you could then edit to your heart's content.

This is what I meant by a "hybrid" approach.

Mark Ramm

unread,
May 30, 2008, 3:15:00 PM5/30/08
to turbogea...@googlegroups.com
> If you wanted to later tweak things, you could issue another
> command, which could selectively start replacing those placeholders
> (model, templates, API, etc) with the actual generated
> code/templates, which you could then edit to your heart's content.
>
> This is what I meant by a "hybrid" approach.

Even if this isn't "automated" it's still worth considering.
Providing insturctions on how to cut/paste is fine.

But if we automate it, we can also keep the "origonal" version of the
plugin (pre-modification) somewhere hidden to facilitate a
three-way-merge when you need to update that particular component.

The success of the Django component system seems to be based on the
fact that people are willing to put up with pain later (difficult
component updates) and some hand-editing up front, if they can get
some thing working quickly. And I don't think we should ignore that.

Sometimes the perfect is the enemy of the good.

If we make the future of 3-way merges possible, we don't nessisarily
need to make it work right now in order to be successful.

--Mark

Mark Ramm

unread,
May 30, 2008, 3:19:57 PM5/30/08
to turbogea...@googlegroups.com
> I'm tempted to suggest code generation as a solution for this, but I
> think that also provides issues when you want to upgrade a component
> for example.

The django way is pure code generation. There are 2 problems with
code-generation:

1) putting not-well understood code inside your project can create problems
2) updating generated code gets complicated

The Django people handle 1 by having reasonably simple "apps" which
aren't hard to understand, and which are reasonably well documented.

And they don't handle 2 at all.

I think documentation is actually a "good enough" answer to the first
problem. And if provide some sort of update tool that does a 3 way
merge that helps reduce the update pain, we'll be in very good shape.

So, I'm not opposed to code-generation on principe. Particularly if
we can help ameliorate the problems that have traditionally made
code-generation hard to handle.

--Mark Ramm

Jorge Godoy

unread,
May 31, 2008, 12:12:14 PM5/31/08
to turbogea...@googlegroups.com, Patrick Lewis
Em Friday 30 May 2008 13:38:11 Patrick Lewis escreveu:
>
> Other than upgrading, are there other issues with injection /
> templating? Is reuse in other projects / apps a concern?
> Are there other issues? From the standpoint of seeing what's going on
> and making changes though, injection is attractive (IMO).

I am concerned when apps / extensions start touching my database. I am
usually doing advanced things in there, with lots of automation, triggers,
functions / stored procedures and views.

I want to get the most out of it and if I can't optimize the extension, it is
a huge drawback to me.

The major concern is that when using something generic it has to be designed
to the lowest common denominator, i.e., it won't use all the advanced
features of the database I chose for the project (mainly because of those
advanced features).

> How would a hybrid approach work? I'm guessing the end developer would
> issue a command to spit the component into the project, but what
> then? Is it totally in the end-developer's hands? From a component
> writer's perspective, it seems like there would be more overhead, but
> maybe you could make tools to help out.

I prefer that the project checks, at start time, for the presence of specific
classes on the model. The documentation would supply "cut & past'able" code
for those but it is the developer's responsibility to have those implemented
at the model with the semantics he / she wants.

For example, I don't want to have "UnicodeCol(length=255)" in any of my
classes. It makes no sense to the database I use and I consider that bad
practice. On the other hand, I understand that it makes sense for who
chooses using MySQL due to its other deficiencies.

So, if we have a generic API to define classes and locations for those and
have them checked just once at startup time, then it would solve the problem
and would allow me to optimize what I want at the database or controller
level.

--
Jorge Godoy <jgo...@gmail.com>

Reply all
Reply to author
Forward
0 new messages