Pylons Objectives

[Ben Bangert]

While I mentioned some of my initial objectives in an email with
James, I thought it might help myself and anyone interested to
clarify and refine the purpose of Pylons. I've been doing a bit of
Rails development lately, and carefully reading many docs on Rails,
so I know what aspects of the system are useful and would be
productive to bring to Python web development.

Rather than a clone of Rails, it would be much more useful to take
successful concepts it introduced and utilize them where applicable.
It also makes sense to clearly define the target audience for such a
framework as well since there are already a dozen or so moderately
used frameworks available.

First, Pylons aims to be a mega-framework, styled somewhat similarly
to TurboGears, such that re-use of existing projects is utilized
where appropriate to avoid duplication of effort and to build on the
success of other projects. Second, Pylons aims to fulfill my own
development needs which I've found cross almost exactly with James
Gardner's needs as well which is why we're collaborating on this
project. Now for a leap slightly off left, but highly relevant.

There is a *huge* amount of contract work out there nowadays. Many
projects are small to moderate in size (or can be broken into this
size), and many contract projects might involve very little actual
coding and mainly integration of other webapps. Many clients have
similar needs, or want to utilize common sets of components into a
cohesive whole for their company site. There's also many companies
supplying niche webapp software meant for integration, almost always
using PHP right now as its easy to deploy.

This is the first niche field Pylons is aimed at meeting. By
leveraging Python Paste, setuptools, WSGI, Routes, and Myghty; Pylons
aims to make web development easier for webapps intended to be
deployed into other integrated websites, and deployed in scenarios
where it could be common to run 30+ instances of the same webapp only
with different configurations (blogs, forums, wiki's, shopping carts,
form packs, etc.).

Pylons current re-use utilization:
- Python Paste
- Structure generation, for controllers, unit tests, project
directory structure, etc.
- Testing for Controllers
- setuptools
- Distribution through egg's
- Dynamic discovery for plug-in frameworks
- WSGI
- Middleware for use with other WSGI
- Ease of Deployment
- Myghty
- Revamped dispatch scheme through custom Advanced Resolver
- Advanced and extremely quick templating
- Routes
- Semi-programmatic routing of URL's to controller actions
- URL generation for use in templates

Currently being debated for use in Pylons to meet the requirements of
a 'full-stack' framework:
- SQLObject, SQLAlchemy, python.web ORM
- Mochikit, Dojo, Prototype, Scriptaculous (one of them likely)
- CRUD stuff possibly utilizing...
- TG inspired form-generation (there's several), Aquarium FormUtil

Since re-use is preferred where possible, SQLObject is tempting,
especially as TurboGears is fueling so much development of SQLObject
+FormEncode projects. For now, we will probably wait on determining
which ORM to officially include until we see what TG produces.

The immediate short-term focus for Pylons:
- Complete Test framework for fully testing Controllers, using
breakpoints, console troubleshooting, mock objects, etc.
- Heavy Structure generation for Controllers, unit tests, etc.
- Documentation for development, distribution, deployment, and
integration
- Authentication and Identity code (modeled on Bricks and/or TG
identity decorators)
- Helper functions (styled on Rails helpers when applicable)

Lessons borrowed from Rails:
- Convention over configuration, or more precisely, everything comes
with a default setting. All docs assume the defaults are used.
Advanced users still have the ability to go in and tweak things on
their own.
- Structure Generation is good. Making it easy for the user to follow
the defaults by generating them helps.
- Testing of Controllers and Models

Sample use-cases of Pylons:
- Making a blog that automatically finds and uses plug-ins and themes
installed system-wide
- Easily adding two dozen slightly different configurations of a web
application
- A Login WSGI Middleware application that dynamically finds new
login schemes available

Obviously, many of the features are available merely be leveraging
Paste, WSGI, and setuptools to the max. These are not terribly
complex, but typically poorly understood Python tools. Making their
use easy and obvious for developers will be a challenge and extensive
documentation and examples of each of the 3 use-case apps will be
needed.

That's where we stand currently on Pylons, I'm working on using the
Paste tools for structure generation and controller unit testing, and
James is writing a CMS-type webapp with Pylons as it stands so far to
get a better idea for ways it should expand.

Everyone on this small list has quite a bit of experience with at
least one (but usually more) web frameworks, and has created their
own web framework. As such, I'd be thrilled to hear feedback on this.

Cheers,
Ben

[James Gardner]

Hi Ben,

+1 on all of this. Spot on!

James

[Ian Bicking]

Ben Bangert wrote:
> The immediate short-term focus for Pylons:
> - Complete Test framework for fully testing Controllers, using
> breakpoints, console troubleshooting, mock objects, etc.

Incidentally, I committed something for debugging into Paste yesterday,
and I'm planning on polishing up the UI today. Maybe make a screencast
too. Anyway, you can test it out with a deployment file like:

[server:main]
use = egg:PasteScript#wsgiutils
port = 9999

[filter-app:main]
use = egg:Paste#evalerror
next = filterthis

[app:filterthis]
use = egg:PasteScript#test


Then go to http://localhost:9999/?error=true; click on the [+]'s, and
you can evaluate expressions in the local frames of the tracebacks. You
can't continue, but then continuing isn't as important in a web app,
since you can always just restart. Actually, maybe just adding a
"re-request" button to the top would be good.


--
Ian Bicking | ia...@colorstudy.com | http://blog.ianbicking.org

[Shannon -jj Behrens]

On 11/6/05, Ben Bangert <b...@groovie.org> wrote:
> While I mentioned some of my initial objectives in an email with
> James, I thought it might help myself and anyone interested to
> clarify and refine the purpose of Pylons. I've been doing a bit of
> Rails development lately, and carefully reading many docs on Rails,
> so I know what aspects of the system are useful and would be
> productive to bring to Python web development.
>
> Rather than a clone of Rails, it would be much more useful to take
> successful concepts it introduced and utilize them where applicable.
> It also makes sense to clearly define the target audience for such a
> framework as well since there are already a dozen or so moderately
> used frameworks available.

Have you taken a look at PHP Horde yet? It sounds like it matches
much of this email.

> First, Pylons aims to be a mega-framework, styled somewhat similarly
> to TurboGears,

By the way, what's wrong with TurboGears and Subway, and how do you
envision Pylons will be better?

> such that re-use of existing projects is utilized
> where appropriate to avoid duplication of effort and to build on the
> success of other projects. Second, Pylons aims to fulfill my own
> development needs which I've found cross almost exactly with James
> Gardner's needs as well which is why we're collaborating on this
> project. Now for a leap slightly off left, but highly relevant.
>
> There is a *huge* amount of contract work out there nowadays. Many
> projects are small to moderate in size (or can be broken into this
> size), and many contract projects might involve very little actual
> coding and mainly integration of other webapps. Many clients have
> similar needs, or want to utilize common sets of components into a
> cohesive whole for their company site. There's also many companies
> supplying niche webapp software meant for integration, almost always
> using PHP right now as its easy to deploy.
>
> This is the first niche field Pylons is aimed at meeting. By
> leveraging Python Paste, setuptools, WSGI, Routes, and Myghty; Pylons
> aims to make web development easier for webapps intended to be
> deployed into other integrated websites, and deployed in scenarios
> where it could be common to run 30+ instances of the same webapp only

I still don't understand this. For instance, why would you run 30
instances of a blog application when you can run a single instance
that can have 30 blogs? Why not just run one good collaboration
application that supports all the different forms of callaboration you
want? Why not write one app that supports plugins for the different
forms of collaboration, which is what Horde is, as far as I can tell.

> with different configurations (blogs, forums, wiki's, shopping carts,
> form packs, etc.).
>
> Pylons current re-use utilization:
> - Python Paste
> - Structure generation, for controllers, unit tests, project
> directory structure, etc.
> - Testing for Controllers
> - setuptools
> - Distribution through egg's
> - Dynamic discovery for plug-in frameworks
> - WSGI
> - Middleware for use with other WSGI
> - Ease of Deployment
> - Myghty
> - Revamped dispatch scheme through custom Advanced Resolver
> - Advanced and extremely quick templating
> - Routes
> - Semi-programmatic routing of URL's to controller actions
> - URL generation for use in templates
>
> Currently being debated for use in Pylons to meet the requirements of
> a 'full-stack' framework:
> - SQLObject, SQLAlchemy, python.web ORM
> - Mochikit, Dojo, Prototype, Scriptaculous (one of them likely)

After a lot of reading, research, and a demo application, I recommend
Dojo. Don't be afraid to use more than one JavaScript library.
There's no need to limit yourself.

Happy Hacking!
-jj

--
Hacking is to climbing Mt. Everest as
software engineering is to building a Denny's there.

[Ben Bangert]

On Nov 7, 2005, at 8:56 AM, Shannon -jj Behrens wrote:

> Have you taken a look at PHP Horde yet? It sounds like it matches
> much of this email.

Yep, though I haven't been too impressed with most of the technical
decisions it made, mainly as a result of using PHP as the language.
It's limitations are very obvious throughout Horde. So mainly I'm
looking at other Python projects and Rails for useful ideas and
concepts.

> By the way, what's wrong with TurboGears and Subway, and how do you
> envision Pylons will be better?

I can't stand TAL based template languages, Kid is better, but still
not as full featured as Myghty for templating. This is mainly a
personal opinion though as many people seem to have a strong reaction
either for or against TAL-styled templating.

TurboGears and Subway both utilize CherryPy, and Routes integration
with CherryPy won't be possible without patches to core CherryPy
itself until they turn their resolver methodology "inside out" as
Myghty's is. CherryPy was also lacking the necessary ability to have
multiple webapps in the same process, though a recent patch has
resolved that issue. Version 2.2 or 2.3 of CherryPy aims to turn the
resolver mechanism inside-out allowing people to more fully customize
the dispatch style as Pylons does with Myghty.

There's also a nice set of features that one gets by using a
Controller that is more aware and integrated with the View part of
the framework. For example, in Rails, ActiveRecord can be used
completely independently of everything else. I've used ActiveRecord
in stand-alone Ruby scripts without a problem. The rest of the
framework is much more tightly coupled, which becomes useful when
doing some rendering activities that require the View to be aware of
the Controller to an extent.

I've offered to add Myghty templating to TurboGears, but it has no
chance of being included in the core as Kevin wants to maintain a
single templating language. That's understandable as we have no
intention of supplying other built-in templating choices with Pylons,
it makes things easier to include only one template language.

Several of the issues I cited here are mainly just personal tastes,
and a few are technical limitations of CherryPy (which they are
working on fixed). I'm not sure Pylons will be 'better', but it will
be aimed at a different niche. Neither of those frameworks were
created to target the same niche (which as you mentioned, Horde
fulfills for PHP).

Other differences:
- Testing from the beginning, not added on later (TG has TestGears,
which is pretty cool though)
- Structure generation via PasteScript (TG makes a directory
template, not sure how far Subway goes in this aspect)
- Extensive documentation and example applications (This generally is
an after-thought or comes later for TG, Django, and Subway)
- Pythonic (For example, rather than use CherryPy's .expose
attribute, use the Python convention of _method for private methods)

But really, as I mentioned, we're aiming for a different niche. :)

> I still don't understand this. For instance, why would you run 30
> instances of a blog application when you can run a single instance
> that can have 30 blogs? Why not just run one good collaboration
> application that supports all the different forms of callaboration you
> want? Why not write one app that supports plugins for the different
> forms of collaboration, which is what Horde is, as far as I can tell.

Because sooner or later, the complexity inherent in making massive
universal platforms results in something like Zope. Rather than
building up to such grand schemes, it makes as much sense to build
the opposite way. Using Paste for easy deployment, it becomes easy to
quickly throw up 30 instances of some webapp and its easier to
develop a single blog webapp than a super blogging platform. By not
coupling an individual blog into a massive platform, its easier to
mingle in with other webapps into a cohesive site that is just what
the client ordered.

While my examples were focused on some of the obvious webapps,
consider other ones that aren't so content-editing sensitive.
Shopping carts, membership subscription software, etc. It makes
little or no sense to build super platforms for each of these, and
developing them would be significantly more time intensive.

> After a lot of reading, research, and a demo application, I recommend
> Dojo. Don't be afraid to use more than one JavaScript library.
> There's no need to limit yourself.

Cool, I'll take a look into Dojo some more. It'd be nice to have
effects and some common things wrapped up in helper functions like
how Rails does it. While Rails won't make something as cool as
aquajax, it can handle some nice little AJAX things without burdening
the user with Javascript.

Cheers,
Ben

[Ian Bicking]

Ben Bangert wrote:
>> By the way, what's wrong with TurboGears and Subway, and how do you
>> envision Pylons will be better?
>
>
> I can't stand TAL based template languages, Kid is better, but still
> not as full featured as Myghty for templating. This is mainly a
> personal opinion though as many people seem to have a strong reaction
> either for or against TAL-styled templating.

Well, Subway does use Cheetah, which isn't TALish, and has some features
similar to Myghty. Actually, reading your post on Myghty, several of
the features Myghty has that Cheetah doesn't don't look too hard to
implement -- like implicit inheritance, %init, etc. Maybe even with
just a Cheetah.Template.Template superclass.

> Other differences:
> - Testing from the beginning, not added on later (TG has TestGears,
> which is pretty cool though)
> - Structure generation via PasteScript (TG makes a directory template,
> not sure how far Subway goes in this aspect)

TG is using Paste Script in 0.9, though with a turbogears-admin wrapper.

> - Extensive documentation and example applications (This generally is
> an after-thought or comes later for TG, Django, and Subway)
> - Pythonic (For example, rather than use CherryPy's .expose attribute,
> use the Python convention of _method for private methods)

I actually like .expose; I think forcing internal naming conventions to
express publicness is a little unclean, because it conflates "public to
the rest of the program" and "public to the world". Though another
alternative to .expose is a naming convention, like action_*, which
means "public to the world".

Another option is building "explicitness" into the routes definitions.
I guess it could be similar to tainted strings. When a string is
exactly matched by a route, the target is explicitly made public. When
you get something that simply matches :action, then it's only implicitly
public. Of course, the implicit :action matching could potentially be
turned into a naming convention. E.g., a rule that bound a "method"
variable would mean an explicit match, and a rule that bound an "action"
variable would be interpreted by the resolver as meaning "action_*".
That's probably what I'd be inclined to do.

Personally I'd be interested in a very open sort of framework. I.e.,
one that keeps the parts separate, since I would like (for myself) to
make an easy sort of transition from old code to new. So a route would
identify a controller, which would in turn be a WSGI application
(probably with adaptation of some sort). With a different level of
abstractness, a controller produces variables; these variables can get
rendered in a template, sent via JSON, etc. -- I think that's a clever
part of TurboGears. But the controllers don't really have to care that
much about what kind of template they are sending to. Dynamically
scoped template providing might be interesting here.

Exactly how you brand the combination or individual pieces is another,
separate issue.

>> After a lot of reading, research, and a demo application, I recommend
>> Dojo. Don't be afraid to use more than one JavaScript library.
>> There's no need to limit yourself.
>
>
> Cool, I'll take a look into Dojo some more. It'd be nice to have
> effects and some common things wrapped up in helper functions like how
> Rails does it. While Rails won't make something as cool as aquajax, it
> can handle some nice little AJAX things without burdening the user with
> Javascript.

Personally I think MochiKit is better than Dojo at the things MochiKit
does -- it's well documented, easy to integrate, no fancy tools or build
scripts required. But Dojo seems quite a bit more ambitious. Luckily
they feel similar enough that the conceptual overhead of using both at
once doesn't seem too bad.

--
Ian Bicking / ia...@colorstudy.com / http://blog.ianbicking.org

[Ben Bangert]

On Nov 7, 2005, at 10:18 AM, Ian Bicking wrote:

> Well, Subway does use Cheetah, which isn't TALish, and has some
> features similar to Myghty. Actually, reading your post on Myghty,
> several of the features Myghty has that Cheetah doesn't don't look
> too hard to implement -- like implicit inheritance, %init, etc.
> Maybe even with just a Cheetah.Template.Template superclass.

Yea, that is true. However the features I mentioned on that post were
just the beginning, the component abilities go deep into the entire
system. Re-implementing it all is possible, and some features would
be easier than others, but that doesn't solve the problem of deep
Routes integration with the Controller.

> TG is using Paste Script in 0.9, though with a turbogears-admin
> wrapper.

Excellent, I'm hoping to see some tasks in Paste Script get easier as
more people use them and contribute patches.

> I actually like .expose; I think forcing internal naming
> conventions to express publicness is a little unclean, because it
> conflates "public to the rest of the program" and "public to the
> world". Though another alternative to .expose is a naming
> convention, like action_*, which means "public to the world".

True, though generally in the context of a Controller, public to the
rest of the program and the world are equivalent. I can't really
imagine any reason to stick functions that are Controller private
inside a Controller class unless the intention is also to have it
private to that class (ie, for use only with that Controller). If it
isn't Controller private, and is intended to be called from elsewhere
in the program it typically belongs in a separate library module, or
at least its own class.

Ruby dodges this problem to an extent by having two realms of
'private'. They have 'protected' and 'private', and stick the non-
exposed actions under 'protected', though 'protected' in Ruby is
equivilant to '_prefix' practically speaking.

> Another option is building "explicitness" into the routes
> definitions. I guess it could be similar to tainted strings. When
> a string is exactly matched by a route, the target is explicitly
> made public. When you get something that simply matches :action,
> then it's only implicitly public. Of course, the implicit :action
> matching could potentially be turned into a naming convention.
> E.g., a rule that bound a "method" variable would mean an explicit
> match, and a rule that bound an "action" variable would be
> interpreted by the resolver as meaning "action_*". That's probably
> what I'd be inclined to do.

Hmm, not sure what you mean about the matching. Perhaps an example?

> Personally I'd be interested in a very open sort of framework.
> I.e., one that keeps the parts separate, since I would like (for
> myself) to make an easy sort of transition from old code to new.
> So a route would identify a controller, which would in turn be a
> WSGI application (probably with adaptation of some sort). With a
> different level of abstractness, a controller produces variables;
> these variables can get rendered in a template, sent via JSON, etc.
> -- I think that's a clever part of TurboGears. But the controllers
> don't really have to care that much about what kind of template
> they are sending to. Dynamically scoped template providing might
> be interesting here.

Yea, that is pretty nice the way TG is bundled up. I've been
considering something like that for Pylons as well, and it wouldn't
be too difficult, just need to have Myghty dump the args it gets
instead of rendering them. Probably using the same .execute intercept
I plan on using when testing the Controller to ensure templates were
passed the right vars. It'd be pretty easy to hook-up a system
whereby people could make their own controller filter, like (I'm not
a JSON expert, so I doubt this is right):

m.connect('*url/jsonify', controller='filter', action='json')

class FilterController(BaseController):
def json(self, m, r, **params):
r.content_type = 'text/xml' # Or whatever is needed
args = m.subexec(url, capture_args=True)
m.out(jsonify(args)

There's no capture_args option right now, but I need to add a wrapper
for testing anyways and I could make it easy to trigger like that.

I'm not sure Routes would work right when dealing with dispatching to
entire WSGI apps instead of just a Controller, though I don't see why
a Filter middleware of some sort couldn't use Routes to do that. I
kind of lost you after that though when it comes to the WSGI app,
adaptation, etc.

I agree that a light coupling of the parts is ideal, though as I
mentioned before it becomes more useful to have the template and
controller somewhat aware of each other for certain functionality.
The only bounds Pylons so far has when it comes to tying the
Controller to the template is through the magic temp attribute of the
controller (though it could be used easily enough without a template).

I'm still not totally happy with this solution, but it does make it
easier than passing all the variables into the template directly. As
we're using a stateless, persistent controller things can't be
attached to the Controller instance. Which leaves 2 options to move
variables into the template,
1) Pass them explicitly (as TG does)
2) Have a special thread-safe variable that's cleared on every
request, and available in the template

We're using option #2 right now, though having a magic container as a
controller instance attribute thats available in the template might
cause confusion. James' tutorial (http://pylons.groovie.org/project/
wiki/JamesNotes) does do an excellent job of clarifying its use.
While they could be passed everywhere, that becomes a hassle in some
cases. For example, if you had some auth code that needed to be run
before 30 actions, but not for 2 of the ones in a controller, in
Rails you put inside your class:
before_filter :authorize_user, :except => ['login', 'logout']
# Call authorize_user action, unless its the login or logout method

In Pylons, we're having a __before__ method that can be used to
implement similar functionality.

Now, this ties back to how variables are moved around, because we
need a way to get variables from __before__ into the actual action
method thats called. Having a self.temp that can be used is very
handy in this case, and others where you need to store variables for
later use without relying on trying to manage some dict yourself and
pass it all over the place. So yea... I'm somewhat torn, I know I
want to be able to whisk a magic container around without explicitly
passing it (thread-safe singleton or magic object attached to
controller instance), yet I'm not too keen on the current 'self.temp'
stuff I put in.

If there's some way to still do this, but in a more clear-cut way,
I'd love to hear some ideas. :)

The odd thing about holding the variables intended for the template,
is that I find myself typically setting up variables with the
specific intention of being used in a rendered template. The utility
of being able to return them in JSON is somewhat perplexing in much
of my code since I'm passing complex objects that wouldn't be useful
as JSON. I'd actually code completely separate functions if I needed
specific data back as JSON, just as I'd code special functions for
SOAP/XML-RPC services (though maybe those two are likely to have a
more similar dataset).

- Ben

[Shannon -jj Behrens]

On 11/7/05, Ben Bangert <b...@groovie.org> wrote:
> On Nov 7, 2005, at 8:56 AM, Shannon -jj Behrens wrote:
>
> > Have you taken a look at PHP Horde yet? It sounds like it matches
> > much of this email.
>
> Yep, though I haven't been too impressed with most of the technical
> decisions it made, mainly as a result of using PHP as the language.
> It's limitations are very obvious throughout Horde. So mainly I'm
> looking at other Python projects and Rails for useful ideas and
> concepts.
>
> > By the way, what's wrong with TurboGears and Subway, and how do you
> > envision Pylons will be better?
>
> I can't stand TAL based template languages, Kid is better, but still
> not as full featured as Myghty for templating. This is mainly a
> personal opinion though as many people seem to have a strong reaction
> either for or against TAL-styled templating.

(playfully)
So we have a case of a templating engine looking for a meta project ;)

> TurboGears and Subway both utilize CherryPy, and Routes integration
> with CherryPy won't be possible without patches to core CherryPy
> itself until they turn their resolver methodology "inside out" as
> Myghty's is.

Yes. You must think of CherryPy as an object publisher. Routes
simply doesn't make sense for CherryPy without changing the core of
what CherryPy is.

> CherryPy was also lacking the necessary ability to have
> multiple webapps in the same process, though a recent patch has
> resolved that issue. Version 2.2 or 2.3 of CherryPy aims to turn the
> resolver mechanism inside-out allowing people to more fully customize
> the dispatch style as Pylons does with Myghty.
>
> There's also a nice set of features that one gets by using a
> Controller that is more aware and integrated with the View part of
> the framework. For example, in Rails, ActiveRecord can be used
> completely independently of everything else. I've used ActiveRecord
> in stand-alone Ruby scripts without a problem. The rest of the
> framework is much more tightly coupled, which becomes useful when
> doing some rendering activities that require the View to be aware of
> the Controller to an extent.
>
> I've offered to add Myghty templating to TurboGears, but it has no
> chance of being included in the core as Kevin wants to maintain a
> single templating language.

That is understandable from his point of view. I think it makes sense
for you to make it possible to use Myghty in TurboGears. If I had
free time, I'd want to make it easy to use Myghty in Aquarium.

> That's understandable as we have no
> intention of supplying other built-in templating choices with Pylons,
> it makes things easier to include only one template language.

That is unfortunate. I really do think templating engines are a
matter of taste, so having one's framework dictated by one's choice of
a templating engine seems limited.

> Several of the issues I cited here are mainly just personal tastes,
> and a few are technical limitations of CherryPy (which they are
> working on fixed). I'm not sure Pylons will be 'better', but it will
> be aimed at a different niche. Neither of those frameworks were
> created to target the same niche (which as you mentioned, Horde
> fulfills for PHP).
>
> Other differences:
> - Testing from the beginning, not added on later (TG has TestGears,
> which is pretty cool though)
> - Structure generation via PasteScript (TG makes a directory
> template, not sure how far Subway goes in this aspect)
> - Extensive documentation and example applications (This generally is
> an after-thought or comes later for TG, Django, and Subway)
> - Pythonic (For example, rather than use CherryPy's .expose
> attribute, use the Python convention of _method for private methods)
>
> But really, as I mentioned, we're aiming for a different niche. :)
>
> > I still don't understand this. For instance, why would you run 30
> > instances of a blog application when you can run a single instance
> > that can have 30 blogs? Why not just run one good collaboration
> > application that supports all the different forms of callaboration you
> > want? Why not write one app that supports plugins for the different
> > forms of collaboration, which is what Horde is, as far as I can tell.
>
> Because sooner or later, the complexity inherent in making massive
> universal platforms results in something like Zope.

I'm not buying that argument ;)

> Rather than
> building up to such grand schemes, it makes as much sense to build
> the opposite way. Using Paste for easy deployment, it becomes easy to
> quickly throw up 30 instances of some webapp and its easier to
> develop a single blog webapp than a super blogging platform. By not
> coupling an individual blog into a massive platform, its easier to
> mingle in with other webapps into a cohesive site that is just what
> the client ordered.

While I respect your opinion, your desire to have 30 instances of the
same Web app reminds me of Ian's (hi, Ian!) obsession for middleware
;) I'm smart enough to know that I don't understand everything, and
I'm sure you have your reasons ;)

> While my examples were focused on some of the obvious webapps,
> consider other ones that aren't so content-editing sensitive.
> Shopping carts, membership subscription software, etc. It makes
> little or no sense to build super platforms for each of these, and
> developing them would be significantly more time intensive.

Given.

> > After a lot of reading, research, and a demo application, I recommend
> > Dojo. Don't be afraid to use more than one JavaScript library.
> > There's no need to limit yourself.
>
> Cool, I'll take a look into Dojo some more. It'd be nice to have
> effects and some common things wrapped up in helper functions like
> how Rails does it. While Rails won't make something as cool as
> aquajax, it can handle some nice little AJAX things without burdening
> the user with Javascript.

[Shannon -jj Behrens]

> Personally I think MochiKit is better than Dojo at the things MochiKit
> does -- it's well documented, easy to integrate, no fancy tools or build
> scripts required. But Dojo seems quite a bit more ambitious. Luckily
> they feel similar enough that the conceptual overhead of using both at
> once doesn't seem too bad.

Yes, I agree with you completely. Personally, Dojo had a bunch of
things that I needed that MochiKit didn't when I wrote aquajax.com.

Best Regards,

[Ian Bicking]

Ben Bangert wrote:
>
> On Nov 7, 2005, at 10:18 AM, Ian Bicking wrote:
>
>> Well, Subway does use Cheetah, which isn't TALish, and has some
>> features similar to Myghty. Actually, reading your post on Myghty,
>> several of the features Myghty has that Cheetah doesn't don't look
>> too hard to implement -- like implicit inheritance, %init, etc.
>> Maybe even with just a Cheetah.Template.Template superclass.
>
>
> Yea, that is true. However the features I mentioned on that post were
> just the beginning, the component abilities go deep into the entire
> system. Re-implementing it all is possible, and some features would be
> easier than others, but that doesn't solve the problem of deep Routes
> integration with the Controller.

I'm not necessarily saying that you should, but there is certainly more
overlap between Cheetah and Myghty than Kid and Myghty, and that relates
to how distinct Pylons niche is.

>> Another option is building "explicitness" into the routes
>> definitions. I guess it could be similar to tainted strings. When a
>> string is exactly matched by a route, the target is explicitly made
>> public. When you get something that simply matches :action, then
>> it's only implicitly public. Of course, the implicit :action
>> matching could potentially be turned into a naming convention. E.g.,
>> a rule that bound a "method" variable would mean an explicit match,
>> and a rule that bound an "action" variable would be interpreted by
>> the resolver as meaning "action_*". That's probably what I'd be
>> inclined to do.
>
>
> Hmm, not sure what you mean about the matching. Perhaps an example?

Well, I'm thinking like:

m.connect('article/view/:id', controller='article.Article',
method='show_article')
m.connect(':controller/:action')

In the first one, article.Article.show_article will be used, with no
access controls, because by creating this explicit route you're
implicitly giving access. This is indicated by the use of "method" (by
some convention). In the second, you are not implicitly giving access
and some explicit acesss (either naming convention or attribute) is
required.

That's the way just using what is there now. Potentially Routes could
include some other data that indicated what strings in the mapping came
from a pattern (like 'show_article'), and what strings came from the
request itself (like action for the second rule). This would be more
like string taintedness, but it would be up to the code that matches
dictionaries to objects to make use of it explicitly.

Though, heck, you could also *always* require explicit access for all
things but __call__ or some default action, and simply omit action from
explicit patterns and use fully-qualified controllers instead. However,
that would require some traversal algorithm, because you probably can't
point deep into objects that don't yet exist (e.g., they need to be
instantiated).

Something pretty much like that. text/x-json or somesuch.

> There's no capture_args option right now, but I need to add a wrapper
> for testing anyways and I could make it easy to trigger like that.
>
> I'm not sure Routes would work right when dealing with dispatching to
> entire WSGI apps instead of just a Controller, though I don't see why a
> Filter middleware of some sort couldn't use Routes to do that. I kind
> of lost you after that though when it comes to the WSGI app,
> adaptation, etc.

Well, mostly that you're controller is just Some Object. Then there's
some routine that converts objects (based on type, context,
configuration, whatever) into WSGI applications. For something that
inherits from paste.webkit.wkservlet.Servlet, you simply call it with no
arguments and you get a WSGI application. For Pylon controllers you
might pass it to some code that wraps controllers.

Other variables that routes captured would become part of the request,
and would be handled by the wrapper in whatever way it thinks is best.

> I agree that a light coupling of the parts is ideal, though as I
> mentioned before it becomes more useful to have the template and
> controller somewhat aware of each other for certain functionality. The
> only bounds Pylons so far has when it comes to tying the Controller to
> the template is through the magic temp attribute of the controller
> (though it could be used easily enough without a template).
>
> I'm still not totally happy with this solution, but it does make it
> easier than passing all the variables into the template directly. As
> we're using a stateless, persistent controller things can't be attached
> to the Controller instance. Which leaves 2 options to move variables
> into the template,
> 1) Pass them explicitly (as TG does)
> 2) Have a special thread-safe variable that's cleared on every request,
> and available in the template

In my code (with ZPTKit) I do something like 2, except because servlets
are instantiated for every request you can assign things like this to
"self" and not use thread-local objects. So I do things like
"self.options.object1 = foo" in my controller methods. I find the
one-instance-per-request model to be very convenient personally. I
think sometimes that's not used for performance reasons, but that's a
false optimization as the instantiation overhead is minimal.

> The odd thing about holding the variables intended for the template, is
> that I find myself typically setting up variables with the specific
> intention of being used in a rendered template. The utility of being
> able to return them in JSON is somewhat perplexing in much of my code
> since I'm passing complex objects that wouldn't be useful as JSON. I'd
> actually code completely separate functions if I needed specific data
> back as JSON, just as I'd code special functions for SOAP/XML-RPC
> services (though maybe those two are likely to have a more similar
> dataset).

I think that's transitional, as eventually you'll start tweaking the
output into two separate bits of code. But I think it can be a nice
starting place. It's a little reminiscent of how Rails makes use of
partials for Ajax stuff, but exposing the internal factoring of the
templates.

This also reminds me a bit of some ideas for RESTish URLs. People want
to be explicit about mapping HTTP methods into the URL resolution, e.g.,
article_GET vs article_POST or whatever. When this is enforced (like in
SimpleHTTPServer) it is obnoxious, because it requires me to dispatch on
HTTP method at a point when I might not have sufficient knowledge to do
so. But if a GET request first looked for action_GET(), then action()
that would be ideal. And maybe action_JSON() could fit in there too
(and action_XMLRPC and whatever else).

[Ian Bicking]

Shannon -jj Behrens wrote:
>>TurboGears and Subway both utilize CherryPy, and Routes integration
>>with CherryPy won't be possible without patches to core CherryPy
>>itself until they turn their resolver methodology "inside out" as
>>Myghty's is.
>
>
> Yes. You must think of CherryPy as an object publisher. Routes
> simply doesn't make sense for CherryPy without changing the core of
> what CherryPy is.

However, it seems possible that Routes could implement CherryPy object
publishing. That's not the complete CherryPy environment, but it's
within spitting distance.

>>That's understandable as we have no
>>intention of supplying other built-in templating choices with Pylons,
>>it makes things easier to include only one template language.
>
>
> That is unfortunate. I really do think templating engines are a
> matter of taste, so having one's framework dictated by one's choice of
> a templating engine seems limited.

I think it's important to keep separate track of branding and
documentation choices, and actual technical choices. "Pylons" may
choose Myghty; but the things it's built on don't have to choose, and
the things it's built on could (should) be most of the implementation.
That's when Pylons becomes more of a brand than software.

>>Rather than
>>building up to such grand schemes, it makes as much sense to build
>>the opposite way. Using Paste for easy deployment, it becomes easy to
>>quickly throw up 30 instances of some webapp and its easier to
>>develop a single blog webapp than a super blogging platform. By not
>>coupling an individual blog into a massive platform, its easier to
>>mingle in with other webapps into a cohesive site that is just what
>>the client ordered.
>
>
> While I respect your opinion, your desire to have 30 instances of the
> same Web app reminds me of Ian's (hi, Ian!) obsession for middleware
> ;) I'm smart enough to know that I don't understand everything, and
> I'm sure you have your reasons ;)

From what I know of what you are doing, I think you are coming at web
development from a different requirements than Ben or me (and I think
the two of us have similar requirements). If you are building a primary
product which you control completely -- which I assume is what you do --
you want to tweak it out as much as possible, customize everything and
make sure it's consistent.

When you are supporting a variety of applications for a variety of
clients, a little sloppiness in terms of consistency is okay. At least
it is to me, if it means that I can do something faster and better, or
use someone else's faster better application. If it means that one
application uses a "error" CSS class for bad fields, and another uses an
"error-field" class, I'm not going to worry about it.

And there's a certain RESTian elegance to lots of applications. Because
when you have that, you have to develop good ways of interaction. If
your shopping cart is separate from your store itself, you have to come
up with good formalized communication channels. There's some good
architectural reasons for that. It's also a lot easier to develop in
parrallel, and based on different schedules, especially given multiple
clients with different needs, some of which may require complete forks
of pieces.

[Shannon -jj Behrens]

> From what I know of what you are doing, I think you are coming at web
> development from a different requirements than Ben or me (and I think
> the two of us have similar requirements). If you are building a primary
> product which you control completely -- which I assume is what you do --
> you want to tweak it out as much as possible, customize everything and
> make sure it's consistent.
>
> When you are supporting a variety of applications for a variety of
> clients, a little sloppiness in terms of consistency is okay. At least
> it is to me, if it means that I can do something faster and better, or
> use someone else's faster better application. If it means that one
> application uses a "error" CSS class for bad fields, and another uses an
> "error-field" class, I'm not going to worry about it.
>
> And there's a certain RESTian elegance to lots of applications. Because
> when you have that, you have to develop good ways of interaction. If
> your shopping cart is separate from your store itself, you have to come
> up with good formalized communication channels. There's some good
> architectural reasons for that. It's also a lot easier to develop in
> parrallel, and based on different schedules, especially given multiple
> clients with different needs, some of which may require complete forks
> of pieces.

Well, I 'spose so. It's definitely interesting how different our needs are.

[Ben Bangert]

On Nov 7, 2005, at 1:12 PM, Shannon -jj Behrens wrote:

(playfully)

So we have a case of a templating engine looking for a meta project ;)

No, we have the case of a toolset and utilities that don't belong inside Myghty, or Paste, or Routes. So the only logical conclusion is to put the bundle out there on its own, and make it more pleasant to use. Myghty is more than a templating engine, though the way it leaves how far it goes open to the user causes confusion amongst those that haven't used it (and even some that do).

Yes.  You must think of CherryPy as an object publisher.  Routes

simply doesn't make sense for CherryPy without changing the core of

what CherryPy is.

CherryPy is more than just an object publisher (though they try not to go too far), though making resources available is part of what it does. Their own desire to make it easier to customize the dispatch process indicates to me that they are interested in allowing Routes integration (one user already did it through CherryPy's default mechanism, which works pretty well).

That is unfortunate.  I really do think templating engines are a

matter of taste, so having one's framework dictated by one's choice of

a templating engine seems limited.

You could return a Cheetah template if you wished from Pylons rather easily. We just don't plan on packaging Cheetah with it, nor will we focus our docs on it. Documentation is important, and trying to document the same example with 3 different template languages would be rather unpleasant and time consuming.

Please note from the original email I posted:

Lessons borrowed from Rails:

- Convention over configuration, or more precisely, everything comes with a default setting. All docs assume the defaults are used. Advanced users still have the ability to go in and tweak things on their own.

The default is rendering with Myghty, and thats what will be documented, just as TurboGears does with Kid. We're taking steps to ensure flexibility for advanced users to change tweak it as much as they want.

Pylons will actually come with many hooks and methods to change its behavior, much of this building on easy Myghty made it to customize. Since Routes is plugged in at the web developers side when they create a new project template, an advanced user could go in and not use it at all, or replace it with a totally different dispatch scheme. There's lots of flexibility around every corner, we're just using defaults to get people started faster, keep the docs clean, and give new users a common starting point.

While I respect your opinion, your desire to have 30 instances of the

same Web app reminds me of Ian's (hi, Ian!) obsession for middleware

;)  I'm smart enough to know that I don't understand everything, and

I'm sure you have your reasons ;)

Yea, I do (have my reasons that is). It really comes from the very different fields of work we're in. Building public-facing websites and/or websites for clients, that ask for software that typically overlaps, but not in enough ways it makes sense to build an uber-system, is the main field Pylons is aimed at. Though this field also overlaps some with "plug-in" webapps that ISP's and hosting providers would like to have for their users.

Middleware makes an incredible amount of sense for some things many people waste time re-implementing in every webapp they create. Using an apache module to do something before handing off to a webapp is a form of middleware too (authentication typically).

There's a few big reasons why uber-apps just don't work for contractors, mainly that different clients have different sites, and even if they're on the same hosted server you can't get away with installing one uber-app for them all. They really do need their own instance of the webapp.

Cheers,

Ben

[Ben Bangert]

As this mainly covers dispatch style, I've changed the subject to
cover that.

On Nov 7, 2005, at 1:44 PM, Ian Bicking wrote:

> I'm not necessarily saying that you should, but there is certainly
> more overlap between Cheetah and Myghty than Kid and Myghty, and
> that relates to how distinct Pylons niche is.

True, I could see having a little doc in the How-To's for Pylons, How-
To use Cheetah templates. Though it isn't the default and not used in
the docs, it'd be handy for the Cheetah-addicts that like every other
aspect of Pylons.

> m.connect('article/view/:id', controller='article.Article',
> method='show_article')
> m.connect(':controller/:action')
> In the first one, article.Article.show_article will be used, with
> no access controls, because by creating this explicit route you're
> implicitly giving access. This is indicated by the use of
> "method" (by some convention). In the second, you are not
> implicitly giving access and some explicit acesss (either naming
> convention or attribute) is required.

Ok, now I see what you're referring to.

> That's the way just using what is there now. Potentially Routes
> could include some other data that indicated what strings in the
> mapping came from a pattern (like 'show_article'), and what strings
> came from the request itself (like action for the second rule).
> This would be more like string taintedness, but it would be up to
> the code that matches dictionaries to objects to make use of it
> explicitly.

That's actually already available since the Route that the Mapper
locates has a defaults attribute that holds a dict of the 'defaults'
for the route. Though to get at that info from a Controller I'd need
to expose the matched Route object to it. I'm not entirely sure what
the user would do with it though, since they can ensure variables
meet their req's by using the requirements keyword on a m.connect.

> In my code (with ZPTKit) I do something like 2, except because
> servlets are instantiated for every request you can assign things
> like this to "self" and not use thread-local objects. So I do
> things like "self.options.object1 = foo" in my controller methods.
> I find the one-instance-per-request model to be very convenient
> personally. I think sometimes that's not used for performance
> reasons, but that's a false optimization as the instantiation
> overhead is minimal.

It's very tempting to switch to a new object for every request, I've
played with that on my own as its a very small change in Pylons to
use that dispatch style. I'll benchmark it and see if there's any
meaningful difference in performance or ram usage, not having to
worry about assigning other things to self is useful. On the other
hand, it can be useful to attach persistent things to a persistent
Controller, how do you deal with variables and objects that should be
kept persistent between requests?

> This also reminds me a bit of some ideas for RESTish URLs. People
> want to be explicit about mapping HTTP methods into the URL
> resolution, e.g., article_GET vs article_POST or whatever. When
> this is enforced (like in SimpleHTTPServer) it is obnoxious,
> because it requires me to dispatch on HTTP method at a point when I
> might not have sufficient knowledge to do so. But if a GET request
> first looked for action_GET(), then action() that would be ideal.
> And maybe action_JSON() could fit in there too (and action_XMLRPC
> and whatever else).

I really like that idea, and it makes it easy to have functionality
that will only work under specific HTTP methods or even as you
mentioned, XML-RPC. It keeps the methods together under the class
that deals with them, and by having a naming scheme for them should
be easy to scan controllers for methods to put together mappings of
all the JSON / XMLRPC / SOAP calls supported.

I'm still under the impression that when a class is acting as a
Controller, private method means private to the world and to the
whole program. In which case, using the _ prefix makes sense, as I
can't see why any other part of your program should be accessing
private Controller methods from outside of that Controller. Is there
an example out there where getting access to a private method in a
Controller class from some other class is used?

- Ben

[Shannon -jj Behrens]

On 11/7/05, Ben Bangert <b...@groovie.org> wrote:

Ok, I'll buy that. But if they have their own instance of the app,
don't you think they also deserve their own hostname and process? I
don't understand why you would want to embed two instances of the same
Web app in the same process :-/ Personally, perhaps I'm naive, but
I've taken a liking for having a bunch of virtual hosts in Apache that
are each proxied to separate instances of Glass (Aquarium's Web
server) running on their own port in their own process. This is
invisible to the user, naturally.

[Ben Bangert]

On Nov 7, 2005, at 5:05 PM, Shannon -jj Behrens wrote:

> Ok, I'll buy that. But if they have their own instance of the app,
> don't you think they also deserve their own hostname and process? I
> don't understand why you would want to embed two instances of the same
> Web app in the same process :-/ Personally, perhaps I'm naive, but
> I've taken a liking for having a bunch of virtual hosts in Apache that
> are each proxied to separate instances of Glass (Aquarium's Web
> server) running on their own port in their own process. This is
> invisible to the user, naturally.

Ah, ok, thats because the blog is just one webapp. So Client X calls
up and wants a blog, a forum, and a shopping cart. You grab Blog
WriterPro, ForumMaster, and CartAHolic (all made up names for these
apps, all the apps are Paste-compatible).

You edit a little conf file, add the 3 webapps at /blog, /forum, /
cart or what have you. You then fire it up, maybe do a little
skinning. Your client is happy....

It runs in a single Fast CGI, or SCGI process behind lighttpd or
Apache (depending on the hosting server). So here you have 3 webapps,
running in a single process.

Is there any good reason a web developer would want to use multiple
frameworks in a single webapp? Probably not. This feature is mostly
intended for cases like I just listed where an end-user (or developer
acting on behalf of a client, or ISP webadmin setting up new webapp X
for user) sets up multiple webapps for a single client. This will run
as a single process (though you could run several of them for scaling
of course), typically under a single domain (the clients).

What we mean by 30 instances of a webapp on a server, is the ease of
merely installing the webapp using an egg, then just having a little
config file for each "site" that tells it what webapp "instances" it
should run and their configration/database options. You're not
necessarily running them all in a single process, but depending on
the clients needs there could be several of the same webapp instances
in a single process.

Does that help?

Cheers,
Ben

[Ian Bicking]

Ben Bangert wrote:
>> That's the way just using what is there now. Potentially Routes
>> could include some other data that indicated what strings in the
>> mapping came from a pattern (like 'show_article'), and what strings
>> came from the request itself (like action for the second rule). This
>> would be more like string taintedness, but it would be up to the code
>> that matches dictionaries to objects to make use of it explicitly.
>
>

> That's actually already available since the Route that the Mapper
> locates has a defaults attribute that holds a dict of the 'defaults'
> for the route. Though to get at that info from a Controller I'd need to
> expose the matched Route object to it. I'm not entirely sure what the
> user would do with it though, since they can ensure variables meet
> their req's by using the requirements keyword on a m.connect.

OK, so a "tainted" string would be one where the actual value of a
variable differs from the default (which may be none, in which case the
variable would always be tainted).

I'm thinking the code that would make use of this would be whatever maps
the Routes response into an invocation. This might be superclass
methods of the controller, or some code that maps objects to WSGI apps,
or whatever. This might be called the "resolver" in some frameworks.
In many ways it is specific to the framework, though the number of ways
it is implemented is fairly small.

>> In my code (with ZPTKit) I do something like 2, except because
>> servlets are instantiated for every request you can assign things
>> like this to "self" and not use thread-local objects. So I do things
>> like "self.options.object1 = foo" in my controller methods. I find
>> the one-instance-per-request model to be very convenient personally.
>> I think sometimes that's not used for performance reasons, but that's
>> a false optimization as the instantiation overhead is minimal.
>
>

> It's very tempting to switch to a new object for every request, I've
> played with that on my own as its a very small change in Pylons to use
> that dispatch style. I'll benchmark it and see if there's any
> meaningful difference in performance or ram usage, not having to worry
> about assigning other things to self is useful.

Considering those tests I was doing comparing the overhead of creating
an object for every row in a result, creating objects is really cheap.
To remind myself or anyone else, the code is here:
http://svn.colorstudy.com/home/ianb/test_obj_overhead.py

The "flyweight" case, where I reused a single object and reassigned
attributes on it. This is like Webware's instance pool. The "mapped"
case is like an instance-per-request. The "null" case is like passing
the values in and letting the function handle the issues.

Time for null1: 0.010000 seconds
Time for null2: 0.170000 seconds
Time for test_flyweight1: 0.050000 seconds x5.0
Time for test_flyweight2: 1.590000 seconds x9.4
Time for test_mapped1: 0.330000 seconds x33.0
Time for test_mapped2: 1.770000 seconds x10.4

Each is the time for looping through 200000 creates, "1" is looping with
only object creation, "2" is with a little work in the loop.

My personal feeling is that mapped isn't much worse than flyweight, and
they are all insignificant when compared to anything involving real work.

> On the other hand, it
> can be useful to attach persistent things to a persistent Controller,
> how do you deal with variables and objects that should be kept
> persistent between requests?

Module globals, or setup at the module scope (instead of __init__).

>> This also reminds me a bit of some ideas for RESTish URLs. People
>> want to be explicit about mapping HTTP methods into the URL
>> resolution, e.g., article_GET vs article_POST or whatever. When this
>> is enforced (like in SimpleHTTPServer) it is obnoxious, because it
>> requires me to dispatch on HTTP method at a point when I might not
>> have sufficient knowledge to do so. But if a GET request first
>> looked for action_GET(), then action() that would be ideal. And
>> maybe action_JSON() could fit in there too (and action_XMLRPC and
>> whatever else).
>
>

> I really like that idea, and it makes it easy to have functionality
> that will only work under specific HTTP methods or even as you
> mentioned, XML-RPC. It keeps the methods together under the class that
> deals with them, and by having a naming scheme for them should be easy
> to scan controllers for methods to put together mappings of all the
> JSON / XMLRPC / SOAP calls supported.
>
> I'm still under the impression that when a class is acting as a
> Controller, private method means private to the world and to the whole
> program. In which case, using the _ prefix makes sense, as I can't see
> why any other part of your program should be accessing private
> Controller methods from outside of that Controller. Is there an example
> out there where getting access to a private method in a Controller
> class from some other class is used?

I use it for subclassing sometimes (in servlets, at least). But then I
blithely use methods irregardless of leading underscores, with no
respect for privacy ;) So it doesn't really effect me, except that
leading underscores have come to bother me aesthetically.

[Shannon -jj Behrens]

On 11/7/05, Ben Bangert <b...@groovie.org> wrote:

> On Nov 7, 2005, at 5:05 PM, Shannon -jj Behrens wrote:
>
> > Ok, I'll buy that. But if they have their own instance of the app,
> > don't you think they also deserve their own hostname and process? I
> > don't understand why you would want to embed two instances of the same
> > Web app in the same process :-/ Personally, perhaps I'm naive, but
> > I've taken a liking for having a bunch of virtual hosts in Apache that
> > are each proxied to separate instances of Glass (Aquarium's Web
> > server) running on their own port in their own process. This is
> > invisible to the user, naturally.
>
> Ah, ok, thats because the blog is just one webapp. So Client X calls
> up and wants a blog, a forum, and a shopping cart. You grab Blog
> WriterPro, ForumMaster, and CartAHolic (all made up names for these
> apps, all the apps are Paste-compatible).
>
> You edit a little conf file, add the 3 webapps at /blog, /forum, /
> cart or what have you. You then fire it up, maybe do a little
> skinning. Your client is happy....
>
> It runs in a single Fast CGI, or SCGI process behind lighttpd or
> Apache (depending on the hosting server). So here you have 3 webapps,
> running in a single process.

Sorry, I still don't buy it for these reasons:

1. Those three things don't have to be separate apps. If they were in
their own hierarchies, that would work trivially in Aquarium by adding
each directory to the packagePath. I wouldn't call them separate apps
though :-/

2. If they're in the same app, they have access to the same globals
and might trample each other. I wouldn't even want them sharing the
same OS process, let alone the same Python interpretter.

3. Since when were all the interesting little apps written in Python?
I know Ian has to deal with many apps that aren't even in Python, let
alone WSGI. :-/

To meet your needs, perhaps it'd be more fruitful to have an
application proxy that does various things, perhaps injecting
JavaScript or doing DOM manipulations. That way the apps don't have
to be written in Python, let alone your framework. :-/ In this way,
you can manipulate the applications at the proxy layer, perhaps
automatically logging the user in, etc., without editing the
applications themselves.

> Is there any good reason a web developer would want to use multiple
> frameworks in a single webapp? Probably not. This feature is mostly
> intended for cases like I just listed where an end-user (or developer
> acting on behalf of a client, or ISP webadmin setting up new webapp X
> for user) sets up multiple webapps for a single client. This will run
> as a single process (though you could run several of them for scaling
> of course), typically under a single domain (the clients).
>
> What we mean by 30 instances of a webapp on a server, is the ease of
> merely installing the webapp using an egg, then just having a little
> config file for each "site" that tells it what webapp "instances" it
> should run and their configration/database options. You're not
> necessarily running them all in a single process, but depending on
> the clients needs there could be several of the same webapp instances
> in a single process.
>
> Does that help?

Thanks for your patience.

[Ben Bangert]

On Nov 7, 2005, at 6:22 PM, Shannon -jj Behrens wrote:

> 1. Those three things don't have to be separate apps. If they were in
> their own hierarchies, that would work trivially in Aquarium by adding
> each directory to the packagePath. I wouldn't call them separate apps
> though :-/

No, they don't. But when you're trying to get someone setup with
something ASAP, taking existing webapps that excel at what they do,
and throwing them into a cohesive whole is better than writing them
all yourself. Your example also just forced everyone to use Aquarium.
Pylons uses Paste for deployment, and you can deploy Pylons apps
alongside any other Paste-compatible framework.

Django is working to be Paste-compatible, CherryPy's recent multi-
process patch means its soon going to be fully Paste compatible (thus
TurboGears will be closer). The webapps I cited above might use any
of those frameworks, the nice thing about using Paste to deploy,
means you don't have to care which framework was used.

> 2. If they're in the same app, they have access to the same globals
> and might trample each other. I wouldn't even want them sharing the
> same OS process, let alone the same Python interpretter.

That's why being Paste-compatible means not polluting the global
space. It's why CherryPy didn't work until just recently. If you call
your webapp/framework "Paste-Compatible", it means there should be no
issues running multiple instances of a app in a single process (and
thread). It is assumed that a thread will handle a single request
from start-to-finish before beginning another one (not async/
stackless Python).

> 3. Since when were all the interesting little apps written in Python?
> I know Ian has to deal with many apps that aren't even in Python, let
> alone WSGI. :-/

They aren't, they're made in PHP because you can do a lot of this
mingling with PHP apps but not Python (or even Ruby). That's exactly
why something like Pylons (utilizing Paste) will help out, it means
there'll be a reason to write these interesting apps in Python. Your
question is sort of like saying we shouldn't use Python unless
there's compelling applications in it. You need the tools before you
get the results.

> To meet your needs, perhaps it'd be more fruitful to have an
> application proxy that does various things, perhaps injecting
> JavaScript or doing DOM manipulations. That way the apps don't have
> to be written in Python, let alone your framework. :-/ In this way,
> you can manipulate the applications at the proxy layer, perhaps
> automatically logging the user in, etc., without editing the
> applications themselves.

Did you read my post about WSGI and middleware? WSGI Middleware is a
proxy of sorts, though the way it "proxies" lets it be far more
useful than just injecting some Javascript or DOM manips. The nice
thing about WSGI and WSGI Middleware is that webapps don't need to be
written in a specific framework. :)

Anyways, if that still didn't help, we can always chat about it this
Thursday at the Python User Group. ;)

Cheers,
Ben

[James Gardner]

Hi all,

> After a lot of reading, research, and a demo application, I recommend
> Dojo. Don't be afraid to use more than one JavaScript library.
> There's no need to limit yourself.

Dojo looks quite powerful. The rich text editing test looks very
interesting as I'm trying to write a CMS and like the idea of in-place
editing.

The other (very untechnical) reason that Myghty is quite good to use is
that it is based on HTML::Mason which Amazon uses. As soon as you tell a
client that you are developing their product in the language Google and
NASA use and the templating is done in the same way Amazon.com do theirs
it is an instant win. Pylons is aimed at people who might need to make
such a case to their clients.

> I actually like .expose;

I like the _private rule too though, seems more natural.

In Bricks you set

myfunc.expose = user(..permissions..)

user() just returns a dictionary of permissions and then the __call__
method determines if the action can be called based on the access
permissions in the dictionary and the permissions set to the .expose
attribute of the method they want to access. Users can use their own
permissions systems by calling a different method in __call__ to check
the permissions. Could be fairly flexible and familiar way of doing it
in Pylons without using decorators?

> Personally I'd be interested in a very open sort of framework. I.e.,
> one that keeps the parts separate, since I would like (for myself) to
> make an easy sort of transition from old code to new. So a route would
> identify a controller, which would in turn be a WSGI application
> (probably with adaptation of some sort). With a different level of
> abstractness, a controller produces variables; these variables can get
> rendered in a template, sent via JSON, etc. -- I think that's a clever
> part of TurboGears. But the controllers don't really have to care that
> much about what kind of template they are sending to. Dynamically
> scoped template providing might be interesting here.

I agree with this really. I think I'd like the controllers to be WSGI
applications. The current way to do that is to put variables like m, r,
self.temp etc into the WSGI environ dictionary. That was why I was
really quite interested in the idea of WSGI services but I never did
quite get to the bottom of how they would work!

I guess ideally I'd like to be able to run my old Bricks and generic
WSGI code on Pylons too together with new Pylons code and that could
best be done if Pylons supports WSGI applications as controllers. It
would encourage others to migrate their code too.

How about this for a comprimise between an upgrade path and a powerful
and obvious system out of the box: We detect whether a particular
controller is derived from a Pylons baseController or not and handle it
accordingly if it is and if it isn't just call it and leave it up to the
controller to work out how to deal with the rest of the route?

There are two benefits:

1. It would make it easier to write a tutorial for a beginner since you
could start out with somehting simple they can play with like:

def application(environ, start_response):
start_response('200 OK', [('Content-type','text/plain')])
return ['Hello World!']

then move onto WSGI application classes then introduce the Pylons class
before talking about decorators. It is always much better if you can
lead a beginner through all of the main processes going on and let them
play at each stage as they stand more chance of understanding the system
and therefore using it for their own projects.

2. If we can develop a plugin architecture other people could write a
plugin to allow Pylons to work better with their WSGI classes so it
might be even easier for people to migrate to Pylons. I'm thinking
integration on a controller level here rather than application level
which is more what Paste is for. You never know, people might come up
with a better Pylons controller plugin than Pylons?!

This links into your ideas of how to manage the routes, why not allow a
few alternatives, with the best chosen by default?

> I think it's important to keep separate track of branding and
> documentation choices, and actual technical choices. "Pylons" may
> choose Myghty; but the things it's built on don't have to choose, and
> the things it's built on could (should) be most of the implementation.

+1

I'm particularly keen to keep track of the technical choices we've made,
explain how you would implement things the alternative way if you really
wanted to using Pylons (if indeed it is possible) and provide users the
oppurtunity to do so through documented examples (maybe on the wiki
rather than main site). That way the default is a good way that works
but you can easily find out how to do things a different way yourself if
you need to, although I agree we can't document every alternative.

I'd like someone with a £20/year shared hosting package which runs
Python CGI scripts but doesn't have command prompt access to be able to
install a Pylons cms/forum/blog/wiki/contact application without any
technical knowledge at all. The same Pylons apps should also be able to
take advantage of more advanced server features through Paste etc. But
at the same time Pylons must be powerful enough to be used for something
very sophisticated. This broad range of appeal really isn't too
unrealistic with the current Paste/Pylons setup I don't think.

From my point of view, what will make Pylons better than other things
out there?

1. Technically as good or better than other products with flexible but
very usable defaults
2. Our completely dedicated focus on usability and making things simple,
whether the end user is someone with nothing but an FTP client
installing a Pylons app or an experienced web hacker who wants to
develop something very clever!

It would be good if:

* The pylons website (when we make it) has an online controller database
where people can upload useful controllers (or paste/pylons controller
templates if that's how we choose to do it) which they can use in their
own appliations e.g. an add user controller, an email form controller etc.
* Everything on the Pylons site should be written in Python and be able
to be deployed with Paste (I found it quite funny that Rails uses both
Python and PHP applications, it doesn't build confidence).
* We have an online application database of Pylons application eggs eg
blog, wiki etc which users can install.
* We write a web based installer which requires the user to simply FTP a
single Python CGI script after modifying the #! line and setting execute
permissions. The script runs a web-based installer which downloads and
installs Pylons and provides an admin interface which allows them to
install other Pylons apps as eggs which they can upload. The server.conf
file is editable through the interface. They can do it the "hard" way on
the command line too if they have command line access. Users can then
use the admin interface to install other eggs that do things like user
management.

Would it be worth specifying in the docs that 3rd party apps shouldn't
start with "Pylons". That way we can create a standard set of
applications named PylonsUsers, PylonsDatabase etc to do useful things
and they can be clearly Pylons apps and not third part ones?

Cheers,

James

P.S. Do you all live in the same place in the states?

[Ben Bangert]

On Nov 10, 2005, at 4:30 AM, James Gardner wrote:

> > After a lot of reading, research, and a demo application, I
> recommend
> > Dojo. Don't be afraid to use more than one JavaScript library.
> > There's no need to limit yourself.
>
> Dojo looks quite powerful. The rich text editing test looks very
> interesting as I'm trying to write a CMS and like the idea of in-
> place editing.

Yep, and for the more basic needs, MochiKit looks great, and
interoperates nicely. I'm still learning Javascript though, so my
opinion here is pretty meaningless. :)

> I agree with this really. I think I'd like the controllers to be
> WSGI applications. The current way to do that is to put variables
> like m, r, self.temp etc into the WSGI environ dictionary. That was
> why I was really quite interested in the idea of WSGI services but
> I never did quite get to the bottom of how they would work!

I can't see any way to avoid this resulting in a much larger learning
curve for new users. Ideally I'd like to keep WSGI low-key in Pylons
so they can get by without knowing much about it until they wanted to
learn more.

> I guess ideally I'd like to be able to run my old Bricks and
> generic WSGI code on Pylons too together with new Pylons code and
> that could best be done if Pylons supports WSGI applications as
> controllers. It would encourage others to migrate their code too.

The optimum way to do that would be to use Paste and run the old apps
side-by-side with the new Pylons webapp, no?

> How about this for a comprimise between an upgrade path and a
> powerful and obvious system out of the box: We detect whether a
> particular controller is derived from a Pylons baseController or
> not and handle it accordingly if it is and if it isn't just call it
> and leave it up to the controller to work out how to deal with the
> rest of the route?

To do this, from inside Myghty, we'd have to be able to get to the
untouched start_response, and pass it into the new WSGI webapp. I'm
not sure if Myghty currently exposes this inside a WSGI app in a way
it can be accessed.

> There are two benefits:
>
> 1. It would make it easier to write a tutorial for a beginner since
> you could start out with somehting simple they can play with like:
>
> def application(environ, start_response):
> start_response('200 OK', [('Content-type','text/plain')])
> return ['Hello World!']
> then move onto WSGI application classes then introduce the
> Pylons class before talking about decorators. It is always much
> better if you can lead a beginner through all of the main processes
> going on and let them play at each stage as they stand more chance
> of understanding the system and therefore using it for their own
> projects.

In the PEP, PJB notes that he doesn't expect people making webapps to
ever do such a thing. This is because its a very low-level detail,
and frameworks are really supposed to handle it. Ideally, if they
should actually want to call another WSGI webapp, it should just call
it, and their WSGI webapp should handle those details.

> 2. If we can develop a plugin architecture other people could write
> a plugin to allow Pylons to work better with their WSGI classes so
> it might be even easier for people to migrate to Pylons. I'm
> thinking integration on a controller level here rather than
> application level which is more what Paste is for. You never know,
> people might come up with a better Pylons controller plugin than
> Pylons?!

I'm starting to get lost on what Pylons is and what isn't in this
scenario. The concept of a full-stack framework seems to run rather
counter to something so decoupled and 'vaporous'. The controller is a
very basic object, that merely has some variables and runs m.subexec
currently. It's more conventions and structure than actual code.

Besides for that though, Myght does a bit on top to ensure that it
recompiles your Controller if you make a change to it, it also makes
the proxy Component on top which is what allows us to call the
Component easily from inside a template. I'm not sure how feasible it
will be to swap this part in and out due to this, nor am I sure how
much could be 'gained'.

> From my point of view, what will make Pylons better than other
> things out there?
>
> 1. Technically as good or better than other products with flexible
> but very usable defaults
> 2. Our completely dedicated focus on usability and making things
> simple, whether the end user is someone with nothing but an FTP
> client installing a Pylons app or an experienced web hacker who
> wants to develop something very clever!

Speaking of which, as there's no official version yet, I'd like the
goal of each new version (the first included obviously) to use as
much 'cutting-edge' Python web stuff as applicable. There are some
drawbacks... like having to wait for other included projects to
become 'finalized', but I think that's tolerable for the pay-offs.

Obviously, the usability has to be retained in whatever new things
are added, but I think it would make it appealing for many just to
see working demonstrations of all the cool stuff, and have it first. :)

> * The pylons website (when we make it) has an online controller
> database where people can upload useful controllers (or paste/
> pylons controller templates if that's how we choose to do it) which
> they can use in their own appliations e.g. an add user controller,
> an email form controller etc.

Absolutely, I'd like to add helper functions to PasteScript or Pylons
(as appropriate) to make more structure generation even easier. That
way we could have 3rd party structure generators like Rails.

> * Everything on the Pylons site should be written in Python and be
> able to be deployed with Paste (I found it quite funny that Rails
> uses both Python and PHP applications, it doesn't build confidence).

Yea, the Ruby ppl are currently in the same fix as Python and Perl
regarding deployability. Though in Rails case, DHH seems to prefer
flat HTML and PHP than running a fully dynamic site in Rails.

> * We have an online application database of Pylons application eggs
> eg blog, wiki etc which users can install.
> * We write a web based installer which requires the user to simply
> FTP a single Python CGI script after modifying the #! line and
> setting execute permissions. The script runs a web-based installer
> which downloads and installs Pylons and provides an admin interface
> which allows them to install other Pylons apps as eggs which they
> can upload. The server.conf file is editable through the interface.
> They can do it the "hard" way on the command line too if they have
> command line access. Users can then use the admin interface to
> install other eggs that do things like user management.
>
> Would it be worth specifying in the docs that 3rd party apps
> shouldn't start with "Pylons". That way we can create a standard
> set of applications named PylonsUsers, PylonsDatabase etc to do
> useful things and they can be clearly Pylons apps and not third
> part ones?

I don't see why not, clearly separating "core" Pylons additions from
3rd party ones is a nice distinction.

Cheers,
Ben

[Michael Bayer]

Ben Bangert wrote:
>
> To do this, from inside Myghty, we'd have to be able to get to the
> untouched start_response, and pass it into the new WSGI webapp. I'm
> not sure if Myghty currently exposes this inside a WSGI app in a way
> it can be accessed.
>

One of Myghty's first users and still hasnt read the docs.....

http://www.myghty.org/docs/programmatic.myt#configuration_programmatic_httphandler_WSGIHandler

[Ben Bangert]

Ok, scratch that reason. :)

[Ian Bicking]

Ben Bangert wrote:
>> I agree with this really. I think I'd like the controllers to be WSGI
>> applications. The current way to do that is to put variables like m,
>> r, self.temp etc into the WSGI environ dictionary. That was why I was
>> really quite interested in the idea of WSGI services but I never did
>> quite get to the bottom of how they would work!
>
>
> I can't see any way to avoid this resulting in a much larger learning
> curve for new users. Ideally I'd like to keep WSGI low-key in Pylons so
> they can get by without knowing much about it until they wanted to
> learn more.

You would never know, for instance, that Webware or Wareweb servlets are
WSGI applications. They just happen to implement __call__. Any
information passed into them from something earlier in the stack is
passed in through special environment keys, and the servlet code uses
those keys to provide nice wrapper functions, like
self.forward(internal_url); none of the keys are exposed to the user.
This is all made possible because all servlets subclass a common
servlet, which I think Myghty does too...?

So I think it's pretty easy to hide the fact they are WSGI applications.
The only issue is how to find the application (or wrapper for the
application) without conflicting with other methods? One thing I've
considered is to expect objects to have a .wsgi_application
attribute/method, that would return the application. Then __call__
wouldn't be overloaded, and you could make it a class method that
instantiated instances, so that the resolver could find a class and use
the method to turn it into an instance (or wrapper on an instance) that
implements WSGI.

>> There are two benefits:
>>
>> 1. It would make it easier to write a tutorial for a beginner since
>> you could start out with somehting simple they can play with like:
>>
>> def application(environ, start_response):
>> start_response('200 OK', [('Content-type','text/plain')])
>> return ['Hello World!']
>> then move onto WSGI application classes then introduce the Pylons
>> class before talking about decorators. It is always much better if
>> you can lead a beginner through all of the main processes going on
>> and let them play at each stage as they stand more chance of
>> understanding the system and therefore using it for their own projects.
>
>
> In the PEP, PJB notes that he doesn't expect people making webapps to
> ever do such a thing. This is because its a very low-level detail, and
> frameworks are really supposed to handle it. Ideally, if they should
> actually want to call another WSGI webapp, it should just call it, and
> their WSGI webapp should handle those details.

While I think it is good for controllers to be WSGI apps, I don't think
WSGI makes a good hello world. Unless you are actually teaching WSGI,
which isn't what Pylons would be doing.

>> 2. If we can develop a plugin architecture other people could write a
>> plugin to allow Pylons to work better with their WSGI classes so it
>> might be even easier for people to migrate to Pylons. I'm thinking
>> integration on a controller level here rather than application level
>> which is more what Paste is for. You never know, people might come up
>> with a better Pylons controller plugin than Pylons?!
>
>
> I'm starting to get lost on what Pylons is and what isn't in this
> scenario. The concept of a full-stack framework seems to run rather
> counter to something so decoupled and 'vaporous'. The controller is a
> very basic object, that merely has some variables and runs m.subexec
> currently. It's more conventions and structure than actual code.

I think there's a concept floating underneath of a bunch of hard-working
but not-pretty components. Some of these don't yet exist, and so it's
unclear where they will go or under what name. I think we shouldn't
confuse that software-to-be with what Pylons wants to be, or to limit
software that's on that level because of particular scope that Pylons has.

>> * Everything on the Pylons site should be written in Python and be
>> able to be deployed with Paste (I found it quite funny that Rails
>> uses both Python and PHP applications, it doesn't build confidence).
>
>
> Yea, the Ruby ppl are currently in the same fix as Python and Perl
> regarding deployability. Though in Rails case, DHH seems to prefer flat
> HTML and PHP than running a fully dynamic site in Rails.

I think deployability is nice, but we shouldn't set up roadblocks in the
form of "eating our own dogfood". However, in many ways the validity of
the techniques (or, maybe just the validity of Paste) is in how
adaptable it is, and if it can embrace lots of different code lineages.
E.g., Pastifying MoinMoin or whatever other software shouldn't be a
heroic effort. Luckily there's lots of good Python software for project
websites, so it's not that necessary to look to other languages.

[Ben Bangert]

On Nov 10, 2005, at 3:31 PM, Ian Bicking wrote:

> You would never know, for instance, that Webware or Wareweb
> servlets are WSGI applications. They just happen to implement
> __call__. Any information passed into them from something earlier
> in the stack is passed in through special environment keys, and the
> servlet code uses those keys to provide nice wrapper functions,
> like self.forward(internal_url); none of the keys are exposed to
> the user. This is all made possible because all servlets subclass a
> common servlet, which I think Myghty does too...?

In Myghty, there's a few different controller styles. One of them
does require subclassing a special module.component type, the rest
are all plain objects or functions. In Pylons we're going to have the
user sub-class the Controller so we can put the functionality into
it, which lets them override parts should they want to.

> So I think it's pretty easy to hide the fact they are WSGI
> applications. The only issue is how to find the application (or
> wrapper for the application) without conflicting with other
> methods? One thing I've considered is to expect objects to have
> a .wsgi_application attribute/method, that would return the
> application. Then __call__ wouldn't be overloaded, and you could
> make it a class method that instantiated instances, so that the
> resolver could find a class and use the method to turn it into an
> instance (or wrapper on an instance) that implements WSGI.
>

> While I think it is good for controllers to be WSGI apps, I don't
> think WSGI makes a good hello world. Unless you are actually
> teaching WSGI, which isn't what Pylons would be doing.

Maybe I'm not getting it or something, but what's the advantage of
making simple, easy to understand Controller objects, into WSGI apps?

Besides for a very small group of advanced users that might want to
do something crazy, or some migration of an existing webapp using
WSGI hooks (which isn't very 'basic' or easy either), what does it
buy us?

Maybe it would help more if I saw an example of what calling a WSGI
app as a Controller would actually look like. Here's my rough sketch
given what I'm hearing so far.

#controller file

import .... # All the imports necessary to create the WSGI app

class MyController(object):
def __init__(self):
.. init stuff?...
self.wsgi_application = True

def __call__(self, environ, start_response):
return somewsgiapp(environ, start_response)

Again, this really feels like its throwing simplicity and usability
out the door to enable more use for the least common user. I don't
mind having certain expectations of a users skill level, but this
starts adding a lot of complexity that other frameworks don't have.
Am I missing some easier, less intimidating way of doing this?

Also, I still don't understand why Pylons should be stepping then
sending requests to a different WSGI app, rather than some WSGI
middleware. Isn't that what WSGI middleware is for?

> I think there's a concept floating underneath of a bunch of hard-
> working but not-pretty components. Some of these don't yet exist,
> and so it's unclear where they will go or under what name. I think
> we shouldn't confuse that software-to-be with what Pylons wants to
> be, or to limit software that's on that level because of particular
> scope that Pylons has.

True, thought I don't see why the components can't be pretty. :)

> I think deployability is nice, but we shouldn't set up roadblocks
> in the form of "eating our own dogfood". However, in many ways the
> validity of the techniques (or, maybe just the validity of Paste)
> is in how adaptable it is, and if it can embrace lots of different
> code lineages. E.g., Pastifying MoinMoin or whatever other
> software shouldn't be a heroic effort. Luckily there's lots of
> good Python software for project websites, so it's not that
> necessary to look to other languages.

Agreed.

Cheers,
Ben

[Ben Bangert]

On Nov 10, 2005, at 4:26 PM, Ben Bangert wrote:

> Also, I still don't understand why Pylons should be stepping then
> sending requests to a different WSGI app, rather than some WSGI
> middleware. Isn't that what WSGI middleware is for?

Oops, left out a word here. That should be "stepping in, then sending
requests...".

I'm going to start working on some Pylons structure generation stuff,
at which point hopefully the tests stuff will come together some.

Ian, nice posting on the web test frameworks, I'm going to be
watching that with interest to see what pops up.

Cheers,
Ben

[James Gardner]

Hi all,

The current SVN version doesn't work unless you add import
myghty.exception to the __init__.py file, but wow, the error handling is
awesome! Especially when often you can do||* print*
inspect*.*getsource*(*self*.*index*) *||to see straight away where
you've gone wrong. What would make this perfect is implementing cursor
up and cursor down too :-)

Just catching up on all the WSGI vs non-WSGI controller thoughts.

>The optimum way to do that would be to use Paste and run the old apps
> side-by-side with the new Pylons webapp, no?

Not really. I'd like to make it easy for other people to drop their
existing code into a Pylons app and slowly migrate their code to Pylons.
No-one wants to restart their project from scratch just to use another
framework and the easier we can make it to migrate the better. I think
allowing some way of running WSGI controllers but also taking advantage
of Pylons features is the best way to do this but I can quite see why
you aren't keen on turning the current request mechanism inside out and
it makes things more complicated for the beginner.

> Ideally, if they
> should actually want to call another WSGI webapp, it should just call
> it, and their WSGI webapp should handle those details.

That sounds like the best solution to me, we just provide a
pylons.runWSGI() method which can be called from within a controller to
call a WSGI app:

class HelloController(BaseController):
def index(self, **params):
pylons.runWSGI(myWSGIAppFromSomewhereElse, self, **params)

def myWSGIAppFromSomewhereElse(environ, start_response):
start_response('200 OK', [['Content-type','text/html']])
yield str(environ['pylons.r'].environ)
yield 'Yup, I work'

That would keep me satisfied anyway as my individual controllers just
have to be modified by 3 lines and I can run them within Pylons and take
advantage of Myghty and all the rest. myWSGIAppFromSomewhereElse()
still gets called on each request so this works fine. Passing self
through also lets me move code into HelloContrllor if I really wanted to.

> While I think it is good for controllers to be WSGI apps, I don't think
> WSGI makes a good hello world. Unless you are actually teaching WSGI,
> which isn't what Pylons would be doing.

OK, fair enough. I guess I find WSGI quite a good way to teach web
programming, it seems quite a natural progression for me in terms of
complexity to teach CGI, then a WSGI function then a WSGI class then a
Pylons class. But you're right, we should just be jumping in with Pylons
controller and explaining that!

> Speaking of which, as there's no official version yet, I'd like the
> goal of each new version (the first included obviously) to use as
> much 'cutting-edge' Python web stuff as applicable. There are some
> drawbacks... like having to wait for other included projects to
> become 'finalized', but I think that's tolerable for the pay-offs.
>Obviously, the usability has to be retained in whatever new things
> are added, but I think it would make it appealing for many just to
> see working demonstrations of all the cool stuff, and have it first. :)

Fine!

> Absolutely, I'd like to add helper functions to PasteScript or Pylons
> (as appropriate) to make more structure generation even easier. That
> way we could have 3rd party structure generators like Rails.

Great stuff,

James

[Ian Bicking]

Ben Bangert wrote:
>
> On Nov 10, 2005, at 3:31 PM, Ian Bicking wrote:
>
>> You would never know, for instance, that Webware or Wareweb servlets
>> are WSGI applications. They just happen to implement __call__. Any
>> information passed into them from something earlier in the stack is
>> passed in through special environment keys, and the servlet code uses
>> those keys to provide nice wrapper functions, like
>> self.forward(internal_url); none of the keys are exposed to the user.
>> This is all made possible because all servlets subclass a common
>> servlet, which I think Myghty does too...?
>
>
> In Myghty, there's a few different controller styles. One of them does
> require subclassing a special module.component type, the rest are all
> plain objects or functions. In Pylons we're going to have the user
> sub-class the Controller so we can put the functionality into it, which
> lets them override parts should they want to.

Then I'd expect the WSGI code to go into Controller...?

>> So I think it's pretty easy to hide the fact they are WSGI
>> applications. The only issue is how to find the application (or
>> wrapper for the application) without conflicting with other methods?
>> One thing I've considered is to expect objects to have a
>> .wsgi_application attribute/method, that would return the
>> application. Then __call__ wouldn't be overloaded, and you could
>> make it a class method that instantiated instances, so that the
>> resolver could find a class and use the method to turn it into an
>> instance (or wrapper on an instance) that implements WSGI.
>>
>> While I think it is good for controllers to be WSGI apps, I don't
>> think WSGI makes a good hello world. Unless you are actually
>> teaching WSGI, which isn't what Pylons would be doing.
>
>
> Maybe I'm not getting it or something, but what's the advantage of
> making simple, easy to understand Controller objects, into WSGI apps?

The advantage being that they are manipulable and dispatchable alongside
any other WSGI application. And vice versa. For instance, I have
frequently been putting StaticURLParser's in my code, configured
programmatically. It's nice they can easily live alongside other
controllers. And like James mentioned, it can mean that -- with maybe a
little tweaking -- a legacy application can be a working Pylons
application, and you can convert it incrementally (or not at all,
leaving the old code alone).

> Besides for a very small group of advanced users that might want to do
> something crazy, or some migration of an existing webapp using WSGI
> hooks (which isn't very 'basic' or easy either), what does it buy us?
>
> Maybe it would help more if I saw an example of what calling a WSGI app
> as a Controller would actually look like. Here's my rough sketch given
> what I'm hearing so far.
>
> #controller file
>
> import .... # All the imports necessary to create the WSGI app
>
> class MyController(object):
> def __init__(self):
> .. init stuff?...
> self.wsgi_application = True
>
> def __call__(self, environ, start_response):
> return somewsgiapp(environ, start_response)

I don't think the controller should subclass from object. Certainly if
it does, it looks unpleasant as a WSGI application. Instead I was
thinking of something like this:

class Controller(object): # Abstract base class
@paste.util.classinstance.classinstancemethod
def wsgi_application(self, cls):
if self is None:
self = cls()
return WSGIWrapper(self)

Or something like that. @paste.util.classinstance.classinstancemethod
is a decorator that allows a function to be simultaneously a class and
instance method -- that way you can have __init__ args and
instantantiate the instance manually, and get something that is just as
usable as the class.

I feel strongly that the controller shouldn't just be a bare object that
happens to fulfill some Pylons-specific protocol. In isolation, it
won't look like what it is -- a Pylons controller. So some subclassing
or something is important.

> Again, this really feels like its throwing simplicity and usability out
> the door to enable more use for the least common user. I don't mind
> having certain expectations of a users skill level, but this starts
> adding a lot of complexity that other frameworks don't have. Am I
> missing some easier, less intimidating way of doing this?

This is what both Paste WebKit and Wareweb do, and you'd never know,
it's just an internal detail. But it means I can defer entering the
framework-proper until much later in the process, and in turn other WSGI
applications can live alongside easily. So, if you did this, then
WebKit servlets should be able to go alongside Pylons controllers
seemlessly. While that's not necessarily an entirely mainstream desire,
it's also not an obscure or hard-to-understand feature.

It might be useful to look at the Wareweb controller (aka servlet):
http://svn.pythonpaste.org/Paste/Wareweb/trunk/wareweb/servlet.py

It has the entire WSGI implementation in __call__. And actually most of
the controller logic is implemented in that method. I haven't had much
time to revisit that code -- if I did I'd change things like how events
work and some other details (maybe add that wsgi_application method) --
but in the case of the WSGIness, it's essentially the same as Paste
WebKit, and I haven't found any problems with that at all (and I've been
using that a lot).

> Also, I still don't understand why Pylons should be stepping then
> sending requests to a different WSGI app, rather than some WSGI
> middleware. Isn't that what WSGI middleware is for?

Often programmatic forwarding is desired. For instance, I might want to
dispatch to a StaticURLParser based on a database lookup.

Also, application and middleware can become a vague distinction in many
cases. The evaldebug middleware is an example of that -- it's actually
an application (mounted at /_debug/), and middleware at the same time.
And that code even has a framework in it, though a very small and silly
little framework. It's very possible to write something that is just on
the other end -- that looks like an application, but underneith is also
middleware. E.g., something that forwards some requests to another
application (maybe another server), then modifies them on the way back.

[Ben Bangert]

On Nov 11, 2005, at 6:17 AM, James Gardner wrote:

> The current SVN version doesn't work unless you add import
> myghty.exception to the __init__.py file, but wow, the error
> handling is awesome! Especially when often you can do||* print*
> inspect*.*getsource*(*self*.*index*) *||to see straight away where
> you've gone wrong. What would make this perfect is implementing
> cursor up and cursor down too :-)

Oops, should've tested that more first. I added the import statement
to svn so it should be all good now. :)

> That sounds like the best solution to me, we just provide a
> pylons.runWSGI() method which can be called from within a
> controller to call a WSGI app:
>
> class HelloController(BaseController):
> def index(self, **params):
> pylons.runWSGI(myWSGIAppFromSomewhereElse, self, **params)
>
> def myWSGIAppFromSomewhereElse(environ, start_response):
> start_response('200 OK', [['Content-type','text/html']])
> yield str(environ['pylons.r'].environ)
> yield 'Yup, I work'
>
> That would keep me satisfied anyway as my individual controllers
> just have to be modified by 3 lines and I can run them within
> Pylons and take advantage of Myghty and all the rest.
> myWSGIAppFromSomewhereElse() still gets called on each request so
> this works fine. Passing self through also lets me move code into
> HelloContrllor if I really wanted to.

Looks good, I just tried the following:

from base import *

class WsgiController(BaseController):
def index(self, **params):
self.r.environ['pylons.r'] = self.r
return myWSGIAppFromSomewhereElse(self.r.environ,
self.r.start_response)

def myWSGIAppFromSomewhereElse(environ, start_response):
start_response('200 OK', [['Content-type','text/html']])
yield str(environ['pylons.r'].environ)
yield 'Yup, I work'

But I get a blank screen for some reason.

Mike, anything I need to add for this to work?

> OK, fair enough. I guess I find WSGI quite a good way to teach web
> programming, it seems quite a natural progression for me in terms
> of complexity to teach CGI, then a WSGI function then a WSGI class
> then a Pylons class. But you're right, we should just be jumping in
> with Pylons controller and explaining that!

The thing to keep in mind, is that the vast majority of users have no
clue what WSGI is, and when they see it, it freaks them out. I'm not
saying there's any reason for this, because there isn't (I wrote an
entire blog post trying to stress how easy WSGI is). Since we'd like
to show people how to make WSGI middleware and such with Pylons, we
obviously need to have docs on it at some point. I think getting them
hooked on the easier stuff first would work great, and keep the
learning curve more even.

For example, we could have a quick-start guide, which gets them
running a basic blog or todo-list, or whatever the hello-world webapp
is. Then the more detailed, I-really-want-to-know-whats-happening
would go over WSGI and migrating webapps, etc. How's that sound?

Cheers,
Ben

[Ben Bangert]

On Nov 11, 2005, at 9:19 AM, Ian Bicking wrote:

>> In Myghty, there's a few different controller styles. One of them
>> does require subclassing a special module.component type, the
>> rest are all plain objects or functions. In Pylons we're going to
>> have the user sub-class the Controller so we can put the
>> functionality into it, which lets them override parts should they
>> want to.
>
> Then I'd expect the WSGI code to go into Controller...?

Yup, I think that'll work fine.

> The advantage being that they are manipulable and dispatchable
> alongside any other WSGI application. And vice versa. For
> instance, I have frequently been putting StaticURLParser's in my
> code, configured programmatically. It's nice they can easily live
> alongside other controllers. And like James mentioned, it can mean
> that -- with maybe a little tweaking -- a legacy application can be
> a working Pylons application, and you can convert it incrementally
> (or not at all, leaving the old code alone).

Ok, that makes some sense. Heck, I can even see it being a quick way
to convert a WSGI capable app into a Paste-compatible app.

> class Controller(object): # Abstract base class
> @paste.util.classinstance.classinstancemethod
> def wsgi_application(self, cls):
> if self is None:
> self = cls()
> return WSGIWrapper(self)
>
> Or something like that.
> @paste.util.classinstance.classinstancemethod is a decorator that
> allows a function to be simultaneously a class and instance method
> -- that way you can have __init__ args and instantantiate the
> instance manually, and get something that is just as usable as the
> class.
>
> I feel strongly that the controller shouldn't just be a bare object
> that happens to fulfill some Pylons-specific protocol. In
> isolation, it won't look like what it is -- a Pylons controller.
> So some subclassing or something is important.

I should probably be a little clearer. The BaseController present in
lib/base.py is used for all the users-classes to inherit from.
However, Pylons currently is doing some magic behind the scenes,
since the BaseController gets things attached to it after its
instantiated. There's also some Component proxy stuff that occurs for
Myghty to be able to call the Controller from within a template. So
it's not really a completely basic object.

> This is what both Paste WebKit and Wareweb do, and you'd never
> know, it's just an internal detail. But it means I can defer
> entering the framework-proper until much later in the process, and
> in turn other WSGI applications can live alongside easily. So, if
> you did this, then WebKit servlets should be able to go alongside
> Pylons controllers seemlessly. While that's not necessarily an
> entirely mainstream desire, it's also not an obscure or hard-to-
> understand feature.
> It might be useful to look at the Wareweb controller (aka servlet):
> http://svn.pythonpaste.org/Paste/Wareweb/trunk/wareweb/servlet.py

Ok, I could see that being useful. Anything wrong with James example
of calling a WSGI app from the Controller? It would remove the need
to figure out how to guess whether a controller should be called as
WSGI, or as normal Pylons controller, etc.

> It has the entire WSGI implementation in __call__. And actually
> most of the controller logic is implemented in that method. I
> haven't had much time to revisit that code -- if I did I'd change
> things like how events work and some other details (maybe add that
> wsgi_application method) -- but in the case of the WSGIness, it's
> essentially the same as Paste WebKit, and I haven't found any
> problems with that at all (and I've been using that a lot).

Yea, makes sense for calling it as a WSGI app. The __init__.py for
the Pylons project is currently filled with a significant amount of
work required to setup the Pylons WSGI app, having to do that again
for the Controller would be pretty clumsy, which is why I wouldn't
want Pylons controllers to be WSGI apps. Since this goes beyond what
Myghty was created to do, I'm not sure how or if you'd be able to
mingle the m.subexec call into an app thats called as a WSGI app from
the controller.

Also, if your WSGI app does its own dispatching, I'm not sure how the
nested WSGI app would create templates with proper URL's. Given that
Routes is handling part of it (and not updating SCRIPT_NAME/PATH),
the WSGI app thats called from the Controller would be pretty
clueless on its location, wouldn't it?

Since Pylons already setup the Routes config object, using url_for
and redirect_to inside the migrated WSGI app should sort of work,
though if something might alter the internal dispatch of the WSGI
app... not sure what would happen. Since Routes dispatches in a style
that doesn't correlate with just knocking the first section off the
URI, there's no way to update the SCRIPT_NAME/PATH_INFO environ to
reflect where it is.

> Also, application and middleware can become a vague distinction in
> many cases. The evaldebug middleware is an example of that -- it's
> actually an application (mounted at /_debug/), and middleware at
> the same time. And that code even has a framework in it, though a
> very small and silly little framework. It's very possible to write
> something that is just on the other end -- that looks like an
> application, but underneith is also middleware. E.g., something
> that forwards some requests to another application (maybe another
> server), then modifies them on the way back.

Yea, I noticed that. I plan on writing a little single-sign on WSGI
middleware with Pylons, which will probably be similar in that regard.

Cheers,
Ben

[Ian Bicking]

Ben Bangert wrote:
>> class Controller(object): # Abstract base class
>> @paste.util.classinstance.classinstancemethod
>> def wsgi_application(self, cls):
>> if self is None:
>> self = cls()
>> return WSGIWrapper(self)
>>
>> Or something like that.
>> @paste.util.classinstance.classinstancemethod is a decorator that
>> allows a function to be simultaneously a class and instance method --
>> that way you can have __init__ args and instantantiate the instance
>> manually, and get something that is just as usable as the class.
>>
>> I feel strongly that the controller shouldn't just be a bare object
>> that happens to fulfill some Pylons-specific protocol. In isolation,
>> it won't look like what it is -- a Pylons controller. So some
>> subclassing or something is important.
>
>
> I should probably be a little clearer. The BaseController present in
> lib/base.py is used for all the users-classes to inherit from. However,
> Pylons currently is doing some magic behind the scenes, since the
> BaseController gets things attached to it after its instantiated.
> There's also some Component proxy stuff that occurs for Myghty to be
> able to call the Controller from within a template. So it's not really
> a completely basic object.

I don't think that should cause a problem.

I guess some of how I deal with this is that I stuff things in the WSGI
environment, and then pull them back out later. This way I can do some
work up-front, without having to actually know what I'm in front of.
I.e., I don't need access to the "real" WSGI app (as opposed to the
middleware that hides the app), I just need some agreement with that
app. An example might be internal redirects, where a redirector
(paste.recursive) puts some stuff in the environment, and the
application can later use that to do a redirect. There's probably others.

>> This is what both Paste WebKit and Wareweb do, and you'd never know,
>> it's just an internal detail. But it means I can defer entering the
>> framework-proper until much later in the process, and in turn other
>> WSGI applications can live alongside easily. So, if you did this,
>> then WebKit servlets should be able to go alongside Pylons
>> controllers seemlessly. While that's not necessarily an entirely
>> mainstream desire, it's also not an obscure or hard-to- understand
>> feature.
>> It might be useful to look at the Wareweb controller (aka servlet):
>> http://svn.pythonpaste.org/Paste/Wareweb/trunk/wareweb/servlet.py
>
>
> Ok, I could see that being useful. Anything wrong with James example of
> calling a WSGI app from the Controller? It would remove the need to
> figure out how to guess whether a controller should be called as WSGI,
> or as normal Pylons controller, etc.

Well, one of the nice things about using forwards and includes and
whatnot, is that you don't need to know what's on the other end. At
least, opposed to calling a function that returns text or something. It
would be nice if you didn't need to know if it was a WSGI app on the
other end of the URL. From what I've seen Myghty does a lot of these
kinds of calls.

>> It has the entire WSGI implementation in __call__. And actually most
>> of the controller logic is implemented in that method. I haven't had
>> much time to revisit that code -- if I did I'd change things like how
>> events work and some other details (maybe add that wsgi_application
>> method) -- but in the case of the WSGIness, it's essentially the same
>> as Paste WebKit, and I haven't found any problems with that at all
>> (and I've been using that a lot).
>
>
> Yea, makes sense for calling it as a WSGI app. The __init__.py for the
> Pylons project is currently filled with a significant amount of work
> required to setup the Pylons WSGI app, having to do that again for the
> Controller would be pretty clumsy, which is why I wouldn't want Pylons
> controllers to be WSGI apps. Since this goes beyond what Myghty was
> created to do, I'm not sure how or if you'd be able to mingle the
> m.subexec call into an app thats called as a WSGI app from the controller.

There should be ways of dealing with that. Maybe you could do work and
put it in the environment, for instance. Or you could do the work
lazily and cache it somewhere. Or you could put it in a metaclass, and
do stuff to the controller on class creation. There's lots of ways,
depending on the particular issue.

> Also, if your WSGI app does its own dispatching, I'm not sure how the
> nested WSGI app would create templates with proper URL's. Given that
> Routes is handling part of it (and not updating SCRIPT_NAME/PATH), the
> WSGI app thats called from the Controller would be pretty clueless on
> its location, wouldn't it?

Yes; Routes -- or whatever code uses what Routes passes back -- should
put everything up to :controller into SCRIPT_NAME, and the rest in
PATH_INFO; putting everything in SCRIPT_NAME if the controller isn't
specifically specified, I think.

> Since Pylons already setup the Routes config object, using url_for and
> redirect_to inside the migrated WSGI app should sort of work, though if
> something might alter the internal dispatch of the WSGI app... not sure
> what would happen. Since Routes dispatches in a style that doesn't
> correlate with just knocking the first section off the URI, there's no
> way to update the SCRIPT_NAME/PATH_INFO environ to reflect where it is.

It is a little unclear how much should be consumed. On one level it
seems like everything should be put into SCRIPT_NAME, because all of the
URL is used to parse, leaving nothing left. Or, perhaps, you could
define the path info as part of the rule, with a default of ''.

[Ben Bangert]

On Nov 11, 2005, at 1:02 PM, Ian Bicking wrote:

> I guess some of how I deal with this is that I stuff things in the
> WSGI environment, and then pull them back out later. This way I
> can do some work up-front, without having to actually know what I'm
> in front of. I.e., I don't need access to the "real" WSGI app (as
> opposed to the middleware that hides the app), I just need some
> agreement with that app. An example might be internal redirects,
> where a redirector (paste.recursive) puts some stuff in the
> environment, and the application can later use that to do a
> redirect. There's probably others.

Myghty also has an internal redirectory, and m.subexec actually acts
as a sub-request within the Myghty scope. As I'm merely using a
custom Resolver with Myghty to implement the Pylons dispatch style
and integration with Routes, Myghty does step in and assume some
stuff. For example, despite you being able to set the start_response,
due to the way the Controller is implemented, the return value of
m.write() or m.out(). So this does work:

from base import *

class WsgiController(BaseController):
def index(self, **params):
self.r.environ['pylons.r'] = self.r

for line in myWSGIAppFromSomewhereElse(self.r.environ,
self.r.start_response):
self.m.write(line)

def myWSGIAppFromSomewhereElse(environ, start_response):
start_response('200 OK', [['Content-type','text/plain']])

yield str(environ['pylons.r'].environ)
yield 'Yup, I work'

A little adaptation layer can be built for the controller so it more
fully populates the environ, and that should do the trick, no?

Now that I'm actually seeing this, it'd only take a few lines of code
in the myghtyroutes stuff to test the Controller for a .wsgi_app
attribute or something, and call it as such in that case. Would that
be preferable?

>> Yea, makes sense for calling it as a WSGI app. The __init__.py
>> for the Pylons project is currently filled with a significant
>> amount of work required to setup the Pylons WSGI app, having to
>> do that again for the Controller would be pretty clumsy, which is
>> why I wouldn't want Pylons controllers to be WSGI apps. Since
>> this goes beyond what Myghty was created to do, I'm not sure how
>> or if you'd be able to mingle the m.subexec call into an app
>> thats called as a WSGI app from the controller.
> There should be ways of dealing with that. Maybe you could do work
> and put it in the environment, for instance. Or you could do the
> work lazily and cache it somewhere. Or you could put it in a
> metaclass, and do stuff to the controller on class creation.
> There's lots of ways, depending on the particular issue.

If the above solution solves your guys desire to call a WSGI app, it
sounds fine to me. Caching, metaclass, and that stuff to create
essentially two WSGI apps when its not actually necessary seems a
little... unnecessary. Could also use some method introspection to
see if the __call__ for the Controller is expecting environ/
start_response (thus is WSGI) and call it like that if it is. That
way you could mingle the WSGI controllers in with normal Pylons ones
without any hassle.

> Yes; Routes -- or whatever code uses what Routes passes back --
> should put everything up to :controller into SCRIPT_NAME, and the
> rest in PATH_INFO; putting everything in SCRIPT_NAME if the
> controller isn't specifically specified, I think.

Not really possible though, a Route might not have :controller in the
path info at all, like here:
m.connect('archives/:year/:month/:day', controller='article',
action='view', year=2005, month=None, day=None)

That's pretty common in blog and CMS's, and action isn't in the path
either. These are still basic examples too.

> It is a little unclear how much should be consumed. On one level
> it seems like everything should be put into SCRIPT_NAME, because
> all of the URL is used to parse, leaving nothing left. Or,
> perhaps, you could define the path info as part of the rule, with a
> default of ''.

Yea, thats the only thing that would make sense, is to clear out the
whole thing. Not sure how a WSGI app would function under such a
condition though, since it no longer has any clue where it is. If you
wanted to have some URL space usable for the WSGI controller, I could
see a route like this:
m.connect('blog/*url', controller='wsgiblog', action='index')

Then the WSGI controller could then use *url as the PATH_INFO. Maybe
stick all the Routes args under environ['pylons.routes.defaults'] or
something.

How's that look?

- Ben

[Ben Bangert]

On Nov 11, 2005, at 1:43 PM, Ben Bangert wrote:

> Yea, thats the only thing that would make sense, is to clear out
> the whole thing. Not sure how a WSGI app would function under such
> a condition though, since it no longer has any clue where it is. If
> you wanted to have some URL space usable for the WSGI controller, I
> could see a route like this:
> m.connect('blog/*url', controller='wsgiblog', action='index')
>
> Then the WSGI controller could then use *url as the PATH_INFO.
> Maybe stick all the Routes args under environ
> ['pylons.routes.defaults'] or something.

I meant that another WSGI app under a controller could then use
that.... anyways, hopes that does the trick.

Btw, I'm heading out of town for the weekend, so no more replies for
me till Sunday. :)

Cheers,
Ben

[Ian Bicking]

James Gardner wrote:
> The current SVN version doesn't work unless you add import
> myghty.exception to the __init__.py file, but wow, the error handling is
> awesome! Especially when often you can do||* print*
> inspect*.*getsource*(*self*.*index*) *||to see straight away where
> you've gone wrong. What would make this perfect is implementing cursor
> up and cursor down too :-)

Implemented!

[James Gardner]

Ian Bicking wrote:

>James Gardner wrote:
>
>
>> What would make this perfect is implementing cursor
>>up and cursor down too :-)
>>
>>
>Implemented!
>
>

Absolutely perfect! Many thanks.