While I mentioned some of my initial objectives in an email with  
James, I thought it might help myself and anyone interested to  
clarify and refine the purpose of Pylons. I've been doing a bit of  
Rails development lately, and carefully reading many docs on Rails,  
so I know what aspects of the system are useful and would be  
productive to bring to Python web development.

Rather than a clone of Rails, it would be much more useful to take  
successful concepts it introduced and utilize them where applicable.  
It also makes sense to clearly define the target audience for such a  
framework as well since there are already a dozen or so moderately  
used frameworks available.

First, Pylons aims to be a mega-framework, styled somewhat similarly  
to TurboGears, such that re-use of existing projects is utilized  
where appropriate to avoid duplication of effort and to build on the  
success of other projects. Second, Pylons aims to fulfill my own  
development needs which I've found cross almost exactly with James  
Gardner's needs as well which is why we're collaborating on this  
project. Now for a leap slightly off left, but highly relevant.

There is a *huge* amount of contract work out there nowadays. Many  
projects are small to moderate in size (or can be broken into this  
size), and many contract projects might involve very little actual  
coding and mainly integration of other webapps. Many clients have  
similar needs, or want to utilize common sets of components into a  
cohesive whole for their company site. There's also many companies  
supplying niche webapp software meant for integration, almost always  
using PHP right now as its easy to deploy.

This is the first niche field Pylons is aimed at meeting. By  
leveraging Python Paste, setuptools, WSGI, Routes, and Myghty; Pylons  
aims to make web development easier for webapps intended to be  
deployed into other integrated websites, and deployed in scenarios  
where it could be common to run 30+ instances of the same webapp only  
with different configurations (blogs, forums, wiki's, shopping carts,  
form packs, etc.).

Pylons current re-use utilization:
  - Python Paste
        - Structure generation, for controllers, unit tests, project  
directory structure, etc.
        - Testing for Controllers
  - setuptools
        - Distribution through egg's
        - Dynamic discovery for plug-in frameworks
  - WSGI
        - Middleware for use with other WSGI
        - Ease of Deployment
  - Myghty
        - Revamped dispatch scheme through custom Advanced Resolver
        - Advanced and extremely quick templating
  - Routes
        - Semi-programmatic routing of URL's to controller actions
        - URL generation for use in templates

Currently being debated for use in Pylons to meet the requirements of  
a 'full-stack' framework:
  - SQLObject, SQLAlchemy, python.web ORM
  - Mochikit, Dojo, Prototype, Scriptaculous (one of them likely)
  - CRUD stuff possibly utilizing...
         - TG inspired form-generation (there's several), Aquarium FormUtil

Since re-use is preferred where possible, SQLObject is tempting,  
especially as TurboGears is fueling so much development of SQLObject
+FormEncode projects. For now, we will probably wait on determining  
which ORM to officially include until we see what TG produces.

The immediate short-term focus for Pylons:
- Complete Test framework for fully testing Controllers, using  
breakpoints, console troubleshooting, mock objects, etc.
- Heavy Structure generation for Controllers, unit tests, etc.
- Documentation for development, distribution, deployment, and  
integration
- Authentication and Identity code (modeled on Bricks and/or TG  
identity decorators)
- Helper functions (styled on Rails helpers when applicable)

Lessons borrowed from Rails:
- Convention over configuration, or more precisely, everything comes  
with a default setting. All docs assume the defaults are used.  
Advanced users still have the ability to go in and tweak things on  
their own.
- Structure Generation is good. Making it easy for the user to follow  
the defaults by generating them helps.
- Testing of Controllers and Models

Sample use-cases of Pylons:
- Making a blog that automatically finds and uses plug-ins and themes  
installed system-wide
- Easily adding two dozen slightly different configurations of a web  
application
- A Login WSGI Middleware application that dynamically finds new  
login schemes available

Obviously, many of the features are available merely be leveraging  
Paste, WSGI, and setuptools to the max. These are not terribly  
complex, but typically poorly understood Python tools. Making their  
use easy and obvious for developers will be a challenge and extensive  
documentation and examples of each of the 3 use-case apps will be  
needed.

That's where we stand currently on Pylons, I'm working on using the  
Paste tools for structure generation and controller unit testing, and  
James is writing a CMS-type webapp with Pylons as it stands so far to  
get a better idea for ways it should expand.

Everyone on this small list has quite a bit of experience with at  
least one (but usually more) web frameworks, and has created their  
own web framework. As such, I'd be thrilled to hear feedback on this.

Cheers,
Ben

Hi Ben,

+1 on all of this. Spot on!

James

Incidentally, I committed something for debugging into Paste yesterday,
and I'm planning on polishing up the UI today.  Maybe make a screencast
too.  Anyway, you can test it out with a deployment file like:

[server:main]
use = egg:PasteScript#wsgiutils
port = 9999

[filter-app:main]
use = egg:Paste#evalerror
next = filterthis

[app:filterthis]
use = egg:PasteScript#test

Then go to http://localhost:9999/?error=true; click on the [+]'s, and
you can evaluate expressions in the local frames of the tracebacks.  You
can't continue, but then continuing isn't as important in a web app,
since you can always just restart.  Actually, maybe just adding a
"re-request" button to the top would be good.

--
Ian Bicking  |  i...@colorstudy.com  |  http://blog.ianbicking.org

On 11/6/05, Ben Bangert <b...@groovie.org> wrote:

Have you taken a look at PHP Horde yet?  It sounds like it matches
much of this email.

By the way, what's wrong with TurboGears and Subway, and how do you
envision Pylons will be better?

I still don't understand this.  For instance, why would you run 30
instances of a blog application when you can run a single instance
that can have 30 blogs?  Why not just run one good collaboration
application that supports all the different forms of callaboration you
want?  Why not write one app that supports plugins for the different
forms of collaboration, which is what Horde is, as far as I can tell.

After a lot of reading, research, and a demo application, I recommend
Dojo.  Don't be afraid to use more than one JavaScript library.
There's no need to limit yourself.

Happy Hacking!
-jj

--
Hacking is to climbing Mt. Everest as
software engineering is to building a Denny's there.

On Nov 7, 2005, at 8:56 AM, Shannon -jj Behrens wrote:

Yep, though I haven't been too impressed with most of the technical  
decisions it made, mainly as a result of using PHP as the language.  
It's limitations are very obvious throughout Horde. So mainly I'm  
looking at other Python projects and Rails for useful ideas and  
concepts.

I can't stand TAL based template languages, Kid is better, but still  
not as full featured as Myghty for templating. This is mainly a  
personal opinion though as many people seem to have a strong reaction  
either for or against TAL-styled templating.

TurboGears and Subway both utilize CherryPy, and Routes integration  
with CherryPy won't be possible without patches to core CherryPy  
itself until they turn their resolver methodology "inside out" as  
Myghty's is. CherryPy was also lacking the necessary ability to have  
multiple webapps in the same process, though a recent patch has  
resolved that issue. Version 2.2 or 2.3 of CherryPy aims to turn the  
resolver mechanism inside-out allowing people to more fully customize  
the dispatch style as Pylons does with Myghty.

There's also a nice set of features that one gets by using a  
Controller that is more aware and integrated with the View part of  
the framework. For example, in Rails, ActiveRecord can be used  
completely independently of everything else. I've used ActiveRecord  
in stand-alone Ruby scripts without a problem. The rest of the  
framework is much more tightly coupled, which becomes useful when  
doing some rendering activities that require the View to be aware of  
the Controller to an extent.

I've offered to add Myghty templating to TurboGears, but it has no  
chance of being included in the core as Kevin wants to maintain a  
single templating language. That's understandable as we have no  
intention of supplying other built-in templating choices with Pylons,  
it makes things easier to include only one template language.

Several of the issues I cited here are mainly just personal tastes,  
and a few are technical limitations of CherryPy (which they are  
working on fixed). I'm not sure Pylons will be 'better', but it will  
be aimed at a different niche. Neither of those frameworks were  
created to target the same niche (which as you mentioned, Horde  
fulfills for PHP).

Other differences:
- Testing from the beginning, not added on later (TG has TestGears,  
which is pretty cool though)
- Structure generation via PasteScript (TG makes a directory  
template, not sure how far Subway goes in this aspect)
- Extensive documentation and example applications (This generally is  
an after-thought or comes later for TG, Django, and Subway)
- Pythonic (For example, rather than use CherryPy's .expose  
attribute, use the Python convention of _method for private methods)

But really, as I mentioned, we're aiming for a different niche. :)

Because sooner or later, the complexity inherent in making massive  
universal platforms results in something like Zope. Rather than  
building up to such grand schemes, it makes as much sense to build  
the opposite way. Using Paste for easy deployment, it becomes easy to  
quickly throw up 30 instances of some webapp and its easier to  
develop a single blog webapp than a super blogging platform. By not  
coupling an individual blog into a massive platform, its easier to  
mingle in with other webapps into a cohesive site that is just what  
the client ordered.

While my examples were focused on some of the obvious webapps,  
consider other ones that aren't so content-editing sensitive.  
Shopping carts, membership subscription software, etc. It makes  
little or no sense to build super platforms for each of these, and  
developing them would be significantly more time intensive.

Cool, I'll take a look into Dojo some more. It'd be nice to have  
effects and some common things wrapped up in helper functions like  
how Rails does it. While Rails won't make something as cool as  
aquajax, it can handle some nice little AJAX things without burdening  
the user with Javascript.

Cheers,
Ben

Well, Subway does use Cheetah, which isn't TALish, and has some features
similar to Myghty.  Actually, reading your post on Myghty, several of
the features Myghty has that Cheetah doesn't don't look too hard to
implement -- like implicit inheritance, %init, etc.  Maybe even with
just a Cheetah.Template.Template superclass.

TG is using Paste Script in 0.9, though with a turbogears-admin wrapper.

I actually like .expose; I think forcing internal naming conventions to
express publicness is a little unclean, because it conflates "public to
the rest of the program" and "public to the world".  Though another
alternative to .expose is a naming convention, like action_*, which
means "public to the world".

Another option is building "explicitness" into the routes definitions.
I guess it could be similar to tainted strings.  When a string is
exactly matched by a route, the target is explicitly made public.  When
you get something that simply matches :action, then it's only implicitly
public.  Of course, the implicit :action matching could potentially be
turned into a naming convention.  E.g., a rule that bound a "method"
variable would mean an explicit match, and a rule that bound an "action"
variable would be interpreted by the resolver as meaning "action_*".
That's probably what I'd be inclined to do.

Personally I'd be interested in a very open sort of framework.  I.e.,
one that keeps the parts separate, since I would like (for myself) to
make an easy sort of transition from old code to new.  So a route would
identify a controller, which would in turn be a WSGI application
(probably with adaptation of some sort).  With a different level of
abstractness, a controller produces variables; these variables can get
rendered in a template, sent via JSON, etc. -- I think that's a clever
part of TurboGears.  But the controllers don't really have to care that
much about what kind of template they are sending to.  Dynamically
scoped template providing might be interesting here.

Exactly how you brand the combination or individual pieces is another,
separate issue.

Personally I think MochiKit is better than Dojo at the things MochiKit
does -- it's well documented, easy to integrate, no fancy tools or build
scripts required.  But Dojo seems quite a bit more ambitious.  Luckily
they feel similar enough that the conceptual overhead of using both at
once doesn't seem too bad.

--
Ian Bicking  /  i...@colorstudy.com  /  http://blog.ianbicking.org

On Nov 7, 2005, at 10:18 AM, Ian Bicking wrote:

Yea, that is true. However the features I mentioned on that post were  
just the beginning, the component abilities go deep into the entire  
system. Re-implementing it all is possible, and some features would  
be easier than others, but that doesn't solve the problem of deep  
Routes integration with the Controller.

Excellent, I'm hoping to see some tasks in Paste Script get easier as  
more people use them and contribute patches.

True, though generally in the context of a Controller, public to the  
rest of the program and the world are equivalent. I can't really  
imagine any reason to stick functions that are Controller private  
inside a Controller class unless the intention is also to have it  
private to that class (ie, for use only with that Controller). If it  
isn't Controller private, and is intended to be called from elsewhere  
in the program it typically belongs in a separate library module, or  
at least its own class.

Ruby dodges this problem to an extent by having two realms of  
'private'. They have 'protected' and 'private', and stick the non-
exposed actions under 'protected', though 'protected' in Ruby is  
equivilant to '_prefix' practically speaking.

Hmm, not sure what you mean about the matching. Perhaps an example?

Yea, that is pretty nice the way TG is bundled up. I've been  
considering something like that for Pylons as well, and it wouldn't  
be too difficult, just need to have Myghty dump the args it gets  
instead of rendering them. Probably using the same .execute intercept  
I plan on using when testing the Controller to ensure templates were  
passed the right vars. It'd be pretty easy to hook-up a system  
whereby people could make their own controller filter, like (I'm not  
a JSON expert, so I doubt this is right):

m.connect('*url/jsonify', controller='filter', action='json')

class FilterController(BaseController):
        def json(self, m, r, **params):
                r.content_type = 'text/xml'    # Or whatever is needed
                args = m.subexec(url, capture_args=True)
                m.out(jsonify(args)

There's no capture_args option right now, but I need to add a wrapper  
for testing anyways and I could make it easy to trigger like that.

I'm not sure Routes would work right when dealing with dispatching to  
entire WSGI apps instead of just a Controller, though I don't see why  
a Filter middleware of some sort couldn't use Routes to do that. I  
kind of lost you after that though when it comes to the WSGI app,  
adaptation, etc.

I agree that a light coupling of the parts is ideal, though as I  
mentioned before it becomes more useful to have the template and  
controller somewhat aware of each other for certain functionality.  
The only bounds Pylons so far has when it comes to tying the  
Controller to the template is through the magic temp attribute of the  
controller (though it could be used easily enough without a template).

I'm still not totally happy with this solution, but it does make it  
easier than passing all the variables into the template directly. As  
we're using a stateless, persistent controller things can't be  
attached to the Controller instance. Which leaves 2 options to move  
variables into the template,
1) Pass them explicitly (as TG does)
2) Have a special thread-safe variable that's cleared on every  
request, and available in the template

We're using option #2 right now, though having a magic container as a  
controller instance attribute thats available in the template might  
cause confusion. James' tutorial (http://pylons.groovie.org/project/
wiki/JamesNotes) does do an excellent job of clarifying its use.  
While they could be passed everywhere, that becomes a hassle in some  
cases. For example, if you had some auth code that needed to be run  
before 30 actions, but not for 2 of the ones in a controller, in  
Rails you put inside your class:
    before_filter :authorize_user, :except => ['login', 'logout']    
# Call authorize_user action, unless its the login or logout method

In Pylons, we're having a __before__ method that can be used to  
implement similar functionality.

Now, this ties back to how variables are moved around, because we  
need a way to get variables from __before__ into the actual action  
method thats called. Having a self.temp that can be used is very  
handy in this case, and others where you need to store variables for  
later use without relying on trying to manage some dict yourself and  
pass it all over the place. So yea... I'm somewhat torn, I know I  
want to be able to whisk a magic container around without explicitly  
passing it (thread-safe singleton or magic object attached to  
controller instance), yet I'm not too keen on the current 'self.temp'  
stuff I put in.

If there's some way to still do this, but in a more clear-cut way,  
I'd love to hear some ideas. :)

The odd thing about holding the variables intended for the template,  
is that I find myself typically setting up variables with the  
specific intention of being used in a rendered template. The utility  
of being able to return them in JSON is somewhat perplexing in much  
of my code since I'm passing complex objects that wouldn't be useful  
as JSON. I'd actually code completely separate functions if I needed  
specific data back as JSON, just as I'd code special functions for  
SOAP/XML-RPC services (though maybe those two are likely to have a  
more similar dataset).

- Ben

On 11/7/05, Ben Bangert <b...@groovie.org> wrote:

(playfully)
So we have a case of a templating engine looking for a meta project ;)

Yes.  You must think of CherryPy as an object publisher.  Routes
simply doesn't make sense for CherryPy without changing the core of
what CherryPy is.

That is understandable from his point of view.  I think it makes sense
for you to make it possible to use Myghty in TurboGears.  If I had
free time, I'd want to make it easy to use Myghty in Aquarium.

That is unfortunate.  I really do think templating engines are a
matter of taste, so having one's framework dictated by one's choice of
a templating engine seems limited.

I'm not buying that argument ;)

While I respect your opinion, your desire to have 30 instances of the
same Web app reminds me of Ian's (hi, Ian!) obsession for middleware
;)  I'm smart enough to know that I don't understand everything, and
I'm sure you have your reasons ;)

Given.

Happy Hacking!
-jj

--
Hacking is to climbing Mt. Everest as
software engineering is to building a Denny's there.