The CherryPy/Middleware debate
Uche Ogbuji
Just wanting to drag the debat beyond IRC and Weblog comments.
See Ian Bicking on writing a Paste module for CherryPy
http://blog.ianbicking.org/pastifying-cherrypy.html
He links to an IRC log that teases out some of Robert Brewer's thoughts
about multi-app deoplyment in CherryPy.
http://www.defuze.org/oss/cpirc/%23cherrypy.2006-01-11.log.html#t2006...
Then see PJE's response to Ian decrying CherryPy's behavior:
http://blog.ianbicking.org/pastifying-cherrypy-comment-1.html
My perspective is that I'm sympathetic to both sides of the argument,
and would love to see some sort of compromise. This thread is just a
hope to trigger such discussion.
--
Uche Ogbuji Fourthought, Inc.
http://uche.ogbuji.net http://fourthought.com
http://copia.ogbuji.net http://4Suite.org
Articles: http://uche.ogbuji.net/tech/publications/
Ian Bicking
> [Moved from cherrypy-users...]
>
> Just wanting to drag the debat beyond IRC and Weblog comments.
>
> See Ian Bicking on writing a Paste module for CherryPy
>
> http://blog.ianbicking.org/pastifying-cherrypy.html
>
> He links to an IRC log that teases out some of Robert Brewer's thoughts
> about multi-app deoplyment in CherryPy.
>
> http://www.defuze.org/oss/cpirc/%23cherrypy.2006-01-11.log.html#t2006...
>
> Then see PJE's response to Ian decrying CherryPy's behavior:
>
> http://blog.ianbicking.org/pastifying-cherrypy-comment-1.html
In CherryPy's defense, it's not that uncommon what it's doing. Webware
wasn't that different -- and I even added a URL parsing system to
Webware that is architecturally similar to CherryPy in some ways (based
on a singleton object). Unfortunately it's also problematic; from the
IRC log I get at least Robert's feeling that Paste should adapt to
CherryPy -- but that's really not feasible, because in doing so Paste
there would be a cascade of problems related to supporting any other
frameworks, and that starts to make Paste pointless. Of course hacks
are possible -- and I'd classify CherryPaste as a hack -- but I strongly
suspect that many applications will get around that hack in ways that
would break CherryPaste, and that a developer combining Paste and
CherryPy documentation is unlikely to stumble upon the safe way to use
the two together.
But I don't want to change what I see as the essential aspects of
CherryPy. The URL parsing would remain, it would just have a
configurable and dynamic starting point instead of a singleton. The
request remains, filters remain (and even if they serve the same purpose
as WSGI middleware, that doesn't meant hey have to be destroyed or
anything). On this API -- a fairly simple refactoring of the current
workings, from what I can tell -- it should be quite easy to build the
current API that works from cherrypy.root. It's really just a single
case of the more general case of a configurable root.
So really what I'm hoping for is a sort of in-between API -- the current
API continues working fine, but many applications written to this API
may not work properly with Paste, because it allows/encourages some
particular practices that are problematci. Then another API becomes
available with a more declarative application instantiation. By
"declarative" I mean that an application indicates its root, it doesn't
insert itself anywhere. This does start to devalue CherryPy as an
application server, but frankly that's the boring side of web
frameworks. I'm just trying to do the application server part in a
flexible and simple way in Paste, it's not that I think it's a very
interesting problem myself (though it is a serious usability problem).
--
Ian Bicking / ia...@colorstudy.com / http://blog.ianbicking.org
Robert Brewer
> So really what I'm hoping for is a sort of in-between API
> -- the current API continues working fine, but many
> applications written to this API may not work properly
> with Paste, because it allows/encourages some
I don't think that's feasible. If CherryPy is going to move to multiple
roots (as the multiapp branch or CherryPaste implementations do it),
then it needs to be called "CherryPy 3.0".
> Of course hacks are possible -- and I'd classify CherryPaste
> as a hack -- but I strongly suspect that many applications
> will get around that hack in ways that would break CherryPaste,
> and that a developer combining Paste and CherryPy documentation
> is unlikely to stumble upon the safe way to use the two together.
Absolutely correct, which is why I don't think it's worth anybody's time
to connect Paste and CherryPy 2.x. I can easily imagine a large group of
CP 2 users also using Paste, and I'm fairly sure none of us (Ian
included) wants to support such hacks or their workarounds. If we
"support" Paste in the 2.x line, it must be in the style that's in trunk
HEAD, and we live with the limitations* for a while. If we decide to do
true multi-roots, then it's got to called 3.0.
In other words, I don't see an opportunity for a "compromise" at this
point, only wise version-management. It's unfortunate that the Paste
challenge has arisen now (so soon after CP 2.1) and is pushing forward
so quickly; we will probably lose users because of it who can't handle
the rapid pace of development.
As an aside, one way for Paste to use CherryPy 2.x would be for Paste to
support spawning a different process for each CP app.
Robert Brewer
System Architect
Amor Ministries
fuma...@amor.org
* As Ben and I discussed on IRC, the only serious limitation with the
current code seems to be that you can't host multiple apps on the same
root (a corner case which I think is survivable until we move to CP 3).
Ian Bicking
>>Of course hacks are possible -- and I'd classify CherryPaste
>>as a hack -- but I strongly suspect that many applications
>>will get around that hack in ways that would break CherryPaste,
>>and that a developer combining Paste and CherryPy documentation
>>is unlikely to stumble upon the safe way to use the two together.
>
>
> Absolutely correct, which is why I don't think it's worth anybody's time
> to connect Paste and CherryPy 2.x. I can easily imagine a large group of
> CP 2 users also using Paste, and I'm fairly sure none of us (Ian
> included) wants to support such hacks or their workarounds. If we
> "support" Paste in the 2.x line, it must be in the style that's in trunk
> HEAD, and we live with the limitations* for a while. If we decide to do
> true multi-roots, then it's got to called 3.0.
I don't see how multiple roots requires any backward incompatible
changes, so long as it is opt-in -- i.e., applications probably have to
be changed to use multi-root, but applications that aren't changed will
work fine in exactly the way they work right now. That seems useful to
me, but also fulfills CherryPy's backward compatibility requirements.
Anyway, at this point I mostly want CherryPaste to work with TurboGears,
which is more constrained than CherryPy in general. That might require
some under-the-cover hacks (under the cover of TurboGears at least), but
I think should be reasonable in CP 2.2.
> As an aside, one way for Paste to use CherryPy 2.x would be for Paste to
> support spawning a different process for each CP app.
I would be interested in supporting that, and would certainly welcome
any code related to that. It's not something I feel that comfortable
writing myself, as I haven't done a lot of coding like that in the past,
so I'm not sure how to move forward on it.
I would envision such code looking like:
def spawn_wsgi_app(initialize_command, communication_parameters): ...
Where on startup it runs initialize_command to start the subserver (or
maybe something with fork), and communicating via some channel, though
I'm not sure what kind of channel -- a pipe, named socket, network
socket, http, ...? Some information can't be passed to subprocesses, of
course, like callbacks. But I would definitely want at least CGI-style
environmental variables to work (e.g., REMOTE_USER), so if it went
through HTTP it should have some trusted way to serialize non-header
environmental variables through HTTP headers, and not lose things ike
HTTP_HOST, SCRIPT_NAME, and PATH_INFO -- i.e., if it is an HTTP proxy it
should be a really clean HTTP proxy. But everything else (i.e., all
non-capitalized environmental variables) can be thrown away. WSGI does
allow for that, and the middleware in Paste tries to act reasonably in
that case.
Uche Ogbuji
> Ian Bicking wrote (in a different order):
> > So really what I'm hoping for is a sort of in-between API
> > -- the current API continues working fine, but many
> > applications written to this API may not work properly
> > with Paste, because it allows/encourages some
> > particular practices that are problematic.
>
> I don't think that's feasible. If CherryPy is going to move to multiple
> roots (as the multiapp branch or CherryPaste implementations do it),
> then it needs to be called "CherryPy 3.0".
I don't think that anyone has insisted that such changes must be in the
2.x branch. I certainly think it should be in 3.0. Maybe the 2.x vs.
3.0 issue is a red herring? There is no reason why work cannot start
on a 3.0 branch while it continues on 2.x.
And let's not be *too* sanctimonious about the backwards compatability
issue. From my plain user's perspectivem the move from 2.0 to 2.1
broke just about everything, and teh move from 2.1 to 2.2 broke a great
deal. I've just sucked it up and fixed things. I agree more stability
would be great within 2.x, but CherryPy is already not doing such a
great job of that.
Uche Ogbuji
> And let's not be *too* sanctimonious about the backwards compatability
> issue. From my plain user's perspectivem the move from 2.0 to 2.1
> broke just about everything, and teh move from 2.1 to 2.2 broke a great
> deal. I've just sucked it up and fixed things. I agree more stability
> would be great within 2.x, but CherryPy is already not doing such a
> great job of that.
I should say "the move from 2.1 to 2.2 is breaking a great deal", since
2.2 is not yet out, but I understand that trunk is currently on the
road to 2.2, so my basic point remains.
Kevin Dangoor
>
> Uche Ogbuji wrote:
>
> > And let's not be *too* sanctimonious about the backwards compatability
> > issue. From my plain user's perspectivem the move from 2.0 to 2.1
> > broke just about everything, and teh move from 2.1 to 2.2 broke a great
> > deal. I've just sucked it up and fixed things. I agree more stability
> > would be great within 2.x, but CherryPy is already not doing such a
> > great job of that.
>
> I should say "the move from 2.1 to 2.2 is breaking a great deal", since
> 2.2 is not yet out, but I understand that trunk is currently on the
> road to 2.2, so my basic point remains.
From what I understand, breakage between 2.1 and 2.2 is considered a
bug. I just shifted TurboGears to 2.2 and the only breakage I saw was
with my monkeypatches (which went away anyhow) and my test helper code
which was using CherryPy internals.
Kevin
Uche Ogbuji
Hmm. Certainly the code I migrated from 2.1 to 2.2beta was some of the
earliest I wrote with CP, and it's possible that I was doing all kinds
of dumb things, but when I mentioned to Robert on IRC that a migration
document was needed, he didn't seem to indicate that no code migration
should have been necessary.
Most of my changes had to do with import paths for filters and
differences in how positional parameters work (I used a lot of exposed
functions with foo(self, *arg).
If I'd known such changes were not to be expected, I'd have kept better
track of my travails for a ticket. As it was, I just hacked and hacked
until it all worked again (mostly having to browse the CP internal code
and squint at changesets to figure out how things stood in 2.2beta
because there was no migration guide--understandable for such an early
use of the 2.2. branch).
Sylvain Hellegouarch
> Hmm. Certainly the code I migrated from 2.1 to 2.2beta was some of the
> earliest I wrote with CP, and it's possible that I was doing all kinds
> of dumb things, but when I mentioned to Robert on IRC that a migration
> document was needed, he didn't seem to indicate that no code migration
> should have been necessary.
>
> Most of my changes had to do with import paths for filters and
> differences in how positional parameters work (I used a lot of exposed
> functions with foo(self, *arg).
>
> If I'd known such changes were not to be expected, I'd have kept better
> track of my travails for a ticket. As it was, I just hacked and hacked
> until it all worked again (mostly having to browse the CP internal code
> and squint at changesets to figure out how things stood in 2.2beta
> because there was no migration guide--understandable for such an early
> use of the 2.2. branch).
As a developer of CP, I really feel sad about failing at keeping documentation
up to date (not just the api, but also tips and tricks). I know that I'm a
fairly eager to release CP 2.2 but I really fera the state of our
documentation. 2.2 might well last as being the stable version for a long time
if we move soon to the CP 3.0 branch, so we do need to make sure its doc is
clear and comprehensive. I'm not sure that is where we are :/
- Sylvain