Werkzeug req-resp/WebOb integration
Chris McDonough
and response methods and properties map to their Werkzeug analogues.
The results are at:
https://bitbucket.org/chrism/webob-py3k/src/c699da0e325f/request_table.rst
https://bitbucket.org/chrism/webob-py3k/src/c699da0e325f/response_table.rst
I spoke briefly with Armin about some sort of try at collaboration, and
he seemed amenable. We both seem to believe that some sort of merge
here would be beneficial to the Python web world.
The req/resp APIs of WebOb and Werkzeug are very similar but obviously
not identical. So I doubt we can do a big-bang-break-everyone merge.
We need to provide some period of backwards compatibility during any
merge.
A strategy that I favor currently is to implement a third package upon
which WebOb and Werkzeug would begin to depend. For sake of brevity, I'm
currently calling that package "Wub"). Wub would implement a superset
of the APIs required by both Werkzeug and WebOb users. We could then
change both Werkzeug and WebOb to depend on Wub (deleting most of the
code from each). Werkzeug and WebOb would then implement their current
APIs unmodified, but we'd start issuing deprecation warnings for
functionality accessed by users that was not in Wub itself. Eventually,
we'd retire WebOb and Werkzeug req/resp and come to depend on Wub only.
Just my thoughts,
- C
Sergey Schetinin
but a reaction to the idea of unifying with "some other" library.
Currently WebOb has no external dependencies and is reasonably
minimal. I was thinking it would be nice to have yet more minimal core
as part of webob itself, say webob.mini. Currently I actually "hack
away" parts of webob in subclasses to make it do less -- no
multidicts, POST being a simple dict and not a view on input stream,
things like that. It would be nice to be able to just inherit from a
more basic class instead of doing that. I'm not sure how many people
care about this and I'm not sure how much disruption this might bring
to projects that depend on webob, so I left it at that -- just an
idea.
But if there were some unification with Werkzeug, I would really
prefer to see a shared dependency which is a shared subset of the
libraries, not their superset.
Also, I know repoze doesn't mind a lot of small packages depending on
each other, but for webob getting a dependency would be a big thing
too. I feel that while webob appeals to developers that want to write
lean code it's already 'too big' for some of them, or at least gives
that impression. Spreading it out in this way, might be a bad idea for
that reason.
I understand that the idea is to eventually deprecate both libraries
and therefore Wub has to be a superset, but I'm skeptical that can
ever happen. If anything, a more realistic goal is to deprecate just
one of WebOb / Werkzeug and add the missing features to the other one,
but I don't see that happening either.
I think there's a great chance here to waste a lot of time for some
idealistic reasons and end up worse off than before. I'm still open to
the idea, so it would interesting to hear what everybody else thinks.
-Sergey
Chris McDonough
> Disclaimer: I never really looked into Werkzeug, so this is nothing
> but a reaction to the idea of unifying with "some other" library.
>
> Currently WebOb has no external dependencies and is reasonably
> minimal. I was thinking it would be nice to have yet more minimal core
> as part of webob itself, say webob.mini. Currently I actually "hack
> away" parts of webob in subclasses to make it do less -- no
> multidicts, POST being a simple dict and not a view on input stream,
> things like that. It would be nice to be able to just inherit from a
> more basic class instead of doing that. I'm not sure how many people
> care about this and I'm not sure how much disruption this might bring
> to projects that depend on webob, so I left it at that -- just an
> idea.
>
> But if there were some unification with Werkzeug, I would really
> prefer to see a shared dependency which is a shared subset of the
> libraries, not their superset.
This is what I meant by "Wub", yes. Apologies if I didn't make that
clear.
> Also, I know repoze doesn't mind a lot of small packages depending on
> each other, but for webob getting a dependency would be a big thing
> too. I feel that while webob appeals to developers that want to write
> lean code it's already 'too big' for some of them, or at least gives
> that impression. Spreading it out in this way, might be a bad idea for
> that reason.
>
> I understand that the idea is to eventually deprecate both libraries
> and therefore Wub has to be a superset, but I'm skeptical that can
> ever happen.
No, it can be a subset.
> If anything, a more realistic goal is to deprecate just
> one of WebOb / Werkzeug and add the missing features to the other one,
> but I don't see that happening either.
I could also imagine that.
> I think there's a great chance here to waste a lot of time for some
> idealistic reasons and end up worse off than before. I'm still open to
> the idea, so it would interesting to hear what everybody else thinks.
The reasons aren't really idealistic; they're practical. We don't need
two almost identical implementations of request/response objects in
Python. It adds both to the testing and to the porting burden across
major Python releases, and having both to choose from prevents
frameworks which base themselves on different ones from collaborating
more closely. A big reason BFG and Pylons were able to merge into
Pyramid is because they shared a common WebOb dependency and folks were
used to its API.
- C
Sergey Schetinin
>> I think there's a great chance here to waste a lot of time for some
>> idealistic reasons and end up worse off than before. I'm still open to
>> the idea, so it would interesting to hear what everybody else thinks.
>
> The reasons aren't really idealistic; they're practical. We don't need
> two almost identical implementations of request/response objects in
> Python. It adds both to the testing and to the porting burden across
> major Python releases, and having both to choose from prevents
> frameworks which base themselves on different ones from collaborating
> more closely. A big reason BFG and Pylons were able to merge into
> Pyramid is because they shared a common WebOb dependency and folks were
> used to its API.
Yes, I understand the practical benefit, and of course in that sense
this idea is practical. But I feel that this migration might be about
as tough to pull off than getting python 2 deprecated in favor of
python 3. Of course Python has more inertia, but it's also too easy to
just stick to some version of webob or fork it or something of that
kind and end up fragmenting the development even more. What I meant by
idealistic reasons is to do something because it's the right thing to
do, even though it might not work out in the end. Or maybe I'm just
being overly pessimistic.
It would be nice to have a more complete list of frameworks that
depend on either library. For some reason I can't remember much off
top of my head.
WebOb: Pylons, BFG / Pyramid
Werkzeug: Flask
--
Best Regards,
Sergey Schetinin
http://self.maluke.com/ -- My articles and open-source stuff
http://www.maluke.com/ -- My commercial software and custom development services
Chris McDonough
Here's the ones I know of:
WebOb: Pylons, Pyramid, Bobo, Restish, TurboGears 2
Werkzeug: Flask, Glashammer
- C
Ian Bicking
Creating a common superclass in a separate library seems like a choice based on political feasibility rather than any technical (or even community or maintenance) advantage.
Some things I'm aware of:
* Form processing. Werkzeug's is superior to cgi.FieldStorage. It would be great to use Werkzeugs, just like WebOb already uses Werkzeug's cookie parser (via copy-and-paste code sharing)
* WebOb exposes FieldStorage in req.POST with file uploads. I considered doing something with this before 1.0, but ended up punting. I think file uploads could be revisited in WebOb. The only concern I have with a separate files accessor is that you lose field ordering for files then, and I've assiduously maintained full field ordering so I'm partial to it ;) But on the other hand it's clear that a separate attribute is generally preferable in practice, so maybe a full ordering could be maintained in a more off-to-the-side manner.
* WebOb is mutable. My response continues to be: fine; if you don't like it, don't use it. But it really bothers Armin. It does though seem feasible to make WebOb objects freezable, which you could largely accomplish by giving it an immutable environ or immutable headerlist/app_iter objects in the case of a response. Or maybe there's another technique. This doesn't seem like a biggy, unless you are offended by the mere existence of mutability in the codebase.
* Some of the more obscure URL-related attributes on requests are different (e.g., req.path), maybe semantically different, or maybe just a different factoring. I don't know... I think WebOb already has too many to keep track of. Some thoughtful design could maybe lead to a small set we all like, and deprecation of the others.
* Names are actually fairly similar, and I think we can either punt on the naming some (allowing subclasses with name aliases) or work it out one-by-one. I know Armin dislikes req.GET and req.POST, which are acknowledged misnomers which I used because it seemed like "convention" (in early versions they had different names). I don't care; but the current names are *already* a concession to common convention.
* "mimetype" is IMHO also kind of a misnomer as its often used, which is why I avoided it. Some people find "charset" to be a misnomer ("encoding" being more accurate"), but it's right there in HTTP so I decided not to be creative and so to reuse the name.
* Some of the body stuff is deliberately symmetric in WebOb between request and response, which I like.
* I believe Werkzeug does more with Authorization/WWW-Authenticate than WebOb does, which I would offer is possible because of the use of mixins which encourage core pluggability, and parsing and understanding these headers is kind of complicated. I tried not to rely on subclassing in WebOb because I always felt it made other people's subclassing more complicated; I basically wanted to let subclassing belong to the framework and/or user. Maybe there's a way to work out these particular headers -- there's no much innovation going on in these headers anymore anyway (https having largely taken away the desire to get fancy).
* I wouldn't want to rely on lots of use of mixins detracts from the concept of a "common request/response object" because if you use a different set of mixins you have a functionally different object.
I'll also note that Paste was in its time intended to be a common library for building request and response objects, much like Wub is being proposed. And it kind of served that purpose, and it kind of didn't... but I didn't build WebOb on Paste for a reason.
--
You received this message because you are subscribed to the Google Groups "Paste Users" group.
To post to this group, send email to paste...@googlegroups.com.
To unsubscribe from this group, send email to paste-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/paste-users?hl=en.
Chris McDonough
Also, pure numbers of frameworks don't really matter. IMO, we're really
still at the very beginning of a very long road in Python web-land.
Flask/Werkzeug is really great code. Armin is an exceptional
programmer. There aren't many of those, so any way we can encourage
cooperation instead of competition between WebOb and Werkzeug is a +1 in
my book, especially when the API differences between them are so minor.
- C
Sergey Schetinin
> * [...] just like WebOb already uses Werkzeug's cookie
> parser (via copy-and-paste code sharing)
Not really, I started from stdlib cookie parser and refactored it
until it seemed sane.
> [...]
> * I believe Werkzeug does more with Authorization/WWW-Authenticate than
> WebOb does, which I would offer is possible because of the use of mixins
> which encourage core pluggability, and parsing and understanding these
> headers is kind of complicated. I tried not to rely on subclassing in WebOb
> because I always felt it made other people's subclassing more complicated; I
> basically wanted to let subclassing belong to the framework and/or user.
> Maybe there's a way to work out these particular headers -- there's no much
> innovation going on in these headers anymore anyway (https having largely
> taken away the desire to get fancy).
Webob parses them into (type, params) tuple, Werkzeug seems to have
special WWWAuthenticate class (
https://github.com/mitsuhiko/werkzeug/blob/master/werkzeug/datastructures.py#L2157
) which makes it a view on the actual header and adds access to some
of the params as properties. Not sure it's worth it -- there's very
little code to save by having that. I would argue that it saves no
code for user at all.
> I'll also note that Paste was in its time intended to be a common library
> for building request and response objects, much like Wub is being proposed.
> And it kind of served that purpose, and it kind of didn't... but I didn't
> build WebOb on Paste for a reason.
Could you please elaborate on this? I think this is an important
point, but maybe I interpret it differently that what you meant by
this.
Sergey Schetinin
https://github.com/mitsuhiko/werkzeug/blob/master/werkzeug/formparser.py#L38
Here's one I wrote for pasteob
https://bitbucket.org/mlk/pasteob/src/e9a641c1ec6e/pasteob/base.py#cl-332
with minor changes it could be used in webob, there's very little
code.
-Sergey
Mike Orr
>> If anything, a more realistic goal is to deprecate just
>> one of WebOb / Werkzeug and add the missing features to the other one,
>> but I don't see that happening either.
>
> I could also imagine that.
>
>> I think there's a great chance here to waste a lot of time for some
>> idealistic reasons and end up worse off than before. I'm still open to
>> the idea, so it would interesting to hear what everybody else thinks.
>
> The reasons aren't really idealistic; they're practical. We don't need
> two almost identical implementations of request/response objects in
> Python. It adds both to the testing and to the porting burden across
> major Python releases, and having both to choose from prevents
> frameworks which base themselves on different ones from collaborating
> more closely. A big reason BFG and Pylons were able to merge into
> Pyramid is because they shared a common WebOb dependency and folks were
> used to its API.
+1 to unify WebOb and Werkzeug somehow. WebOb has been being adopted
[grammar?] by more and more frameworks, and is the closest thing
Python to a common Request/Response object across frameworks.
For a few years I've been advocating a WebOb-based stack to replace
WSGI, or at least as an alternative because WSGI middleeware is so
clunky to write and maintain that people either avoid writing
middleware or make their middleware use WebOb, which means you're
re-creating Request/Response objects at every level. Why not just pass
a single pair of Request/Response objects through the whole stack?
So having one common Request/Response object pair would not only make
it easier to interoperate and switch between framework packages, but
it would also allow new possibilities across the entire stack.
--
Mike Orr <slugg...@gmail.com>
Jim Fulton
...
> Here's the ones I know of:
>
> WebOb: Pylons, Pyramid, Bobo, Restish, TurboGears 2
also Google App Engine.
Jim
Ian Bicking
On 20 March 2011 01:03, Ian Bicking <ia...@colorstudy.com> wrote:
> * [...] just like WebOb already uses Werkzeug's cookie
> parser (via copy-and-paste code sharing)Not really, I started from stdlib cookie parser and refactored it
until it seemed sane.
Huh, I remember Ben saying he was copying in that code from Werkzeug, but maybe it was something else or didn't get integrated.
> [...]
> * I believe Werkzeug does more with Authorization/WWW-Authenticate thanWebob parses them into (type, params) tuple, Werkzeug seems to have
> WebOb does, which I would offer is possible because of the use of mixins
> which encourage core pluggability, and parsing and understanding these
> headers is kind of complicated. I tried not to rely on subclassing in WebOb
> because I always felt it made other people's subclassing more complicated; I
> basically wanted to let subclassing belong to the framework and/or user.
> Maybe there's a way to work out these particular headers -- there's no much
> innovation going on in these headers anymore anyway (https having largely
> taken away the desire to get fancy).
special WWWAuthenticate class (
https://github.com/mitsuhiko/werkzeug/blob/master/werkzeug/datastructures.py#L2157
) which makes it a view on the actual header and adds access to some
of the params as properties. Not sure it's worth it -- there's very
little code to save by having that. I would argue that it saves no
code for user at all.
Well, I assume you could handle things like base64 encoding username:password, etc. It's not major, but it could be handy.
> I'll also note that Paste was in its time intended to be a common libraryCould you please elaborate on this? I think this is an important
> for building request and response objects, much like Wub is being proposed.
> And it kind of served that purpose, and it kind of didn't... but I didn't
> build WebOb on Paste for a reason.
point, but maybe I interpret it differently that what you meant by
this.
Well, at the time I was writing Paste there was generally less agreement and reuse than there is now in the web framework world. I didn't feel comfortable, or at least confident enough to try to replace what people were using, but thought it would be possible to reuse underlying plumbing while letting frameworks do a kind of value-add on top of that, in the form of aesthetic APIs. So Paste generally tried to be minimal and direct. That might have been the right choice for the time, or not... I'm not really sure. But I don't think it served well as a good way for libraries to interact on a lower level, and I don't think there was ever any uptake to refactor things to use it on a lower level while maintaining higher-level APIs (unless maybe you count Repoze, but that was a late comer).
When I wrote WebOb it was not a refutation of the code in Paste as a concept (at least insofar as Paste overlapped with what WebOb was doing). It's not to say the code was great in Paste, and I was satisfied to rewrite it more carefully, but that wasn't the motivation. Rather I felt clear about exactly what a request and response object should be. It seems relatively obvious, but it wasn't at all clear when I was writing Paste, and even now there are some outliers (e.g., CherryPy I believe still includes methods related to URL traversal in the request, must like Zope).
But anyway, I digress. If we want to move towards a common request/response, to me the point is specifically *not* to embrace diversity. If we have Werkzeug-flavored-Wub and WebOb-flavored-Wub all we've really accomplished is to refactor some code out, and I don't see any real purpose to that. Code is a liability... but it's not that much of a liability. And hey, we even have 100% test coverage! Refactoring could be used as a way to try to defer hard choices while moving forward in terms of infrastructure. But it doesn't seem like it saves any real effort, as changing names and doing deprecations inside WebOb itself is not particularly hard. If modularity itself is a feature we're trying to add then it would be helpful, but modularity is in direct contrast to a common request/response object. To me one of the biggest advantages is that everyone can put "request" in their code, and everyone else will know what that means. If you start splitting it up that becomes unclear.
Ian
Jim Fulton
...
> If we want to move towards a common
> request/response, to me the point is specifically *not* to embrace
> diversity. If we have Werkzeug-flavored-Wub and WebOb-flavored-Wub all
> we've really accomplished is to refactor some code out, and I don't see any
> real purpose to that. Code is a liability... but it's not that much of a
> liability. And hey, we even have 100% test coverage! Refactoring could be
> used as a way to try to defer hard choices while moving forward in terms of
> infrastructure. But it doesn't seem like it saves any real effort, as
> changing names and doing deprecations inside WebOb itself is not
> particularly hard. If modularity itself is a feature we're trying to add
> then it would be helpful, but modularity is in direct contrast to a common
> request/response object. To me one of the biggest advantages is that
> everyone can put "request" in their code, and everyone else will know what
> that means. If you start splitting it up that becomes unclear.
Well said.
Jim
Chris McDonough
> Code is a liability... but it's not that much of a liability. And
> hey, we even have 100% test coverage! Refactoring could be used as a
> way to try to defer hard choices while moving forward in terms of
> infrastructure. But it doesn't seem like it saves any real effort, as
> changing names and doing deprecations inside WebOb itself is not
> particularly hard. If modularity itself is a feature we're trying to
> add then it would be helpful, but modularity is in direct contrast to
> a common request/response object. To me one of the biggest advantages
> is that everyone can put "request" in their code, and everyone else
> will know what that means. If you start splitting it up that becomes
> unclear.
The goals are:
- get rid of code, so that any porting and maintenance effort can be
shared.
- bring at least one more web framework into the set of frameworks that
use the same req/resp code.
If we can do this without the "third thing" that is Wub, that'd be ok by
me. If we're unwilling to create the third thing, then someone just has
to convince Armin to use WebOb in Flask. I'll assume that effort has
not been successful in the past.
- C
Sergey Schetinin
It would be great to know what kind of thing Armin would like Wub to
be and then we can see if WebOb can be that thing.
Chris Withers
> I spoke briefly with Armin about some sort of try at collaboration, and
> he seemed amenable.
Did you ask him if he would mind moving to WebOb if it grew support for
anything he needed that wasn't there? (Kinda like the stuff Pyramid grew
to make people familiar with Pylons more comfortable...)
> We both seem to believe that some sort of merge
> here would be beneficial to the Python web world.
Very strongly agree.
> A strategy that I favor currently is to implement a third package upon
> which WebOb and Werkzeug would begin to depend. For sake of brevity, I'm
> currently calling that package "Wub").
If it's needed, please give it a good name. "Wub" is not a good name.
"WebOb" *is* a good name and maybe Armin would be amenable to a "WebOb
2.0" that supported both what current WebOb users need along with what
Werkzeug needs?
2.1 could then see the deprecations you refer to, with 2.2 seeing the
extraneous methods and interfaces removed, with 3.0 being left for
Python 3 compatibility.
cheers,
Chris
--
Simplistix - Content Management, Batch Processing & Python Consulting
- http://www.simplistix.co.uk
Chris McDonough
> On 19/03/2011 21:09, Chris McDonough wrote:
> > I spoke briefly with Armin about some sort of try at collaboration, and
> > he seemed amenable.
>
> Did you ask him if he would mind moving to WebOb if it grew support for
> anything he needed that wasn't there? (Kinda like the stuff Pyramid grew
> to make people familiar with Pylons more comfortable...)
Nope. I assume that has been tried, otherwise there wouldn't have been
both Werkzeug and WebOb in the first place.
I think maybe Armin's major issue with WebOb is that Werkzeug's req/resp
code really is much cleaner than WebOb's. Now that we have better test
coverage, it's maybe not outside the realm of possibility to clean WebOb
up a bit. And maybe doing so would make a difference. But I'm guessing
really. I'll point Armin here to see if he'd be willing to respond.
- C
Sergey Schetinin
> I think maybe Armin's major issue with WebOb is that Werkzeug's req/resp
> code really is much cleaner than WebOb's. Now that we have better test
> coverage, it's maybe not outside the realm of possibility to clean WebOb
> up a bit.
Actually I've cleaned up a code quite a bit over the past year or two,
what areas do you feel need a cleanup?
Ian Bicking
On Sun, 2011-03-20 at 11:35 -0500, Ian Bicking wrote:The goals are:
> Code is a liability... but it's not that much of a liability. And
> hey, we even have 100% test coverage! Refactoring could be used as a
> way to try to defer hard choices while moving forward in terms of
> infrastructure. But it doesn't seem like it saves any real effort, as
> changing names and doing deprecations inside WebOb itself is not
> particularly hard. If modularity itself is a feature we're trying to
> add then it would be helpful, but modularity is in direct contrast to
> a common request/response object. To me one of the biggest advantages
> is that everyone can put "request" in their code, and everyone else
> will know what that means. If you start splitting it up that becomes
> unclear.
- get rid of code, so that any porting and maintenance effort can be
shared.
- bring at least one more web framework into the set of frameworks that
use the same req/resp code.
Just bringing one more framework into the family doesn't IMHO warrant major refactoring, especially when if it means actually increasing the total number of libraries, and potentially creating a championless underlying library. If we're going to do major refactoring I think it's only worth it if we're shooting for universality. Getting some Django people in the conversation would be nice too; it's too far off to consider what would be involved in Django actually using WebOb, but it would be nice to clear up what we can with that in mind.
If we can do this without the "third thing" that is Wub, that'd be ok by
me. If we're unwilling to create the third thing, then someone just has
to convince Armin to use WebOb in Flask. I'll assume that effort has
not been successful in the past.
We haven't tried that hard. Armin and I talked during PyCon, and while we didn't come to a conclusion there weren't really big blockers. Werkzeug and Flask have a lot more going on than the request/response objects, and if anything this would allow them to concentration more on those parts, to everyone's benefit. I think Armin is also more comfortable with dependencies than he would have been when originally writing Werkzeug.
Ian
Chris McDonough
> On 20 March 2011 21:35, Chris McDonough <chr...@plope.com> wrote:
> > I think maybe Armin's major issue with WebOb is that Werkzeug's req/resp
> > code really is much cleaner than WebOb's. Now that we have better test
> > coverage, it's maybe not outside the realm of possibility to clean WebOb
> > up a bit.
>
> Actually I've cleaned up a code quite a bit over the past year or two,
> what areas do you feel need a cleanup?
- methods with many deeply nested conditionals (not easy to test):
https://bitbucket.org/chrism/webob-py3k/src/c699da0e325f/webob/response.py#cl-63
https://bitbucket.org/chrism/webob-py3k/src/c699da0e325f/webob/response.py#cl-290
https://bitbucket.org/chrism/webob-py3k/src/c699da0e325f/webob/response.py#cl-970
https://bitbucket.org/chrism/webob-py3k/src/c699da0e325f/webob/etag.py#cl-105
https://bitbucket.org/chrism/webob-py3k/src/c699da0e325f/webob/byterange.py#cl-98
- An import-time monkeypatch to the CGI module (maybe not to be helped):
https://bitbucket.org/chrism/webob-py3k/src/c699da0e325f/webob/request.py#cl-1175
- Paste monkeypatching:
https://bitbucket.org/chrism/webob-py3k/src/c699da0e325f/webob/exc.py#cl-1063
- Import time side effects:
https://bitbucket.org/chrism/webob-py3k/src/c699da0e325f/webob/exc.py#cl-1078
https://bitbucket.org/chrism/webob-py3k/src/c699da0e325f/webob/__init__.py#cl-41
Sergey Schetinin
That's a great list actually :)
Chris McDonough
>
> Was there any reaction from Armin BTW?
I spoke to him. I think he may be preparing some sort of design defense
for Werkzeug before commenting.
- C
Ian Bicking
Cool; that could help us focus and discuss specifics, which I think would be a good next step.
Ian
Sergey Schetinin
Also OpenStack.
Was there any reaction from Armin BTW?
-Sergey
Jorge Vargas
Hi,
GothAlice seems to have troubles replying here so she asked me to post
for here, this is a forward of said email.
---------- Forwarded message ----------
From: Alice Bevan-McGregor <al...@gothcandy.com>
Date: Mon, Mar 28, 2011 at 10:11 PM
Subject: Re: Werkzeug req-resp/WebOb integration
To: jorge....@gmail.com
On 2011-03-19 14:09:02 -0700, Chris McDonough said:
> As part of the PyCon sprints, a few of us mapped out how WebOb request
> and response methods and properties map to their Werkzeug analogues.
> The results are at:
>
> https://bitbucket.org/chrism/webob-py3k/src/c699da0e325f/request_table.rst
>
> https://bitbucket.org/chrism/webob-py3k/src/c699da0e325f/response_table.rst
That's a very cool list; handy not just for attempts to merge but also
as a quick reference or porting guide.
I've begun work on a spreadsheet version, better organized, and with
additional information like RFC chapter references. The Response
sheet is by far the most complete thus far, and includes a
transcription of the resopnse table linked above.
This spreadsheet should be editable by anyone, so it'd be great if
someone could add descriptions to the miscelaneous items at the bottom
of that sheet and double-check my transcription. (I noticed WebOb
spelt WebBob in the original, plus some confusion over read/write due
to the order of those columns.)
Additionally, I've added the API from marrow.wsgi.objects, both the
current state of implementation (in black) and goal (in red). I'll
prepare a "defending m.w.object's design" document in the near future,
as well.
I'd have joined this discussion earlier, but the first two messages I
sent went to /dev/null, it appears. Gmane Usenet access + Paste's
mailing list = bad mojo.
— Alice.
Jorge Vargas
Ian Bicking
Probably it should include stuff like form values, query strings, content_type parameters, and uh other things... but I'm too lazy for the moment ;)
Not sure what's up with posting with the group; most first posts go into moderation, but I haven't rejected any messages lately. Maybe posting through the googlegroups web interface would work?
Armin Ronacher
Sorry it took me so long to reply to this thread. Both
university and my game hacking obsession came in the way of
replying earlier to this. I spend some time now on getting
my design defense of Werkzeug written, so without further
ado: here it is.
Werkzeug Recap
--------------
TLDR: I would love to see a new library for both Python 2 and Python
3 which implements functions for parsing cookies, form data, urls and
a bunch of other things as well as a read-only request object and a
read-write response object.
I personally am not too stubborn on having that library my way, I
would just like to present my view the status of Werkzeug and WebOb
that you have some understanding of how I approached Werkzeug and why
it's different to WebOb.
Why Werkzeug
````````````
Why the hell was Werkzeug written in the first place? When I started
working on my blogging engine zine I wanted to start without an actual
framework and had the component architecture of trac in mind. With
that I started working with Paste originally but dropped that plan
along the way because I ended up wrapping all of the Paste utilities
in order to get unicode support going. I found out that I barely need
anything in there and like so many I wrote my own minimal request
object based on the cgi module and a bunch of other things.
For that matter I recycled code I wrote for Colubrid earlier,
thankfully a project that no longer exists (it basically was my
introduction to both Python web development and actual real web
development in general).
Unfortunately at the time I unbundled the code from the unreleased
textpress into a separate package, Ian Bicking already released WebOb
which in many parts is what I was actually aiming at.
With WebOb there and WebOb building up on cgi and a bunch of standard
library stuff as well I went into a different direction with Werkzeug
and removed magic (except for the stupid import hack I wrote) from the
interfaces and instead wrote parsing functions.
I mainly did this because I missed these functions in Django projects
a lot and it was easy to just import the required header parsing
function from werkzeug and using it in combination with the Django
request objects (eg: parsing accept headers and stuff like that).
Over the time a bunch of other things entered the Werkzeug code base:
- secure cookie, a hash protected cookie (unfortunatley with a
terrible implementation to later implementations by other people).
This was based on the ideas I saw in Ruby on Rails and suffers
from using Pickle internally and trying different protocols on a
per-key basis to get the smallest output possible. In hindsight,
urlencoded JSON would have been a smarter choice.
- a bunch of read-only and writable data structures. Among other
things a MultiDict that stores more than one value per key with
the same access times as the standard dict. Later another dict
that additionally keeps the values linked with a double linked
list to preserve the same access complexity for all operations by
also preserving the ordering.
- Basic session support
- A WSGI test application
- A basic WSGI server that fixes some of the problems that wsgiref
has (like emitting the wrong value for wsgi.multithreaded)
- A debugger middleware that drops the user into an interactive
Python session right in the browser when an error occurs.
- A routing system with smart ordering of rules and strict trailing
slash behavior.
For me Werkzeug was (as the name means) a toolkit used to write web
applications and frameworks in Python.
What went Well
``````````````
For the most part I am happy with the actual implementation of request
and response objects. Because my main motivation for doing this was
the Django request object and my previous PHP and MoinMoin experience
I made some changes that I found where useful.
- There is no request.POST like in WebOb and Django. Instead there
is request.form which is a multidict of all submitted form data
minus files. Why is this good? Because POST is not the only HTTP
method that can transmit form data.
- Separating uploaded files from text based form data as well as not
using cgi.FieldStorage. Werkzeug implements its own multipart
form parser instead of falling back to cgi.FieldStorage. The main
motivation was that cgi.FieldStorage was not designed for WSGI in
mind. For instance it was calling readline() which in WSGI is
technically incorrect as it might stall the server on input data
that is not terminated by a trailing newline.
On top of that cgi.FieldStorage is an awkward object and also
supports nested file uploads which technically were in HTML 3.2 or
something but never implemented in practice by a browser. In fact
multi-file uploads as implemented by Firefox 4 and other browsers
do not use this method but instead transmit more than one key.
Werkzeug like Django provides a nicer interface for uploaded files
which also directly allows you to save it somewhere
(request.files['foo'].save(...))
You can trigger some internal server errors in many webob based
applications if you transmit a file where a string is expected or
the other way round. Or a nested file where a regular file was
expected. This is solved in Django and Werkzeug by separating
uploads and regular form data into two keys (request.files and
request.form in Werkzeug for instance).
I think that was a good decision.
- Unicode implementation. Werkzeug uses unicode internally for
everthing and decodes directly when the data is fed into the
multidict. This was a bad decision under 2.x because it adds some
overhead but on 3.x it's probably the best one as bytes work so
different. In fact porting the majority of interfaces over to
Python 3 should be pretty trivial based on this.
Regarding the additonal memory consumption many people brought up:
Python 3 will (as part of the stable ABI) probably have a string
type that dynamically switches between latin1/ucs2/ucs4 internal
encoding based on the highest code point. At leat that is MVL's
plan and it sounds like a terribly good one to me. (Correct me if
I'm wrong there).
- Splitting request object and parsing / formatting. All the
functionality available on the request and response objects is
also available as separate functions. That helped me a lot in the
past because it's easy and straightforward to use these parsing
capabilities in WSGI middlewares without having to be afraid of
the request object doing something funky with the WSGI environ or
in Django applications where you not always had a WSGI env in the
past (I ran it on top of mod_python there). Nowadays Twisted,
Tornado and some other asychronous environments are perfect
examples of why WSGI might not always be there.
At the language summit at PyCon it was discussed that Deferreds
might eventually land in the Python stdlib and this might be the
start of a async-framework abstraction that might at one point be
WSGI for asynchronous HTTP. Werkzeug would very well work for
that because you could just forward the data from whatever format
you have to the parsing without having to reconstruct a full WSGI
environment.
- Non-magical, read only request object. When I played with WebOb
the magical request object freaked me out a little bit because
it's non obvious what modification on the request object has which
consequence behind the scenes. I love that it supports that, but
I would prefer if the modifications were on a separate object
(like a TestRequest object). In Werkzeug for this purpose a
EnvironBuilder exists instead which I currently use for testing
and it works pretty well as far as I can tell.
What's Bad
``````````
Of course I learned along the way and Werkzeug is not perfect, far
from it.
1. Exceptions. They suck. Because with 2.4 you could not raise new
style objects they are emulating part of the response API but not
all. Not that we do not have to care about 2.4 that much any more
it would make sense to make the exceptions actual response
objects.
2. Too much stuff in there. Really, that's too much for being a
reference implementation.
3. The demand import system. Do not look into __init__.py, do not
ever attempt to make something like that. I took the inspiration
from the py package and that was not a clever idea.
4. Most of the stuff in contrib should have gone to a separate
package. Inspired by the flaskext system I will probably try to
move them into separate packages if they make sense or deprecate
in favor of other things. I did unbundle the markup class from
Jinja2 recently into a separate package and that turned out to be
a good decision which encourages me to try that more.
5. utf-8 Header encoding. I decided to use utf-8 as default encoding
for headers and that was not the best decision in hindsight due to
the encoding in Python 3 being latin1. As it only affects the
cookie header in practice I might be able to change this even in
Werkzeug itself without causing havor.
6. Missing support for byte ranges. I never thought they were useful
until I found out that it's a commonly used feature on GAE for
their blobstore.
7. Special key errors. Sounded like a good idea at the time and I
still use it, but it's a bad idea for a general library. The idea
is that a MultiDict does not just raise a key error, it raises a
key error that is also a bad request error. That means if a user
does not catch it down, it's automatically converted into a bad
request response. I would love to see an API where this would
still be sortof possible though. It makes for very concicse code
that does not result in 500 internal server errors.
8. ignore as default error handling for decoding for incoming data.
It would have been nicer if this was chosen to be 'replace' and
was a huge mistake. I was not yet man enough to make this
transition in Werkzeug itself but I should.
Why I would like to see
```````````````````````
I think both webob and Werkzeug made enough mistakes in the past that
it would be amazing to learn from both and come up with a small and
useful core library that can be used as the basis for frameworks.
It's without question that WebOb and Werkzeug will both have to be
independently maintained for a while because nobody of us wants to
force people to do another hard upgrade without much value.
My ideal upgrade path would be starting to subclass from the new core
request class and adapt the interface to match the old one.
I know that Werkzeug has the lower adoption compared to WebOb which
inherited many of the former paste users, but I still think that it
would make sense to look at what worked in Werkzeug for some input.
Keep in mind that the Python web ecosystem is larger than just WebOb /
Werkzeug and we also should keep the requirements for Django and other
frameworks in mind. Django is special because they don't want to
depend on stuff :) However Django in general leads the way in many
regards. The Python packaging world is now in the hands of Tarek, but
virtualenv and pip are mostly in the hands of Django people which
means good things. First of all they are not only willing to
reconsider their decisions eventually, but also that they are working
on making the infrastructure better.
Also, a common request base implementation might just be shipped with
Django hidden somewhere like they are currently shipping simplejson
and pydispatch as well as a bunch of stdlib backports.
GSOC as a Chance
````````````````
Google Summer of Code is here and it would be amazing to have a
student working on having a look how a merge of WebOb and Werkzeug
with best intentions in mind could look like. Porting to Python 3
alone is probably not worth GSOC, but also merging these two libs
might be an amazing GSOC project and for the best of the whole Python
community.
Regards,
Armin
Ian Bicking
I've been reluctant to implement things in WebOb that are in some sense application policy, signed cookies being an example of that. Even URL-encoding all cookie values by default made me uncomfortable, even though I would consider it generally good practice.
- a bunch of read-only and writable data structures. Among other
things a MultiDict that stores more than one value per key with
the same access times as the standard dict. Later another dict
that additionally keeps the values linked with a double linked
list to preserve the same access complexity for all operations by
also preserving the ordering.
I had considered this, but felt it was a premature optimization. A list with a handful of items performs very similarly to a dict, and the case of a form submission or query string with a large number of keys seems implausible (and the performance while linear with the number of items, I doubt will ever be large enough to matter even in outlying cases).
- Basic session support
Way too much application policy here, IMHO. (Of course Werkzeug has a somewhat different scope than WebOb, being more a framework while WebOb tries hard not to be one.)
- A WSGI test application
- A basic WSGI server that fixes some of the problems that wsgiref
has (like emitting the wrong value for wsgi.multithreaded)
- A debugger middleware that drops the user into an interactive
Python session right in the browser when an error occurs.
- A routing system with smart ordering of rules and strict trailing
slash behavior.
For me Werkzeug was (as the name means) a toolkit used to write web
applications and frameworks in Python.
What went Well
``````````````
For the most part I am happy with the actual implementation of request
and response objects. Because my main motivation for doing this was
the Django request object and my previous PHP and MoinMoin experience
I made some changes that I found where useful.
- There is no request.POST like in WebOb and Django. Instead there
is request.form which is a multidict of all submitted form data
minus files. Why is this good? Because POST is not the only HTTP
method that can transmit form data.
It's a naming issue, sure; and many RESTful POST requests won't form-encode the request body. Lacking any particular consensus around other names I chose to stick with an inaccurate but common name.
- Separating uploaded files from text based form data as well as not
using cgi.FieldStorage. Werkzeug implements its own multipart
form parser instead of falling back to cgi.FieldStorage. The main
motivation was that cgi.FieldStorage was not designed for WSGI in
mind. For instance it was calling readline() which in WSGI is
technically incorrect as it might stall the server on input data
that is not terminated by a trailing newline.
On top of that cgi.FieldStorage is an awkward object and also
supports nested file uploads which technically were in HTML 3.2 or
something but never implemented in practice by a browser. In fact
multi-file uploads as implemented by Firefox 4 and other browsers
do not use this method but instead transmit more than one key.
I agree, FieldStorage is not a particularly good parser or representation of file uploads. I would be happy to remove it.
My only concern (which is not a very major concern) with separating out file uploads from non-file-uploads is that you might lose the full ordering of all submitted fields, since file uploads can be interleaved in order with other fields. This is a fairly obscure need, but I have tried very hard not to throw away any information from the HTTP request. But maybe there's a way of providing a full ordering in a more out-of-the-way location while separating out files from other fields in the normal case; I agree it is problematic in practice to accidentally encounter a file upload (and FieldStorage compounds those problems with its peculiar interface).
Werkzeug like Django provides a nicer interface for uploaded files
which also directly allows you to save it somewhere
(request.files['foo'].save(...))
I looked at Django's implementation, but felt there were parts of the implementation that relied on application policy (even configuration), which is something I did not want. But yes, these conveniences are useful.
You can trigger some internal server errors in many webob based
applications if you transmit a file where a string is expected or
the other way round. Or a nested file where a regular file was
expected. This is solved in Django and Werkzeug by separating
uploads and regular form data into two keys (request.files and
request.form in Werkzeug for instance).
I think that was a good decision.
- Unicode implementation. Werkzeug uses unicode internally for
everthing and decodes directly when the data is fed into the
multidict. This was a bad decision under 2.x because it adds some
overhead but on 3.x it's probably the best one as bytes work so
different. In fact porting the majority of interfaces over to
Python 3 should be pretty trivial based on this.
Regarding the additonal memory consumption many people brought up:
Python 3 will (as part of the stable ABI) probably have a string
type that dynamically switches between latin1/ucs2/ucs4 internal
encoding based on the highest code point. At leat that is MVL's
plan and it sounds like a terribly good one to me. (Correct me if
I'm wrong there).
I made the decision to decode lazily again to be sure to preserve all the information in the request, which would be lost for instance if you decode non-UTF-8 data as UTF-8. A default charset is again a kind of application policy, which while WebOb supports, it does so non-destructively; so for instance you can interact safely with the environment in middleware while being encoding neutral (which is often reasonable).
- Splitting request object and parsing / formatting. All the
functionality available on the request and response objects is
also available as separate functions. That helped me a lot in the
past because it's easy and straightforward to use these parsing
capabilities in WSGI middlewares without having to be afraid of
the request object doing something funky with the WSGI environ or
in Django applications where you not always had a WSGI env in the
past (I ran it on top of mod_python there). Nowadays Twisted,
Tornado and some other asychronous environments are perfect
examples of why WSGI might not always be there.
It would be interesting to have a request object that did not (very) specifically handle the request body, as that's the only real difference between a synchronous and async environment. The response can be a bit more tricky, though even then usually a response object would be fine -- it's the actual WSGI call that is where async is really different. OTOH I haven't seen any much indication of interest from most of those communities; or at least, there's the pretend-it's-WSGI group (e.g., gunicorn's async) who can use everything anyway, and the Twisted group which always does their own thing. And then Tornado... and I don't know their community really, though they seem to avoid dependencies.
WebOb does have importable parsing and serialization routines (the descriptor system encourages these to be separate routines) but they are not documented publicly.
At the language summit at PyCon it was discussed that Deferreds
might eventually land in the Python stdlib and this might be the
start of a async-framework abstraction that might at one point be
WSGI for asynchronous HTTP. Werkzeug would very well work for
that because you could just forward the data from whatever format
you have to the parsing without having to reconstruct a full WSGI
environment.
- Non-magical, read only request object. When I played with WebOb
the magical request object freaked me out a little bit because
it's non obvious what modification on the request object has which
consequence behind the scenes. I love that it supports that, but
I would prefer if the modifications were on a separate object
(like a TestRequest object). In Werkzeug for this purpose a
EnvironBuilder exists instead which I currently use for testing
and it works pretty well as far as I can tell.
The underlying environment dict is writable, so there are no guarantees of immutability. I don't think anything that writes to the environment in WebOb is non-obvious -- it either involves setting an attribute (which seems like an obvious write to me) or methods that IMHO are fairly clear (path_info_pop, remove_conditional_headers). The request body handling is pretty complex, arguably too complex, but if we want to focus on that specifically we can.
What's Bad
``````````
Of course I learned along the way and Werkzeug is not perfect, far
from it.
1. Exceptions. They suck. Because with 2.4 you could not raise new
style objects they are emulating part of the response API but not
all. Not that we do not have to care about 2.4 that much any more
it would make sense to make the exceptions actual response
objects.
I'm fairly okay with the compromise in WebOb, which is to make Python 2.4 exceptions slightly awkward (e.g., raise HTTPNotFound().exception). Now that 2.4 is largely a non-issue this compatibility doesn't cause clutter.
2. Too much stuff in there. Really, that's too much for being a
reference implementation.
3. The demand import system. Do not look into __init__.py, do not
ever attempt to make something like that. I took the inspiration
from the py package and that was not a clever idea.
4. Most of the stuff in contrib should have gone to a separate
package. Inspired by the flaskext system I will probably try to
move them into separate packages if they make sense or deprecate
in favor of other things. I did unbundle the markup class from
Jinja2 recently into a separate package and that turned out to be
a good decision which encourages me to try that more.
5. utf-8 Header encoding. I decided to use utf-8 as default encoding
for headers and that was not the best decision in hindsight due to
the encoding in Python 3 being latin1. As it only affects the
cookie header in practice I might be able to change this even in
Werkzeug itself without causing havor.
6. Missing support for byte ranges. I never thought they were useful
until I found out that it's a commonly used feature on GAE for
their blobstore.
You mean the Range set of headers?
7. Special key errors. Sounded like a good idea at the time and I
still use it, but it's a bad idea for a general library. The idea
is that a MultiDict does not just raise a key error, it raises a
key error that is also a bad request error. That means if a user
does not catch it down, it's automatically converted into a bad
request response. I would love to see an API where this would
still be sortof possible though. It makes for very concicse code
that does not result in 500 internal server errors.
I can see how a subclass of KeyError could be useful; though actually making that a Bad Request response does seem a bit too complex (and is something I considered but chose not to do). Code that chooses to could catch the special KeyError subclass and turn it into BadRequest if it wanted to.
8. ignore as default error handling for decoding for incoming data.
It would have been nicer if this was chosen to be 'replace' and
was a huge mistake. I was not yet man enough to make this
transition in Werkzeug itself but I should.
Why I would like to see
```````````````````````
I think both webob and Werkzeug made enough mistakes in the past that
it would be amazing to learn from both and come up with a small and
useful core library that can be used as the basis for frameworks.
It's without question that WebOb and Werkzeug will both have to be
independently maintained for a while because nobody of us wants to
force people to do another hard upgrade without much value.
Well... I don't feel like WebOb has a lot of mistakes, and what issues there are (like exposed FieldStorage) could be resolved.
My ideal upgrade path would be starting to subclass from the new core
request class and adapt the interface to match the old one.
I know that Werkzeug has the lower adoption compared to WebOb which
inherited many of the former paste users, but I still think that it
would make sense to look at what worked in Werkzeug for some input.
I'd also be happy to do so even in the context of WebOb.
What specifically is your reluctance to do this work inside WebOb? Of course it does give WebOb the benefit of precedence, but I really am not excited about redesigning everything from scratch, and the likely errors of omission that would result. I was thinking *very specifically* about all these issues *as I designed WebOb* -- I already went through that process. I read the docs for all the other frameworks I could find, and made changes when appropriate. And yes, it was just me, and I might have gotten things wrong, and there might be points of compromise where there are several possible valid implementations. But I am open to discussion, and I don't think I've shown a disinclination to compromise when concrete issues are being brought up (well, except for separating out the implementation of writable requests; though adding the notion of freezing a request object seems quite reasonable, rendering the write code unusable).
Keep in mind that the Python web ecosystem is larger than just WebOb /
Werkzeug and we also should keep the requirements for Django and other
frameworks in mind. Django is special because they don't want to
depend on stuff :) However Django in general leads the way in many
regards. The Python packaging world is now in the hands of Tarek, but
virtualenv and pip are mostly in the hands of Django people which
means good things. First of all they are not only willing to
reconsider their decisions eventually, but also that they are working
on making the infrastructure better.
Also, a common request base implementation might just be shipped with
Django hidden somewhere like they are currently shipping simplejson
and pydispatch as well as a bunch of stdlib backports.
I am interested in how to bring Django into this. I think the packaging issue is up to them, and probably resolvable. If they don't want dependencies they can copy the code over or whatever, and they are clever enough to figure out how to handle that. It does add some possible nasty stuff with that pattern, if for instance they always use their copied code and then isinstance(django_request, CommonRequest) will be false. That WebOb keeps its information in the environment does mitigate some of the problems that can occur with multiple implementations. But it would certainly be better all around if they could just handle packaging in a reasonable way.
I am not as excited about something that's just an underlying base implementation. It defers the political challenge of finding a consensus interface, but in doing so avoids all the hard choices that are what make a common implementation interesting.
GSOC as a Chance
````````````````
Google Summer of Code is here and it would be amazing to have a
student working on having a look how a merge of WebOb and Werkzeug
with best intentions in mind could look like. Porting to Python 3
alone is probably not worth GSOC, but also merging these two libs
might be an amazing GSOC project and for the best of the whole Python
community.
I generally don't think this sort of thing makes sense for a GSOC project (unless the student is very familiar with the task; I guess you might count ;) -- the hard work is all design, and it's not something that can be easily delegated. Even the Python 3 aspects are likely to involve decisions that the student would not be able to make, things like process issues about how to maintain the two lines of code.
Ian
wobsta
On 29 Mrz., 23:16, Ian Bicking <i...@colorstudy.com> wrote:
> I agree, FieldStorage is not a particularly good parser or representation of
> file uploads. I would be happy to remove it.
FieldStorage implementation on Python 2.5+ and Python 3, namely
https://github.com/defnull/multipart
It was implemented last summer by the bottle author due to the major
problems with cgi.FieldStorage on Python 3.1. (While the major bugs in
the FieldStorage seem to have been addressed in Python 3.2, IMHO it is
still a much favorable solution for parsing multipart headers and
content.) The code is heavily inspired by werkzeug, has 100% test code
coverage and even reuses test data from werkzeug. IMHO it could be
used for implementing a joint foundation for request objects as the
code concentrates on this very single issue and was designed to run on
Python 2.5+ and Python 3 from the very beginning. While it results in
a forms and files multidict similar to what werkzeug does, this is
trivial to adjust to other needs.
Best,
André
GothAlice
Trying to post directly from the Google Groups interface; we'll see
how it goes! (I usually utilize Gmane and a Usenet news reader.)
The spreadsheet is split into two sheets, one for Request, one for
Response. The Response sheet was transcribed from the re-structured
text table. Thank you, Ian, for adding additional information. I've
added additional columns for Django.
http://bit.ly/http-object-reference
— Alice.