https://bitbucket.org/chrism/webob-py3k/src/c699da0e325f/request_table.rst
https://bitbucket.org/chrism/webob-py3k/src/c699da0e325f/response_table.rst
I spoke briefly with Armin about some sort of try at collaboration, and
he seemed amenable. We both seem to believe that some sort of merge
here would be beneficial to the Python web world.
The req/resp APIs of WebOb and Werkzeug are very similar but obviously
not identical. So I doubt we can do a big-bang-break-everyone merge.
We need to provide some period of backwards compatibility during any
merge.
A strategy that I favor currently is to implement a third package upon
which WebOb and Werkzeug would begin to depend. For sake of brevity, I'm
currently calling that package "Wub"). Wub would implement a superset
of the APIs required by both Werkzeug and WebOb users. We could then
change both Werkzeug and WebOb to depend on Wub (deleting most of the
code from each). Werkzeug and WebOb would then implement their current
APIs unmodified, but we'd start issuing deprecation warnings for
functionality accessed by users that was not in Wub itself. Eventually,
we'd retire WebOb and Werkzeug req/resp and come to depend on Wub only.
Just my thoughts,
- C
Currently WebOb has no external dependencies and is reasonably
minimal. I was thinking it would be nice to have yet more minimal core
as part of webob itself, say webob.mini. Currently I actually "hack
away" parts of webob in subclasses to make it do less -- no
multidicts, POST being a simple dict and not a view on input stream,
things like that. It would be nice to be able to just inherit from a
more basic class instead of doing that. I'm not sure how many people
care about this and I'm not sure how much disruption this might bring
to projects that depend on webob, so I left it at that -- just an
idea.
But if there were some unification with Werkzeug, I would really
prefer to see a shared dependency which is a shared subset of the
libraries, not their superset.
Also, I know repoze doesn't mind a lot of small packages depending on
each other, but for webob getting a dependency would be a big thing
too. I feel that while webob appeals to developers that want to write
lean code it's already 'too big' for some of them, or at least gives
that impression. Spreading it out in this way, might be a bad idea for
that reason.
I understand that the idea is to eventually deprecate both libraries
and therefore Wub has to be a superset, but I'm skeptical that can
ever happen. If anything, a more realistic goal is to deprecate just
one of WebOb / Werkzeug and add the missing features to the other one,
but I don't see that happening either.
I think there's a great chance here to waste a lot of time for some
idealistic reasons and end up worse off than before. I'm still open to
the idea, so it would interesting to hear what everybody else thinks.
-Sergey
This is what I meant by "Wub", yes. Apologies if I didn't make that
clear.
> Also, I know repoze doesn't mind a lot of small packages depending on
> each other, but for webob getting a dependency would be a big thing
> too. I feel that while webob appeals to developers that want to write
> lean code it's already 'too big' for some of them, or at least gives
> that impression. Spreading it out in this way, might be a bad idea for
> that reason.
>
> I understand that the idea is to eventually deprecate both libraries
> and therefore Wub has to be a superset, but I'm skeptical that can
> ever happen.
No, it can be a subset.
> If anything, a more realistic goal is to deprecate just
> one of WebOb / Werkzeug and add the missing features to the other one,
> but I don't see that happening either.
I could also imagine that.
> I think there's a great chance here to waste a lot of time for some
> idealistic reasons and end up worse off than before. I'm still open to
> the idea, so it would interesting to hear what everybody else thinks.
The reasons aren't really idealistic; they're practical. We don't need
two almost identical implementations of request/response objects in
Python. It adds both to the testing and to the porting burden across
major Python releases, and having both to choose from prevents
frameworks which base themselves on different ones from collaborating
more closely. A big reason BFG and Pylons were able to merge into
Pyramid is because they shared a common WebOb dependency and folks were
used to its API.
- C
Yes, I understand the practical benefit, and of course in that sense
this idea is practical. But I feel that this migration might be about
as tough to pull off than getting python 2 deprecated in favor of
python 3. Of course Python has more inertia, but it's also too easy to
just stick to some version of webob or fork it or something of that
kind and end up fragmenting the development even more. What I meant by
idealistic reasons is to do something because it's the right thing to
do, even though it might not work out in the end. Or maybe I'm just
being overly pessimistic.
It would be nice to have a more complete list of frameworks that
depend on either library. For some reason I can't remember much off
top of my head.
WebOb: Pylons, BFG / Pyramid
Werkzeug: Flask
--
Best Regards,
Sergey Schetinin
http://self.maluke.com/ -- My articles and open-source stuff
http://www.maluke.com/ -- My commercial software and custom development services
Here's the ones I know of:
WebOb: Pylons, Pyramid, Bobo, Restish, TurboGears 2
Werkzeug: Flask, Glashammer
- C
--
You received this message because you are subscribed to the Google Groups "Paste Users" group.
To post to this group, send email to paste...@googlegroups.com.
To unsubscribe from this group, send email to paste-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/paste-users?hl=en.
Also, pure numbers of frameworks don't really matter. IMO, we're really
still at the very beginning of a very long road in Python web-land.
Flask/Werkzeug is really great code. Armin is an exceptional
programmer. There aren't many of those, so any way we can encourage
cooperation instead of competition between WebOb and Werkzeug is a +1 in
my book, especially when the API differences between them are so minor.
- C
Not really, I started from stdlib cookie parser and refactored it
until it seemed sane.
> [...]
> * I believe Werkzeug does more with Authorization/WWW-Authenticate than
> WebOb does, which I would offer is possible because of the use of mixins
> which encourage core pluggability, and parsing and understanding these
> headers is kind of complicated. I tried not to rely on subclassing in WebOb
> because I always felt it made other people's subclassing more complicated; I
> basically wanted to let subclassing belong to the framework and/or user.
> Maybe there's a way to work out these particular headers -- there's no much
> innovation going on in these headers anymore anyway (https having largely
> taken away the desire to get fancy).
Webob parses them into (type, params) tuple, Werkzeug seems to have
special WWWAuthenticate class (
https://github.com/mitsuhiko/werkzeug/blob/master/werkzeug/datastructures.py#L2157
) which makes it a view on the actual header and adds access to some
of the params as properties. Not sure it's worth it -- there's very
little code to save by having that. I would argue that it saves no
code for user at all.
> I'll also note that Paste was in its time intended to be a common library
> for building request and response objects, much like Wub is being proposed.
> And it kind of served that purpose, and it kind of didn't... but I didn't
> build WebOb on Paste for a reason.
Could you please elaborate on this? I think this is an important
point, but maybe I interpret it differently that what you meant by
this.
Here's one I wrote for pasteob
https://bitbucket.org/mlk/pasteob/src/e9a641c1ec6e/pasteob/base.py#cl-332
with minor changes it could be used in webob, there's very little
code.
-Sergey
+1 to unify WebOb and Werkzeug somehow. WebOb has been being adopted
[grammar?] by more and more frameworks, and is the closest thing
Python to a common Request/Response object across frameworks.
For a few years I've been advocating a WebOb-based stack to replace
WSGI, or at least as an alternative because WSGI middleeware is so
clunky to write and maintain that people either avoid writing
middleware or make their middleware use WebOb, which means you're
re-creating Request/Response objects at every level. Why not just pass
a single pair of Request/Response objects through the whole stack?
So having one common Request/Response object pair would not only make
it easier to interoperate and switch between framework packages, but
it would also allow new possibilities across the entire stack.
--
Mike Orr <slugg...@gmail.com>
also Google App Engine.
Jim
On 20 March 2011 01:03, Ian Bicking <ia...@colorstudy.com> wrote:
> * [...] just like WebOb already uses Werkzeug's cookie
> parser (via copy-and-paste code sharing)Not really, I started from stdlib cookie parser and refactored it
until it seemed sane.
> [...]
> * I believe Werkzeug does more with Authorization/WWW-Authenticate thanWebob parses them into (type, params) tuple, Werkzeug seems to have
> WebOb does, which I would offer is possible because of the use of mixins
> which encourage core pluggability, and parsing and understanding these
> headers is kind of complicated. I tried not to rely on subclassing in WebOb
> because I always felt it made other people's subclassing more complicated; I
> basically wanted to let subclassing belong to the framework and/or user.
> Maybe there's a way to work out these particular headers -- there's no much
> innovation going on in these headers anymore anyway (https having largely
> taken away the desire to get fancy).
special WWWAuthenticate class (
https://github.com/mitsuhiko/werkzeug/blob/master/werkzeug/datastructures.py#L2157
) which makes it a view on the actual header and adds access to some
of the params as properties. Not sure it's worth it -- there's very
little code to save by having that. I would argue that it saves no
code for user at all.
> I'll also note that Paste was in its time intended to be a common libraryCould you please elaborate on this? I think this is an important
> for building request and response objects, much like Wub is being proposed.
> And it kind of served that purpose, and it kind of didn't... but I didn't
> build WebOb on Paste for a reason.
point, but maybe I interpret it differently that what you meant by
this.
Well said.
Jim
> Code is a liability... but it's not that much of a liability. And
> hey, we even have 100% test coverage! Refactoring could be used as a
> way to try to defer hard choices while moving forward in terms of
> infrastructure. But it doesn't seem like it saves any real effort, as
> changing names and doing deprecations inside WebOb itself is not
> particularly hard. If modularity itself is a feature we're trying to
> add then it would be helpful, but modularity is in direct contrast to
> a common request/response object. To me one of the biggest advantages
> is that everyone can put "request" in their code, and everyone else
> will know what that means. If you start splitting it up that becomes
> unclear.
The goals are:
- get rid of code, so that any porting and maintenance effort can be
shared.
- bring at least one more web framework into the set of frameworks that
use the same req/resp code.
If we can do this without the "third thing" that is Wub, that'd be ok by
me. If we're unwilling to create the third thing, then someone just has
to convince Armin to use WebOb in Flask. I'll assume that effort has
not been successful in the past.
- C
It would be great to know what kind of thing Armin would like Wub to
be and then we can see if WebOb can be that thing.
Did you ask him if he would mind moving to WebOb if it grew support for
anything he needed that wasn't there? (Kinda like the stuff Pyramid grew
to make people familiar with Pylons more comfortable...)
> We both seem to believe that some sort of merge
> here would be beneficial to the Python web world.
Very strongly agree.
> A strategy that I favor currently is to implement a third package upon
> which WebOb and Werkzeug would begin to depend. For sake of brevity, I'm
> currently calling that package "Wub").
If it's needed, please give it a good name. "Wub" is not a good name.
"WebOb" *is* a good name and maybe Armin would be amenable to a "WebOb
2.0" that supported both what current WebOb users need along with what
Werkzeug needs?
2.1 could then see the deprecations you refer to, with 2.2 seeing the
extraneous methods and interfaces removed, with 3.0 being left for
Python 3 compatibility.
cheers,
Chris
--
Simplistix - Content Management, Batch Processing & Python Consulting
- http://www.simplistix.co.uk
Nope. I assume that has been tried, otherwise there wouldn't have been
both Werkzeug and WebOb in the first place.
I think maybe Armin's major issue with WebOb is that Werkzeug's req/resp
code really is much cleaner than WebOb's. Now that we have better test
coverage, it's maybe not outside the realm of possibility to clean WebOb
up a bit. And maybe doing so would make a difference. But I'm guessing
really. I'll point Armin here to see if he'd be willing to respond.
- C
Actually I've cleaned up a code quite a bit over the past year or two,
what areas do you feel need a cleanup?
On Sun, 2011-03-20 at 11:35 -0500, Ian Bicking wrote:The goals are:
> Code is a liability... but it's not that much of a liability. And
> hey, we even have 100% test coverage! Refactoring could be used as a
> way to try to defer hard choices while moving forward in terms of
> infrastructure. But it doesn't seem like it saves any real effort, as
> changing names and doing deprecations inside WebOb itself is not
> particularly hard. If modularity itself is a feature we're trying to
> add then it would be helpful, but modularity is in direct contrast to
> a common request/response object. To me one of the biggest advantages
> is that everyone can put "request" in their code, and everyone else
> will know what that means. If you start splitting it up that becomes
> unclear.
- get rid of code, so that any porting and maintenance effort can be
shared.
- bring at least one more web framework into the set of frameworks that
use the same req/resp code.
If we can do this without the "third thing" that is Wub, that'd be ok by
me. If we're unwilling to create the third thing, then someone just has
to convince Armin to use WebOb in Flask. I'll assume that effort has
not been successful in the past.
- methods with many deeply nested conditionals (not easy to test):
https://bitbucket.org/chrism/webob-py3k/src/c699da0e325f/webob/response.py#cl-63
https://bitbucket.org/chrism/webob-py3k/src/c699da0e325f/webob/response.py#cl-290
https://bitbucket.org/chrism/webob-py3k/src/c699da0e325f/webob/response.py#cl-970
https://bitbucket.org/chrism/webob-py3k/src/c699da0e325f/webob/etag.py#cl-105
https://bitbucket.org/chrism/webob-py3k/src/c699da0e325f/webob/byterange.py#cl-98
- An import-time monkeypatch to the CGI module (maybe not to be helped):
https://bitbucket.org/chrism/webob-py3k/src/c699da0e325f/webob/request.py#cl-1175
- Paste monkeypatching:
https://bitbucket.org/chrism/webob-py3k/src/c699da0e325f/webob/exc.py#cl-1063
- Import time side effects:
https://bitbucket.org/chrism/webob-py3k/src/c699da0e325f/webob/exc.py#cl-1078
https://bitbucket.org/chrism/webob-py3k/src/c699da0e325f/webob/__init__.py#cl-41
That's a great list actually :)
I spoke to him. I think he may be preparing some sort of design defense
for Werkzeug before commenting.
- C
Also OpenStack.
Was there any reaction from Armin BTW?
-Sergey
Hi,
GothAlice seems to have troubles replying here so she asked me to post
for here, this is a forward of said email.
---------- Forwarded message ----------
From: Alice Bevan-McGregor <al...@gothcandy.com>
Date: Mon, Mar 28, 2011 at 10:11 PM
Subject: Re: Werkzeug req-resp/WebOb integration
To: jorge....@gmail.com
On 2011-03-19 14:09:02 -0700, Chris McDonough said:
> As part of the PyCon sprints, a few of us mapped out how WebOb request
> and response methods and properties map to their Werkzeug analogues.
> The results are at:
>
> https://bitbucket.org/chrism/webob-py3k/src/c699da0e325f/request_table.rst
>
> https://bitbucket.org/chrism/webob-py3k/src/c699da0e325f/response_table.rst
That's a very cool list; handy not just for attempts to merge but also
as a quick reference or porting guide.
I've begun work on a spreadsheet version, better organized, and with
additional information like RFC chapter references. The Response
sheet is by far the most complete thus far, and includes a
transcription of the resopnse table linked above.
This spreadsheet should be editable by anyone, so it'd be great if
someone could add descriptions to the miscelaneous items at the bottom
of that sheet and double-check my transcription. (I noticed WebOb
spelt WebBob in the original, plus some confusion over read/write due
to the order of those columns.)
Additionally, I've added the API from marrow.wsgi.objects, both the
current state of implementation (in black) and goal (in red). I'll
prepare a "defending m.w.object's design" document in the near future,
as well.
I'd have joined this discussion earlier, but the first two messages I
sent went to /dev/null, it appears. Gmane Usenet access + Paste's
mailing list = bad mojo.
— Alice.
- a bunch of read-only and writable data structures. Among other
things a MultiDict that stores more than one value per key with
the same access times as the standard dict. Later another dict
that additionally keeps the values linked with a double linked
list to preserve the same access complexity for all operations by
also preserving the ordering.
- Basic session support
- A WSGI test application
- A basic WSGI server that fixes some of the problems that wsgiref
has (like emitting the wrong value for wsgi.multithreaded)
- A debugger middleware that drops the user into an interactive
Python session right in the browser when an error occurs.
- A routing system with smart ordering of rules and strict trailing
slash behavior.
For me Werkzeug was (as the name means) a toolkit used to write web
applications and frameworks in Python.
What went Well
``````````````
For the most part I am happy with the actual implementation of request
and response objects. Because my main motivation for doing this was
the Django request object and my previous PHP and MoinMoin experience
I made some changes that I found where useful.
- There is no request.POST like in WebOb and Django. Instead there
is request.form which is a multidict of all submitted form data
minus files. Why is this good? Because POST is not the only HTTP
method that can transmit form data.
- Separating uploaded files from text based form data as well as not
using cgi.FieldStorage. Werkzeug implements its own multipart
form parser instead of falling back to cgi.FieldStorage. The main
motivation was that cgi.FieldStorage was not designed for WSGI in
mind. For instance it was calling readline() which in WSGI is
technically incorrect as it might stall the server on input data
that is not terminated by a trailing newline.
On top of that cgi.FieldStorage is an awkward object and also
supports nested file uploads which technically were in HTML 3.2 or
something but never implemented in practice by a browser. In fact
multi-file uploads as implemented by Firefox 4 and other browsers
do not use this method but instead transmit more than one key.
Werkzeug like Django provides a nicer interface for uploaded files
which also directly allows you to save it somewhere
(request.files['foo'].save(...))
You can trigger some internal server errors in many webob based
applications if you transmit a file where a string is expected or
the other way round. Or a nested file where a regular file was
expected. This is solved in Django and Werkzeug by separating
uploads and regular form data into two keys (request.files and
request.form in Werkzeug for instance).
I think that was a good decision.
- Unicode implementation. Werkzeug uses unicode internally for
everthing and decodes directly when the data is fed into the
multidict. This was a bad decision under 2.x because it adds some
overhead but on 3.x it's probably the best one as bytes work so
different. In fact porting the majority of interfaces over to
Python 3 should be pretty trivial based on this.
Regarding the additonal memory consumption many people brought up:
Python 3 will (as part of the stable ABI) probably have a string
type that dynamically switches between latin1/ucs2/ucs4 internal
encoding based on the highest code point. At leat that is MVL's
plan and it sounds like a terribly good one to me. (Correct me if
I'm wrong there).
- Splitting request object and parsing / formatting. All the
functionality available on the request and response objects is
also available as separate functions. That helped me a lot in the
past because it's easy and straightforward to use these parsing
capabilities in WSGI middlewares without having to be afraid of
the request object doing something funky with the WSGI environ or
in Django applications where you not always had a WSGI env in the
past (I ran it on top of mod_python there). Nowadays Twisted,
Tornado and some other asychronous environments are perfect
examples of why WSGI might not always be there.
At the language summit at PyCon it was discussed that Deferreds
might eventually land in the Python stdlib and this might be the
start of a async-framework abstraction that might at one point be
WSGI for asynchronous HTTP. Werkzeug would very well work for
that because you could just forward the data from whatever format
you have to the parsing without having to reconstruct a full WSGI
environment.
- Non-magical, read only request object. When I played with WebOb
the magical request object freaked me out a little bit because
it's non obvious what modification on the request object has which
consequence behind the scenes. I love that it supports that, but
I would prefer if the modifications were on a separate object
(like a TestRequest object). In Werkzeug for this purpose a
EnvironBuilder exists instead which I currently use for testing
and it works pretty well as far as I can tell.
What's Bad
``````````
Of course I learned along the way and Werkzeug is not perfect, far
from it.
1. Exceptions. They suck. Because with 2.4 you could not raise new
style objects they are emulating part of the response API but not
all. Not that we do not have to care about 2.4 that much any more
it would make sense to make the exceptions actual response
objects.
2. Too much stuff in there. Really, that's too much for being a
reference implementation.
3. The demand import system. Do not look into __init__.py, do not
ever attempt to make something like that. I took the inspiration
from the py package and that was not a clever idea.
4. Most of the stuff in contrib should have gone to a separate
package. Inspired by the flaskext system I will probably try to
move them into separate packages if they make sense or deprecate
in favor of other things. I did unbundle the markup class from
Jinja2 recently into a separate package and that turned out to be
a good decision which encourages me to try that more.
5. utf-8 Header encoding. I decided to use utf-8 as default encoding
for headers and that was not the best decision in hindsight due to
the encoding in Python 3 being latin1. As it only affects the
cookie header in practice I might be able to change this even in
Werkzeug itself without causing havor.
6. Missing support for byte ranges. I never thought they were useful
until I found out that it's a commonly used feature on GAE for
their blobstore.
7. Special key errors. Sounded like a good idea at the time and I
still use it, but it's a bad idea for a general library. The idea
is that a MultiDict does not just raise a key error, it raises a
key error that is also a bad request error. That means if a user
does not catch it down, it's automatically converted into a bad
request response. I would love to see an API where this would
still be sortof possible though. It makes for very concicse code
that does not result in 500 internal server errors.
8. ignore as default error handling for decoding for incoming data.
It would have been nicer if this was chosen to be 'replace' and
was a huge mistake. I was not yet man enough to make this
transition in Werkzeug itself but I should.
Why I would like to see
```````````````````````
I think both webob and Werkzeug made enough mistakes in the past that
it would be amazing to learn from both and come up with a small and
useful core library that can be used as the basis for frameworks.
It's without question that WebOb and Werkzeug will both have to be
independently maintained for a while because nobody of us wants to
force people to do another hard upgrade without much value.
My ideal upgrade path would be starting to subclass from the new core
request class and adapt the interface to match the old one.
I know that Werkzeug has the lower adoption compared to WebOb which
inherited many of the former paste users, but I still think that it
would make sense to look at what worked in Werkzeug for some input.
Keep in mind that the Python web ecosystem is larger than just WebOb /
Werkzeug and we also should keep the requirements for Django and other
frameworks in mind. Django is special because they don't want to
depend on stuff :) However Django in general leads the way in many
regards. The Python packaging world is now in the hands of Tarek, but
virtualenv and pip are mostly in the hands of Django people which
means good things. First of all they are not only willing to
reconsider their decisions eventually, but also that they are working
on making the infrastructure better.
Also, a common request base implementation might just be shipped with
Django hidden somewhere like they are currently shipping simplejson
and pydispatch as well as a bunch of stdlib backports.
GSOC as a Chance
````````````````
Google Summer of Code is here and it would be amazing to have a
student working on having a look how a merge of WebOb and Werkzeug
with best intentions in mind could look like. Porting to Python 3
alone is probably not worth GSOC, but also merging these two libs
might be an amazing GSOC project and for the best of the whole Python
community.