About WSGIProxy

[Gael Pasgrimaud]

Hi Ian, Paste users,

Today I'd like to talk about WSGIProxy.

A year ago I've helped Benoit Chesneau to implement a WSGI proxy in
restkit[1]. It basically has the same interfaces as WSGIProxy's
applications.

Yesterday I wrote requests_proxy[2] which is a simple wsgi proxy based
on requests.

Since WSGIProxy is not maintained at the moment, I'd like to take the
ownership of the package on pypi to be able to release a totally new
implementation.
The idea is to provide some common interfaces in order to be able to use
both requests and restkit as backends for Proxies applications/middlewares.
I'll keep the same api as the current WSGIProxy so it won't break
existing code. (maybe just removing the middlewares. They look useless
to me).

requests will be the default backend since it's a pure python
implementation.

Benoit is ok to move restkit's code to the new WSGIProxy package.

Advantages: Both restkit and requests are more solid than httplib. Both
requests and restkit are compatible with python2 and 3. Using third
party libraries will reduce the amount of code.

Disadvantages: This will add requests as a dependency

I can also create a new package, but the name is just perfect, and ...
WSGIProxy may die if nobody does anything :)

Your thoughts ?

Regards,

Ga�l

[1]
https://github.com/benoitc/restkit/blob/master/restkit/contrib/wsgi_proxy.py
[2] https://github.com/gawel/requests_proxy

[Ian Bicking]

Well, some of what's in WSGIProxy is now in webob.client: https://github.com/Pylons/webob/blob/master/webob/client.py – pretty much everything that was in wsgiproxy.exactproxy.  webob.client also had some support for urllib3, but it didn't really work – but it wasn't particularly hard if you wanted to revisit that, as urllib3 had some httplib-like interfaces.  I think much of the more durable request stuff is in urllib3, which requests uses?

The actually routing stuff is not in webob.client, stuff like setting X-Forwarded-* headers, or translating URLs.  That stuff doesn't really belong in WebOb.  It would be nice if there was a home for that kind of code, but it's not a lot of code.

Gaël

[Gael Pasgrimaud]

On 18/12/2012 17:23, Ian Bicking wrote:
> Well, some of what's in WSGIProxy is now in webob.client:
> https://github.com/Pylons/webob/blob/master/webob/client.py – pretty
> much everything that was in wsgiproxy.exactproxy. webob.client also
> had some support for urllib3, but it didn't really work – but it
> wasn't particularly hard if you wanted to revisit that, as urllib3 had
> some httplib-like interfaces. I think much of the more durable
> request stuff is in urllib3, which requests uses?
>
> The actually routing stuff is not in webob.client, stuff like setting
> X-Forwarded-* headers, or translating URLs. That stuff doesn't really
> belong in WebOb. It would be nice if there was a home for that kind
> of code, but it's not a lot of code.
>

Good to know. There's some good ideas like passing some options through
environ. But my idea is more about using restkit and requests high level
apis. I don't want to take care about httplib api. And I'm not sure that
webob is the right place for that. Mostly because it will require to add
a dependency.

requests use urllib3 but it also has a lot of configuration options
(verify/allow_redirects/etc.)
restkit has a killer feature: it stream both request and response body.
This mean that you can send/receive a large video without problems.

(and webob.client, like wsgiproxy.exactproxy, return [body] as the
iterator, ouch ;)

So I think this will end with a WSGIProxy2 :)

[Ian Bicking]

On Tue, Dec 18, 2012 at 11:31 AM, Gael Pasgrimaud <gael.pa...@gmail.com> wrote:

On 18/12/2012 17:23, Ian Bicking wrote:

Well, some of what's in WSGIProxy is now in webob.client: https://github.com/Pylons/webob/blob/master/webob/client.py – pretty much everything that was in wsgiproxy.exactproxy.  webob.client also had some support for urllib3, but it didn't really work – but it wasn't particularly hard if you wanted to revisit that, as urllib3 had some httplib-like interfaces.  I think much of the more durable request stuff is in urllib3, which requests uses?

The actually routing stuff is not in webob.client, stuff like setting X-Forwarded-* headers, or translating URLs.  That stuff doesn't really belong in WebOb.  It would be nice if there was a home for that kind of code, but it's not a lot of code.

Good to know. There's some good ideas like passing some options through environ. But my idea is more about using restkit and requests high level apis. I don't want to take care about httplib api. And I'm not sure that webob is the right place for that. Mostly because it will require to add a dependency.

You can have optional dependencies though.  Just because webob.client could use urllib3, does not mean that it must require urllib3.  There might be little conditionals necessary, like _timeout_supported.

 

requests use urllib3 but it also has a lot of configuration options (verify/allow_redirects/etc.)

Things like allow_redirects wouldn't make sense in webob, or IMHO any WSGI proxy system – it should pass HTTP through.  I'm not sure what other options might make sense, the only one I put in webob.client is timeout.  If there was a number of them, then webob.client would probably need a simple adapter system, so that HTTP(S)Connection were wrapped in something that formed a superset of all the connection objects.

 

restkit has a killer feature: it stream both request and response body. This mean that you can send/receive a large video without problems.

Yes – adding streaming support for the response body to webob.client wouldn't be too hard, since it'd just be an iterator that reads chunks.  I don't think any of the libraries have a particularly good way to stream the request body though, do they?  At least, I'd guess the technique to do that would be to poke around with private socket objects, which is no fun.  If there was a proper way to stream bodies to the connection then it shouldn't be too hard to chunk reads from wsgi.input.

[Gael Pasgrimaud]

Right. But verify (ssl) can be useful. And I'm totally agree with the
adapter thing.

> restkit has a killer feature: it stream both request and response
> body. This mean that you can send/receive a large video without
> problems.
>
>
> Yes – adding streaming support for the response body to webob.client
> wouldn't be too hard, since it'd just be an iterator that reads
> chunks. I don't think any of the libraries have a particularly good
> way to stream the request body though, do they? At least, I'd guess
> the technique to do that would be to poke around with private socket
> objects, which is no fun. If there was a proper way to stream bodies
> to the connection then it shouldn't be too hard to chunk reads from
> wsgi.input.
>

Ok, so I'll create a WSGIProxy2 as a proof of concept. I'll try to deal
with the httplib api too.. And I'll ask for a review before the first
public release and we'll see if we can merge that in webob.

Looks like a good compromise :)

Thanks!

[Gael Pasgrimaud]

Here it is: https://github.com/gawel/wsgi_proxy

I've added a few docs: https://wsgi_proxy.readthedocs.org/en/latest/

You can use httplib, urllib3, requests and restkit as http client. You
can also use your own http client.

httplib and urllib3 responses are not streamed (but this can be done by
adding a response iterator).

Test coverage is at 100%.

It can be added as webob.proxy but I'm still not sure that this is a
good idea. A WSGI proxy is only a WSGI application. It as nothing to do
with a "Web Object". Also you've extracted some stuff from Paste to
create webob and wsgiproxy. Not sure we want webob to become the new
Paste :)

Oh, and I thought that restkit support python3 but it's not.

>
> Thanks!
>