Including a little wsgiproxy in webob

[Ian Bicking]

I think I mentioned this briefly before, but I had thought about moving wsgiproxy.exactproxy.proxy_exact_request into WebOb.  This made me think about it again:
  http://kennethreitz.com/the-future-of-python-http.html
obviously reactionary of me though ;)

The relevant code is here:
  https://bitbucket.org/ianb/wsgiproxy/src/3f05b7b2b414/wsgiproxy/exactproxy.py#cl-44
it's not that much code, probably should get a review, and then copied into the WebOb source tree.

Then we can rename/alias Request.send to Request.get_response (since it reads more nicely) and perhaps make proxy_exact_request the default app.

It's not a big change, and has some limited utility, but it does pull things together for example for testing.

Also maybe a few things that are lacking setters and getters should get them, e.g., Request.json_body should have a setter.

[Thomas G. Willis]

I've been using TransparentProxy from paste. I would love to have something like this in webob. 

It's handy on appengine where something like requests wont work due to it opening sockets(I think).

Would a pull request be welcome?

Thomas G. Willis

--
You received this message because you are subscribed to the Google Groups "Paste Users" group.
To post to this group, send email to paste...@googlegroups.com.
To unsubscribe from this group, send email to paste-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/paste-users?hl=en.

[Thomas G. Willis]

Hello Ian, Sergey and the rest.

I spent the afternoon yesterday going through the quickstart for requests and seeing what I would do that would be equivalent in webob. It was a fun exercise(I didn't know about encode_content/decode_content) and I'm at the point now where I am wondering whether there is any interest in taking this further to make some sort of library or webob subpackage that focuses more on client interaction. Pasted below is the experiment expressed as unit tests. It shouldn't take too much to imagine hitting a live website this way via proxy_exact_request or paste.proxy.TransparentProxy.

I guess primarily I'm wondering if there's any thoughts opinion on this.

"""

examples for every example in requests quickstart

http://docs.python-requests.org/en/latest/user/quickstart/

"""

from webob import Request, Response

import urllib

import unittest

def status_200(environ, start_response):

    return Response("OK")(environ, start_response)

def echo_data(environ, start_response):

    req = Request(environ)

    return Response(str(req.params))(environ, start_response)

def crap_server(environ, start_response):

    response_body = u"I am unicode \u0107".encode("utf-8")

    status = "200 OK"

    headers = [("Content-type", "text/plain"),

               ("Content-Length", len(response_body))]

    start_response(status, headers)

    return [response_body]

def filter_charset(app):

    def m(environ, start_response):

        request = Request(environ)

        res = request.get_response(app)

        if not res.charset:

            res.charset = "utf8"

        return res(environ, start_response)

    return m

def gzip_server(environ, start_response):

    response = Response("this is some text that will be compressed")

    response.encode_content(encoding="gzip")

    return response(environ, start_response)

def filter_decode(app):

    def m(environ, start_response):

        request = Request(environ)

        response = request.get_response(app)

        response.decode_content()

        return response(environ, start_response)

    return m

class TestQuickStart(unittest.TestCase):

    def testGet(self):

        r = Request.blank("/").get_response(status_200)

        self.assertEqual(r.status_int, 200)

    def testPayload(self):

        """

        urlencoding yourself seems unexpected

        """

        query_string = urllib.urlencode(dict(foo=1, bar=2))

        r = Request.blank("/", query_string=query_string).get_response(echo_data)

        self.assertEqual(r.status_int, 200)

        self.assert_("foo" in r.body)

        self.assert_("bar" in r.body)

    def testPayloadPOST(self):

        """

        but POST takes a dict and the param is all caps

        """

        POST = dict(foo=1, bar=2)

        r = Request.blank("/", POST=POST).get_response(echo_data)

        self.assertEqual(r.status_int, 200)

        self.assert_("foo" in r.body)

        self.assert_("bar" in r.body)

    def testCharSet(self):

        """

        magically figuring out the charset of the response??

        """

        r = Request.blank("/").get_response(filter_charset(crap_server))

        self.assertEqual(r.status_int, 200)

        self.assert_(r.charset, "no charset")

        self.assert_(r.text)

    def testBinary(self):

        """

        magically figures out transfer encodings and gzip deflates is necessary

        """

        r = Request.blank("/").get_response(filter_decode(gzip_server))

        self.assert_("compressed" in r.body, r.body)

    def testRawResponse(self):

        """

        why would you want to do this if you are using a client lib

        that emphasizes its easy to use?

        """

        pass

    # headers

    # cookies

    # auth

    # redirection

    # timeout

Thomas G. Willis

To unsubscribe from this group, send email to paste-users+unsubscribe@googlegroups.com.

[Ian Bicking]

Cool - I had thought about doing something like this myself, at least to see what it would look like and how many differences there would be.  Using Request's own tests would be interesting, though perhaps only academically so.

I can imagine a client library that looks something like:

class Session(object):
    RequestClass = webob.Request
    def __init__(self):
        self.proxy = ProxyApp()
    def get(self, url, **kw):
        return self.request('GET', url, **kw)
    def post(self, url, data, **kw):
        return self.request('POST', url, POST=data, **kw)
    def put(self, url, data, **kw):
        return self.request('PUT', url, body=data, **kw)
    # a couple others...
    def request(self, method, url, **kw):
        req = self.RequestClass.blank(url, method=method, **kw)
        req = self.transform_request(req)
        resp = self.proxy.get_response(req)
        return self.transform_response(resp)
    def transform_request(self, req):
        # Maybe load cookies, or set auth, here
        return req
    def transform_response(self, resp):
        resp.decode_content()
        # maybe handle the cookie jar here

And you'd probably want the ProxyApp() to be able to do things like cache resources locally, handle preconditions, timeouts, maybe some other stuff.  You might subclass webob.Request to allow some info go through that into the app, or you might capture those settings in request().  But I could imagine for instance allowing req.timeout = 10 and setting environ['client.timeout'] = 10, and then having ProxyApp look for that setting and adjust the timeout.  transform_request could apply "default" settings in this fashion.  Attaching it to the request actually seems reasonable.  But some other stuff like the cookie jar belongs to the session.

Of course it's all just an experiment as such, I'm not sure what else it might or should be.

[Thomas G. Willis]

Yeah my next step was to experiment with friendlier syntax and see how far I got. Though maybe I'm just used to it, but the syntax doesn't seem  all that bad as is. I guess a good "pain test" would be "would I type this in the console?" .

So, then I started looking at the cookie jar logic and at first I thought that could be in middleware(and I would borrow webtests cookie handling), but now I'm not so sure. It seems more appropriate to be a part of the proxy or whatever piece gets to call urllib. 

I haven't quite thought it through yet but a jumble of components(not calling it a framework) seems to be emerging in theory that could be composed into a nice client http library but also with all the pieces available for you to assemble your own for whatever needs you may have, and with established patterns on how/what to extend when/why. 

I have a lot of potential uses for this kind of thing myself. 

About the ProxyApp, what is the difference if any between proxy_exact_request and paste.proxy.TransparentProxy ?

[Ian Bicking]

On Mon, Apr 9, 2012 at 2:06 PM, Thomas G. Willis <tom.w...@gmail.com> wrote:

Yeah my next step was to experiment with friendlier syntax and see how far I got. Though maybe I'm just used to it, but the syntax doesn't seem  all that bad as is. I guess a good "pain test" would be "would I type this in the console?" .

Well, simply copying the Requests API verbatim seemed like a reasonable starting point.  There are some minor differences that might or might not be worth papering over.  status_int is actually kind of a terrible name, changing/adding "status_code" as a name in webob seems reasonable to me.

Auth is funny, because it often has to operate on the request itself, as many auth techniques sign the request.

So, then I started looking at the cookie jar logic and at first I thought that could be in middleware(and I would borrow webtests cookie handling), but now I'm not so sure. It seems more appropriate to be a part of the proxy or whatever piece gets to call urllib. 

I think the proxy should be relatively opaque – it might have settings, but I think holding the cookie jar would be wrong.  It would make it hard to tell what cookies you actually sent.  I think the instantiated request object should be as accurate as possible.
 

I haven't quite thought it through yet but a jumble of components(not calling it a framework) seems to be emerging in theory that could be composed into a nice client http library but also with all the pieces available for you to assemble your own for whatever needs you may have, and with established patterns on how/what to extend when/why. 

I have a lot of potential uses for this kind of thing myself. 

About the ProxyApp, what is the difference if any between proxy_exact_request and paste.proxy.TransparentProxy ?

Not much; I believe proxy_exact_request gets rid of the tiny bit of statefulness that TransparentProxy has, by using SERVER_NAME/SERVER_PORT instead of the force* settings with TransparentProxy.

[Thomas G. Willis]

On Monday, April 9, 2012 4:24:13 PM UTC-4, Ian Bicking wrote:

On Mon, Apr 9, 2012 at 2:06 PM, Thomas G. Willis <tom.w...@gmail.com> wrote:

Yeah my next step was to experiment with friendlier syntax and see how far I got. Though maybe I'm just used to it, but the syntax doesn't seem  all that bad as is. I guess a good "pain test" would be "would I type this in the console?" .

Well, simply copying the Requests API verbatim seemed like a reasonable starting point.  There are some minor differences that might or might not be worth papering over.  status_int is actually kind of a terrible name, changing/adding "status_code" as a name in webob seems reasonable to me.
 

Auth is funny, because it often has to operate on the request itself, as many auth techniques sign the request.

Yeah I saw auth as being middleware to tweak the request before sending it along and maybe watch out for challenge responses and handle it.

 

So, then I started looking at the cookie jar logic and at first I thought that could be in middleware(and I would borrow webtests cookie handling), but now I'm not so sure. It seems more appropriate to be a part of the proxy or whatever piece gets to call urllib. 

I think the proxy should be relatively opaque – it might have settings, but I think holding the cookie jar would be wrong.  It would make it hard to tell what cookies you actually sent.  I think the instantiated request object should be as accurate as possible.

That's a good point. Ideally I want to know for sure. But I guess I was trying to figure out how to fit into existing api's. Doesn't the cookielib.CookieJar expect a urllib request? 

Is there an identical API that doesn't depend on that? 

 

I haven't quite thought it through yet but a jumble of components(not calling it a framework) seems to be emerging in theory that could be composed into a nice client http library but also with all the pieces available for you to assemble your own for whatever needs you may have, and with established patterns on how/what to extend when/why. 

I have a lot of potential uses for this kind of thing myself. 

About the ProxyApp, what is the difference if any between proxy_exact_request and paste.proxy.TransparentProxy ?

Not much; I believe proxy_exact_request gets rid of the tiny bit of statefulness that TransparentProxy has, by using SERVER_NAME/SERVER_PORT instead of the force* settings with TransparentProxy.

Ah ok, so proxy_exact_request = stateless and TransparentProxy = stateful. 

Well I will keep cranking away at it in my spare time, if anything materializes to be more than a test file, I'll throw it up on github. Thanks for the input. 

 

[Thomas G. Willis]

On Monday, April 9, 2012 5:24:57 PM UTC-4, Thomas G. Willis wrote:

On Monday, April 9, 2012 4:24:13 PM UTC-4, Ian Bicking wrote:

On Mon, Apr 9, 2012 at 2:06 PM, Thomas G. Willis <tom.w...@gmail.com> wrote:

... But I guess I was trying to figure out how to fit into existing api's. Doesn't the cookielib.CookieJar expect a urllib request? 

Is there an identical API that doesn't depend on that? 

Well sometimes my past life as a .net developer sneaks back in and makes me think everything is incredibly hard unless the vendor provides it. Alas, it was pretty easy to adapt webob to what cookie jar is expecting.

class RequestCookieAdapter(object):

    """

    this class merely provides the methods required for a

    cookielib.CookieJar to work on a webob.Request

    potential for yak shaving...very high

    """

    def __init__(self, request):

        self._request = request

    def is_unverifiable(self):

        return True  # sure? Why not?

    def get_full_url(self):

        return self._request.url

    def get_origin_req_host(self):

        return self._request.host

    def add_unredirected_header(self, key, header):

        self._request.headers[key] = header

    def has_header(self, key):

        return key in self._request.headers

class ResponseCookieAdapter(object):

    """

    this class merely provides methods required for a

    cookielib.CookieJar to work on a webob.Response

    """

    def __init__(self, response):

        self._response = response

    def info(self):

        return self

    def getheaders(self, header):

        return self._response.headers.getall(header)

def cookie_middleware(app):

    """

    intercepts req/res and keeps track of cookies

    """

    jar = CookieJar()

    def m(environ, start_response):

        request = Request(environ)

        jar.add_cookie_header(RequestCookieAdapter(request))

        response = request.get_response(app)

        cookies = jar.make_cookies(ResponseCookieAdapter(response),

                                   RequestCookieAdapter(request))

        for c in cookies:

            jar.set_cookie(c)

        return response(environ, start_response)

    return m

It's days like this that I really love python. 

I might wrap this all up in a client API in the next couple of days. Ian, are you on IRC at all anymore? 

[Thomas G. Willis]

Ok, I went ahead and threw up on github. :)

https://github.com/twillis/webobtoolkit


pull requests welcome. :)

[Sergey Schetinin]

I don't expect to be using this myself, so I don't think I have
anything to add to the discussion, but this seems useful and I have no
objections to merging the necessary parts into webob.

+0.5 from me =)

-Sergey

[Thomas G. Willis]

Are you thinking something like webob.proxy.proxy_exact_request ? or something?

[Sergey Schetinin]

I think calling it "proxy" could be a little confusing, maybe webob.client?

> --
> You received this message because you are subscribed to the Google Groups
> "Paste Users" group.

> To view this discussion on the web visit
> https://groups.google.com/d/msg/paste-users/-/YAiCq9dzYvMJ.

>
> To post to this group, send email to paste...@googlegroups.com.
> To unsubscribe from this group, send email to

> paste-users...@googlegroups.com.

[Ian Bicking]

+1 on client.  And maybe rename proxy_exact_request to send_request_app.

[Thomas G. Willis]

OK, well right now I'm toggling between docs and test coverage. 

But Sergey, I'm not sure I'm clear on what you think should be in WebOb, are we talking this whole "toolkit" I'm writing? or just the proxy->send_request_app piece? I'm cool with either at the moment if that matters.

if you all want to chat about it in a more realtime-y way I am on irc usually in the #pyramid room (twillis)

On Thursday, April 12, 2012 6:46:10 PM UTC-4, Ian Bicking wrote:

+1 on client.  And maybe rename proxy_exact_request to send_request_app.

On Thu, Apr 12, 2012 at 5:20 PM, Sergey Schetinin <mal...@gmail.com> wrote:

I think calling it "proxy" could be a little confusing, maybe webob.client?

On 12 April 2012 20:02, Thomas G. Willis <tom.w...@gmail.com> wrote:
> Are you thinking something like webob.proxy.proxy_exact_request ? or
> something?
>
>
>
> On Wednesday, April 11, 2012 5:31:30 PM UTC-4, Sergey Schetinin wrote:
>>
>> I don't expect to be using this myself, so I don't think I have
>> anything to add to the discussion, but this seems useful and I have no
>> objections to merging the necessary parts into webob.
>>
>> +0.5 from me =)
>>
>> -Sergey
>
> --
> You received this message because you are subscribed to the Google Groups
> "Paste Users" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/paste-users/-/YAiCq9dzYvMJ.
>
> To post to this group, send email to paste...@googlegroups.com.
> To unsubscribe from this group, send email to

> paste-users+unsubscribe@googlegroups.com.

> For more options, visit this group at
> http://groups.google.com/group/paste-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "Paste Users" group.
To post to this group, send email to paste...@googlegroups.com.

To unsubscribe from this group, send email to paste-users+unsubscribe@googlegroups.com.

[Gaël Pasgrimaud]

On Fri, Apr 13, 2012 at 1:02 AM, Thomas G. Willis <tom.w...@gmail.com> wrote:
> OK, well right now I'm toggling between docs and test coverage.
>
> But Sergey, I'm not sure I'm clear on what you think should be in WebOb, are
> we talking this whole "toolkit" I'm writing? or just the
> proxy->send_request_app piece? I'm cool with either at the moment if that
> matters.

Just a reminder: restkit contain some wsgi proxy

http://benoitc.github.com/restkit/api/restkit.contrib.wsgi_proxy-module.html

Advantages:
- use some C code to parse http response (http-parser)
- stream content (does not return a 1Go string if you proxying a divx)
- allow to use gevent to maintain a pool of socket

>>> > paste-users...@googlegroups.com.

>>> > For more options, visit this group at
>>> > http://groups.google.com/group/paste-users?hl=en.
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "Paste Users" group.
>>> To post to this group, send email to paste...@googlegroups.com.
>>> To unsubscribe from this group, send email to

>>> paste-users...@googlegroups.com.

>>> For more options, visit this group at
>>> http://groups.google.com/group/paste-users?hl=en.
>>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Paste Users" group.

> To view this discussion on the web visit

> https://groups.google.com/d/msg/paste-users/-/6Nsoz7X34d4J.

>
> To post to this group, send email to paste...@googlegroups.com.
> To unsubscribe from this group, send email to

> paste-users...@googlegroups.com.

[Mike Orr]

What is a WSGI proxy and who will use it? The "client" terminology
makes it sound like a browser emulator like Mechanize. Is that what it
is?

--
Mike Orr <slugg...@gmail.com>

[Thomas G. Willis]

I use it all the time for calling webservices from my appengine app.
And testing websites using webtest, and when I need to poke around on
my HTTP endpoints in IPython

It's nice because I can deal with the same data structure for client
calls as what i'm using on the server. So getting familiar with one
API for both sides.

I don't have to learn requests.Request or urllib.Request

Thomas G. Willis

[Ian Bicking]

It takes a webob request (or WSGI request) and makes an actual HTTP request based on that, and in turn creates a WSGI response and WebOb Response object from the result.  The name "proxy" came about in the past because you can use a WSGI server and this client code to proxy a request through a server.

[Mike Orr]

Does it have a cookie jar; e.g., to log into sites that have a longin
form and then do something that requires a session cookie?

--
Mike Orr <slugg...@gmail.com>

[Thomas G. Willis]

Mike,

As part of my messing around I managed some middleware to do that.

https://github.com/twillis/webobtoolkit/blob/master/webobtoolkit/filters.py#L149

It wasn't much code at all.

since send_request_app is just a wsgi application, you can apply all
the middleware you want to build up the functionality you would want
in a client.

My stab at what a functional client might be is to automatically
decode gzip responses and ask for gzip, cookie support, and logging
the request/response I find handy often. which resulted in this
function

https://github.com/twillis/webobtoolkit/blob/master/webobtoolkit/client.py#L10

Thomas G. Willis