using mechanize with Turbogears
Andrew Dalke
The TG book mentions mechanize. I didn't like how it talks to the
TG server through the web interface. My other unittests use the
testutil module to issue requests directly though createRequest,
and I liked that, instead of starting the server. For example, I
get to recreate my test data set every time.
I wrote an adapter to replace Mechanize's HTTPHandler. If
the URL request is made to a given host/port it is intercepted
by the adapter, converted into a TG request, processed by
the web app, and the TG response converted into the form
that Mechanize expects.
Here it is
import httplib
import mechanize
from mechanize import _http, _response
from turbogears import testutil
import cherrypy
import rfc822
try:
from cStringIO import StringIO
except ImportError:
from StringIO import StringIO
class TGHTTPHandler(_http.HTTPHandler):
"""intercept requests to the given http host:port and call
TurboGears directly"""
def __init__(self,
intercept_host = "localhost",
intercept_port = "8080",
intercept_address = "127.0.0.1",
debuglevel=0):
_http.HTTPHandler.__init__(self, debuglevel)
self.intercept_host = intercept_host
self.intercept_port = intercept_port
self.intercept_address = intercept_address
def http_open(self, req):
# The request is either the hostname or the hostname:port
combination
# Normalize to make it easier to compare.
host = req.get_host()
if ":" in host:
host, port = host.split(":", 1)
else:
port = "80"
# if self.intercept_port is None then should I intercept all
ports
# to the intercept machine?
if (host != self.intercept_host or port !=
self.intercept_port):
# Let other requests go through. Useful when testing that
the
# TG app correctly links to external URLs
return _http.HTTPHandler.http_open(self, req)
# Pretty much copied this from what mechanize does to make the
request
headers = dict(req.headers)
headers.update(req.unredirected_hdrs)
headers["Connection"] = "close" # do not support HTTP 1.1
pipelining
# normalize request header keys
headers = dict( [(k.title(), v) for (k,v) in
headers.items()] )
# convert a POST document into a file-like object
rfile = None
if req.data:
rfile = StringIO(req.data)
# Call the local TG server. Here is the createRequest
signature
# create_request(request, method='GET', protocol='HTTP/1.1',
headers={}, rfile=None,
# clientAddress='127.0.0.1', remoteHost='localhost',
scheme='http')
# XXX TG has no way to override the port?
testutil.createRequest(request = req.get_selector(),
method = req.get_method(),
headers = headers,
rfile = rfile,
clientAddress = self.intercept_address,
remoteHost = host)
# Convert the TG response to the expected form for Mechanize
response = cherrypy.response
fp = StringIO(response.body[0])
code, msg = response.status.split(None, 1)
code = int(code)
# Mechanize uses an rfc822.Message and not a dictionary of
items
headers = rfc822.Message(StringIO(""))
for (k,v) in response.headers.sorted_list():
headers[k] = v
# This is a bit of a waste because downstream processors end
# up making this seekable, even though it already is seekable
return _response.closeable_response(
fp, headers, req.get_full_url(), code, msg)
def Browser(*args, **kwargs):
b = mechanize.Browser(*args, **kwargs)
# remove the old HTTPHandler -- this is a hack and I don't know
the
# right way to remove an old handler.
b.handlers = [h for h in b.handlers if not isinstance(h,
_http.HTTPHandler)]
# add the intercepting handler
b.add_handler(TGHTTPHandler())
return b
###### Example use -- I have a project called "pachy3"
from pachy3.controllers import Root
from turbogears import startup
def go():
cherrypy.root = Root()
startup.startTurboGears()
b = Browser()
b.set_handle_robots(False)
b.open("http://localhost:8080/")
print b.title()
startup.stopTurboGears()
go()
Andrew Dalke
da...@dalkescientific.com
Andrew Dalke
> I've been working on understanding how to do testing of TG apps.
> The TG book mentions mechanize.
In passing it mentions twill, and I've seen Titus' blogs on that
package.
I looked at twill just now. It supports calling a server through
wsgi,
which seems better than my hacked-up adapter.
Is that how people usually use twill to talk to a TG app for doing
unit tests?
Andrew
dalke @ dalke scientific . com
Bastian
just yesterday I solved the same problem you had with mechanize. Titus
wrote wsgi_intercept (http://darcs.idyll.org/~t/projects/
wsgi_intercept/README.html). He gave me some advice on how to setup
wsgi interception with mechanize as the old version of wsgi_intercept
doesn't work with current mechanize. I'm now using the
wsgi_interception stuff from the current twill egg.
I'm planning to write a short wiki entry how to do the setup. My code
is in a very raw/hacked state and I might need some free afternoons to
find a clean solution to post. I will later try playing with
wsgi_intercepted twill to find out which one works better for me:
twill or mechanize.
See also http://ivory.idyll.org/articles/twill-and-wsgi_intercept.html
if you haven't already.
Bastian
Andrew Dalke
> just yesterday I solved the same problem you had with mechanize. Titus
> wrote wsgi_intercept (http://darcs.idyll.org/~t/projects/
> wsgi_intercept/README.html). He gave me some advice on how to setup
> wsgi interception with mechanize as the old version of wsgi_intercept
> doesn't work with current mechanize. I'm now using the
> wsgi_interception stuff from the current twill egg.
I tried installing from the latest egg but there was a problem with
forms.
Perhaps only when there are multiple forms? It took a while to figure
out.
The twill archives show the problem was with the latest release and
Titus suggested trying an updated version. That seems to work, but
I haven't had the time to try making twill control TurboGears through
WSGI,
and only hints on the web of how I might do it.
> I'm planning to write a short wiki entry how to do the setup.
Please do. :)
Bastian
wsgi interception.
#wsgi_testutil.py
# Many code snippets stolen from Titus Brown
# http://www.advogato.org/article/874.html
from turbogears import testutil
from tgjob.controllers import Root
import cherrypy
from twill import wsgi_intercept
class WSGITest(testutil.DBTest):
def setUp(self):
testutil.DBTest.setUp(self)
_cached_app = {}
### dynamically created function to build & return a WSGI app
### for a CherryPy Web app.
def get_wsgi_app(_cached_app=_cached_app):
if not _cached_app:
cherrypy.root = Root()
# configure cherrypy to be quiet ;)
#cherrypy.config.update({ "server.logToScreen" :
False })
testutil.start_cp()
# get WSGI app.
from cherrypy._cpwsgi import wsgiApp
_cached_app['app'] = wsgiApp
return _cached_app['app']
wsgi_intercept.add_wsgi_intercept('localhost', 80,
get_wsgi_app)
def tearDown(self):
wsgi_intercept.remove_wsgi_intercept('localhost', 80)
# shut down the cherrypy server.
#cherrypy.server.stop()
testutil.DBTest.tearDown(self)
And here is a example using the WSGITest class.
#test_views.py
# Some code snippets stolen from Titus Brown
# http://www.advogato.org/article/874.html
from turbogears import testutil, database
import tgjob.model
from data import setup_model_data
import cherrypy
import wsgi_testutil
database.set_db_uri("sqlite:///:memory:")
class TestPages(wsgi_testutil.WSGITest):
model = tgjob.model
def test_open_job_using_mechanize(self):
"The new job offer page saves job details"
from twill._browser import PatchedMechanizeBrowser as Browser
b = Browser()
b.open("http://localhost/")
# setup_model_data() initializes some records. For unknown
reasons it only works after b.open was called
setup_model_data()
# load the page with the set up data
b.open("http://localhost/")
r = b.follow_link(text=r"Senior Python Developer")
assert b.viewing_html()
def test_save_job_using_twill(self):
"The new job offer page saves job details"
import twill
# while we're at it, snarf twill's output.
from StringIO import StringIO
outp = StringIO()
twill.set_output(outp)
from twill.commands import *
go("http://localhost/")
setup_model_data()
go("http://localhost/")
follow(u'Place a new job offer')
formvalue(1, "title", "fooTitle")
formvalue(1, "company", "fooCompany")
formvalue(1, "place", "fooPlace")
formvalue(1, "homepage", "fooHomepage")
formvalue(1, "description", "fooDescription")
formvalue(1, "contact", "fooContact")
submit()
url("http://localhost")
# from twill import get_browser
# b = get_browser()
# b.go("http://localhost/")
# setup_model_data()
# b.go("http://localhost/")
# b.follow_link(b.find_link(u'Place a new job offer'))
# ...
Hope that helps someone.
Bastian
On 28 Mrz., 13:10, "Bastian" <bastian.b...@gmail.com> wrote:
> Hi Andrew,
>
> just yesterday I solved the same problem you had with mechanize. Titus
> wrote wsgi_intercept (http://darcs.idyll.org/~t/projects/
> wsgi_intercept/README.html). He gave me some advice on how to setup
> wsgi interception with mechanize as the old version of wsgi_intercept
> doesn't work with current mechanize. I'm now using the
> wsgi_interception stuff from the current twill egg.
>
> I'm planning to write a short wiki entry how to do the setup. My code
> is in a very raw/hacked state and I might need some free afternoons to
> find a clean solution to post. I will later try playing with
> wsgi_intercepted twill to find out which one works better for me:
> twill or mechanize.
>
> See alsohttp://ivory.idyll.org/articles/twill-and-wsgi_intercept.html
Marco Mariani
> This is what I hacked so far to get my TurboGears pages tested using
> wsgi interception.
>
Thank you, I need it too..
I've just tried it but got it working for the first test method only.
I took out the DBTest part because I need to have my own
> from twill import wsgi_intercept
> from project.controllers import Root
> import cherrypy, unittest
> from turbogears import testutil
> from twill._browser import PatchedMechanizeBrowser as Browser
>
> class WSGITest(unittest.TestCase):
> def setUp(self):
> _cached_app = {}
> def get_wsgi_app(_cached_app=_cached_app):
> if not _cached_app:
> cherrypy.root = Root()
> testutil.start_cp()
> from cherrypy._cpwsgi import wsgiApp
> _cached_app['app'] = wsgiApp
> return _cached_app['app']
>
> wsgi_intercept.add_wsgi_intercept('localhost', 80, get_wsgi_app)
>
> def tearDown(self):
> wsgi_intercept.remove_wsgi_intercept('localhost', 80)
> # cherrypy.server.stop()
>
>
> class RootControllerTest(WSGITest):
>
> def test_index(self):
> b = Browser()
> b.open('http://localhost/')
>
> def test_index1(self):
> b = Browser()
> b.open('http://localhost/')
The first test method works and has the expected content. I can also
load other pages and do everything I am supposed to.
The second and subsequent methods have no such luck:
File
"/usr/lib/python2.4/site-packages/twill-0.9b1-py2.4.egg/twill/other_packages/mechanize/_mechanize.py",
line 156, in open
return self._mech_open(url, data)
File
"/usr/lib/python2.4/site-packages/twill-0.9b1-py2.4.egg/twill/other_packages/mechanize/_mechanize.py",
line 207, in _mech_open
raise response
httperror_seek_wrapper: HTTP Error 500: Internal error
Calling server.stop() makes no difference.
I guess I need to look closer at the _cached_app magic..
Bastian
cherrypy.server.stop() it worked for me. Anyway, I didn't try your
code yet. Don't know if it is important, but I'm starting my tests
using nosetest.
Bastian
Marco Mariani
> I had the the same problem yesterday. After commenting
> cherrypy.server.stop() it worked for me.
Makes no difference to me.
What solved the problem for me, is removing the cache thing altogether:
> cherrypy.root = Root()
> testutil.start_cp()
>
> class WSGITest(unittest.TestCase):
> def setUp(self):
> wsgi_intercept.add_wsgi_intercept('localhost', 80, lambda:
> wsgiApp)