I've been working on understanding how to do testing of TG apps.
The TG book mentions mechanize.  I didn't like how it talks to the
TG server through the web interface.  My other unittests use the
testutil module to issue requests directly though createRequest,
and I liked that, instead of starting the server.  For example, I
get to recreate my test data set every time.

I wrote an adapter to replace Mechanize's HTTPHandler.  If
the URL request is made to a given host/port it is intercepted
by the adapter, converted into a TG request, processed by
the web app, and the TG response converted into the form
that Mechanize expects.

Here it is

import httplib
import mechanize
from mechanize import _http, _response
from turbogears import testutil
import cherrypy
import rfc822

try:
    from cStringIO import StringIO
except ImportError:
    from StringIO import StringIO

class TGHTTPHandler(_http.HTTPHandler):
    """intercept requests to the given http host:port and call
TurboGears directly"""
    def __init__(self,
                 intercept_host = "localhost",
                 intercept_port = "8080",
                 intercept_address = "127.0.0.1",
                 debuglevel=0):
        _http.HTTPHandler.__init__(self, debuglevel)
        self.intercept_host = intercept_host
        self.intercept_port = intercept_port
        self.intercept_address = intercept_address

    def http_open(self, req):
        # The request is either the hostname or the hostname:port
combination
        # Normalize to make it easier to compare.
        host = req.get_host()
        if ":" in host:
            host, port = host.split(":", 1)
        else:
            port = "80"

        # if self.intercept_port is None then should I intercept all
ports
        # to the intercept machine?
        if (host != self.intercept_host or port !=
self.intercept_port):
            # Let other requests go through.  Useful when testing that
the
            # TG app correctly links to external URLs
            return _http.HTTPHandler.http_open(self, req)

        # Pretty much copied this from what mechanize does to make the
request
        headers = dict(req.headers)
        headers.update(req.unredirected_hdrs)
        headers["Connection"] = "close"  # do not support HTTP 1.1
pipelining
        # normalize request header keys
        headers = dict( [(k.title(), v) for (k,v) in
headers.items()] )

        # convert a POST document into a file-like object
        rfile = None
        if req.data:
            rfile = StringIO(req.data)

        # Call the local TG server.  Here is the createRequest
signature
        #   create_request(request, method='GET', protocol='HTTP/1.1',
headers={}, rfile=None,
        #     clientAddress='127.0.0.1', remoteHost='localhost',
scheme='http')
        # XXX TG has no way to override the port?
        testutil.createRequest(request = req.get_selector(),
                               method = req.get_method(),
                               headers = headers,
                               rfile = rfile,
                               clientAddress = self.intercept_address,
                               remoteHost = host)

        # Convert the TG response to the expected form for Mechanize
        response = cherrypy.response
        fp = StringIO(response.body[0])
        code, msg = response.status.split(None, 1)
        code = int(code)
        # Mechanize uses an rfc822.Message and not a dictionary of
items
        headers = rfc822.Message(StringIO(""))
        for (k,v) in response.headers.sorted_list():
            headers[k] = v

        # This is a bit of a waste because downstream processors end
        # up making this seekable, even though it already is seekable
        return _response.closeable_response(
            fp, headers, req.get_full_url(), code, msg)

def Browser(*args, **kwargs):
    b = mechanize.Browser(*args, **kwargs)
    # remove the old HTTPHandler -- this is a hack and I don't know
the
    # right way to remove an old handler.
    b.handlers = [h for h in b.handlers if not isinstance(h,
_http.HTTPHandler)]
    # add the intercepting handler
    b.add_handler(TGHTTPHandler())
    return b

###### Example use -- I have a project called "pachy3"

from pachy3.controllers import Root
from turbogears import startup

def go():
    cherrypy.root = Root()
    startup.startTurboGears()
    b = Browser()
    b.set_handle_robots(False)

    b.open("http://localhost:8080/")
    print b.title()

    startup.stopTurboGears()

go()
        Andrew Dalke
        da...@dalkescientific.com

On Mar 28, 8:59 am, "Andrew Dalke" <andrewda...@gmail.com> wrote:

In passing it mentions twill, and I've seen Titus' blogs on that
package.

I looked at twill just now.  It supports calling a server through
wsgi,
which seems better than my hacked-up adapter.

Is that how people usually use twill to talk to a TG app for doing
unit tests?

        Andrew
        dalke @ dalke scientific . com

Hi Andrew,

just yesterday I solved the same problem you had with mechanize. Titus
wrote wsgi_intercept (http://darcs.idyll.org/~t/projects/
wsgi_intercept/README.html). He gave me some advice on how to setup
wsgi interception with mechanize as the old version of wsgi_intercept
doesn't work with current mechanize. I'm now using the
wsgi_interception stuff from the current twill egg.

I'm planning to write a short wiki entry how to do the setup. My code
is in a very raw/hacked state and I might need some free afternoons to
find a clean solution to post. I will later try playing with
wsgi_intercepted twill to find out which one works better for me:
twill or mechanize.

See also http://ivory.idyll.org/articles/twill-and-wsgi_intercept.html
if you haven't already.

Bastian

On Mar 28, 9:47 am, "Andrew Dalke" <andrewda...@gmail.com> wrote:

On Mar 28, 1:10 pm, "Bastian" <bastian.b...@gmail.com> wrote:

I tried installing from the latest egg but there was a problem with
forms.
Perhaps only when there are multiple forms?  It took a while to figure
out.
The twill archives show the problem was with the latest release and
Titus suggested trying an updated version.  That seems to work, but
I haven't had the time to try making twill control TurboGears through
WSGI,
and only hints on the web of how I might do it.

Please do.  :)
         Andrew
         dalke @ dalke scientific . com

This is what I hacked so far to get my TurboGears pages tested using
wsgi interception.

#wsgi_testutil.py
# Many code snippets stolen from Titus Brown
# http://www.advogato.org/article/874.html
from turbogears import testutil
from tgjob.controllers import Root
import cherrypy
from twill import wsgi_intercept

class WSGITest(testutil.DBTest):
    def setUp(self):
        testutil.DBTest.setUp(self)

        _cached_app = {}
        ### dynamically created function to build & return a WSGI app
        ### for a CherryPy Web app.
        def get_wsgi_app(_cached_app=_cached_app):
            if not _cached_app:
                cherrypy.root = Root()
                # configure cherrypy to be quiet ;)
                #cherrypy.config.update({ "server.logToScreen" :
False })
                testutil.start_cp()
                # get WSGI app.
                from cherrypy._cpwsgi import wsgiApp
                _cached_app['app'] = wsgiApp
            return _cached_app['app']

        wsgi_intercept.add_wsgi_intercept('localhost', 80,
get_wsgi_app)

    def tearDown(self):
        wsgi_intercept.remove_wsgi_intercept('localhost', 80)
        # shut down the cherrypy server.
        #cherrypy.server.stop()
        testutil.DBTest.tearDown(self)

And here is a example using the WSGITest class.

#test_views.py
# Some code snippets stolen from Titus Brown
# http://www.advogato.org/article/874.html
from turbogears import testutil, database
import tgjob.model
from data import setup_model_data
import cherrypy
import wsgi_testutil

database.set_db_uri("sqlite:///:memory:")

class TestPages(wsgi_testutil.WSGITest):

    model = tgjob.model

    def test_open_job_using_mechanize(self):
        "The new job offer page saves job details"
        from twill._browser import PatchedMechanizeBrowser as Browser
        b = Browser()
        b.open("http://localhost/")
        # setup_model_data() initializes some records. For unknown
reasons it only works after b.open was called
        setup_model_data()
        # load the page with the set up data
        b.open("http://localhost/")
        r = b.follow_link(text=r"Senior Python Developer")
        assert b.viewing_html()

    def test_save_job_using_twill(self):
        "The new job offer page saves job details"
        import twill
        # while we're at it, snarf twill's output.
        from StringIO import StringIO
        outp = StringIO()
        twill.set_output(outp)
        from twill.commands import *
        go("http://localhost/")
        setup_model_data()
        go("http://localhost/")
        follow(u'Place a new job offer')
        formvalue(1, "title", "fooTitle")
        formvalue(1, "company", "fooCompany")
        formvalue(1, "place", "fooPlace")
        formvalue(1, "homepage", "fooHomepage")
        formvalue(1, "description", "fooDescription")
        formvalue(1, "contact", "fooContact")
        submit()
        url("http://localhost")
#         from twill import get_browser
#         b = get_browser()
#         b.go("http://localhost/")
#         setup_model_data()
#         b.go("http://localhost/")
#         b.follow_link(b.find_link(u'Place a new job offer'))
#         ...

Hope that helps someone.

Bastian

On 28 Mrz., 13:10, "Bastian" <bastian.b...@gmail.com> wrote:

Thank you, I need it too..

I've just tried it but got it working for the first test method only.

I took out the DBTest part because I need to have my own

The first test method works and has the expected content. I can also
load other pages and do everything I am supposed to.
The second and subsequent methods have no such luck:

  File
"/usr/lib/python2.4/site-packages/twill-0.9b1-py2.4.egg/twill/other_package s/mechanize/_mechanize.py",
line 156, in open
    return self._mech_open(url, data)
  File
"/usr/lib/python2.4/site-packages/twill-0.9b1-py2.4.egg/twill/other_package s/mechanize/_mechanize.py",
line 207, in _mech_open
    raise response
httperror_seek_wrapper: HTTP Error 500: Internal error

Calling server.stop() makes no difference.
I guess I need to look closer at the _cached_app magic..

I had the the same problem yesterday. After commenting
cherrypy.server.stop() it worked for me. Anyway, I didn't try your
code yet. Don't know if it is important, but I'm starting my tests
using nosetest.

Bastian

On Mar 29, 12:37 pm, Marco Mariani <marco.mari...@prometeia.it> wrote:

Makes no difference to me.

What solved the problem for me, is removing the cache thing altogether: