Testing form submittal

[Ken Kuhlman]

Is it possible/reasonable to test form submission using testutil?  I was thinking that I could just emulate the form submission by passing the appropriate args on the url as query parameters with testutil.create_request. . but that approach doesn't replicate the problem I'm seeing with my code when I hit the page with a form submit from a browser. 

 

Should I be using twill or paste.fixture instead?  If so, which is superior?  From some light blog reading, it looks to me like twill has the edge [1], but I'd be interested in the opinion of someone with more experience in these two tools.

 

thanks,

-Ken

 

[1]

Ian Bicking's 2005 test framework comparison & responses

http://blog.ianbicking.org/best-of-the-web-app-test-frameworks.html

http://www.advogato.org/person/titus/diary/119.html

http://ivory.idyll.org/articles/twill-and-wsgi_intercept.html

 

Using wsgi interception testing with TG (from "Bastian")

http://www.mail-archive.com/turbo...@googlegroups.com/msg25479.html

 

[Ken Kuhlman]

Answering my own questions, for posterity & the curious.

On Dec 29, 2007 5:05 PM, Ken Kuhlman <ksku...@gmail.com> wrote:

Is it possible/reasonable to test form submission using testutil?  I was thinking that I could just emulate the form submission by passing the appropriate args on the url as query parameters with testutil.create_request . . but that approach doesn't replicate the problem I'm seeing with my code when I hit the page with a form submit from a browser. 

My working theory on why I couldn't reproduce the problem with testutil is that the transaction wasn't getting committed (it was a DB integrity violation).   It would probably have been possible to track that down, but I feel like I'm on more solid ground with twill or paste.fixture and have decided to pursue that route [1].

 

Should I be using twill or paste.fixture instead?  If so, which is superior?  From some light blog reading, it looks to me like twill has the edge, but I'd be interested in the opinion of someone with more experience in these two tools.

Here's a brief comparison of twill & paste.fixture based on my current understanding:

twill:
+ good documentation (there's even a short book)
+ wide adoption. 78k vs 2k hits in a simplistic googlefight [2].  I've encountered examples of using twill with CherryPy, Turbogears, Django, Zope, Pylons, etc. during my wanderings.
+ twill-sh can be used as a test recorder.  There's also a maxq extension, but I'm not sure what it's current state is.

paste.fixture:
+ access to wsgi environment makes testing controller results possible.
+ nice pythonic API.
+ pylons uses it, and tg2 will be based on pylons.


Other notes:
  * Several times using twill, I've felt that the exceptions haven't been as informative as they could be.   I haven't used Paste enough to draw a comparison there.

  * I've hacked up twill to support access to the wsgi environment using wsgi_intercept [3], but Titus naturally wants me to clean it up before accepting it.  I plan on working on it more this weekend.

thanks,

-Ken

-Ken


[1] Another reason for going with a 'form aware' test framework is the ability to test identity-controlled apps.

[2] http://www.googlefight.com/index.php?lang=en_GB&word1=python+twill&word2=python+paste.fixture

[3]
http://lists.idyll.org/pipermail/twill/2008-January/000835.html

[Mark Ramm]

> Here's a brief comparison of twill & paste.fixture based on my current
> understanding:
>
> twill:
> + good documentation (there's even a short book)
> + wide adoption. 78k vs 2k hits in a simplistic googlefight [2]. I've
> encountered examples of using twill with CherryPy, Turbogears, Django, Zope,
> Pylons, etc. during my wanderings.
> + twill-sh can be used as a test recorder. There's also a maxq extension,
> but I'm not sure what it's current state is.
>
> paste.fixture:
> + access to wsgi environment makes testing controller results possible.
> + nice pythonic API.
> + pylons uses it, and tg2 will be based on pylons.
>
>
> Other notes:
> * Several times using twill, I've felt that the exceptions haven't been as
> informative as they could be. I haven't used Paste enough to draw a
> comparison there.
>
> * I've hacked up twill to support access to the wsgi environment using
> wsgi_intercept [3], but Titus naturally wants me to clean it up before
> accepting it. I plan on working on it more this weekend.

I definitely think we should support both Twill and paste.fixture in
TurboGears 2, but we need to pick a default, and I think that
paste.fixture offers the path of least resistance for TG2. And I've
been working with it quite a bit recently, and it seems to be very
pythonic, reasonable, and I'm sure that we will be able to make
testing a lot nicer in tg2 than tg1 was.

Having a good set of twill+wsgi intercept recipies will be a huge boon
to TurboGears and Pylons users, as it would be nice to be able to
write some tests that sometimes run in process and sometimes run
against the full server stack -- for instance when writing "deployment
tests" that verify that a given deployment setup actually works ;)

--Mark

[Kumar McMillan]

On Jan 7, 2008 11:48 AM, Mark Ramm <mark.mch...@gmail.com> wrote:
>
> > Here's a brief comparison of twill & paste.fixture based on my current
> > understanding:
> >
> > twill:
> > + good documentation (there's even a short book)
> > + wide adoption. 78k vs 2k hits in a simplistic googlefight [2]. I've
> > encountered examples of using twill with CherryPy, Turbogears, Django, Zope,
> > Pylons, etc. during my wanderings.
> > + twill-sh can be used as a test recorder. There's also a maxq extension,
> > but I'm not sure what it's current state is.
> >
> > paste.fixture:
> > + access to wsgi environment makes testing controller results possible.
> > + nice pythonic API.
> > + pylons uses it, and tg2 will be based on pylons.
> >
> >
> > Other notes:
> > * Several times using twill, I've felt that the exceptions haven't been as
> > informative as they could be. I haven't used Paste enough to draw a
> > comparison there.
> >
> > * I've hacked up twill to support access to the wsgi environment using
> > wsgi_intercept [3], but Titus naturally wants me to clean it up before
> > accepting it. I plan on working on it more this weekend.
>
> I definitely think we should support both Twill and paste.fixture in
> TurboGears 2, but we need to pick a default, and I think that
> paste.fixture offers the path of least resistance for TG2.

why not jump right into webtest?
http://pythonpaste.org/webtest/

It adds yet another requirement but this is the next generation of
paste.fixture so it will be better maintained and already offers some
nicer features plus an even more pythonic interface. One nicety is
how it integrates with lxml.html to provide xpath and other useful
access to HTML docs (even tries hard to support broken or janky
markup). This is not required, btw, and will "not work" gracefully if
you don't have lxml.html installed. Note that this specific feature
also requires lxml2, which may still be in beta.

Kumar

[Ken Kuhlman]

On Jan 7, 2008 11:48 AM, Mark Ramm <mark.mch...@gmail.com> wrote:

> Here's a brief comparison of twill & paste.fixture based on my current
> understanding:
>
> twill:
> + good documentation (there's even a short book)
> + wide adoption. 78k vs 2k hits in a simplistic googlefight [2].  I've
> encountered examples of using twill with CherryPy, Turbogears, Django, Zope,
> Pylons, etc. during my wanderings.
> + twill-sh can be used as a test recorder.  There's also a maxq extension,
> but I'm not sure what it's current state is.
>
> paste.fixture:
> + access to wsgi environment makes testing controller results possible.
> + nice pythonic API.
> + pylons uses it, and tg2 will be based on pylons.
>
>
> Other notes:
>

>   * I've hacked up twill to support access to the wsgi environment using
> wsgi_intercept [3], but Titus naturally wants me to clean it up before
> accepting it.  I plan on working on it more this weekend.

I definitely think we should support both Twill and paste.fixture in
TurboGears 2, but we need to pick a default, and I think that
paste.fixture offers the path of least resistance for TG2.   And I've
been working with it quite a bit recently, and it seems to be very
pythonic, reasonable, and I'm sure that we will be able to make
testing a lot nicer in tg2 than tg1 was.

That's cool... I realize that tg2 needs to pick a default & right now the sane choice is paste.fixture.   On the other hand, you can't blame me for wanting to change the dynamic by bringing some of Ian Bicking's great ideas to the masses with improvements to twill.   Today, access to the wsgi environment.  Tomorrow, perhaps a more pythonic API.  :-)

I started down this path because of the problem above, and because testing identity-driven sites in tg1 is more difficult than it needs to be (yes, I've seen thesamet's blog).   If I'm successful and it becomes a part of TG 2 eventually, I'll be ecstatic, but my primary purpose is to scratch an itch that I have today.
 

Having a good set of twill+wsgi intercept recipies will be a huge boon
to TurboGears and Pylons users, as it would be nice to be able to
write some tests that sometimes run in process and sometimes run
against the full server stack -- for instance when writing "deployment
tests" that verify that a given deployment setup actually works ;)

--Mark

Agreed.  wsgi_intercept is a brilliant idea (another one of Ian's), with a nice implementation from Titus.


A few more bits of randomness:

1) I'm working toward a testutil replacement using the twill enhancement I mentioned.  I hope to be able to use it in tg1, tg2, and the cherrypy3 branch. 

2) Since twill enjoys wide adoption, users of other frameworks have probably been exposed to it, and many are using it already.  I'd like for them to be able to keep their tests intact as possible as they move to TG.

3) I'm concerned that there are a lot of people that don't have unit tests for their TG 1 apps, and they'll perhaps have a harder time switching to TG 2 than they should.  twill-script provides a quick mechanism for adding tests to an existing site.

4) I'm going to be hacking around on this during the sprint this weekend.

 

Thanks for the response!
-Ken

[Mark Ramm]

> 1) I'm working toward a testutil replacement using the twill enhancement I
> mentioned. I hope to be able to use it in tg1, tg2, and the cherrypy3
> branch.
>
> 2) Since twill enjoys wide adoption, users of other frameworks have
> probably been exposed to it, and many are using it already. I'd like for
> them to be able to keep their tests intact as possible as they move to TG.
>
> 3) I'm concerned that there are a lot of people that don't have unit tests
> for their TG 1 apps, and they'll perhaps have a harder time switching to TG
> 2 than they should. twill-script provides a quick mechanism for adding
> tests to an existing site.
>
> 4) I'm going to be hacking around on this during the sprint this weekend.

This sounds like a great idea. If we had a stable test interface
between TG1 and 2 (and some hopeful CherryPy3 version of TG) that
would be amazingly helpful for people as they migrate forward. And I
was never super-excited about the way we monkeyed arround with
cherrypy internals to get the TG1 test interface, and it's slowed us
up more than once when we tried to do the CherryPy 3/1.1 work, and
again when thinking about the TG2 upgrade path.

--Mark Ramm