Testing the db? ( and more )
iain duncan
interactions are actually doing what you think in the db? I've been
doing a lot of reading but the options are certainly overwhelming.
Do you use Nose to set up a blank in memory db for running the
integrations tests? And if so, when testing model methods, do you look
at the db some other way than through the model? For example, I have
some complex association objects in my SA model, and it's easy for me to
be off base on what a model operation is actually doing to the
interrelated tables. So it seems like I should verify those with sql
query based assertions. I haven't seen examples of that, so I don't know
if I'm crazy here.
Ie:
order.add_product( basket_1, 1 )
doesn't just add one product it has a bunch of business rules that
depend on what's already in it and what's going in. So I should be
testing to look at the orders-baskets-product_associations to see
whether they are doing what I think. I'm not sure what the best way to
run raw selects on the in memory db would be in order to keep tests
quick to write and run.
And feedback appreciated. My subcontractor/partner and I are embarking
on a big fix-up-our-shoddy-operations kick, so perhaps we could make a
wiki aimed at beginners on testing tg apps. Would this be useful? Most
of our docs on testing are good, but a bit inaccessible to the new
developer.
My thoughts so far is that I will be using
- nose with puunit
- tg testutils
- twill with and without the wsgi interceptor
- ?? for db hits
- selenium for browse with js?
Thanks
Iain
Alberto Valverde
I've been developing a test framework for a couple of TG apps lately so
I can share what has been working well for me so far:
1) Models are tested independently of the controllers. You must make
sure though that you call turbogears.update_config before the model
module(s) are loaded so the metadata is properly bound to the engine.
You can do this with a nose plugin if using nose (preferred, since it
lets you use doctests) or by updating the config in tests/__init__.py so
it's updated before running the tests.
Turbogears has infrastructure to test models but I'm not sure if it has
been updated to use SA so you might need to implement all this by
yourself (or help me lobby for its inclusion in TG 1,1 ;)
To test them I use a base unittest.TestCase subclass that:
i) In setUp() creates a sqlite database in-memory, creates all tables,
clears session.
ii) In tearDown() Drop all tables.
Sqlite with an in-memory database is *highly* recommended because
dropping tables and recreating them for every singe test becomes quite
expensive quite fast.
You can also use transactions for test-isolation and roll them back in
tearDown but I'd advice against this because you won't be able to
subclass the ModelTestCase for the controller unit tests since the
controller will commit the transaction and test-isolation bye, bye :)
2) Controller unit tests. I don't use TG's infrastructure for this but
use TG's app as an WSGI app and test it with paste.fixture since I like
its API much more (can follow redirects, saves cookies (for tests that
need an authenticated user or persist state between requests)), parses
forms, etc. and, most importantly, treats the app it is testing as a
black-box, unlike the current testing API which depends on peeking
inside cherrypy's internals (assert "something" in
cherrypy.response.body[0], yuck!).
I usually break the "purity" of blackbox testing for convenience by
directly creating the business objects I need for each test by accessing
the model directly. This is why advice against transactions for test
isolation.
The benefit of this approach is that it can be used with twill too or
any wsgi app testing framework (WebTest, Windmill?). Another benefit is
that tests are independent of TG so, theoretically, you could even swap
TG under the hood and use the same functional tests to check the
migration went ok. This could be very useful if you ever plan to port a
1.X app to TG 2.0.
If there's interest, I could commit some changes to TG 1.1 (Florent? are
you following this? ;) to make this "style" of functional testing
available. As a side effect, it will be easier to use (some [1]) WSGI
middleware with TG. 1.1
Alberto
[1] I'm saying "some" because the way CP 2.2 makes its WSGI app
available to the outer world is so funky (IMHO) that not all middleware
can be used, more precisely, middleware which expects to be executed in
the same thread as the app it's wrapping. But I'm digressing now... :)
iain duncan
> I've been developing a test framework for a couple of TG apps lately so
> I can share what has been working well for me so far:
Much appreciated!
>
> 1) Models are tested independently of the controllers. You must make
> sure though that you call turbogears.update_config before the model
> module(s) are loaded so the metadata is properly bound to the engine.
> You can do this with a nose plugin if using nose (preferred, since it
> lets you use doctests) or by updating the config in tests/__init__.py so
> it's updated before running the tests.
> Turbogears has infrastructure to test models but I'm not sure if it has
> been updated to use SA so you might need to implement all this by
> yourself (or help me lobby for its inclusion in TG 1,1 ;)
Lobbying, right here!
>
> To test them I use a base unittest.TestCase subclass that:
> i) In setUp() creates a sqlite database in-memory, creates all tables,
> clears session.
> ii) In tearDown() Drop all tables.
>
> Sqlite with an in-memory database is *highly* recommended because
> dropping tables and recreating them for every singe test becomes quite
> expensive quite fast.
If you have the time, I would love to see code examples of you do the
above. What do you use to test the data in the tables? I was wondering
whether it would be worth having my model ORM object tests confirmed by
sqlalchemy query language hits.
>
> You can also use transactions for test-isolation and roll them back in
> tearDown but I'd advice against this because you won't be able to
> subclass the ModelTestCase for the controller unit tests since the
> controller will commit the transaction and test-isolation bye, bye :)
>
> 2) Controller unit tests. I don't use TG's infrastructure for this but
> use TG's app as an WSGI app and test it with paste.fixture since I like
> its API much more (can follow redirects, saves cookies (for tests that
> need an authenticated user or persist state between requests)), parses
> forms, etc. and, most importantly, treats the app it is testing as a
> black-box, unlike the current testing API which depends on peeking
> inside cherrypy's internals (assert "something" in
> cherrypy.response.body[0], yuck!).
I would love to see examples of that too.
>
> I usually break the "purity" of blackbox testing for convenience by
> directly creating the business objects I need for each test by accessing
> the model directly. This is why advice against transactions for test
> isolation.
>
> The benefit of this approach is that it can be used with twill too or
> any wsgi app testing framework (WebTest, Windmill?). Another benefit is
> that tests are independent of TG so, theoretically, you could even swap
> TG under the hood and use the same functional tests to check the
> migration went ok. This could be very useful if you ever plan to port a
> 1.X app to TG 2.0.
That was the direction I was leaning too. Glad to hear it's being
approved by those in the know. ;)
>
> If there's interest, I could commit some changes to TG 1.1 (Florent? are
> you following this? ;) to make this "style" of functional testing
> available. As a side effect, it will be easier to use (some [1]) WSGI
> middleware with TG. 1.1
>
> Alberto
Thanks again Alberto
Iain
Florent Aide
>
>
> > I've been developing a test framework for a couple of TG apps lately so
> > I can share what has been working well for me so far:
>
> Much appreciated!
sure.
> > Turbogears has infrastructure to test models but I'm not sure if it has
> > been updated to use SA so you might need to implement all this by
> > yourself (or help me lobby for its inclusion in TG 1,1 ;)
>
> Lobbying, right here!
okay okay... let's see the code then :)
> > To test them I use a base unittest.TestCase subclass that:
> > i) In setUp() creates a sqlite database in-memory, creates all tables,
> > clears session.
> > ii) In tearDown() Drop all tables.
I have dbs which can cause pb on sqlite... SA fixed one issue lately
regarding decimal usage in sqlite I need to see if this would help
test some of my models with an sqlite db then...
> > Sqlite with an in-memory database is *highly* recommended because
> > dropping tables and recreating them for every singe test becomes quite
> > expensive quite fast.
yep sure if above problem is fixed.
> > If there's interest, I could commit some changes to TG 1.1 (Florent? are
> > you following this? ;) to make this "style" of functional testing
> > available. As a side effect, it will be easier to use (some [1]) WSGI
> > middleware with TG. 1.1
Yes. and Yes. :-)
I have a project to completely refactor the config of TG. It is too
coupled with the CP config. I'd like to keep the exact same API as we
have now but have our own config instance instance of using the config
instance from CP. The idea would be to then "instrumentize" the CP
instance in some specific methods that would make it easier to
instrument any other engine config ...
I'll try to commit some things this week end, I hope this won't clash
with your own commits :)
Cheers,
Florent.
Alberto Valverde
> On Nov 16, 2007 11:19 PM, iain duncan <iaind...@telus.net> wrote:
>>
>>> I've been developing a test framework for a couple of TG apps lately so
>>> I can share what has been working well for me so far:
>> Much appreciated!
>
> sure.
>
>>> Turbogears has infrastructure to test models but I'm not sure if it has
>>> been updated to use SA so you might need to implement all this by
>>> yourself (or help me lobby for its inclusion in TG 1,1 ;)
>> Lobbying, right here!
>
> okay okay... let's see the code then :)
Ok, I'll be breaking the 1.1 branch quite badly soon then... :)
>
>>> To test them I use a base unittest.TestCase subclass that:
>>> i) In setUp() creates a sqlite database in-memory, creates all tables,
>>> clears session.
>>> ii) In tearDown() Drop all tables.
>
> I have dbs which can cause pb on sqlite... SA fixed one issue lately
> regarding decimal usage in sqlite I need to see if this would help
> test some of my models with an sqlite db then..
It's true that there are times when sqlite is not a possible replacement
for the "real" database because of "propietaty" features of some
databases. However, the ModelTestCase base class is quite thin so it
would be feasible to include two of them, eg a
SATransactionModelTestCase and a SAFreshDBModelTestCase (ideas for
better names please)
.
>
>
>>> Sqlite with an in-memory database is *highly* recommended because
>>> dropping tables and recreating them for every singe test becomes quite
>>> expensive quite fast.
>
> yep sure if above problem is fixed.
>
>>> If there's interest, I could commit some changes to TG 1.1 (Florent? are
>>> you following this? ;) to make this "style" of functional testing
>>> available. As a side effect, it will be easier to use (some [1]) WSGI
>>> middleware with TG. 1.1
>
> Yes. and Yes. :-)
Ok
>
> I have a project to completely refactor the config of TG. It is too
> coupled with the CP config. I'd like to keep the exact same API as we
> have now but have our own config instance instance of using the config
> instance from CP. The idea would be to then "instrumentize" the CP
> instance in some specific methods that would make it easier to
> instrument any other engine config ...
> I'll try to commit some things this week end, I hope this won't clash
> with your own commits :)
I don't think we should step into each others toes as long as
turbogears.update_config has the same behavior and API.
The changes I'm talking about are:
1) Create a wsgiapp module in the quickstarted's app package where a
paste.app_factory entrypoint is implemented. Here a WSGI app factory is
implemented that builds a WSGI app from the Root controller in a similar
way that it is done inside turbogears.startup when tg.fancy_exception is
enabled.
2) start-myapp.py becomes a thin wrapper that loads the mentioned WSGI
app and servers it with the CherryPy3 wsgiserver bundled with
PasteScript (obviously the one in CherryPy3 itself cannot be used since
it will collide with CP2)
3) Remove the now unneeded turbogears.startup.start_server and
SimpleWSGIServer.
4) Now that a TG 1.1 WSGI app and be easily consumed, it is loaded via a
nose plugin when nosetests starts and placed somewhere (eg:
turbogears.wsgi_app) so the ControllerTestCase (implemented in
turbogears.testutil or myapp.tests.__init__) can wrap it with
paste.fixtures TestApp.
It will also be useful to provide the app configured at the same
location inside tg-admin shell so shell sessions can be copied and
pasted into doctests.
I hope to get something done this weekend :)
Alberto
Alberto Valverde
>
>> (...)
>
> If you have the time, I would love to see code examples of you do the
> above. What do you use to test the data in the tables? I was wondering
> whether it would be worth having my model ORM object tests confirmed by
> sqlalchemy query language hits.
I don't think it's necessary to confirm via raw SQL that the ORM is
doing the right thing, why not trust the ORM's own testsuite?
What I do to confirm that the objects are in fact being persisted (what
I really care about) is to flush the session, clear it, retrieve the
objects back and assert they're in the state I expect them to be.
Regarding the examples you asked for, Florent has given me the green
light so you'll hopefully be able to see the code soon inside TG 1.1.
The quickstarted app will have its sample unittests written in this
fashion to serve as example. I'll be using paste.fixture (which is the
test framework TG2.0 uses at the moment for controller functional tests)
and I have no experience with other WSGI app testing frameworks so I
cannot be of much help documenting alternatives.
To give some ideas here's a simple base class for model tests I use:
from turbogears.database import metadata
from unittest import TestCase
class ModelTestCase(TestCase):
def setUp(self):
metadata.create_all()
session.clear()
def tearDown(self):
metadata.drop_all()
As you can see, it makes sure the database and session are in a blank
state before each test. Then each specific test case subclassextends
those methods, specially setUp, to populate the database with data
needed by the test methods inside the test case. For example, a test
case that tests that a user can create, delete and buy books will have
create the user inside setUp and have a test method that creates,
another that buys, etc...
However,the above sample is of little use on its own since it assumes
turbogears.config is populated and the metadata is therefore bound to
the engine. Loading the app's environment is done in a nose plugin which
I cannot copy & paste since it's very specific to the app it's being
used on and is propietary code. This is part of what I plan to
generalize for inclusion in TG.
Alberto
iain duncan
>
> 1) Create a wsgiapp module in the quickstarted's app package where a
> paste.app_factory entrypoint is implemented. Here a WSGI app factory is
> implemented that builds a WSGI app from the Root controller in a similar
> way that it is done inside turbogears.startup when tg.fancy_exception is
> enabled.
>
> 2) start-myapp.py becomes a thin wrapper that loads the mentioned WSGI
> app and servers it with the CherryPy3 wsgiserver bundled with
> PasteScript (obviously the one in CherryPy3 itself cannot be used since
> it will collide with CP2)
>
> 3) Remove the now unneeded turbogears.startup.start_server and
> SimpleWSGIServer.
>
> 4) Now that a TG 1.1 WSGI app and be easily consumed, it is loaded via a
> nose plugin when nosetests starts and placed somewhere (eg:
> turbogears.wsgi_app) so the ControllerTestCase (implemented in
> turbogears.testutil or myapp.tests.__init__) can wrap it with
> paste.fixtures TestApp.
That sounds somewhat similar to the wsgi_intercepter for Twill, maybe it
would be nice to have a template in the tests directory for doing the
same thing with Twill?
Thanks
Iain
iain duncan
> iain duncan wrote:
> >
> >> (...)
> >
> > If you have the time, I would love to see code examples of you do the
> > above. What do you use to test the data in the tables? I was wondering
> > whether it would be worth having my model ORM object tests confirmed by
> > sqlalchemy query language hits.
>
> I don't think it's necessary to confirm via raw SQL that the ORM is
> doing the right thing, why not trust the ORM's own testsuite?
In this case the issue is testing my understanding of _what_ exactly the
orm is doing when the relationships get complicated and I'm using some
multi-column associations
Thanks! Looking forward to seeing it.
Iain
Alberto Valverde
Yes, this is one of the reasons I believe that making TG expose a wsgi
app is very useful... one can then use many tools to test them, serve
them, wrap them, slice them and dice them... :)
However, I have no experience with twill so I'll leave that task to
someone else who has a better understanding.
Alberto
iain duncan
It's on my todo list this week, but it would help me if you have time to
post an example of how you get the wsgi app from tg. I don't understand
all the config stuff well enough yet to get through it quickly. If you
don't have time, no worries. I'd like to make a twill example to put up
though, especially as Grig and Titus have written some great articles
and a mini book on using it.
Iain
Kumar McMillan
>
> How do people here like to test whether their model and controller-model
> interactions are actually doing what you think in the db? I've been
> doing a lot of reading but the options are certainly overwhelming.
I've been working on a module for a while that aims to solve this problem:
http://code.google.com/p/fixture/
I haven't used it in a TG app, mostly for data processing modules, but
I have started using it in a Pylons/SQLAlchemy project.
I'll say right now however that getting fixture to work with
sqlalchemy is a *huge* pain because it wants to use a session to
setup/teardown data and this can easily conflict with your
application's session when you are trying to put fixture in a test
setup/teardown *around* your application. sqlobject is a bit more
forgiving although I am still having problems getting transactions to
work right.
I've actually beat my head up against the sqlalchemy issues in Pylons
for so long now that I've decided fixture needs to support a raw form
of loading, one that simply takes DataSet classes and builds insert
statements, then fires them off with a psycopg cursor (or whatever).
Such a version of fixture is vaporware unfortunately but if enough
people speak up about it I will be much more motivated to get a work
branch started for it ;) I will get it started soon regardless for
the Pylons app I need this for. Also, if someone wants to help on
this, let me know!
-Kumar