Test framework for end-user applications
Russell Keith-Magee
Hi all,
I've got a project at work that requires a formal test harness, so I thought I would kill two birds with one stone and spend some time adding a formal test framework for all Django applications.
This came up a few weeks ago [1], and although I didn't comment at the time, it struck me as a good idea. Plus, as Simon pointed out, it is a feature that Rails has but Django does not, and we can't let those dirty Rails-using savages have something that we don't :-) I have created ticket #2333 to track the feature.
As a first pass, I have been concentrating on adding the framework to find and run unittests and doctests within a Django application. This is essentially what tests/runtests.py already does for the Django tests, except that I am using the unittest framework to find and execute tests.
In my sandbox, I have made modifications such that:
- Any classes in models.py that extend unittest.TestCase will be identified as a test case.
- Any doctests in models.py will be extracted as test cases (using normal doctest extraction strategies - any module, class, or method level doctests, plus anything listed in a __test__ attribute)
- If a mysite.myapp.tests module exists, it will be searched for unittests and doctests in the same way as models.py
I have also added two targets to django-admin:
- 'test' which runs all the unit tests for all installed applications.
- 'testapp' which runs the unit tests for all applications named on the command line
With this new framework in place, it is possible to refactor the django tests; to this end, I have moved 'othertests' into the 'regressiontests' package, so that it looks like an 'application' within the regressiontests 'site'.
When a test targets is executed, django-admin:
1) creates a test database named 'test_' + settings.DATABASE_NAME
2) Installs all the applications in INSTALLED_APPS
3) Constructs a test suite from the test cases found in the requested apps
4) Runs the suite
5) Destroys the test database
Note that at the moment, I'm just looking at getting a basic testing framework in place - after that, I will be looking at other problems such as:
- installation of 'fixtures' (YAML/JSON/Python data for testing purposes).
- integration with selenium type tests that require a running server
I figure that once we have a generic testing framework, we can wrap just about anything else into it (for example, using purpose-specific subclasses of unittest.TestCase for different test setup/teardown sequences).
I'm almost ready to submit some patches (I still need to do some code cleanup and write some documentation), but before I do:
1) Are there any comments on the approach that I have suggested?
2) Is this a big enough change that people want to see the patch before I commit it? It shouldn't actually break anything already in place, just expose some new functionality that might be useful.
Russ Magee %-)
[1] 'Site Testing how-to', http://groups.google.com/group/django-users/browse_thread/thread/24a50006c43f43b5/eee75d48505d1af5
Michael Radziej
> I've got a project at work that requires a formal test harness, so I thought
> I would kill two birds with one stone and spend some time adding a formal
> test framework for all Django applications.
Russell, that sounds great! It looks like everyone's reinventing
the wheel here. Being a newbie in testing for python, it took me
quite some time to put all the stuff together, especially with
tests that require a test database.
I can't comment much on the proposed structure, I'm now using
py.test (but, hey, it really doesn't matter that much, and it's
always a problem to include non-released software). It's probably
very good that you include doctests AND unittests, it makes the
life easy if you don't need the power of unittest.
For my own project, I put together a testing middleware for views
and manipulators, which saves the context given to a template and
allows tests that are solely context based. The middleware can
also write out test cases while you use the browser. I do this
instead of selenium tests as long as I don't use Javascript. It
doesn't depend too much on py.test. Please send me a mail if
you're interested.
Michael
Malcolm Tredinnick
>
> Hi all,
>
> I've got a project at work that requires a formal test harness, so I
> thought I would kill two birds with one stone and spend some time
> adding a formal test framework for all Django applications.
Hmmm .. deja vu. When Adrian said he was going to refactor db/models/
bits and pieces, I put the m-i work on hold last weekend and was
thinking about Simon's post as well. I had a need to rewrite some
templates and wanted some regression tests for them.
I approached things from a different direction, though, more
concentrating on how to unit test templates and views (as in not
requiring a full end-to-end application setup in order to test them). So
it's actually a orthogonal to what you are proposing here and fits in
nicely.
I'll write up what I was thinking about in another mail; I started to
post it in response to this, but it adds no value to your proposal.
> This came up a few weeks ago [1], and although I didn't comment at the
> time, it struck me as a good idea. Plus, as Simon pointed out, it is a
> feature that Rails has but Django does not, and we can't let those
> dirty Rails-using savages have something that we don't :-) I have
> created ticket #2333 to track the feature.
>
> As a first pass, I have been concentrating on adding the framework to
> find and run unittests and doctests within a Django application. This
> is essentially what tests/runtests.py already does for the Django
> tests, except that I am using the unittest framework to find and
> execute tests.
>
> In my sandbox, I have made modifications such that:
> - Any classes in models.py that extend unittest.TestCase will be
> identified as a test case.
> - Any doctests in models.py will be extracted as test cases (using
> normal doctest extraction strategies - any module, class, or method
> level doctests, plus anything listed in a __test__ attribute)
> - If a mysite.myapp.tests module exists, it will be searched for
> unittests and doctests in the same way as models.py
We are going to see more and more cases of a models/ sub-directory, not
just models.py. So hopefully we can search through those as well
eventually. Putting all models in one file leads to very large files.
On the other hand, I'm pretty unlikely to put my tests in the model
files anyway, since I prefer them in a separate directory (again, to
keep the file sizes manageable). So if I'm being selfish, I don't care
what you do with models.py. :-) However, if I was to be professional
about it, we should maybe consider that case.
> I have also added two targets to django-admin:
> - 'test' which runs all the unit tests for all installed
> applications.
> - 'testapp' which runs the unit tests for all applications named on
> the command line
It seems that Django tradition, as handed down from the ancients across
time, is that if a command is given arguments, we run the command
against those apps. Otherwise it runs against all applications. So you
could just add "test" and it either takes the applications to test as
arguments or nothing (=> test all).
> With this new framework in place, it is possible to refactor the
> django tests; to this end, I have moved 'othertests' into the
> 'regressiontests' package, so that it looks like an 'application'
> within the regressiontests 'site'.
>
> When a test targets is executed, django-admin:
> 1) creates a test database named 'test_' + settings.DATABASE_NAME
> 2) Installs all the applications in INSTALLED_APPS
> 3) Constructs a test suite from the test cases found in the requested
> apps
> 4) Runs the suite
> 5) Destroys the test database
I wonder how much debate this will cause. :-)
This makes the tests order-dependent and possibly hard to run in
isolation. Ideally, flushing and restoring the database between each
test (i.e. it's a part of setup and tearDown, or maybe just a pre- and
post-function on each TestCase-derived class) will keep things "clean".
It will be mean the tests will take ages if you are using a disk-based
database, though.
I don't know if there's a right answer here, or even what the preferred
choice would be.
> Note that at the moment, I'm just looking at getting a basic testing
> framework in place - after that, I will be looking at other problems
> such as:
> - installation of 'fixtures' (YAML/JSON/Python data for testing
> purposes).
If you haven't played with it yet, the "python" serialisation target in
the stuff Jacob recently committed is really useful here.
> - integration with selenium type tests that require a running server
This is where you start brushing up against the areas I was thinking
about. Although I was looking at it from isolated unit testing
perspectives.
> I figure that once we have a generic testing framework, we can wrap
> just about anything else into it (for example, using purpose-specific
> subclasses of unittest.TestCase for different test setup/teardown
> sequences).
>
> I'm almost ready to submit some patches (I still need to do some code
> cleanup and write some documentation), but before I do:
> 1) Are there any comments on the approach that I have suggested?
I'm +1 (or, at worst, +0) on everything you wrote.
My personal philosophy is to provide a means without imposing a "you
must do it this way" because testing is such a religious issue that
people will want the flexibility to do it their own way. I think what
you've proposed works in comfortably with that.
I'll post a follow-up later this evening with my ideas coming at this
from the other end. They should be looked at as a "next step" beyond
what you are doing and probably not really clashing with what you are
planning.
> 2) Is this a big enough change that people want to see the patch
> before I commit it? It shouldn't actually break anything already in
> place, just expose some new functionality that might be useful.
Best wishes,
Malcolm
jpel...@gmail.com
I've been working on a plugin for my test runner (nose) that is quite
similar in operation to what you've described above. If you want to
check it out, you can get nose via easy install: easy_install nose and
the plugin here:
svn co http://svn.nose.python-hosting.com/nose-django/trunk/
If you're scratching your head and saying, what's this nose business,
take a quick look at the homepage:
http://somethingaboutorange.com/mrl/projects/nose/
Or Titus Brown's intro article here:
http://ivory.idyll.org/articles/nose-intro.html
In summary, it's py.test without the magic, plus support for plugins
that can be used to change test selection, running and reporting.
JP
Jyrki Pulliainen
> I've got a project at work that requires a formal test harness, so I
> thought I would kill two birds with one stone and spend some time adding a
> formal test framework for all Django applications.
>
> This came up a few weeks ago [1], and although I didn't comment at the time,
> it struck me as a good idea. Plus, as Simon pointed out, it is a feature
> that Rails has but Django does not, and we can't let those dirty Rails-using
> savages have something that we don't :-) I have created ticket #2333 to
> track the feature.
>
Great, I think there're many people in need of a proper unit testing framework!
> In my sandbox, I have made modifications such that:
> - Any classes in models.py that extend unittest.TestCase will be identified
> as a test case.
> - Any doctests in models.py will be extracted as test cases (using normal
> doctest extraction strategies - any module, class, or method level doctests,
> plus anything listed in a __test__ attribute)
> - If a mysite.myapp.tests module exists, it will be searched for unittests
> and doctests in the same way as models.py
I'd suggest using spesific test directories so that (for example)
every directory would have a "test" subdirectory where all the test to
that directory's objects etc. would reside.
> When a test targets is executed, django-admin:
> 1) creates a test database named 'test_' + settings.DATABASE_NAME
This is created to the database server being used? Have you thought of
spawning a new sqlite database to the memory and using that. That way
you could be sure that no extra data is left behind (when you
terminate the application, the sqlite goes down too). This would also
enable testing in enviroinments where there is only one postgres db
available for some reason.
--
Jyrki // jyrki.pu...@gmail.com
Russell Keith-Magee
For my own project, I put together a testing middleware for views
and manipulators, which saves the context given to a template and
allows tests that are solely context based. The middleware can
also write out test cases while you use the browser. I do this
instead of selenium tests as long as I don't use Javascript. It
doesn't depend too much on py.test. Please send me a mail if
you're interested.
Certainly sounds interesting. If you've got something that you want to contribute, attach it as a patch to the ticket. IMHO, the more testing flavours we can offer, the better.
Russ %-)
Russell Keith-Magee
I'll write up what I was thinking about in another mail; I started to
post it in response to this, but it adds no value to your proposal.
Baited breath, and all that :-)
We are going to see more and more cases of a models/ sub-directory, not
just models.py. So hopefully we can search through those as well
eventually. Putting all models in one file leads to very large files.
On the other hand, I'm pretty unlikely to put my tests in the model
files anyway, since I prefer them in a separate directory (again, to
keep the file sizes manageable). So if I'm being selfish, I don't care
what you do with models.py. :-) However, if I was to be professional
about it, we should maybe consider that case.
Initially I was in two minds about allowing tests in models.py. However, I came to the position that allowing tests in models.py hurts nobody, it seems the more natural case for doctests, and tests can always be moved if you end up with a lot of them.
So you
could just add "test" and it either takes the applications to test as
arguments or nothing (=> test all).
Valid point, and an easy fix. Done
> When a test targets is executed, django-admin:
> 1) creates a test database named 'test_' + settings.DATABASE_NAME
> 2) Installs all the applications in INSTALLED_APPS
> 3) Constructs a test suite from the test cases found in the requested
> apps
> 4) Runs the suite
> 5) Destroys the test database
I wonder how much debate this will cause. :-)
I wondered if someone would pick up on this :-)
This makes the tests order-dependent and possibly hard to run in
isolation. Ideally, flushing and restoring the database between each
test (i.e. it's a part of setup and tearDown, or maybe just a pre- and
post-function on each TestCase-derived class) will keep things "clean".
It will be mean the tests will take ages if you are using a disk-based
database, though.
I don't know if there's a right answer here, or even what the preferred
choice would be.
When I came up on this, I thought of three options:
1) A TestCase subclass that creates/destroy the db in setup/teardown - DB heavy
2) The test target does creation/destroy - tests are order dependent.
3) A TestCase subclass that overrides run() to do a 'global' setup/teardown - less DB create/destroys, and tests are only order dependent within any given test case. Middle ground solution.
I initially opted for (2) because it is what is already present. However, I will admit a slight personal preference for 1 or 3 (in that order). Having the TestCase subclass handle db creation means that the generic test framework remains generic, so if you need a test class that _doesn't_ need a database backend (or needs a different db backend setup), you can integrate it by writing a different type of testcase extension. To this end, 1 and 3 are not mutally exclusive - we could support both setup strategies in the django test libraries (documenting the differences as appropriate).
However, I acknowledge that these options will impose a big honking performance hit, and I'll agree that this is not necessarily desirable. Although, that said, it shouldn't _really_ matter how long it takes to run the unit test suite, as long as you get a consistent answer that gives you confidence in a modification before taking it live.
> Note that at the moment, I'm just looking at getting a basic testing
> framework in place - after that, I will be looking at other problems
> such as:
> - installation of 'fixtures' (YAML/JSON/Python data for testing
> purposes).
If you haven't played with it yet, the "python" serialisation target in
the stuff Jacob recently committed is really useful here.
I saw it in passing, but haven't played with it yet. Worth keeping in mind though - thanks.
> - integration with selenium type tests that require a running server
This is where you start brushing up against the areas I was thinking
about. Although I was looking at it from isolated unit testing
perspectives.
Ok; I'll wait to see what you have to say.
Russ %-)
Russell Keith-Magee
Just as an additional data point, or point of discussion or whatever,
...
In summary, it's py.test without the magic, plus support for plugins
that can be used to change test selection, running and reporting.
Unless you can make a particularly convincing case for using an alternative, based upon some deficiency of unittest that will adversely affect django testing, I'm inclined to stick with whats in the standard library.
If the powers-that-be want to override on this issue and declare some alternative testing framework, feel free to let me know.
That said, I am not aiming to set up a django testing framework that actively impedes the use of py.test or nose; if there is anything we can do to make the two compatible/complimentary, let me know.
Russ %-)
Malcolm Tredinnick
> On 7/12/06, Russell Keith-Magee <freakb...@gmail.com> wrote:
> > When a test targets is executed, django-admin:
> > 1) creates a test database named 'test_' + settings.DATABASE_NAME
>
> This is created to the database server being used? Have you thought of
> spawning a new sqlite database to the memory and using that. That way
> you could be sure that no extra data is left behind (when you
> terminate the application, the sqlite goes down too). This would also
> enable testing in enviroinments where there is only one postgres db
> available for some reason.
If one wants to run the tests against in-memory SQLite, then you can use
a settings file with the appropriate settings (there is nothing that
says tests and "main" need to use the same settings file, in fact
multiple settings files are easy and to be encouraged). However, any
framework have to be able to test against all possible database
configurations because of the need to test things like referential
integrity, trigger behaviour, stored procedure actions, portable SQL,
etc.
Regards,
Malcolm
Jay Parlar
> based upon some deficiency of unittest that will adversely affect django
> testing, I'm inclined to stick with whats in the standard library.
>
> If the powers-that-be want to override on this issue and declare some
> alternative testing framework, feel free to let me know.
>
> That said, I am not aiming to set up a django testing framework that
> actively impedes the use of py.test or nose; if there is anything we can do
> to make the two compatible/complimentary, let me know.
>
Well, I'll step in and give my own +1 (for whatever it's worth) to the
use of nose.
I've used nose a good deal in the past, and am completely enamored
with it. One of the fantastic things about it is that it wraps around
unittest, so it's compatible with any unittest suites people might
have.
Of course, you just using unittest by itself also gets you that, but
using nose will save you re-implementing a whole lot of stuff.
I'd suggest at least taking a look at it. All the relevant info about
it (and some pertinent examples) are pretty concisely described on the
project's main page
(http://somethingaboutorange.com/mrl/projects/nose/)
Jay P.
Adrian Holovaty
> cleanup and write some documentation), but before I do:
> 1) Are there any comments on the approach that I have suggested?
> 2) Is this a big enough change that people want to see the patch before I
> commit it? It shouldn't actually break anything already in place, just
> expose some new functionality that might be useful.
Yes, please post the patch before committing -- this is definitely a big thing.
Adrian
--
Adrian Holovaty
holovaty.com | djangoproject.com
Simon Willison
On 12 Jul 2006, at 02:28, Malcolm Tredinnick wrote:
> I approached things from a different direction, though, more
> concentrating on how to unit test templates and views (as in not
> requiring a full end-to-end application setup in order to test
> them). So
> it's actually a orthogonal to what you are proposing here and fits in
> nicely.
I've been thinking about unit testing templates and views on-and-off
for the past few months. Here's the bulk of my thinking.
The best way of running tests against a Django application, to my
mind, is to run a "fake" web server (i.e. one that doesn't actually
bind to a port) around the application. This fake server acts as a
test harness. Tests can then create HttpRequest objects (probably a
subclass called something like TestHttpRequest), pass them to the
Django application, get back an HttpResponse object and run
assertions against that.
There's just one problem with this approach: while running assertions
against an HttpResponse object is adequate, the ideal situation would
be to run assertions against the template context as well. If you
just run assertions against the HttpResponse object you end up having
to process the HTML in some way to work out if the test has passed.
What you really want to be able to do is this (pseudocode):
send_get('/polls/1/')
assert_response_status(200)
assert_response_contains('This is poll 1')
assert_template_rendered('poll_detail.html')
assert_template_context_match('poll.id', 1)
assert_template_context_match('poll.title', 'poll 1')
(These function names are deliberately horrible - I haven't thought
about a nice API for this yet).
The first line above fakes sending a GET request to /polls/1/. The
next line checks that the response status code was 200, and the line
after checks that the response body contained 'This is poll 1'. So
far, all of these things can be done just using an HttpResponse object.
The next three lines assert that the correct template was rendered,
and check that the template context contained the expected data. This
isn't possible with just an HttpResponse - the template and context
have been rendered and forgotten by the time the HttpResponse is
returned.
TurboGears and Rails don't have this problem (in fact Rails is where
the idea for running assertions against the template context comes
from) because they couple their template systems to the view. We
don't want to do that in Django.
The solution I've been thinking about involves Django's event
dispatch system. I think we should be firing an event when a template
is rendered. The testing harness can then listen out for that event
and use it to capture the template and context, meaning you can run
assertions against them. This keeps our template system decoupled
while letting us write tests against the template activity within the
view functions.
I hope the above makes sense. I'll try to follow up later with a
better idea of how I think the assertions API should look.
Cheers,
Simon
Gábor Farkas
>
> On 12 Jul 2006, at 02:28, Malcolm Tredinnick wrote:
>
>> I approached things from a different direction, though, more
>> concentrating on how to unit test templates and views (as in not
>> requiring a full end-to-end application setup in order to test
>> them). So
>> it's actually a orthogonal to what you are proposing here and fits in
>> nicely.
>
> I've been thinking about unit testing templates and views on-and-off
> for the past few months. Here's the bulk of my thinking.
>
> The best way of running tests against a Django application, to my
> mind, is to run a "fake" web server (i.e. one that doesn't actually
> bind to a port) around the application. This fake server acts as a
> test harness. Tests can then create HttpRequest objects (probably a
> subclass called something like TestHttpRequest), pass them to the
> Django application, get back an HttpResponse object and run
> assertions against that.
regarding running a "fake" webserver.. did you look at twill?
it can connected directly to django (without any ports, webservers and
such) using wsgi, like this:
http://blogs.translucentcode.org/mick/2006/02/26/basic-twill-intercept-testing-django/
gabor
Michael Radziej
Simon Willison wrote:
> The best way of running tests against a Django application, to my
> mind, is to run a "fake" web server (i.e. one that doesn't actually
> bind to a port) around the application. This fake server acts as a
> test harness. Tests can then create HttpRequest objects (probably a
> subclass called something like TestHttpRequest), pass them to the
> Django application, get back an HttpResponse object and run
> assertions against that.
>
> ...
I have implemented exactly that: assertions agains the context.
(And, what irony, I would have liked to show it to you during
EuroPython, but time was just too short ...)
I've attached my stuff to the ticket #2333. It'll certainly need
some clean-up for public use.
I didn't want to patch Django to fetch the context, so I resorted
to define a template tag (which has access to the context) to
store the context away in the thread local store.
A TestRunner class is used to run a test case (i.e., setting up
environment, getting response, calling middleware, storing the
cookies like a browser would). It's only a minimal browser
simulation, of course. It also contains the logic for comparing
actual with expected output.
It doesn't work too well to just write out the repr() of the
context and then write asserts that compare it to a string:
doctests show that this is limited, e.g. it does not work for
dicts. So I wrote a "mockup" utility that tries to mimic the
structure of containers. As opposed to what Simon wrote, I'm
usually testing the whole context, but this can be changed.
TestGhostWriter is a middleware that can create test cases for
you. It writes using the standard python logging interface.
A single test looks like this:
#
--------------------------------------------------------------------
# GET /mailadmin/mailbox-qsearch/
GET = MultiValueDict({'search': ['bert'], 'kunde': ['']})
POST = MultiValueDict({})
response, context =
runner.get_response('/mailadmin/mailbox-qsearch/', 'GET', GET, POST)
assert runner.check_response(response, {'status_code': 200,
'cookies': SimpleCookie('')})
assert runner.check_context(response, context, {'form':
{'_inline_collections': None,
'advisory_dict': {},
'data': MultiValueDict({'search': ['bert'], 'kunde':
['']}),
'edit_inline': True,
'error_dict': {},
'manipulator':
'<dj_kunde.mailadmin.forms.search.MailboxQuickSearchForm object>'},
'mailboxes': ['<Person: slu1, Adalbert>', '<Person: slu2,
Berta>'],
'messages': [],
'person': '<Person: bla, Herr X.Y.>',
'user': '<User: bla>'})
I'm not sure if my writing makes sense. I'll be happy to give
more information, just ask.
Michael
Michael Radziej
> regarding running a "fake" webserver.. did you look at twill?
But it does not allow to test against the context!
Michael
Gábor Farkas
yes, that's why i said 'regarding running a "fake" webserver' :-)
but it offers the advantage of being able to simulate the web request in
a more browser-like fashion (follow the third link, submit this form
etc..)...
of course, you don't have to use it.. it's just an option.. maybe
combined with some other way to solve the context-testing problem...
gabor
Malcolm Tredinnick
nicely with Simon's thinking, so I don't have to rewrite all my notes.
This is a brain dump and can probably be safely ignored if it sounds
like rubbish, for reasons I mention at the end.
On Thu, 2006-07-13 at 01:41 -0700, Simon Willison wrote:
>
> On 12 Jul 2006, at 02:28, Malcolm Tredinnick wrote:
>
> > I approached things from a different direction, though, more
> > concentrating on how to unit test templates and views (as in not
> > requiring a full end-to-end application setup in order to test
> > them). So
> > it's actually a orthogonal to what you are proposing here and fits in
> > nicely.
>
> I've been thinking about unit testing templates and views on-and-off
> for the past few months. Here's the bulk of my thinking.
>
> The best way of running tests against a Django application, to my
> mind, is to run a "fake" web server (i.e. one that doesn't actually
> bind to a port) around the application. This fake server acts as a
> test harness. Tests can then create HttpRequest objects (probably a
> subclass called something like TestHttpRequest), pass them to the
> Django application, get back an HttpResponse object and run
> assertions against that.
>
> There's just one problem with this approach: while running assertions
> against an HttpResponse object is adequate, the ideal situation would
> be to run assertions against the template context as well. If you
> just run assertions against the HttpResponse object you end up having
> to process the HTML in some way to work out if the test has passed.
Agreed.
I was working out how to divorce each of the three components (M/V/T)
from each other so that I could unit test each in isolation. Part of the
logic here is to provide a way for template designers to test invariants
of the template without having to worry about views. Similarly, view
refactoring should not require delving deep inside the templates to work
out what happens: that makes the test results too fragile in the face of
any template changes. If it takes too long to maintain the tests, people
don't do it, because it's just a chore, rather than pure benefit.
Views first: we can make a TestHttpRequest style object and some utility
functions to make simulating a request simple. We pass this plus canned
parameters to the view function we are testing. We can override the
template class (create a TestTemplate class) that does not actually
render to a string (although it might need to behave string-like). This
way we can test this object to verify that it was constructed from the
right source and given the right context. We can also validate that the
right properties were set on the HttpResponse object that is returned.
Testing whether or not the template would have generated the right
result is not part of "view" testing, that is "template" testing. In
view testing we are verifying that the right input returns the right
result.
Initially, I think it will be hard to have the views entirely divorced
from the model framework. To do that (separate them for testing
purposes) would require making it very easy to creat mock objects. This
isn't too painful, but capturing all the necessary data via all the
function calls a view can take can be tricky. I have a sketch of how to
capture this via the profiling hooks in Python (as a dev server runtime
option) for seeding tests, but it's not implemented at the moment, so
there are no doubt hidden problems (I don't think we can use signal
dispatches here as Simon wants to for the view/template interaction,
because we would need to add so many of them). So figure that we
probably, unfortunately, need a seeded database and model setup for this
to work initially.
It's still useful, though, since now when we are working on views, we
can check that we are still passing the necessary information into the
template layer and back to the client and we can do this in a way that
is independent of the particular way the template has been written.
For template testing, I was thinking about passing in an object that
supported the necessary attribute accesses (the beauty of everything
being accessed as "foo.bar.baz" is that faking it is reasonably
straightforward). The test takes the template source filename (or source
string), the canned input data and we can run assertions against the
result.
The hard bit, that I haven't worked out a solution for -- and it may not
exist nicely -- is how to handle custom template tags. We could test
them as opaque, mock objects that return a canned response for testing
purposes, but that only provides a way to test the templates they live
in. How do you test the tags themselves? Maybe template tags can be
tested similarly to views. Not sure about this part, it hasn't
crystallised in my brain yet.
> What you really want to be able to do is this (pseudocode):
>
> send_get('/polls/1/')
>
> assert_response_status(200)
> assert_response_contains('This is poll 1')
>
> assert_template_rendered('poll_detail.html')
> assert_template_context_match('poll.id', 1)
> assert_template_context_match('poll.title', 'poll 1')
>
> (These function names are deliberately horrible - I haven't thought
> about a nice API for this yet).
I can't help here. All my thinking has been along the lines of how to
duck imposing much API policy at all. Testing frameworks are such a
religious issue (and not without reason) that I wanted to supply the
tools so that people could easy run assertions against Django results,
without us imposing too much in the way of "you have to use this
framework".
So, view test results would contain a way to access everything in the
context, everything in the response and the template source (name).
Django can provide a framework for making it easy to test for invariants
there, but it also gives everybody a way to use these things from there
own preferred harness, be it nose or test.py or
crazy-test-thing-from-outer-space.py.
> The first line above fakes sending a GET request to /polls/1/. The
> next line checks that the response status code was 200, and the line
> after checks that the response body contained 'This is poll 1'. So
> far, all of these things can be done just using an HttpResponse object.
>
> The next three lines assert that the correct template was rendered,
> and check that the template context contained the expected data. This
> isn't possible with just an HttpResponse - the template and context
> have been rendered and forgotten by the time the HttpResponse is
> returned.
>
> TurboGears and Rails don't have this problem (in fact Rails is where
> the idea for running assertions against the template context comes
> from) because they couple their template systems to the view. We
> don't want to do that in Django.
>
> The solution I've been thinking about involves Django's event
> dispatch system. I think we should be firing an event when a template
> is rendered. The testing harness can then listen out for that event
> and use it to capture the template and context, meaning you can run
> assertions against them. This keeps our template system decoupled
> while letting us write tests against the template activity within the
> view functions.
I was going to poke about under the covers during test setup. For
example, a TestTemplate class would have all the functions of a Template
class, except that render() would just store the context for later
referral (and maybe return a stub string in case somebody really
expected a string back).
Your way is probably simpler, though, and a much better first step. If
somebody does not want to test the template contents in one particular
suite, they can just ignore those bits and focus on the results. Since
the signal you want to dispatch probably isn't that useful in production
(and signal dispatches do add overhead), we can poke in a "test" render
method as part of the test setup that adds the new dispatch call.
> I hope the above makes sense. I'll try to follow up later with a
> better idea of how I think the assertions API should look.
Now that I've written all that, I think I might be stepping back from
this a bit. Whilst eating lunch today, I sat down and looked at
everything I have going on professionally and in my Django sandboxes and
I think that since there are going to be a million people wanting to
work on the testing, I can spend my efforts elsewhere. There are a bunch
of medium-sized bits in Django I want to work on as well as getting the
open ticket count down and this part is already in good hands. It was
just fortuitous timing that Russell spoke up not three days after I had
spent an afternoon writing stuff down.
Regards,
Malcolm
jpel...@gmail.com
> based upon some deficiency of unittest that will adversely affect django
> testing, I'm inclined to stick with whats in the standard library.
To me, possibly due to my own moral failings :), the most compelling
case is the one that speaks to developer laziness. With the nose-django
plugin, nose does right now everything in your proposal, with the
exception of finding non-doctests in models.py. In addition it already
implements a whole bunch of other things that users are going to ask
you for, sooner or later: coverage reports; profiling; package- and
module-level setup and teardown; output capture; generator-based tests;
test functions; tests in classes that don't descend from
unittest.TestCase; skipped and deprecated tests; deep, easy and
flexible test selection; integration with setuptools ("python setup.py
test")... and then there's the plugin system, which allows you to
easily output test results in html or xml, write custom test discovery,
and hopefully lots of other things that I haven't thought of.
nose is LGPLed and easy_installable, and its really just a layer on top
of unittest. FWIW, TurboGears uses nose and integrates it just by
adding the requirement to its setup file and letting setuptools do the
work.
> That said, I am not aiming to set up a django testing framework that
> actively impedes the use of py.test or nose; if there is anything we can do
> to make the two compatible/complimentary, let me know.
The core django test runner is pretty hostile to outside use, mainly
because it wants to run the show and provides no hooks for anybody else
to get in there. So don't do that. :) The most helpful thing I can
think of right now would be to put the test db functions (create,
reset, install app, flush data, drop) someplace accessible so that the
core django test runner can use them and your runner can use them and
the nose-django plugin can use them and anybody else can use them, so
we're all setting up and tearing down the test environment in the same
way. Then if your runner is finding unittests and doctests in
predictable locations, I can tune the nose-django plugin to behave in
the same way, so that the same test suites will be runnable with either
tool.
JP
Michael Radziej
>
> To me, possibly due to my own moral failings :), the most compelling
> case is the one that speaks to developer laziness. With the nose-django
> plugin, nose does right now everything in your proposal, with the
> exception of finding non-doctests in models.py. In addition it already
> implements a whole bunch of other things that users are going to ask
> you for, sooner or later: coverage reports; profiling; package- and
> module-level setup and teardown; output capture; generator-based tests;
> test functions; tests in classes that don't descend from
> unittest.TestCase; skipped and deprecated tests; deep, easy and
> flexible test selection; integration with setuptools ("python setup.py
> test")... and then there's the plugin system, which allows you to
> easily output test results in html or xml, write custom test discovery,
> and hopefully lots of other things that I haven't thought of.
It makes coverage reports? And they work with Django? Whow!
Michael
jpel...@gmail.com
Sure.
It uses Ned Batchelder's coverage module:
http://www.nedbatchelder.com/code/modules/coverage.html
Grab that, nose and the nose-django plugin, install the plugin with
python setup.py develop (may need sudo, depending on your platform). Go
to the examples directory in the plugin distribution, and run like
this:
nosetests -v --with-coverage --cover-package=project --with-doctest
--with-django --django-settings=project.settings
You should get something like:
Doctest: project.zoo.models ... ok
Doctest: project.zoo.models.Zoo ... ok
Doctest: project.zoo.models.Zoo.__str__ ... ok
Doctest: project.zoo.models.func ... ok
project.tests.test_views.test_view_index ... ok
Name Stmts Exec Cover Missing
--------------------------------------------------
project 0 0 100%
project.settings 18 18 100%
project.urls 2 2 100%
project.zoo 0 0 100%
project.zoo.models 7 5 71% 37, 45
project.zoo.views 2 2 100%
--------------------------------------------------
TOTAL 29 27 93%
----------------------------------------------------------------------
Ran 5 tests in 0.135s
OK
You can also stick all 1000 of those command-line opts in ENV, if long
command lines aren't your thing.
JP
(My usual warning on the nose-django plugin: it's early alpha, it
creates and drops databases or schemas, PLEASE don't use it with a
settings file that points at your production database. I don't believe
there are any data destroying bugs, but ... you know.)
Russell Keith-Magee
Yes, please post the patch before committing -- this is definitely a big thing.
Ok; i've just attached a first pass set of patches to ticket #2333.
The objectives of these patches:
- Add a 'test' target to django-admin that will start the test suite
- Provide a set of testing utilities of capability equivalent to those currently offered by runtests
- Refactor the existing Django tests to use the new framework
This is not yet attempting to add fixtures, context testing etc - thats the next step, once there is a basic test framework to work in.
I've attached a walkthrough of the patches to the ticket - there are a lot of changes, but they're not that complex, mostly mechanical.
Also - I've used unittest here to replicate the capabilities of tests/runtests.py, but I've tried to keep the door open for those that want to use nose (or anything else for that matter). The method that is used by syncdb to kickstart testing is user-configurable, and I've kept the 'useful utilities', like db setup/teardown factored and available for others to use.
Russ Magee %-)
Russell Keith-Magee
> It makes coverage reports? And they work with Django? Whow!
Sure.
Now - that's what I call a convincing reason to switch.
I'm still hesitant to make the _default_ testing scheme dependent on an external package, simply because Django doesn't have any dependencies on anything other than the Python core libraries and a DB integration library.
However, if you look at the patches that I have submitted, the approach I have taken allows end users to configure their own testing scheme to be invoked by './manage.py test'. If there are utilities or a 'nose run_test' implementation that we can add to a ' django.test.nose' package, then adding nose testing will be as simple as installing nose and changing a configuration setting for the site.
Russ Magee %-)
Jacob Kaplan-Moss
On Jul 14, 2006, at 12:13 AM, Russell Keith-Magee wrote:
> On 7/13/06, jpel...@gmail.com <jpel...@gmail.com> wrote:
>>
>>
>>> It makes coverage reports? And they work with Django? Whow!
>>
>> Sure.
>
>
> Now - that's what I call a convincing reason to switch.
Indeed :)
> I'm still hesitant to make the _default_ testing scheme dependent
> on an
> external package, simply because Django doesn't have any
> dependencies on
> anything other than the Python core libraries and a DB integration
> library.
I completely agree -- having a testing framework that depends on an
external package being installed is almost as bad as not having one
at all.
Batteries included, and all that...
Now, what's the license of nose? Perhaps we could bundle it with
Django (as we've done with PyDispatcher and Simplejson)? In my
(limited) messing with Django & nose I've been pretty happy so far
(but I've not played with Russell's patch for comparison).
Jacob
Russell Keith-Magee
Now, what's the license of nose? Perhaps we could bundle it with
Django (as we've done with PyDispatcher and Simplejson)? In my
(limited) messing with Django & nose I've been pretty happy so far
(but I've not played with Russell's patch for comparison).
Just to be clear - I'm not even going to pretend that what I have implemented is a replacement/comparable with nose. It is an attempt at a very basic test running framework extending unittest and doctest. Along the way, it also acts as a testbed for factoring out the pieces of code that are needed to set up and run tests (such as DB setup/teardown), with hooks for more complex functionality. It uses the existing Django test cases as a proof of concept.
Russ %-)
jpel...@gmail.com
> external package being installed is almost as bad as not having one
> at all.
>
> Batteries included, and all that...
Very true. I like the approach that Russell has taken in the patches
attached to the ticket. Having a configurable test module and runner
gives other test runners a place to hook in, which is all that's really
needed for folks to use whatever they want. This is similar to how the
setuptools test command works, but more open-ended (which is good). The
only thing I'd change would be to let the test_runner get all of
sys.argv that manage doesn't care about.
> Now, what's the license of nose? Perhaps we could bundle it with
> Django (as we've done with PyDispatcher and Simplejson)?
It's LGPL. If that's not liberal enough, and you do decide that you'd
like to bundle it, I'd be happy to talk about contributing it, or
whatever parts you want, under another license.
JP
Russell Keith-Magee
The best way of running tests against a Django application, to my
mind, is to run a "fake" web server (i.e. one that doesn't actually
bind to a port) around the application.
Agreed; although I'm not sure that you actually need to run a server in the 'start a process/thread' sense. A TestHandler() that operates much like the mod_python or wsgi Handler objects (or a light wrapper around one of those handlers that provides a nice testing interface) should be sufficient.
There's just one problem with this approach: while running assertions
against an HttpResponse object is adequate, the ideal situation would
be to run assertions against the template context as well. If you
just run assertions against the HttpResponse object you end up having
to process the HTML in some way to work out if the test has passed.
I'm inclined to agree with Malcolm here - imposing the assertion API isn't the right approach. We should let the user's testing framework provide the assert/fail methods. Django's responsibilty should be to provide variables and methods that can be asserted. So, (again, exact names up for debate), a python unittest test should end up looking something like:
def test_stuff(self):
response = self.testServer.get('/polls/1')
self.assertEqual(response.status, 200)
self.assert_('This is poll 1' in response)
self.assertEqual(response.template , 'poll_detail.html')
self.assertEqual(response.context['poll.title'], 'poll 1')
But a simple hand cranked test file might use:
if response.status != 200:
raise 'Test Error'
and so on.
The solution I've been thinking about involves Django's event
dispatch system.
The are only two other solutions that come to mind for me:
1) The template tag based approach that Michael Radziej demonstrated in his patch. While this is a good solution from Michael's perspective (solution that works without having to modify Django source code), I'm not sure it's the right solution for the 'official' test tools.
2) Adding special cases so that the template renderer/context processing phases check 'if TestRequest', add attach the template and context as extra data to the response as appropriate; however, special cases make my eyeballs itch.
Neither of these are particularly attractive, IMHO. A TestServer that attaches to the (various?) signals during the response process and captures context/template detail looks like the best approach to me. +1.
Russ Magee %-)
Michael Radziej
Am 17.07.2006 um 06:26 schrieb Russell Keith-Magee:
> On 7/13/06, Simon Willison <swil...@gmail.com> wrote:
> > The best way of running tests against a Django application, to my
> > mind, is to run a "fake" web server (i.e. one that doesn't actually
> > bind to a port) around the application.
>
> Agreed; although I'm not sure that you actually need to run a
> server in the 'start a process/thread' sense. A TestHandler() that
> operates much like the mod_python or wsgi Handler objects (or a
> light wrapper around one of those handlers that provides a nice
> testing interface) should be sufficient.
It's nicer if you don't have an additional process since you can
start the debugger in your tests.
> The are only two other solutions that come to mind for me:
>
> 1) The template tag based approach that Michael Radziej
> demonstrated in his patch. While this is a good solution from
> Michael's perspective (solution that works without having to modify
> Django source code), I'm not sure it's the right solution for the
> 'official' test tools.
To repeat myself, I really don't propose to use this template tag
hack if you build it into Django. It's just been a hack to work
around the restrictions. I'd say: go and use the event dispatcher,
it's much cleaner and looks like the proper solution.
Michael
Jyrki Pulliainen
>
>
> On 7/13/06, Simon Willison <swil...@gmail.com> wrote:
> >
> > The best way of running tests against a Django application, to my
> > mind, is to run a "fake" web server (i.e. one that doesn't actually
> > bind to a port) around the application.
>
>
> Agreed; although I'm not sure that you actually need to run a server in the
> 'start a process/thread' sense. A TestHandler() that operates much like the
> mod_python or wsgi Handler objects (or a light wrapper around one of those
> handlers that provides a nice testing interface) should be sufficient.
>
> > There's just one problem with this approach: while running assertions
> > against an HttpResponse object is adequate, the ideal situation would
> > be to run assertions against the template context as well. If you
> > just run assertions against the HttpResponse object you end up having
> > to process the HTML in some way to work out if the test has passed.
> >
>
I've been using on some none-django projects Twill testing framework
(http://twill.idyll.org/). It's been a bit buggy though, but on the
other hand all the bugs we've fixed at our company have been sent to
the developer.
Anyway, Twill's been an ideal tool for web testing, because you can
give follow links, fill forms etc.
--
Jyrki // jyrki.pu...@gmail.com
Russell Keith-Magee
I know Malcolm said he was going to step away from this discussion, so this is mostly for the benefit of everyone else.
Initially, I think it will be hard to have the views entirely divorced
from the model framework. To do that (separate them for testing
purposes) would require making it very easy to creat mock objects.
While I appreciate what you're trying to say here - test one thing at a time, etc - I can't help but feel that this is overkill. We implicitly trust the Django models and query tools, so end user model tests don't require tests of basic query functionality, but then we put in a complex framework to mock the Django models so that we don't have to use the actual models during testing.
A fixture/db seeding capability provides the same testing capability with very low development effort on our part. While this testing regime is _theoreticallly_ weaker because it relies upon the validity of the models, in practice I don't see that being a very large source of potential error - certainly no larger than the confidence in the mocking framework itself, which will only ever exercised during the test process.
For template testing, I was thinking about passing in an object that
supported the necessary attribute accesses (the beauty of everything
being accessed as "foo.bar.baz" is that faking it is reasonably
straightforward). The test takes the template source filename (or source
string), the canned input data and we can run assertions against the
result.
Agreed, but the question that I have is 'what assertions?'
1) We could test that template + sample context -> template output == known template output
3) We could parse 1 into a DOM tree and validate the contents of various template tags.
The problem is:
(1) is extremely fragile;
(2) is less fragile, but not especially helpful IMHO,
(3) can only be applied to DOM-like documents.
(2) can be made more helpful with lots of string comparison methods ('n instances of string', 'string 1 occurs before string 2'). However, I'm not sure that it actually serves to validate anything useful. The Django policy of preventing complex logic from being placed in the template means that if you are able to validate at the view level that the correct context has been created and the correct template has been used, the only real test that applies to the template is 'does this page render correctly'. Testing substrings of the generated output doesn't achieve this validation - correct rendering can only be answered by 1) direct comparison with a known correct example, or 2) visual inspection.
Of course, there is the need to test custom template tags and filters, for which (1) is an appropriate solution. However, this requires no additional framework to test - as evidenced by the fact that template tags are already tested as part of the 'othertests' package.
Russ Magee %-)
Russell Keith-Magee
Anyway, Twill's been an ideal tool for web testing, because you can
give follow links, fill forms etc.
Much like the unittest vs py.test vs nose discussion earlier, I think we should be avoiding discussions about what to bless as the 'official Django testing framework'.
In addition, the kind of testing that Twill (and Selenium, for that matter) perform is limited to validating against generated HTML; while this is a valid and useful class of test, it does not allow for validation of views independent of the rendering framework - essentially every error found in a Twill test could have two sources - the template or the view. To make matters worse, it is possible that an error in the view could be hidden by a complementary error in the template (or vice versa).
However, that said, IMHO the goal of this testing effort should be to provide a set of tools that enables any developer to use whatever style of testing they think will provide helpful validation of a system.
If there is some facility or feature that can be added as to Django that makes Twill testing much easier, let us know; similarly for any other framework.
Russ Magee %-)