Unittest - How do I code lots of simple tests
Paul Moore
number of adhoc testing tricks I've used in the past, and to Perl's
"standard" testing framework) is that the testcase-as-a-class model
tends to imply a relatively high granularity in testing.
A good example of this comes from "Dive Into Python"
(http://diveintopython.org) section 7.3. Here, the author has written
a module which converts numbers to Roman numerals. The test case is
then
class KnownValues(unittest.TestCase):
knownValues = ( (1, 'I'), ... # 50-plus specific cases
def testToRomanKnownValues(self):
"""toRoman should give known result with known input"""
for integer, numeral in self.knownValues:
result = roman.toRoman(integer)
self.assertEqual(numeral, result)
Now, to my mind, the big problem here is that this *isn't* one test,
but rather 50 (or more). OK, the same checks are done if everything
succeeds, but
1. If a test fails, the rest are skipped! If there's a pattern to the
failures (the code handles numbers ending in 4 and 9 wrongly, for
example) it's much easier to find if all of the checks are
reported.
2. Psychologically, "52 tests succeeded" is a much bigger boost than
"1 test succeeded" (albeit this test covered 52 cases). And it's
not just psychology - we really *did* test 52 distinct conditions.
The only way I can see of producing the "true" number of tests is via
some form of ugly hack like
test_values = ((1, 'I'), ...)
class KnownValues(unittest.TestCase):
pass
for arabic, roman in test_values:
def test(self):
result = roman.toRoman(arabic)
self.assertEqual(roman, result)
setattr(KnownValues, 'test_%s_%s' % (arabic, roman), test)
But I can't really see that as the "right approach".
Can anyone suggest a more reasonable way of running this sort of
table-driven test via unittest?
Maybe I should use doctest instead, but to be honest, I prefer the
overall infrastructure of unittest (for real tests). It's just this
particular issue that really bugs me...
Paul.
--
This signature intentionally left blank
Peter Hansen
>
> Can anyone suggest a more reasonable way of running this sort of
> table-driven test via unittest?
Why not just extend self.assertEqual() and use your own check, with
additional logic as required to increment counters or add items
to the list of passing tests. Then put a final check of the number
of passing tests or something like that at the end to make sure
things worked overall.
For example:
class KnownValues(unittest.TestCase):
def setUp(self):
self.passCount = 0
def checkValue(self, expected, result):
if expected == result:
self.passCount += 1
else:
# left as exercise to the reader, but pass would work...
def testToRomanKnownValues(self):
for integer, numeral in self.knownValues:
result = roman.toRoman(integer)
self.checkValue(numeral, result)
self.assertEqual(len(self.knownValues), self.passCount)
No, you don't get the psychologically affirming "52 tests passed!"
without changes to the TestRunner, but I assume the non-cosmetic part
of this is more your concern right now...
-Peter
Ian Bicking
This will still abort if toRoman raises an exception (you'll just get
one exception, not all exceptions or failures, and you won't know if
previous conversions didn't pass but didn't cause an exception). Also,
you won't be able to run a specific test, you'll have to run the whole
set -- obviously not a big deal for this, but other tests might be more
time consuming.
You can do something like:
class RomanTest(unittest.TestCase):
def __init__(self, source, expected):
self.expected = expected
self.result = result
def test(self):
self.assertEqual(expected, roman.toRoman(source))
def createSuite():
suite = unittest.TestSuite()
for i, v in knownValues:
suite.addTest(RomanTest(i, v))
if __name__ == '__main__':
unittest.main(defaultTest='createSuite')
You still can't name tests this way. I've tried to name tests, but
unittest.main won't pay attention to my names. unittest is not written
with subclassing in mind, except for the limited subclassing that is
documented. (And it uses double-underscore variables, like it's just
*trying* to piss me off! Double-underscore variables are so arrogant
and patronizing. But maybe I'm a little more bothered by this right
now because I'm not in a good mood).
--
Ian Bicking | ia...@colorstudy.com | http://blog.ianbicking.org
David Goodger
>>> Can anyone suggest a more reasonable way of running this sort of
>>> table-driven test via unittest?
Take a look at Docutils' test/DocutilsTestSupport.py for ideas. The
CustomTestSuite and CustomTestCase classes provide support for named
data-driven tests. Most of Docutils' tests are data-driven.
Ian Bicking wrote:
> unittest is not written with subclassing in mind, except for the
> limited subclassing that is documented. (And it uses
> double-underscore variables, like it's just *trying* to piss me off!
> Double-underscore variables are so arrogant and patronizing.
All very true. Double-underscores ought to be banned from the
standard library. They inevitably get in the way because no matter
how well a class is written, somebody is going to want to subclass it
in a way the original author never considered.
--
David Goodger http://starship.python.net/~goodger
For hire: http://starship.python.net/~goodger/cv
Docutils: http://docutils.sourceforge.net/
(includes reStructuredText: http://docutils.sf.net/rst.html)
Jeremy Fincher
> 1. If a test fails, the rest are skipped! If there's a pattern to the
> failures (the code handles numbers ending in 4 and 9 wrongly, for
> example) it's much easier to find if all of the checks are
> reported.
That's true, but most of the time when I test, I prefer that behavior.
Many of my asserts depend on the success of the assert prior to
them, so I want the test to fail as soon as one of the asserts has
failed.
It'd be nice to have both possibilities, though.
Jeremy
Jeremy Fincher
> (And it uses double-underscore variables, like it's just
> *trying* to piss me off! Double-underscore variables are so arrogant
> and patronizing.
Don't look at it like that. Double-underscore variables are for
averting namespace problems, not for protection. Unittest uses them
not to protect itself, but to keep from stomping all over your
namespace when you subclass it. In that way, it's very much built
with subclassing in mind.
Jeremy
Miki Tebeka
> test_values = ((1, 'I'), ...)
>
> class KnownValues(unittest.TestCase):
> pass
>
> for arabic, roman in test_values:
> def test(self):
> result = roman.toRoman(arabic)
> self.assertEqual(roman, result)
> setattr(KnownValues, 'test_%s_%s' % (arabic, roman), test)
>
> But I can't really see that as the "right approach".
On reason that it won't do what you want :-)
All the tests will check the last value in test_values. (Something to
do with binding rules, can't recall how to solve)
Try:
--- i2r.py ---
#!/usr/bin/env python
from unittest import TestCase, makeSuite, main
class Roman:
def toRoman(self, i):
return { 1 : "I",
2 : "II",
3 : "III",
4 : "IV",
5 : "V"}[i]
roman = Roman()
class KnownValues(TestCase):
pass
test_values = ((1, "I"), (2, "II"), (3, "III"), (4, "IV"), (5, "V"))
for a, r in test_values:
def test(self):
print a, r
result = roman.toRoman(a)
self.assertEqual(r, result)
setattr(KnownValues, "test_%s_%s" % (a, r), test)
test_suite = makeSuite(KnownValues, "test_")
if __name__ == "__main__":
main()
--- i2r.py ---
$ python i2r.py
5 V
.5 V
.5 V
.5 V
.5 V
.
----------------------------------------------------------------------
Ran 5 tests in 0.000s
OK
HTH.
Miki
Duncan Booth
> But I can't really see that as the "right approach".
>
> Can anyone suggest a more reasonable way of running this sort of
> table-driven test via unittest?
Ok, how about the file below.
It uses a metaclass to generate dynamic tests from a table. I deliberately
wrote the tests so that one fails, when run it will tell you that
test_roman_v failed so you can see that the names get handled properly.
Output:
D:\temp>tabletest.py --verbose
testMe (__main__.MyTests) ... ok
test_roman_i (__main__.MyTests) ... ok
test_roman_v (__main__.MyTests) ... FAIL
test_roman_x (__main__.MyTests) ... ok
======================================================================
FAIL: test_roman_v (__main__.MyTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "D:\temp\tabletest.py", line 21, in test
self.assertEquals(roman, 'x')
File "D:\Python23\lib\unittest.py", line 302, in failUnlessEqual
raise self.failureException, \
AssertionError: 'v' != 'x'
----------------------------------------------------------------------
Ran 4 tests in 0.020s
FAILED (failures=1)
Note that the metaclass itself is reusable. The factory
function tableDrivenTests, although defined within the class is a function,
not a method, and cannot access any members of the class (since the class
does not yet exist at the time it is called). The table itself either has
to be created inside the tableDrivenTests function, or global.
The factory function simply returns a dictionary of functions which are
added to the class, note that the keyname in the dictionary is important so
far as the unittest code is concerned, not the original name of the
function.
Also be sure to pass parameters into the test function using default
parameters as nested scopes will get the values left at the end of the loop
(so you might end up with lots of tests that all do the same thing). I
mention that here because I did exactly that writing the code.
---- begin tabletest.py ----
class MetaTableTest(type):
def __new__(metacls, name, bases, dict):
factory = dict['tableDrivenTests']
dict.update(factory())
return super(MetaTableTest, metacls).__new__(metacls, name, bases,
dict)
import unittest
class MyTests(unittest.TestCase):
__metaclass__ = MetaTableTest
def tableDrivenTests():
'''Return a dictionary of additional test functions'''
knownValues = (1,'i'), (5, 'v'), (10, 'x')
table = {}
for arabic, roman in knownValues:
def test(self, arabic=arabic, roman=roman):
if arabic==1:
self.assertEquals(roman, 'i')
else:
self.assertEquals(roman, 'x')
table['test_roman_%s' % roman] = test
return table
def testMe(self):
self.assert_(True)
if __name__=='__main__':
unittest.main()
---- end tabletest.py ----
--
Duncan Booth dun...@rcp.co.uk
int month(char *p){return(124864/((p[0]+p[1]-p[2]&0x1f)+1)%12)["\5\x8\3"
"\6\7\xb\1\x9\xa\2\0\4"];} // Who said my code was obscure?
Peter Otten
Change the above line to
def test(self, a=a, r=r):
or you will perform the test five times with (5, "V").
> print a, r
> result = roman.toRoman(a)
> self.assertEqual(r, result)
> setattr(KnownValues, "test_%s_%s" % (a, r), test)
>
> test_suite = makeSuite(KnownValues, "test_")
>
> if __name__ == "__main__":
> main()
> --- i2r.py ---
I like the idea, once the little error is removed. In general, I think the
unit test code should be as simple as possible. Otherwise we would need
unit tests for unit tests for...
Peter
Paul Moore
I don't see the remainder of the problem as "non-cosmetic". The error
report I get (or rather the information it offers) is
1 test failed - pass count is 25 instead of 52.
But that doesn't tell me *which* tests failed.
The key point here is that I'm NOT running one test - I really am
running 52 distinct tests. OK, they are all very similar, differing
only in the data - but for pity's sake, isn't that what an object
oriented structure is supposed to make easy???
Paul.
Paul Moore
> Paul Moore wrote:
> >>> Can anyone suggest a more reasonable way of running this sort of
> >>> table-driven test via unittest?
>
> Take a look at Docutils' test/DocutilsTestSupport.py for ideas. The
> CustomTestSuite and CustomTestCase classes provide support for named
> data-driven tests. Most of Docutils' tests are data-driven.
Urk. That's hairy stuff. Thanks for the pointer, I'll do some research.
But I still think that this sort of thing should be easy :-(
Paul.
Anthony Briggs
>Peter Hansen <pe...@engcorp.com> wrote in message
>news:<3F95A5DC...@engcorp.com>...
>> without changes to the TestRunner, but I assume the non-cosmetic part
>> of this is more your concern right now...
>
>I don't see the remainder of the problem as "non-cosmetic". The error
>report I get (or rather the information it offers) is
>
> 1 test failed - pass count is 25 instead of 52.
>
>But that doesn't tell me *which* tests failed.
Are you running the tests verbosely? eg. with a -v argument under
UNIX, or as specified in the docs
<http://www.python.org/doc/current/lib/minimal-example.html>? I get
the following output when using Miki's code (with a deliberate error
thrown in, and the print statement commented out):
bash-2.05b$ ./test.py -v
test_1_I (__main__.KnownValues) ... ok
test_2_II (__main__.KnownValues) ... ok
test_3_II (__main__.KnownValues) ... FAIL
test_4_IV (__main__.KnownValues) ... ok
test_5_V (__main__.KnownValues) ... ok
======================================================================
FAIL: test_3_II (__main__.KnownValues)
----------------------------------------------------------------------
Traceback (most recent call last):
File "./test.py", line 21, in test
self.assertEqual(r, result)
File "/usr/local/lib/python2.3/unittest.py", line 292, in failUnlessEqual
raise self.failureException, \
AssertionError: 'II' != 'III'
----------------------------------------------------------------------
Ran 5 tests in 0.055s
FAILED (failures=1)
bash-2.05b$
>The key point here is that I'm NOT running one test - I really am
>running 52 distinct tests. OK, they are all very similar, differing
>only in the data - but for pity's sake, isn't that what an object
>oriented structure is supposed to make easy???
Well, you're testing one aspect of the code. It's really just a
question of how you think about your tests.
Anthony
--
----------------------------------------------------
HyPEraCtiVE? HeY, WhO aRE YoU cALliNg HypERaCtIve?!
aBR...@wEStNeT.cOm.aU
----------------------------------------------------
John J. Lee
[...]
> You still can't name tests this way. I've tried to name tests, but
> unittest.main won't pay attention to my names. unittest is not
> written with subclassing in mind, except for the limited subclassing
When I looked at it, it certainly seemed that just about everything
*except* unittest.main was designed for subclassing. Just regard
unittest.main as a convenience function or example code -- that's all
it's indended as, IIRC.
Having said that, that didn't stop me writing a hack to abuse
unittest.main, instead of bothering to figure out what and how to
subclass to do what I wanted.
John
John J. Lee
[grumbles about unittest]
BTW, Ian, did you notice that somewhere on one of the Zope sites
there's a reference to your blog entry about this? Looked like they
have a unittest.main-replacement.
John
Peter Hansen
>
> Peter Hansen <pe...@engcorp.com> wrote in message news:<3F95A5DC...@engcorp.com>...
> > Paul Moore wrote:
> > >
> > > Can anyone suggest a more reasonable way of running this sort of
> > > table-driven test via unittest?
> > No, you don't get the psychologically affirming "52 tests passed!"
> > without changes to the TestRunner, but I assume the non-cosmetic part
> > of this is more your concern right now...
>
> I don't see the remainder of the problem as "non-cosmetic". The error
> report I get (or rather the information it offers) is
>
> 1 test failed - pass count is 25 instead of 52.
>
> But that doesn't tell me *which* tests failed.
>
> The key point here is that I'm NOT running one test - I really am
> running 52 distinct tests. OK, they are all very similar, differing
> only in the data - but for pity's sake, isn't that what an object
> oriented structure is supposed to make easy???
Well, look at it this way. Using the built-in assertEquals() and
similar functions is a way of explicitly asking for a test method to
abort, and for the framework to continue on with the next test
method. If you don't want that behaviour, nothing's forcing you
to use assertEqual().
Instead, just write up your own comparison routine, which doesn't
abort, and have verbose output for each failure. Something I've
done repeatedly in the past is to have a routine which says simply
(upon failure) "case %s: expected %s, result %s" and then substitute
in the test case input, the expected value, and the actual result.
The part that is "cosmetic" is insisting that this has to result
in the framework reporting the as individual "test" failures. To
do that, you need more extensive modifications, because unittest
has a clear, simple definition of what constitutes a test, and
individual comparisons inside a testMethod() are not it... it's the
whole test method that is a test.
After all, just because a test has two self.assertEquals() and a single
self.assert_() doesn't necessarily mean it's *three* tests.
As Anthony B. wrote, you're testing one "aspect" or something... don't
think of tests as calls to assertXxxx() methods, think of them as
collections of such calls.
-Peter
Emile van Sebille
"Paul Moore" <paul....@atosorigin.com> wrote in message
news:182bcf76.0310...@posting.google.com...
If you find unnittest too heavy handed, you may want to take a look at
doctest. Essentially, doctest will test code found in docstrings and
compare results. This allows you to paste an interactive session into your
module.
HTH,
Emile van Sebille
em...@fenx.com
Ian Bicking
> Well, look at it this way. Using the built-in assertEquals() and
> similar functions is a way of explicitly asking for a test method to
> abort, and for the framework to continue on with the next test
> method. If you don't want that behaviour, nothing's forcing you
> to use assertEqual().
>
> Instead, just write up your own comparison routine, which doesn't
> abort, and have verbose output for each failure. Something I've
> done repeatedly in the past is to have a routine which says simply
> (upon failure) "case %s: expected %s, result %s" and then substitute
> in the test case input, the expected value, and the actual result.
>
> The part that is "cosmetic" is insisting that this has to result
> in the framework reporting the as individual "test" failures. To
> do that, you need more extensive modifications, because unittest
> has a clear, simple definition of what constitutes a test, and
> individual comparisons inside a testMethod() are not it... it's the
> whole test method that is a test.
What we really want isn't granularity inside a testMethod(). A
TestCase can currently have more than one test already (different test*
methods) -- what we want is to generalize that. Instead of just def
testA(self), def testB(self), etc., we want a testCompare(self, input,
output), where the input and output comes from some list of inputs and
known outputs (bonus if that list also contains names so we can address
particular tests).
Ian Bicking
Yes, I've been experimenting with it. (For others, the URL is
http://cvs.zope.org/Zope/test.py ) -- it's a bit hard to figure out, it
has some documentation but only about how to run the script, not how to
organize your tests.
After much struggling I've gotten it to run -- it seems picky about
where you put your tests, and it just ignored my tests until I more
carefully pointed the script to where they were (but there's several
ways to point the script to a directory, only --libdir seemed to work
for me). I'm still not sure how to specify which tests I want to run
-- it has some regex control there, which seems like both overkill and
underkill (I don't need the flexibility of regexes to match modules,
but I want better ways to indicate specific tests in those modules). I
actually *like* the basic interface of unittest.main.
What appealed to me about Zope's test.py was pychecker and pdb options
(ontop of code coverage, which several frameworks have). Though once I
looked at how pychecker is invoked, that's easy to add to any
framework. Code coverage is supported by several frameworks -- I added
it to my own unittest-based tests pretty easily too. Maybe if I looked
at pdb it would also be easy enough as well, at which point I may or
may not find the Zope test.py appealing enough to try to figure out how
it finds (or does not find) files (though now that I've gotten this
far...)
Ian Bicking
I never have problems with namespaces like that. If you really do have
significant likelihood of a namespace clash then you shouldn't be using
inheritance -- some of those instance variables should be shifted into
a separate object.
I generally assume that a subclass is made with the superclass in mind,
and that subclasses will be respectful of the requirements of the
superclass. If I'm not sure I trust the subclassing programmer, then
I'll add lots of asserts. The asserts cover the *actual* requirements.
Double-underscore says "I won't even tell you the requirements,
because I don't believe you can be trusted with them. Instead I'll
hide it all from you." You can still unmangle the names -- which I
frequently do -- but it's annoying.
This is open source -- sure, things may change in some later version,
and if I go playing around with undocumented interfaces then I won't be
surprised if some future version breaks my code. I'm okay with that,
and it should be my choice to make fragile code if I want.
Garth T Kidd
> CustomTestSuite and CustomTestCase classes provide support for named
> data-driven tests. Most of Docutils' tests are data-driven.
Ha! You beat me to it. And here I was ready to post my article on self-
loading test suites:
http://www.pycs.net/users/0000088/stories/7.html
Apart from the default TestLoader not picking up TestSuite subclasses,
which can be fixed with a modified main test running script, it all
works pretty well.
> Ian Bicking wrote:
> > unittest is not written with subclassing in mind, except for the
> > limited subclassing that is documented. (And it uses
> > double-underscore variables, like it's just *trying* to piss me off!
> > Double-underscore variables are so arrogant and patronizing.
>
> All very true. Double-underscores ought to be banned from the
> standard library. They inevitably get in the way because no matter
> how well a class is written, somebody is going to want to subclass it
> in a way the original author never considered.
Amen to that. I complain in my article about having to hack around
unittest's double-underscored attributes. Worse, I never found a way to
call double-underscored methods. Bah humbug.
Back to tests, though; I imagine it wouldn't take much to merge a few of
these wheel implementations and produce a new, improved unittest.
Regards,
Garth.