Scheme Test Suites?
Robert Uhl <ruhl@4dv.net>
<http://travtrack.sf.net/>, a C library with Scheme bindings (using
Guile), and it seems to me that there _must_ be a regular,
well-ordered way to write a test suite.
What I'm thinking is some way of specifying preconditions and
postconditions, and verifying that an operation performs only the
tasks it must.
My general pattern is:
(set-func)
(if (= (get-func) expected-value)
(display "test succeeded")
(error "test failed"))
But sometimes, rather than display success right away, I chain several
together. Sometimes I must perform a bit more set-up, but not
normally a whole lot.
None of this is very advanced, I think. It'd be awfully nice if
someone could point me to a (hopefully online) reference which might
have some good thoughts on how to implement and automate such a
scheme.
I'm thinking of something like:
(define-test test
"galaxy child validation"
trav-mapobject-children!
trav-mapobject-children
(trav-sector-new))
(apply-test test (trav-galaxy-new))
But I'm not attached to that pattern--and indeed, it doesn't suffice
for some of the more complex stuff I plan on doing. I figure that
this is probably a solved problem, and I'd like to avoid re-inventing
the wheel if possible.
Thanks much for any help which can be given.
--
Robert Uhl <ru...@4dv.net>
It is impossible to defend perfectly against the attack of those who want
to die.
Peter Keller
: I'm in the midst of writing various tests for travlib
: <http://travtrack.sf.net/>, a C library with Scheme bindings (using
: Guile), and it seems to me that there _must_ be a regular,
: well-ordered way to write a test suite.
: What I'm thinking is some way of specifying preconditions and
: postconditions, and verifying that an operation performs only the
: tasks it must.
The way do it is with an "expectation" of the outcome of the test. You
can "expect" either a success or a failure and that is the main point.
Example:
Suppose you define some functions like this:
(define num-failed 0)
(define good #t)
(define passed
(lambda (fname)
(begin
(print fname " PASSED!")
#t)))
(define failed
(lambda (fname)
(begin
(print fname " FAILED!")
(set! good #f)
(set! num-failed (+ num-failed 1))
#f)))
(define expect_zero
(lambda (val msg)
(if (zero? val)
(passed msg)
(failed msg))))
(define expect_nonzero
(lambda (val msg)
(if (not (zero? val))
(passed msg)
(failed msg))))
(define expect_true
(lambda (val msg)
(if (equal? #t val)
(passed msg)
(failed msg))))
(define expect_false
(lambda (val msg)
(if (equal? #f val)
(passed msg)
(failed msg))))
(define expect_equal
(lambda (val arg msg)
(if (equal? val arg)
(passed msg)
(failed msg))))
(define expect_near
(lambda (val arg msg)
(if (< (abs (- val arg)) .00001)
(passed msg)
(failed msg))))
(define expect_positive
(lambda (val msg)
(if (> val 0)
(passed msg)
(failed msg))))
(define expect_negative
(lambda (val msg)
(if (< val 0)
(passed msg)
(failed msg))))
Now, you'd use them like this:
(expect_nonzero (func arg0 arg1 arg2) "func")
(expect_equal (+ 1 2) 3 "+")
(expect_near (pi) 3.14159 "pi")
;; you might not always want to pass the name of current test, but maybe some
;; past test that might have failed in a side effecting way.
(let ((fro (thingy)))
(expect_zero (func fro) "thingy"))
If you wanted to do a series of things, but stop at the first failure:
(if (and
(expect ...)
(expect ...)
(expect ...)
(expect ...)
(expect ...)
(expect ...))
(true case)
(false case))
Code written this way is highly regular and nice to read, IMHO.
In fact, here is an example of such code I wrote to test my GNU MP foreign
function interface in Chicken:
(let ( (mpf1 (make-mpf_t))
(mpf2 (make-mpf_t)))
(expect_zero (mpf_init_set_str mpf1 "-4.44444" 10) "mpf_init_set_str")
(mpf_init_set mpf2 mpf1)
(expect_near (mpf_get_d mpf2) -4.44444 "mpf_init_set")
(mpf_init_set_ui mpf1 42)
(expect_zero (mpf_cmp_ui mpf1 42) "mpf_init_set_ui")
(mpf_init_set_si mpf1 -42)
(expect_zero (mpf_cmp_si mpf1 -42) "mpf_init_set_si")
(mpf_init_set_d mpf1 3.14159)
(mpf_init_set_d mpf2 3.14159)
(expect_nonzero (mpf_eq mpf1 mpf2 10) "mpf_init_set_d")
(mpf_clear mpf1)
(mpf_clear mpf2))
Some of the above functions like mpf_get_d had already been tested previously
so I wasn't testing them again here.
Hope this idiomatic construction helps you.
--
-pete
E-mail address corrupted to stop spam.
Reply mail: psilord at cs dot wisc dot edu
I am responsible for what I say, noone else.
Friedrich Dominicus
Friedrich
Thien-Thi Nguyen
> I think this is really nice done.
it is also done in guile distributions (OP take note). and nicely, too, in
the sense that outcomes can be one of PASS, FAIL, XPASS, XFAIL, UNRESOLVED,
UNTESTED, UNSUPPORTED, ERROR; and there is support for customizing the report.
(to see this in action, do "make check".)
now somebody just needs to do the work and package it up so that it can be
installed for general use (by guile users).
i'm sure mzscheme has good infrastructure to share (especially after reading
the "compilable and composable macros" paper, i'm thinking mzscheme *surely*
must have a good test suite! :-).
thi
Jussi Piitulainen
> i'm sure mzscheme has good infrastructure to share (especially after
> reading the "compilable and composable macros" paper, i'm thinking
> mzscheme *surely* must have a good test suite! :-).
There is one called SchemeUnit, by noel Welsh, downloadable from
SourceForge. One writes named test cases like so:
(make-test-case "Successor"
(assert = (+ 7 1) 8))
and collects them into named test suites like so:
(make-test-suite "Addition"
(make-test-case "Successor"
(assert = (+ 7 1) 8))
(make-test-case "Commutativity"
(assert = (+ 7 4) (+ 4 7)))
(make-test-case "Right Zero"
(assert = (+ 7 0) 7)))
and finally runs a suite with test/text-ui to get a report of
successes, failures and errors. Test suites nest. There are some
redundant variants of assert, and some that are not well documented; I
haven't studied the source yet; it seems to me that test case bodies
can be arbitrary expressions, and one can have several assertions in
one test case. There is a mechanism to set up a test case and to clean
it up afterwards.
It comes as a .plt file that apparently can only be opened by
installing it directly into the PLT collects tree.
--
Jussi
Noel Welsh
> There are some
> redundant variants of assert, and some that are not well documented;
The documentation is up-to-date with the current release. What are
you referring to? The documentation is at
http://schematics.sourceforge.net/schemeunit/index.htm
The next version of SchemeUnit (still in development) is discussed in
the paper linked from the Schematics home page:
http://schematics.sourceforge.net/
I'm quite excited about the new version, but then I would be.
> I haven't studied the source yet; it seems to me that test case bodies
> can be arbitrary expressions, and one can have several assertions in
> one test case. There is a mechanism to set up a test case and to clean
> it up afterwards.
The make-test-case macro has the following forms:
(make-test-case expr)
(make-test-case expr teardown)
(make-test-case expr setup teardown)
so if you want to test multiple expressions put a begin around them or
they'll be interpreted as setup and teardown expressions.
> It comes as a .plt file that apparently can only be opened by
> installing it directly into the PLT collects tree.
Correct. We plan to SRFI SchemeUnit and port it other implementations
in the very near future.
HTH,
Noel
Jussi Piitulainen
> Jussi Piitulainen wrote:
>> There are some redundant variants of assert, and some that are not
>> well documented;
>
> The documentation is up-to-date with the current release. What are
> you referring to?
What if a tested form throws an exception? What is the meaning of
(assert-exn form)?
What about, say (let (...) (assert ...) (assert ...) (assert ...)) as
a tested form? Does this count as one or three tests, or maybe two if
the middle assertion throws an exception?
What if I (let (...) (make-test-case "Test" (assert ...))) instead of
(make-test-case "Test" (let (...) (assert ...)))?
I did not find the answers in the documentation. I guess that variants
of assert catch exceptions, or maybe make-test-case wraps the whole
form in something that catches exceptions...
I have only had the package for a week or two, and it was dated, what?
2001? So I did not expect it to change just now.
As for redundancy, I see no merit to (assert-eqv? ...) over (assert
eqv? ...), and very little to (assert-false form) over (assert-true
(not form)).
> The next version of SchemeUnit (still in development) is discussed
> in the paper linked from the Schematics home page:
I have to go and look again. I did not notice any paper.
> so if you want to test multiple expressions put a begin around them
> or they'll be interpreted as setup and teardown expressions.
But are the outcomes of the separate assertions inside one test case
counted separately?
> We plan to SRFI SchemeUnit and port it other implementations in the
> very near future.
Go for it.
--
Jussi
Ray Blaak
> But are the outcomes of the separate assertions inside one test case
> counted separately?
My advice, having written a few testing frameworks myself, is not to count
them separetely.
Instead, let the user have control as to what is the thing that is passing or
failing. In this case, that is clearly the test case itself, via
(make-test-case ...).
The proper way to view (assert ...) is to consider it as a single verification
that either succeeds or else contributes to the failure of the current test
case. Unexpected exceptions also would fail the test, albeit for a different
reason.
With this kind of view, then the composition of test logic is straightforward
-- one can decide the granularity of the test cases being officially reported
separately from the implementations of each test case. In particular, we would
have this kind of equivalence:
(make-test-case "some test"
(assert (and this that the-other) "someting did not occur"))
is the same (in terms of what is being tested) as:
(make-test-case "some test"
(let
(assert this "this did not occur")
(assert that "that did not occur")
(assert the-other "the-other did not occur")))
If you really want to have these has separately reportable test cases, then do
this instead:
(make-test-case "this test" (assert this "this did not occur"))
(make-test-case "that test" (assert that "that did not occur"))
(make-test-case "the-other test" (assert the-other "the-other did not occur"))
BTW, (assert...) should have the ability to let user-specified explanations of
failures be given (if it doesn't already). Otherwise, things like (assert (= a
b)) aren't very descriptive if it fails.
Whether a single assert halts the current test case or lets execution continue
is a different question. I have used frameworks that do either style, and am
of two minds as to which approach is better.
On the one hand I have observed that halting the current test case via an
exception is the simplest thing to do for most tests and is often adequate.
On the other hand, having (assert...) to merely report and set the current
test as failed and only having some sort of explicit (abort-test-case) form
actually halt the test prematurely allows some more complete testing. E.g.,
assuming that (assert...) does *not* halt the test, then:
(make-test-case "complex test"
(assert required-measurement)
(assert another-required-measurement)
(unless critical-measurement
(assert #f "critical measurement failed, cannot proceed")
;; or just: (fail "critical measurement failed, cannot proceed")
(abort-test-case))
(assert subsequent-measurement))
this way we can measure a number of failures at once, and yet still have
control as to which failures mean the rest of the test is no longer even
relevant.
If (assert...) halted the current test immediately, one would be forced to fix
each failure and rerun before being aware of all the failures, which can be
tedious.
Caveat: I have not used or reviewed the framework being discussed in any
way. I am only commenting from general principles.
--
Cheers, The Rhythm is around me,
The Rhythm has control.
Ray Blaak The Rhythm is inside me,
bl...@telus.net The Rhythm has my soul.
MJ Ray
><http://travtrack.sf.net/>, a C library with Scheme bindings (using
> Guile), and it seems to me that there _must_ be a regular,
> well-ordered way to write a test suite.
ISTR that greg is a test system for guile programs, linked from their web
site. You might also like to look at Schematics.sf.net and the SchemeUnit
system. Hopefully they will converge (or at least SchemeUnit's interface
gets ported).
MJR
Robert Uhl <ruhl@4dv.net>
>
> The way do it is with an "expectation" of the outcome of the test.
> You can "expect" either a success or a failure and that is the main
> point.
[snip]
> Hope this idiomatic construction helps you.
Thanks--I'll take a look at this.
--
Robert Uhl <ru...@4dv.net>
To murder a man is much odious, to kill a woman is in manner unnatural,
but to slay and destroy innocent babes and young infants, the whole
world abhorreth, and their blood from the earth crieth for vengeance to
almighty God. --Edward Hall, c. 1480
Noel Welsh
> What if a tested form throws an exception? What is the meaning of
> (assert-exn form)?
Ok, I can see the need for better documentation. The paper gives more
details on this. We'll plagarize it for the documentation of the
upcoming release.
Ray Blaak is generally on the money. To parameterise his
explaination: A single failed assertion halts a test case. Assertions
in the dev version can take an optional message.
> I have only had the package for a week or two, and it was dated, what?
> 2001? So I did not expect it to change just now.
There have been releases of Scheme in May and Jan 2002 so it sounds
like you have an old version. Perhaps I should publicise the releases
more.
> As for redundancy, I see no merit to (assert-eqv? ...) over (assert
> eqv? ...), and very little to (assert-false form) over (assert-true
> (not form)).
To each their own. You get better messages if you use the form
closest to what you're testing. This is particularly applicable in
the dev version where users can easily build libraries of assertions.
> But are the outcomes of the separate assertions inside one test case
> counted separately?
Ray gave a good description of this. A test-case is the unit of
testing. Assertions are placed inside a test-case. A single
assertion failure halts testing of a test-case.
HTH
Noel
Jussi Piitulainen
> Jussi Piitulainen wrote:
>> What if a tested form throws an exception? What is the meaning of
>> (assert-exn form)?
>
> Ok, I can see the need for better documentation. The paper gives
> more details on this. We'll plagarize it for the documentation of
> the upcoming release.
I went to the SourceForge site - or is it better to call it Schematics
site? - but still could not find a paper. Do you mean one titled
"Lambda tge Ultimate SchemeUnit and SchemeQL: Two Little Languages"?
Google hit that, too. (I cannot read Postscript easily just now, but
maybe tomorrow.)
Here's a couple more things that you might be sure to document:
1. The test cases are not run in the order they are written. At
least I think they aren't, and I only learnt this when I ran
my tests. (I'm the weird kind who actually reads documentation
before trying things out. Hence my concern.)
2. Do assertions need to be physically inside test cases? I do not
even know. Is (let ((ass (lambda (x) (assert = x 3))))
(make-test-case "Huh?" (ass 3)))
all right? Not terribly important, of course.
3. Does it need to be an equality predicate? Is (assert < 3 17)
guaranteed to succeed?
4. What about (make-test-case "" "Bee or no bee?")? Is it a syntax
error or a failed case or a success case or what, when there is
no assertion at all?
> Ray Blaak is generally on the money. To parameterise his
> explaination: A single failed assertion halts a test case.
Ok. I am using the system happily. I only hope the documentation to
tell me these things.
> There have been releases of Scheme in May and Jan 2002 so it sounds
> like you have an old version. Perhaps I should publicise the
> releases more.
I think I was mistaken. The (HTML) docs are indeed from May.
When I decided that I need such a suite, I remembered your message in
this group some time ago, and I searched for it. I'm not sure if I
looked in schemers.org first; if I did, I did not find it there.
>> As for redundancy, I see no merit to (assert-eqv? ...) over (assert
>> eqv? ...), and very little to (assert-false form) over (assert-true
>> (not form)).
>
> To each their own. You get better messages if you use the form
> closest to what you're testing. This is particularly applicable in
> the dev version where users can easily build libraries of
> assertions.
So I could add assert-= and assert-array-equal?? In that case, I stop
feeling that the provided three are arbitrary. I think I will still
continue to use just assert. Either it is enough to know which test
case failed, or else I'll have to make the test cases finer. My taste.
>> But are the outcomes of the separate assertions inside one test
>> case counted separately?
>
> Ray gave a good description of this. A test-case is the unit of
> testing. Assertions are placed inside a test-case. A single
> assertion failure halts testing of a test-case.
Ok. Yes, his message was sensible, but of course it described general
principles, not this particular system. One of my questions above
remains: what if a test case lacks any assertions? Is that a success?
(I suggest that it should be a failure, to protect a careless test
case author agains a false sense of security.)
Just to be sure, I'm a happy user despite all my complaints,
--
Jussi
Ray Blaak
> One of my questions above remains: what if a test case lacks any assertions?
> Is that a success? (I suggest that it should be a failure, to protect a
> careless test case author agains a false sense of security.)
Theoretically you would be right, forcing the author to specify success
constructively "guarantees" that passing cannot occur accidentally.
However, in practice, it is almost always the case a typical test case wants
no failures == passed. That is it usually more useful to have "no assertions"
== "only passed assertions" == "no failed assertions".
Thus, (make-test-case "nothing special" (begin)) passes, trivially.
Consider also the case where you are testing that no unexpected exceptions
occur:
(make-test-case "ensure no exceptions" (thunk-that-has-no-assertions))
Granted, you can always code an explicit pass criteria, but in practice
forcing that to be always required tends to be tedious. There is no loss of
generality either way, so it is best to pick the approach that is most
convenient for the usual case.
Thien-Thi Nguyen
> However, in practice, it is almost always the case [...]
if the test framework doesn't handle (aka allow expression) of all cases, that
incompleteness then becomes part of the test burden (aka another thing to test
for). for this reason, "almost always the case" should be codified in user
land; the framework should provide primitives to support but not enforce this.
[insert classic separation of interests spew here.]
thi
Jussi Piitulainen
> Granted, you can always code an explicit pass criteria, but in
> practice forcing that to be always required tends to be tedious.
> There is no loss of generality either way, so it is best to pick the
> approach that is most convenient for the usual case.
I find the difference in tedium very slight here. Also, I think I want
to make test cases stronger than just "no exceptions", so there would
usually be some natural predicate to assert anyway.
But "no failed assertions => success" is certainly a pleasant, simple
rule to live with. Just make it explicit in the documentation.
--
Jussi
Ray Blaak
> if the test framework doesn't handle (aka allow expression) of all cases, that
> incompleteness then becomes part of the test burden (aka another thing to test
> for). for this reason, "almost always the case" should be codified in user
> land; the framework should provide primitives to support but not enforce this.
Certainly. One could have this:
(raw-test-case ...)
that fails if one does *not* set an explicit pass.
Then, you can build on top of it:
(make-test-case test-case-name ...)
which expands into something like:
(raw-test-case test-case-name
(begin
...
(pass-if-not-failed-yet)))
then you have full generality, plus convenience.
Thien-Thi Nguyen
> then you have full generality, plus convenience.
yes, that's the best.
(why can't life be like this? :-)
thi