clojure.contrib.test-is changes
Stuart Sierra
1. *test-out* now defaults to standard output (instead of standard
error). This should be better for SLIME users.
2. Based on Frantisek Sodomka's suggestions, I've added two
convenience macros, each= and all-true:
user=> (doc all-true)
-------------------------
clojure.contrib.test-is/all-true
([& body])
Macro
Convenience macro; every body expression is tested as with 'is'.
nil
user=> (doc each=)
-------------------------
clojure.contrib.test-is/each=
([& forms])
Macro
Convenience macro for doing a bunch of equality tests. Same as
doing (is (= ...)) on each pair.
(each= (test-expr-1) expected-value1
(test-expr-2) expected-value2
(test-expr-3) expected-value3)
I also modified test_clojure/numbers to demonstrate use of all-true.
I though about calling it "are", since it's like a plural form of
"is", but I thought that might be confusing for non-native-English
speakers.
I have committed these changes ONLY in the new, no-namespace-directory
files.
-Stuart
Paul Drummond
Cool additions - I will certainly use them.
Regarding test-is additions, did you ever get a chance to check out the patch I submitted a while back?
http://groups.google.com/group/clojure/browse_thread/thread/c509d589e181df1e/4319c02c9930d12e?lnk=gst&q=PATCH+test-is#4319c02c9930d12e
It was my first patch so appologies if there are some glaring errors or bad-style in there (but it was a very simple addition so I'd hope not!).
I realised just after posting that catching Errors in the general case could be viewed as a no-no because the JVM could be in an unstable state after the Error (depending on which error was thrown) so the desired outcome would be to terminate. Is that your view?
For my particular scenario, I wanted test-is to support StackOverflowError. I was working on a SICP exercise that caused StackOverflowErrors and I wanted to have a test that passed when a StackOverflowError occurred, then continue processing other tests. So in my specific case, I felt the patch made sense but maybe supporting *all* Errors is going to far?
Hmmm........
Cheers,
Paul.
--
Iode Software Ltd, registered in England No. 6299803.
Registered Office Address: 12 Sancroft Drive, Houghton-le-Spring, Tyne & Wear, DH5 8NE.
This message is intended only for the use of the person(s) ("the intended recipient(s)") to whom it is addressed. It may contain information which is privileged and confidential within the meaning of applicable law. If you are not the intended recipient, please contact the sender as soon as possible. The views expressed in this communication may not necessarily be the views held by The Company.
Frantisek Sodomka
It will certainly make writing tests more enjoyable :-)
Inspiration for :equal-pairs/each= came from the test framework I wrote
for newLISP:
http://newlisp-on-noodles.org/wiki/index.php/Function_Testing
Tests there are written as each= with 2 exceptions:
'->' evaluates the next expression without testing it
'err' works as 'throws'
Function 'test-fn' evaluates tests and returns the list of results.
Function 'print-test-fn' can take these results and just print them out.
Feel free to get inspired from it. If you see something nice, we can use
it.
Frantisek
PS: newLISP is changing fast. Test framework was written for older version
so many things might not work at this moment.
Stuart Sierra
Thanks, glad you like it. I hadn't seen your earlier post -- wasn't
following the group for a couple weeks there. My first inclination is
not to catch any Error subclasses, as Sun says,
http://java.sun.com/javase/6/docs/api/java/lang/Error.html
But, StackOverflowError might be an exception (no pun intended) to the
rule, at least according to this guy:
http://blog.igorminar.com/2008/05/catching-stackoverflowerror-and-bug-in.html
In any case, if you want to assert that a particular form *always*
throws an error, you can use the "throws" macro:
user=> (use 'clojure.contrib.test-is)
nil
user=> (defn foo ([] (foo))
{:test (fn [] (throws java.lang.StackOverflowError (foo)))})
#'user/foo
user=> (run-tests)
Testing user
Ran 1 tests with 1 assertions.
0 failures, 0 exceptions.
nil
user=> (foo)
java.lang.StackOverflowError (NO_SOURCE_FILE:0)
So I probably won't catch Error, unless there's another scenario in
which it's needed.
-Stuart Sierra
On Nov 17, 6:33 am, "Paul Drummond" <paul.drumm...@iode.co.uk> wrote:
> Hi Stuart,
>
> Cool additions - I will certainly use them.
>
> Regarding test-is additions, did you ever get a chance to check out the
> patch I submitted a while back?
>
Frantisek Sodomka
Few more ideas for you :)
each= could be extended to allow more freedom. The first parameter could
say what operation we want to perform:
each =
each not=
each <
each >
...
(each =
a b
c d)
=>
(all-true
(= a b)
(= c d))
(each not=
a b
c d)
=>
(all-true
(not= a b)
(not= c d))
Since 'each' might be too general (could it be part of Clojure core?), I
also like the name is-each which fits nicely with 'is'.
is-each =
is-each not=
...
It might be a good idea to check if count of forms passed to each/is-each
is even:
(if (odd? (count forms)) (throw ... "even number of forms needed!"))
Many thanks, Frantisek
Frantisek Sodomka
Lets look at this test for minus:
(deftest test-minus
(all-true
(number? (- 1 2))
(integer? (- 1 2))
(float? (- 1.0 2))
(ratio? (- 2/3 1))
(float? (- 2/3 (/ 1.0 3))))
(throws IllegalArgumentException (-))
(each=
(- 1) -1
(- 1 2) -1
(- 1 2 3) -4
(- -2) 2
(- 1 -2) 3
(- 1 -2 -3) 6
(- 1 1) 0
(- -1 -1) 0
(- 2/3) -2/3
(- 2/3 1) -1/3
(- 2/3 1/3) 1/3
(- 1.2) -1.2
(- 2.2 1.1) 1.1
(- 6.6 2.2 1.1) 3.3)
; no underflow :)
(is (< (- Integer/MIN_VALUE 10) Integer/MIN_VALUE)))
There are currently 4 elements - all-true, throws, each=, is. I am
scratching my head, trying to figure out, how to merge these together or
to make it read better...
Ideas:
A) Lets merge 'is' and 'all-true'
Would it be possible to abandon the optional message 'is' macro has? I am
sure, I will not use it. Then 'is' could accept multiple params and we are
down to 3! (all-true is out)
B) What about 'throws' macro? Could this become a function returning
true/false? Then we could stick it inside 'is' or 'all-true'. (I guess it
doesn't matter that much, does it?)
C) Lets name them more similar, so it reads better: 'is', 'is-each',
'is-true'.
Just what came to my mind :-)
Greetings, Frantisek
Meikel Brandmeyer
Am 18.11.2008 um 00:13 schrieb Frantisek Sodomka:
> B) What about 'throws' macro? Could this become a function returning
> true/false? Then we could stick it inside 'is' or 'all-true'. (I
> guess it
> doesn't matter that much, does it?)
I have a is-like construct, which is build-up slightly different.
Instead of checking, what I get, I immediatelly dispatch to a
multimethod.
(defmacro is [t msg] (is* t msg))
(defmulti is* (fn [t _] (first t)))
(defmethod is* :default `(simply-run-t-here ...))
(defmethod is* nil `(always-fail ...))
(defmethod is* '= `(do-test-here ...))
(defmethod is* 'not= `(do-test-here ...))
(defmethod is* 'instance? `(do-test-here ...))
(defmethod is* 'throwing? `(do-test-here ...))
So it is easy to extend is with other tests, while still being able
to provide diagnostics in case a test should fail.
Furthermore I use a test driver function, which reduces the repition to
a minimum. It takes care to run the tests in a try, report the result
and provide diagnostics in case of a failure. Then each methods above
just has to specify in a callback how the actual test is carried out and
what kind diagnostics should be printed. (eg. compare assert-expr
for = and instance?. They are very similar.)
What do you think about such a structure?
Sincerely
Meikel
Stuart Sierra
Good suggestions, all. I'll have to spend some time looking at these
and figure out if I can make them work with the existing test-is. Two
thoughts first:
1. I want to keep optional messages per-assertion. These are very
useful in the RSpec testing framework for Ruby; they're like comments
explaining what each assertion is supposed to demonstrate.
2. The current 'is' macro works more or less like the one Meikel
described. The multimethod is "assert-expr", but it's complicated and
could be simplified.
In general, I want to make the library more oriented towards
functional programming, and less reliant on macros. That should both
simplify the interface and make it easier to add new kinds of
assertions. Stay tuned.
-Stuart Sierra
> 5KViewDownload
mb
On 18 Nov., 15:58, Stuart Sierra <the.stuart.sie...@gmail.com> wrote:
> 1. I want to keep optional messages per-assertion. These are very
> useful in the RSpec testing framework for Ruby; they're like comments
> explaining what each assertion is supposed to demonstrate.
implementation for Clojure, which I find nice for communicating test
results to external processes. In TAP also the messages are used
as some kind of documentation.
This leads me to another question: Is it possible to look into
pluggable
harnesses? That is: can we separate the tests from the result
reporting?
In my TAP implementation I currently have two harnesses, one
produces TAP output, one can be used inside a test to allow recursive
tests. One application is for example the ClojureCheck library I am
working on.
(holds?
(for-all [x Integer
y Integer]
(is (= (+ x y) (+ y x))))
"addition commutes")
for-all sets up a batch-harness, so the body of the for-all may
contain
any number or form of tests. The result of the for-all, is then the
result
of the internal tests.
> 2. The current 'is' macro works more or less like the one Meikel
> described. The multimethod is "assert-expr", but it's complicated and
> could be simplified.
example with working code later on today.
Sincerely
Meikel
Stuart Sierra
> This leads me to another question: Is it possible to look into
> pluggable harnesses? That is: can we separate the tests
> from the result reporting?
be a reporting function that can be dynamically rebound.
-S
Meikel Brandmeyer
as previously threatened here the anatomy of my TAP library.
The testing and reporting is split in separate parts. The testing
part provides is as main interface. I first followed Perl's Test::More,
but thought it would be better to be closer to test-is. So I adopted
is, but - as I think - in slightly easier fashion.
The core is (as with test-is) the is macro, which is implemented
as a multimethod. Since there is no multimacro, we use is* for
the method.
(defmulti is* (fn [x & _] (if (seq? t) (first t) t)))
(defmacro is [t & desc] (is* t (first desc)))
Now we can start to define tests.
; (is (= actual expected) "description")
(defmethod is* '=
[t desc]
(let [actual (nth t 1)
exptd (nth t 2)]
`(test-driver (fn [] ~actual)
(quote ~actual)
(fn [] ~exptd)
~desc
(fn [e# a#] (= e# a#))
(fn [e# a# r#]
(diag (.concat "Expected: " a#))
(diag (.concat "to be: " e#))
(diag (.concat "but was: " r#))))))
Now this looks scary. So please let me explain. The first argument
packages about the "actual" expression. The second quotes it. The
third is the "expected" expression. It is also packaged up in a
closure. The fourth is the test description and the fifth the actual
test. The last argument is a callback, which is called in case the
test fails and which might be used to provide specialised diagnostic
message. I like having my tests tell me, why they failed.
While this seems quite complicated, I think it really is this essence
of a test. Everything else is boilerplate which is handled in the
test-driver function.
(defn test-driver
[actual qactual exp desc pred diagnose]
(try
(let [e (exp)
a (actual)
r (pred e a)]
(report-result r desc)
(when-not r
(let [es (pr-str e)
as (pr-str qactual)
rs (pr-str a)]
(diagnose es as rs)))
a)
(catch Exception e
(report-result false desc)
(diag (str "Exception was thrown: " e))
`test-failed)))
So this is pretty straightforward. Fire up a try, evaluate the
expected and actual expression, run the predicate and check the
result. In case the test fails or an Exception is thrown, the
failure is reported and some diagnostics are printed.
So that completes the testing part. From the user point of
view one has function - is -, which handles all the testing.
To provide a new test form, one simply defines a new method.
In the method, one simply passes the work to test-driver which
takes care for reporting and proper test execution.
The reporting side already showed up here and there in form of
the diag and the report-result functions. Others are plan (for
TAP), get-result to retrieve the test results. They act on a
global Var *the-harness*. They can also be implemented as
multimethods. A TAP harness would produce TAP output, an
"interactive" harness would only report failures and maybe some
statistics, a "batch" harness just keeps book of failing tests
and diagnostics for recursive use.
Changing a harness is a simple matter of
(binding [*the-harness* (make-some-harness)]
(do-some tests here))
Although this looks terribly complicated and is a hard to
digest bunch of stuff, I'd appreciate your comments. However,
how this can be made more functional... I have no clue.
Sincerely
Meikel