fixing, LIFTing the test suite
Stephen Compall
translations. I cloned the writers, so if you have write perms on
dev, you can write to this as well. Test translation and test fixes
are welcome.
http://bitbucket.org/S11001001/wb-test-fixes
There are many, many tests, and while Slava is developing a solution
to the problem of lots of literal HTML in tests, there are many tests
that don't require lots of HTML, and these can be changed now. So
here is a brief guide to testing in Weblocks, then and now.
Then (RT)
*********
RT is a simple test system. Forms look like this:
(deftest SYMBOL
FORM
VALUE ...)
When this form is evaluated, the test is stored in its global map
under the name SYMBOL. When you run tests, you run all RT tests
registered by any package.
Each of the values resulting from evaluating FORM is compared to a
VALUE, by something like EQUAL. The VALUEs are not evaluated; this is
why you see #. all over the place in the existing tests.
RT doesn't provide fixtures (setup and teardown before every test), so
every test that needs some environment set up has to do it itself.
This is what with-webapp and with-request are for.
It also inconveniently doesn't provide ASSERT-like testing -- you have
to save each actual value and make sure it's in the values answered by
FORM.
Now (LIFT)
**********
LIFT is a CLOS-heavy test framework with composable suites, fixtures,
various assertions, and all that good stuff.
At the top of every file (why will be clear in a moment) should be a
unique, somewhat defclass-compatible `deftestsuite' form¹; for
simplicity, extend weblocks-suite to pull in request, session, and
application fixtures; you may also create your own fixtures.² Here's a
sample from dependencies.lisp, which I have recently converted.
(deftestsuite dependencies-suite (weblocks-suite)
())
While this isn't a convention yet, I propose that your suite be named
after the filepath to it. So dataform-suite would be renamed to
widgets/dataform-suite.
Now, let's translate a small test:
(deftest dependencies-2
(with-webapp ()
(dependencies "some-string"))
nil)
with-webapp's features are now provided by the fixture, so this
becomes:
(addtest dependencies-2
(ensure-same (dependencies "some-string") 'nil))
Now I can run the test with
weblocks-test> (run-test :test 'dependencies-2).
#<dependencies-suite.dependencies-suite passed>
addtest adds to the last-defined test; you need the testsuite
definition in every file because otherwise your test may end up in
some other random suite when you press C-c C-k or something.
Remember: *never define or redefine a test with C-M-x, always reload
or recompile the whole file.*³
I quoted the nil to emphasize that the "expected" part is now
evaluated. Obviously, barring any other niceties, all of the tests
can be trivially translated to combinations of `addtest' and
`ensure-same'. (When using ensure-same, remember that the first arg
should be the "actual" values, and the second the "expected".) But we
can do better:
(addtest dependencies-2
(ensure-null (dependencies "some-string")))
There are many ensure-* variants; consult the user guide⁴ for the
built-ins. As the ensure-* things are just macros, they are in a
normal evaluation position, so you can put them inside let forms, in a
lambda passed to mapc, and so on. Every test can have as many of them
as you like; when one fails, LIFT terminates that test as a failure,
and proceeds to the next test.
LIFT walks subclasses, so you can easily run your test with the rest
of the suite:
weblocks-test> (run-tests :suite 'weblocks-suite)
Here you get a very small report of number of failures and such. For
the full story, run `describe' on the result of that form:
weblocks-test> (describe *)
Comparing structures
********************
RT locks you into its EQUAL variant for all comparison, so anything
else required either string coercion, or just running a predicate form
and comparing it to T or NIL. `ensure-same', and its counterpart
`ensure-different', have some more features:
(deftest make-local-dependency-3
(format nil "~A" (dependency-url (make-local-dependency :stylesheet "main")))
"/pub/stylesheets/main.css")
This translates to:
(addtest make-local-dependency-3
(ensure-same (dependency-url (make-local-dependency :script "weblocks"
:do-not-probe t))
(puri:uri "/pub/scripts/weblocks.js")
:test puri:uri=))
If you have many of these, you can simply rebind *lift-equality-test*
and do them all in there:
(addtest probe-dependencies
(let ((*lift-equality-test* 'puri:uri=))
(ensure-same (dependency-url (make-local-dependency :stylesheet "non-existing-file-name"
:do-not-probe t))
(puri:uri "/pub/stylesheets/non-existing-file-name.css"))
(ensure-same (dependency-url (make-local-dependency :stylesheet "main"))
(puri:uri "/pub/stylesheets/main.css"))
(ensure-same (dependency-url (make-local-dependency :script "weblocks"))
(puri:uri "/pub/scripts/weblocks.js"))
(ensure-same (dependency-url (make-local-dependency :script "weblocks" :do-not-probe t))
(puri:uri "/pub/scripts/weblocks.js"))))
Of course you don't want to collapse too many tests together, even for
this convenience; if you can get your tests grouped into a suite, you
can use the :dynamic-variables deftestsuite option to bind
*lift-equality-test* for every test in your suite.
When you have a list of values to compare, remember that ensure-same
compares all multiple values received, so consider using values-list.
That way LIFT will point out the item that failed, instead of just
writing larger list structures in the report.
Even more comparison
********************
I've already taken advantage of the ability to extend the set of
`ensure-*' macros to aid in HTML test conversion. For large string
comparison, `ensure-nodiff' is like `ensure-same', but if you have
trivial-shell loaded, you'll get the output of running `diff -du' on
the strings.
A level of wrapping later and we come to `ensure-same-html':
weblocks-test> (ensure-same-html "<a><b><c>" "<a><d><c>")
; Warning: Ensure-same: "<a>
; <b>
; <c>" is not equal to "<a>
; <d>
; <c>" (Differences found:
; --- /tmp/file0x4L8p 2008-08-31 22:13:27.000000000 -0500
; +++ /tmp/fileKYTk5b 2008-08-31 22:13:27.000000000 -0500
; @@ -1,3 +1,3 @@
; <a>
; -<d>
; +<b>
; <c>
(The newlines are inserted because this would otherwise be useless. I
don't think any tests will be negatively affected by this.)
Another layer of wrapping in the form of `ensure-html-output',
provides all of the features of deftest-html, so I can translate the
last forms in dependencies.lisp:
(deftest-html render-dependency-in-page-head-1
(render-dependency-in-page-head (make-instance 'script-dependency :url "http://boing.com/abc.js"))
(:script :src "http://boing.com/abc.js" :type "text/javascript" ""))
becomes
(addtest render-dependency-in-page-head-1
(ensure-html-output
(render-dependency-in-page-head
(make-instance 'script-dependency :url "http://boing.com/abc.js"))
(:script :src "http://boing.com/abc.js" :type "text/javascript" "")))
and for large HTML content I get the benefit of full diff information.
Style
*****
I wrote the basics above, but Slava really put more style into
translating tests. Look in test/widgets/dataform.lisp: interlacing
ensure forms and regular forms to do stuff, came up with new, good
names for the tests, uses a slot (check out the references to the
`dataform' variable in the tests—where does it come from?), and is all
around just neat.
Footnotes
*********
¹ http://common-lisp.net/project/lift/user-guide.html#header3-86
² One obscure point is that dynamic variables defined in the
deftestsuite form are propagated to subclasses when those subclasses
are defined. So if you change application-suite's
:dynamic-variables argument, you must reevaluate all subclasses.
This is unfortunate, but the state of things as I write this.
³ Unless you, like, you know, really know what you're doing.
⁴ http://common-lisp.net/project/lift/user-guide.html#header3-88
--
I write stuff at http://failex.blogspot.com/ now. But the post
formatter and themes are terrible for sharing code, the primary
content, so it might go away sooner or later.
Vyacheslav Akhmechet
is slow, seems to eat a lot of memory, and appears to be buggy in
quite frustrating ways. Other than Lift, there isn't a single CL unit
test framework that handles fixture setup well (IMO), so I've spent
the past couple of days working on a test framework that fixes these
issues. I think that hierarchical fixture setup (just like
hierarchical suite definitions) is a key feature for testing modern
software.
It is very similar to Lift in the user interface (with some minor
differences), so porting Lift tests to it is a trivial, almost a
mechanical exercise. Tests look exactly the same, and suite definition
look only slightly differently. So, if you're porting existing tests
to Lift, I'll trivially port them to this framework later.
It's almost ready for prime time, and it runs existing Lift tests with
almost no modifications. I'll release it soon, I just have to polish
it around the edges. I hope it will make unit testing for weblocks
(and many other projects) a breeze.
Stephen Compall
> It is very similar to Lift in the user interface (with some minor
> differences), so porting Lift tests to it is a trivial, almost a
> mechanical exercise. Tests look exactly the same, and suite definition
> look only slightly differently. So, if you're porting existing tests
> to Lift, I'll trivially port them to this framework later.
Will it also be easy to port my custom ensure-* macros?
Vyacheslav Akhmechet
> Will it also be easy to port my custom ensure-* macros?
Leslie P. Polzer
> almost no modifications. I'll release it soon, I just have to polish
> it around the edges. I hope it will make unit testing for weblocks
> (and many other projects) a breeze.
Does it have a functional interface, too?
Also, it would be nice if it assumed a default test suite if no IN-
SUITE statement was specified.
Stephen Compall
> Also, it would be nice if it assumed a default test suite if no IN-
> SUITE statement was specified.
This would be inherited from LIFT—assuming, as seems likely, that Slava
is inheriting the basic semantics in order to support essentially the
same deftestsuite/addtest/ensure-* flow.
Vyacheslav Akhmechet
> Great! It's YACLUTF! ;)
>
> Does it have a functional interface, too?
custom error handlers (so that you can debug the test if it fails).
Saying (run-test 'some-test-foo) runs it with the error handlers in
place, so that you just get a report instead of being dropped in a
debugger. It turns out to be quite a nice system.
> Also, it would be nice if it assumed a default test suite if no IN-
> SUITE statement was specified.
Yes, that's in.
avodo...@gmail.com
BTW, in the test/weblocks-test.lisp:
(defun test-weblocks (&optional (verbose t))
"[...]"
;; XXX better results combination
(let ((results (list (run-tests :suite 'weblocks-suite)
(run-tests :suite 'weblocks-store-test::store-suite))))
(when verbose
(mapc #'describe results))
(values-list results)))
lift test suites seems to be are named by strings, at least the same
test suites may be run by (run-tests :suite :weblocks-suite)
(run-tests :suite :store-suite). If so, it may be better to
rename :store-suite to :weblocks-store-suite.
To combine results, you may combine test suties. Define a super-suite
which includes both test suites:
(deftestsuite weblocks-super-ensure () ())
(deftestsuite store-suite () (weblocks-super-suite))
(deftestsuite weblocks-suite () (weblocks-super-suite))
(run-tests :suite :weblocks-super-suite)