clojure.test - Test fixtures and ordering

[Matt Revelle]

Hi group,

I recently noticed that fixtures were not being called for some tests
in my code and it turns out (thanks, Chouser) the use of test-ns-hook
is incompatible with fixtures. This isn't necessarily undesirable
behavior, but it raises an issue: does anyone want high-level support
for grouping and ordering tests while using fixtures to setup and tear
down test resources?

Currently, if one wishes to specify which tests are called while using
fixtures, a test-ns-hook function needs to be implemented that
directly calls fixtures with the appropriate tests as arguments.

The test namespace that brought the issue to my attention can be found
at:
http://github.com/mattrepl/clojure-cassandra/blob/ff4ed1c03fb5e9888419198eb8f3a57c7d69beac/src/cassandra/test.clj

In this example, the test-ns-hook would become:
http://gist.github.com/146934

-Matt

[Stuart Sierra]

On Jul 14, 9:41 am, Matt Revelle <mreve...@gmail.com> wrote:
> I recently noticed that fixtures were not being called for some tests  
> in my code and it turns out (thanks, Chouser) the use of test-ns-hook  
> is incompatible with fixtures.  This isn't necessarily undesirable  
> behavior, but it raises an issue: does anyone want high-level support  
> for grouping and ordering tests while using fixtures to setup and tear  
> down test resources?

Hi Matt,
You're right; it doesn't work. Fixtures were added later, long after
test-ns-hook, and I never used them together.

Something I've been meaning to do is replace "test-ns-hook" with
metadata on the namespace. But that still won't solve the fixture
problem.

Namespace-wide fixtures ("once-fixtures") are easy -- they should just
run around the top-level test function. That's something I can fix,
and it will be sufficient for your example.

But per-test fixtures ("each-fixtures") present a problem. If the
fixtures are run around every test function, then they will be called
multiple times when test functions are nested. Should they be called
just around the top-level test functions, or just around the bottom-
level test functions? And how do you determine where the "bottom" of
the test function hierarchy is?

I don't know the answers to these questions. So for now, my solution
is: if you want to compose tests by calling them as functions, then
you should implement your own fixtures using macros, like this:

(defmacro with-db [& body]
`(... set-up ...
~@body
... tear-down ...)))

(deftest foo
(with-db ...))

That way you can decide exactly where the fixture should be run. This
is how I used to do fixtures before implementing them in the testing
library.

-Stuart Sierra

[Jarkko Oranen]

On Jul 14, 6:58 pm, Stuart Sierra <the.stuart.sie...@gmail.com> wrote:
> Namespace-wide fixtures ("once-fixtures") are easy -- they should just
> run around the top-level test function.  That's something I can fix,
> and it will be sufficient for your example.
>
> But per-test fixtures ("each-fixtures") present a problem.  If the
> fixtures are run around every test function, then they will be called
> multiple times when test functions are nested.  Should they be called
> just around the top-level test functions, or just around the bottom-
> level test functions?  And how do you determine where the "bottom" of
> the test function hierarchy is?

I don't think ns-test-hook should really care about this. Perhaps it
would be best to consider test-ns-hook a low-level construct that can
do whatever it wants with the defined tests and fixtures, and provide
some other means for specifying which tests will be run.

Alternatively (or additionally), provide a facility for tagging tests
and defining fixtures that are run around tests with a specific tag.
Then, you could simply use a tag that has no fixtures defined:

; no explicit tag implies "each"?
(deftest test1 ...)
(deftest test2 ...)

(deftest testgroup {:fixtures 'no-fixtures}
(test1)
(test2))

To get the fixtures called, though, deftest probably needs to return
something like:
(def testname
(fn [] (execute-test test-info-map ; we should have the map at
this point so we can form a closure, right?
(fn [] (actual-test-content-here)))

where execute-test would by default be bound to a function that runs
whatever fixtures the test has. A custom test-ns-hook could rebind
execute-test at will.

This could all be done with fixture macros manually too of course, but
wrapping functions are probably somewhat easier to write.

--
Jarkko


PS. For whatever reason, this stuff makes me think of monads. :)

[Stuart Sierra]

On Jul 14, 1:23 pm, Jarkko Oranen <chous...@gmail.com> wrote:
> Perhaps it
> would be best to consider test-ns-hook a low-level construct that can
> do whatever it wants with the defined tests and fixtures, and provide
> some other means for specifying which tests will be run.

Yes, I'm leaning this way too. If you want to control the order in
which tests run, you should also control when set-up / tear-down
occurs, and not use the fixture mechanism at all. For example:

(defn test-ns-hook
... global set-up ...
(test-group-one)
(test-group-two)
... global tear-down ...)

(deftest test-group-one
... set-up specific to group one ...
(test-a)
(test-b))

-SS