observations and questions on clojure testing
Stuart Halloway
Feel free to start new threads with responses, since there are a lot of different things in here...
---------
(1) Invariants are nice, and neither assertions nor clojure.test/is are ideal for capturing them. Do we need a new thing? Should assertions get smarter?
The assert-valid-hiearchy function captures are bunch of invariants about a clojure derive hierarchy, such as "the ancestors relationship is the transitive closure of the parents relationship." I am using test/is, but neither assertions nor test/is are a perfect fit for capturing invariants, because:
(1a) Invariants may be quite expensive to calculate, so you don't want to assert them inline
(1b) Even despite 1a, invariants are useful in production code. So, you don't want them hidden away in tests.
(1c) Assertions currently do not report enough context.
(1d) test/is is closer to reporting the right amount of context, but the driver functions are not easily accessible behind their macros, so they are hard to compose. E.g., I found it easier to write 'for-all' then to use test/are.
---------
(2) Tests-as-usage-examples are nice, and thinking in invariants makes it easier to write such tests.
Once I had the invariants in place, it was easier to write tests that read more like usage examples. The global-hiearachy-test shows a running usage example. The inline in the example are minimal, and function more as documentation. But, the calls to assert-valid-hiearchy do a ton of validation without impairing the readability of the tests. (Compare this with the fairly opaque tests in the first commit, which do not attempt to function as usage examples.)
---------
(3) There are a bunch of helpers you need to know about in order to write isolated tests. The global derivation hierarchy uses alter-var-root, so an isolated test needs a with-var-root. There are several functions like this in clojure/test-clojure/helpers.clj. At least some of these need to be in a public test API so that people do not reinvent the wheel.
---------
(4) Thinking about invariants flushes out incompleteness in the API. In trying to write the invariants, I discovered that the derivation API provides no way to bootstrap reflection over a hierarchy. I had to write my own hierarchy-tags. (Will consumers of the derivation API need this function as well?)
---------
(5) It is easy for tests to overspecify. This is true in all languages, but it is particularly noticeable in Clojure because Rich is generally careful to avoid committing to too much in docstrings for API functions. Two examples here:
(5a) The original tests asked a lot of questions about the data structure used internally by derive: the {:parents ... :ancestors ... :descendants} map. Since this is not clearly public, the revised tests treat is as private, and ask questions only through the public API functions. Note that this flies in the face on whitebox unit testing, where querying such implementation details is expected. I think that with a good set of invariant tests, and with good contextual reporting of errors, we can often omit such whitebox tests entirely. But it is surely worth a discussion...
(5b) Error handling has often been overspecified in the Clojure test suite. If a docstring leaves some scenarios unspecified, or says something is not allowed but doesn't specify the consequences, then the tests should typically go no farther than the docstring. In other words, test that an error is thrown, but don't test the type of the error. Only test the details of the error message if the error message is worth caring about, e.g. helpful and specific. If you are testing the message, test the one that matters -- usually the root cause, not a wrapper. The clojure.test macros do not help you do the right thing, but helpers.clj has a 'fails-with-cause? hack that is probably closer to right.