Writing library tests for cl-test-grid compatibility

[Vladimir Sedach]

Hello,

What is the recommended way to write test suites for inclusion with cl-
test-grid? Looking at the code, the test frameworks supported are Eos,
FiveAM, lift, rt, and Stefil. Is there a way to define and/or name
test suites to make it easier for cl-test-grid?

Also, is there a way to automate this? asdf:test-op is supposed to be
such a mechanism, but I don't see anyone using it.

Vladimir

[Robert Goldman]

FWIW, we have an ASDF extension that lets you specify FiveAM tests
conveniently for the TEST-OP. Possibly I should push it to the ASDF
contrib/ directory....

[Anton Vodonosov]

21.03.2012, 01:43, "Vladimir Sedach" <vse...@gmail.com>:

> Hello,
>
> What is the recommended way to write test suites for inclusion with cl-
> test-grid? Looking at the code, the test frameworks supported are Eos,
> FiveAM, lift, rt, and Stefil. Is there a way to define and/or name
> test suites to make it easier for cl-test-grid?

Libraries having test suites written in these frameworks
is the most convenient to add to test grid - there are helper
functions for these, so adding such a library is tree lines of code.

Some libraries or test frameworks are less convenient.

For example ptester framework doesn't encourage using of
named test cases, so I only distinguish OK/FAIL for such
test suites, and do not collect list of failed testcases (cl-base64
and puri use ptester; imho it's enough to have just OK/FAIL
for them, because they usually pass the tests, and the regressions
will be detected if OK will change to FAIL).

Also cl-test-more framework does not return any lisp value
after test, but it produces TAP (the Test Anything Protocol)
output - it's a test format. So it's necessary to parse test output.
It's relatively easy to parse, but I limited the handling to
determine if all the tests passed, or not (i.e. just OK/FAIL
for the whole library, without detalization to test cases).

Some libraries contain copy of the test framework sources
(named-readtables and ironclad include rt.lisp, out of desire
to simplify installation). Not very convenient for me.

Some libraries have their own test frameworks (flexi-streams,
cl-ppcre), or just use cl:assert. Again, we collect just OK/FAIL
for the whole library; and the failure details may be found
in logs.

But these inconveniences are not real blockers, it's all solvable.

> Also, is there a way to automate this? asdf:test-op is supposed to be
> such a mechanism, but I don't see anyone using it.

Almost all the libraries having test suites provide asdf:test-op.
But asdf:test-op does not return any value indicating the result
of testing. It's meant to be run by human who read results from
terminal I think.

But it's very convenient that the libraries provide asdf:test-op, because
that's where I look for how to run the library test suite - I usually
open .asdf file and see how asdf:test-op is implemented.

If speak of introduction of a similar mechanism, e.g. some extended
form of asdf:test-op, I am not ready to suggest the interface
for such thing.

Initially, I thought it's enough for test suite to just return a simple OK/FAIL status.
But then it become apparent, that list of failed tests is necessary too
(that way we can see if new tests start failing in new library version,
and so prevent further regression of the library code, even if the library had
FAIL status before)

Another aspect, is that some libraries have some requirements to the
environment, which are out of scope of CL standard and may absent.

For example presence of some native library (like OpenSSL is needed for
cl+ssl); or C compiler and some C include files available (osicat and
static-vectors use cffi-grovel, which uses C compiler). And similrar
requirements. It will be good to accumulate some convenience
code for such cases and be able to return the :no-resource status
for the test suite of such libraries.

To summarize, currently cl-test-grid model of test suite is that
test suite is a collection of named test cases, each may succeed
or fail.

If the test suite uses one of the test frameworks
already met by cl-test-grid, it's the most convenient. If another
test framework is used, it's usually not difficult to support.

Best regards,
- Anton

[Anton Vodonosov]

> it produces TAP (the Test Anything Protocol)
> output - it's a test format.

I mean teXt format.