Testworks changes

[Carl Gay]

Testworks doesn't need half of its complexity. We don't need suites and the tedium of adding every test to a suite and every suite to the top-level suite. We don't need test app wrapper libraries. We can run each test library as an executable that writes results to a standard location. It scales better.  It's easier to learn. An external tool can gather results and summarize them if needed. A test library can consist of a single library that has a number of "define test" forms and calls run-test-application() with no arguments.

I will start by making these changes:

• Make "define test" and "define benchmark" register each test/benchmark so that they can be found automatically.
• Make run-test-application run all predefined tests and benchmarks by default, without need to pass a suite.
• Output test results in json format (i.e., --report=json).

I plan to add a test runner to the dylan-tool (maybe later deft) that

• makes a workspace with all libraries in the package catalog (--revision=head|latest)
• builds all libraries named .*-test
• runs all those libraries as executables
• reports all failures

I don't anticipate anyone has any objections to the above, per se, but it depends on the following....

What I'm really wondering about is whether anyone has a strong objection to completely getting rid of suites? I just don't see any real need for them. They require a lot of additional maintenance, they're finicky about order (that's my fault, but it came with benefits), they make the Testworks code and documentation a lot more complex, and they seem mostly to exist to facilitate combining test libraries into a single giant library that can be run as one executable, which doesn't scale well and a hard crash in one test kills the whole thing.

If we decide to keep both models then I guess I'll have to use some hueristics to run the *-test-app library if it exists or the *-test library if it doesn't, and black-list certain *-test-app libraries that are known to combine several library test suites into one.

So does anyone (Peter Housel, I guess I'm looking at you here since you're the other main user) object if my long term goal is to get rid of suites completely?  Have I missed something important?

-Carl

[peter...@gmail.com]

Sounds OK, but is is still possible to select a subset of tests - for example the dylan-library-test-suite-app currently has a lot of tests, some of which are slow. Currently it's convenient to use a suite name to just run (for example) the file system tests without running the thread synchronization ones.

Also - cheeky feature request - would it be possible to have a 're-run failing tests' option? I can do this with CIDER and I like it.

Peter 'not-Housel' Hull

[Carl Gay]

On Monday, November 26, 2018 at 12:44:59 PM UTC-5, peter...@gmail.com wrote:

Sounds OK, but is is still possible to select a subset of tests - for example the dylan-library-test-suite-app currently has a lot of tests, some of which are slow. Currently it's convenient to use a suite name to just run (for example) the file system tests without running the thread synchronization ones.

There are a couple of possibilities. One is to use naming and to add a --match=<regex> command-line option.  I'm planning to do something like that anyway, since I'd like to replace these four options:

--test

--suite

--skip-test

--skip-suite

with these:

--match=<regex>

--skip=<regex>

I think that provides even more flexibility but again with less complexity.  It does require a bit of discipline in use of naming conventions.

You can also use tags. We haven't done a lot with these yet, but the idea was that we would get in the habit of tagging tests that are slow or that we often want to run as a group. At work we have a concept of test "size", small, medium, large, huge. We could do that with tags also.  (The sizes are well-defined in terms of time and a warning can be issued if the test takes more or less time than it's stated size.)

dylan-test --tags=-huge

dylan-test --skip=threads

My goal is to make the common case as simple as possible, and keep the difficult cases (the dylan library?) possible.

 

Also - cheeky feature request - would it be possible to have a 're-run failing tests' option? I can do this with CIDER and I like it.

Yes! https://github.com/dylan-lang/testworks/issues/85