Recursive testsuites in tcltest

[konrad....@gmail.com]

Hi,

on the one hand, tcltest::runAllTests invokes all files in
[configure -testdir] matching
[configure -file] and not matching [configure -notfile] -- by default,
files matching *.test in the current directory.

On the other hand, runAllTests recursively searches through
sub-directories of -testdir matching -relateddir and not
matching -asidefromdir, by default all sub-directories,
and their sub-sub-directories, and so on,
executing any file all.tcl found via this search.
It does not, however, directly execute any test files
in these sub-directories.

To actually execute tests in these sub-directories and obtain test
summary information requires all.tcl files in sub-directories
to execute runAllTests. However, as the workingdirectory has not
been changed by the top-level runAllTests, the prologue of
all.tcl in sub-directories must contain a call to
configure -testdir [file dirname [info script]].

Additionally, to prevent all.tcl in sub-directories from recursing
their sub-directories (and sub-sub-directories)
-- since the top-level runAllTests is already doing this --
the prologue must additionally call
configure -asidefromdir *.
(Obviously, configure -testdir ... -asidefromdir ...
can be combined into one call to configure).

The tcltest man page suggests that the top-level file that calls
runAllTests should be called all.tcl.

This means the top-level all.tcl needs to be coded differently
from all.tcl files in sub-directories: it must not call configure
-asidefromdir *, and typically does not
need to call configure -testdir ... .

Honestly, this seems fairly boneheaded.

If runAllTests only iterated across the direct sub-directories of
-testdir, no call to -asidefromdir would be necessary
in all.tcl of sub-directories. This would mean that
directly invoking an all.tcl of a true sub-directory
would recursively invoke tests in that part of the file hierarchy,
an intuitive expectation.

If runAllTests changed configure -testdir automatically to
the sub-directory before calling all.test in that sub-directory,
the boilerplate configure -testdir [file dirname [info script]]
could be avoided.

What is the rationale for the current design?

Konrad Schwarz

[Brad Lanam]

I didn't get quite as far as you.
But this:

# Checking for subdirectories in which to run tests
foreach directory [GetMatchingDirectories [testsDirectory]] {

has no sort, so running my tests on different machines came out in
a different order. Not terribly useful. So I wrote my own top level
tester to traverse the directories and run runAllTests in each (my
hierarchy is flat).

[Gerald W. Lester]

Best practices is that unit test should not care what order they are run in.

--
+------------------------------------------------------------------------+
| Gerald W. Lester |
|"The man who fights for his ideals is the man who is alive." - Cervantes|
+------------------------------------------------------------------------+

[Brad Lanam]

On Saturday, June 25, 2016 at 8:19:43 AM UTC-7, Gerald W. Lester wrote:
> On 6/24/16 7:34 PM, Brad Lanam wrote:
> > I didn't get quite as far as you.
> > But this:
> >
> > # Checking for subdirectories in which to run tests
> > foreach directory [GetMatchingDirectories [testsDirectory]] {
> >
> > has no sort, so running my tests on different machines came out in
> > a different order. Not terribly useful. So I wrote my own top level
> > tester to traverse the directories and run runAllTests in each (my
> > hierarchy is flat).
>
> Best practices is that unit test should not care what order they are run in.

That's not the issue here (though my tests were not independent at the time).
The issue is that I cannot compare output from one machine to the next.

[Don Porter]

On 06/24/2016 01:17 AM, konrad....@gmail.com wrote:
> Honestly, this seems fairly boneheaded.

Yes. Yes it is.

> What is the rationale for the current design?

These tcltest package options are essentially unchanged since they
were introduced in 1999 as part of the effort to transform a
collection of utility scripts into a formal package.

http://core.tcl.tk/tcl/info/f368875e9c462dbc

At that time, Tcl development was done by Scriptics and the checkin
is labeled as coming from a user "jenn". With that data, you might
track down the person and they might recall what the design thinking
was.

It's possible that the design matched some coding practice that was
in use within Scriptics. That would be the kind assumption.

But given just how boneheaded to the point of unusability you
demonstrated it to be, I suspect these features were just added without
any use cases or test cases or actual need, just because it seemed
like the sort of thing needed to flesh out a collection of utility
scripts into a 'finished' package.

After that, our community ethic of really trying not to break compat
just in case someone's counting on things being as they are has done
the rest. As it happens, Tk does make use of this facility to add
in the Ttk set of tests, but only by using the default operations.

Not recommended.

--
| Don Porter Applied and Computational Mathematics Division |
| donald...@nist.gov Information Technology Laboratory |
| http://math.nist.gov/~DPorter/ NIST |
|______________________________________________________________________|