I've used tcltest for exercising procedures and objects.  But I had
never really figured out how to use it to functionally test a whole
application.  If you [source main.tcl] in a test, it will load the
code, build the GUI and start the application.  But shutting down the
application is more difficult.  There is typically a lot of state left
in the interpreter, and if you destroy the Tk application (by wm
deleting "."), then it gets really hard to start over again.

I considered a GUI-driving test program like android or tkreplay, but
I don't necessarily want to drive the GUI widgets.  I want to exercise
the application's API, which is called by the widgets' -command
options.  Instead of "black box" testing the GUI, I want to "white
box" test the application logic, peeking and poking at the internal
state of the application.

Suddenly, it is obvious to me that the answer is to use slave
interpreters.  Each test creates a slave interpreter and uses it to
source the mainline code.  The slave loads Tk and other extensions,
builds the GUI, and initializes the application.  The test code can
then use [$slave eval] to command and probe the state of the
application.  Cleanup is done by simply deleting the slave
interpreter.

For example, for the TNT application, I can create a standard -setup
and -cleanup script like this:

  set mainProgram ../tnt.tcl

  set startTNT {
      set app [interp create]
      $app alias exit return "application called 'exit'"
      $app eval source $mainProgram
      update
  }

  set stopTNT {
      catch {interp delete $app}
  }

Now I can write tests which start the application, twiddle its
innards, and clean it up, no matter what state it is left in.

  test tnt-1.1 {run application} -body {
      # if the application started, it set this global variable
      $app eval info exists TNT_VERSION
  } -setup $startTNT -cleanup $stopTNT -result {1}

  test tnt-1.2 {normal exit, as called from File->Exit menu} -body {
      $app eval _exit
  } -setup $startTNT -cleanup $stopTNT -result {application called
'exit'}

There are some "gotchas."  My setup script should alias [exit] to
prevent the application from aborting the tests.  It must set argv0,
argv, and argc.  And I also need to be aware that the environment
array (::env) is shared between all interpreters.  But are there other
things that are going to sneak up and bite me?  Or is this a good
approach for application testing?

I think the slave interpreter approach is a good one.  It stops your
test from polluting your test control environment and as you say allows
an easy gauranteed cleanup for each test.

We've been using an in house developed test exec for several years that
runs our test scripts in slave interpreters.

I tried to cure this by loading tcltest also into the slave interp, but
to no avail. Appended you find a small script that exercises the failure.

Greetings,

Erik Leunissen
==============

package require tcltest 2.2
namespace import -force tcltest::*
tcltest::configure -verbose {body pass error}

set setupScript {
        interp create slave
        # This did not help:
        # slave eval [list package require tcltest 2.2]

test errorOutput-1.2 {writes to stderr from slave interp} -body {
        slave eval [list puts stderr "Do you read stderr (slave)?"]

Greetings,

Erik Leunissen
==============

package require tcltest 2.2
namespace import -force tcltest::*
tcltest::configure -verbose {body pass error}

set setupScript {
        interp create slave
        # But this does help:
        slave alias puts puts; # <------- FIX

test errorOutput-1.2 {writes to stderr from slave interp} -body {
        slave eval [list puts stderr "Do you read stderr (slave)?"]

                      PITA

The following example test "writeToFile" will cause an error because the
call to puts in [writeToFile] requires the channel $fd to exist. While
that is the case in a normal execution environment, it is not in a slave
interpreter unless you make it explicitly available using [interp
transfer]. Since the [puts] immediately follows the opening of the file,
there is no way to do that in a timely fashion when you're writing the
test script. You could of course insert it into the proc to take into
account that in the future it may be executed in a test environment that
uses slave interpreters, but ... good heavens, NO.

Are there any people who see a workable solution here?

Or should we conclude that the best thing is not to use slave
interpreters when the code that you want to test accesses files other
than the standard ones?

set proc writeToFile {txt} {
     set fd [open ./testFile w]
     puts $fd $txt
     close $fd

Thanks in advance for sharing your experience,

In general, channels are accessible in slave interps. But apparently it
is not in my case.

But I don't see that as a serious problem for many file I/O tests.
The example you provide (proc writeToFile) could be handled with
regular tcltests, without the use of slave interps.  My proposed use
of slave interps was to facilitate full Tk applications.  Stdin/stdout
could be a significant issue.  But most file I/O shouldn't be a
problem, because all the code will be run by the slave.  (It won't
have to straddle interpreters.)

Did you see the fix that I suggested in an earlier posting in this
thread?  (look for "<----- FIX")

That fix does what you propose here.

In the meantime, I resolved the issues with non-standard files too and
fixed a problem that had been created when defining the alias for puts.

In case you're interested: I've attached the complete setup for using
slave interps with tcltest to this message. It contains a self test also.

Greetings,

Did you see the fix that I suggested in an earlier posting in this
thread?  (look for "<----- FIX")

That fix does what you propose here.

In the meantime, I resolved the issues with non-standard files too and
fixed a problem that had been created when defining the alias for puts.

In case you're interested: I've attached the complete setup for using
slave interps with tcltest to this message. It contains a self test also.

Greetings,

Did you see the code that I suggested in an earlier posting in this
thread?  (look for "slave alias puts puts")

That line does what you propose here.

In the meantime, I resolved the issues with non-standard files too and
fixed a problem that had been created when defining the alias for puts.

In case you're interested: I've attached the complete setup for using
slave interps with tcltest to this message. It contains a self test also.

Greetings,

Erik Leunissen

But your code that opens and shares between master and slave should
also work.  (And its clever, too.)