tcltest exit code

[Olе Streicher]

Hi,

I am trying to write some unit tests to be run when a tcl module (with
some C code) is built. I use a simple "all.tcl":

---------------8<------------------
package require tcltest
tcltest::verbose {pass body error}
tcltest::runAllTests
---------------8<------------------

But, how do I get the test result as the exit code? The tclsh
interpreter always returns exit code 0. Foofle found some "if { [info
exists tcltest::testFileFailures] } ..." but this seems not to work.

Best regards

Ole

[Don Porter]

On 08/22/2012 09:53 AM, Olе Streicher wrote:
> ---------------8<------------------
> package require tcltest
> tcltest::verbose {pass body error}
> tcltest::runAllTests
> ---------------8<------------------
>
> But, how do I get the test result as the exit code?

The tcltest package doesn't support any programmatic access to its test
stats; it's written to do its own reporting.

You can cheat and peek into tcltest's private operations, and use this:

exit [expr {$::tcltest::numTests(Failed) > 0}]

The risk is that someday, those private parts of the tcltest package
could change and you'd need to update your test suite to deal with it.
Empirically, that's not been a problem on the scale of years.

--
| Don Porter Applied and Computational Mathematics Division |
| donald...@nist.gov Information Technology Laboratory |
| http://math.nist.gov/~DPorter/ NIST |
|______________________________________________________________________|

[Olе Streicher]

Hi,

Don Porter <d...@nist.gov> writes:
> On 08/22/2012 09:53 AM, Olе Streicher wrote:
>> ---------------8<------------------
>> package require tcltest
>> tcltest::verbose {pass body error}
>> tcltest::runAllTests
>> ---------------8<------------------
>>
>> But, how do I get the test result as the exit code?
>
> The tcltest package doesn't support any programmatic access to its test
> stats; it's written to do its own reporting.

Is there a better way to do unit tests for module compilation?

> You can cheat and peek into tcltest's private operations, and use this:
> exit [expr {$::tcltest::numTests(Failed) > 0}]

Unfortunately, this does not work. I always get zero here.

> The risk is that someday, those private parts of the tcltest package
> could change and you'd need to update your test suite to deal with it.
> Empirically, that's not been a problem on the scale of years.

What is the current state of tcl development? It looks if almost nothing
happens anymore there (at least, when compared to f.e. python). This
problem is there since at least 7 years, without a clean solution yet.

Best regards

Ole

[Don Porter]

>> You can cheat and peek into tcltest's private operations, and use this:
>> exit [expr {$::tcltest::numTests(Failed)> 0}]

> Unfortunately, this does not work. I always get zero here.

Aha! You're right. [tcltest::cleanupTests 1] resets the values before
you get to them. Here's the workaround version of all.tcl:

package require tcltest
tcltest::verbose {pass body error}

proc tcltest::cleanupTestsHook {} {
variable numTests
set ::code [expr {$numTests(Failed) > 0}]
}
tcltest::runAllTests
exit $code

[Olе Streicher]

Hi Don,

Don Porter <d...@nist.gov> writes:
>>> You can cheat and peek into tcltest's private operations, and use this:
>>> exit [expr {$::tcltest::numTests(Failed)> 0}]
>> Unfortunately, this does not work. I always get zero here.

> Aha! You're right. [tcltest::cleanupTests 1] resets the values before
> you get to them. Here's the workaround version of all.tcl:
>
> package require tcltest
> tcltest::verbose {pass body error}
> proc tcltest::cleanupTestsHook {} {
> variable numTests
> set ::code [expr {$numTests(Failed) > 0}]
> }
> tcltest::runAllTests
> exit $code

Thank you. However, it still does not work completely: if the test
crashes because of a segmentation fault in the module, I still get a
success.

Best regards

Ole

[Don Porter]

> Thank you. However, it still does not work completely: if the test
> crashes because of a segmentation fault in the module, I still get a
> success.

package require tcltest
tcltest::verbose {pass body error}
proc tcltest::cleanupTestsHook {} {
variable numTests

upvar 2 testFileFailures crashed

set ::code [expr {$numTests(Failed) > 0}]

if {[info exists crashed]} {
set ::code [expr {$::code || [llength $crashed]}]

[Olе Streicher]

Don Porter <d...@nist.gov> writes:
> package require tcltest
> tcltest::verbose {pass body error}
> proc tcltest::cleanupTestsHook {} {
> variable numTests
> upvar 2 testFileFailures crashed
> set ::code [expr {$numTests(Failed) > 0}]
> if {[info exists crashed]} {
> set ::code [expr {$::code || [llength $crashed]}]
> }
> }
> tcltest::runAllTests
> exit $code

Thank you very much! Works well for me.

Best regards

Ole

[Andreas Kupries]

ole-use...@gmx.net (Olе Streicher) writes:

> What is the current state of tcl development? It looks if almost nothing
> happens anymore there (at least, when compared to f.e. python).

Where are you looking ?

While our bug trackers are still at

http://tcl.sourceforge.net/

the source code repositories have moved to

http://core.tcl.tk/

--
So long,
Andreas Kupries <akup...@shaw.ca>
<http://www.purl.org/NET/akupries/>
Developer @ <http://www.activestate.com/>

http://www.tcl.tk/community/tcl2012/ - Tcl'2012, Nov 12-16, Chicago, IL, USA.
-------------------------------------------------------------------------------

[Olе Streicher]

Hi Andreas,

Andreas Kupries <akup...@shaw.ca> writes:
> ole-use...@gmx.net (Olе Streicher) writes:
>> What is the current state of tcl development? It looks if almost nothing
>> happens anymore there (at least, when compared to f.e. python).

> Where are you looking ?

I don't mean Tcl itself but the whole ecosystem. In the last months, I
tried to push some older (astrophysical) programs for the use of the
current Tcl versions and found a good nightmare; there are many
libraries that are stuck at older versions. Most prominent is BLT
(current version 2.4z which is ~10 years old).

Another example is the tcltest (Tcl core) package: you find the exit
code question in the usenet already 6 years ago, but nothing seems to
happened since then: just slowly changing workarounds. This is not what
I would call a living development. Therefore my question about its
current state.

Best regards

Ole

[Andreas Kupries]

ole-use...@gmx.net (Olе Streicher) writes:

> Hi Andreas,
>
> Andreas Kupries <akup...@shaw.ca> writes:
>> ole-use...@gmx.net (Olе Streicher) writes:
>>> What is the current state of tcl development? It looks if almost nothing
>>> happens anymore there (at least, when compared to f.e. python).
>
>> Where are you looking ?

> I don't mean Tcl itself but the whole ecosystem. In the last months,
> I tried to push some older (astrophysical) programs for the use of
> the current Tcl versions and found a good nightmare; there are many
> libraries that are stuck at older versions. Most prominent is BLT
> (current version 2.4z which is ~10 years old).

A fork for Tcl 8.5 exists at

http://pdqi.com/w/pw/pdqi/Wize/Blt/

Mentioned at http://wiki.tcl.tk/199
Note also http://wiki.tcl.tk/24533

> Another example is the tcltest (Tcl core) package: you find the exit
> code question in the usenet already 6 years ago, but nothing seems to
> happened since then: just slowly changing workarounds.

Has it only been discussed on usenet ?

Has anybody ever found the issue important enough to actually file a
bug report on it ? Plus maybe the patches they have?

(On my quick search right now it seems not)

> This is not what I would call a living development. Therefore my
> question about its current state.

Most people in the Tcl area, including maintainers, have a day jobs
taking much of their time. Then comes family.

Even I, without much family time, find myself not reading
comp.lang.tcl as much as I did in the past. I am mostly on the
tcl-core list where core development is talked about. Tracking the bug
tracker more than here.

[Don Porter]

On 08/26/2012 09:59 PM, Andreas Kupries wrote:
>> Another example is the tcltest (Tcl core) package: you find the exit
>> code question in the usenet already 6 years ago, but nothing seems to
>> happened since then: just slowly changing workarounds.
>
> Has it only been discussed on usenet ?
>
> Has anybody ever found the issue important enough to actually file a
> bug report on it ? Plus maybe the patches they have?
>
> (On my quick search right now it seems not)

<semi-disorganized brain dump follows>

The SF Tracker has 16 open feature requests for the tcltest package.
Most relevant to this discussion are

http://sf.net/tracker/?func=detail&aid=623787&group_id=10894&atid=360894
http://sf.net/tracker/?func=detail&aid=2794560&group_id=10894&atid=360894

I would be quite pleased for someone interested to open a fossil branch
to work on some/all of these, or other improvements to tcltest.

The other constraint to keep in mind for tcltest enhancements is the
existing body of test suites that you'd want to avoid breaking as much
as reasonably possible. To avoid that constraint (and to shuffle off
the burden of "tcltest 1" compatibility still woven through the existing
package), someone might also consider going to work directly on either
tcltest 3, or a successor package of testing facilities.

For quite a while now, tcltest edits are mostly small bug fixes or
small accommodations to the changing facilities of Tcl. Substantial
new features haven't been done. This is due to a number of things.

A large factor is that the maintainer, me, has formed an impression that
tcltest is largely "done". As in, it's good enough, and the proposed
improvements are all tinkerings around the edge that might be nice, but
aren't a big deal. So working on them never rises above all the other
pending Tcl improvements which seem more important at the time. Only
occasionally does someone dissatisfied interact with me enough like this
to bring a deficiency into focus.

A contributing factor is that tcltest started existence as a set of
utility scripts forming a testing application, rather than a testing
package, and it's never really completed the transformation. It still
contains, and even prefers, facilities that reach out and read $::argv
for itself and write directly to stdout and stderr for its own
reporting, rather than specify interfaces that would allow a client
application to get at these things and use them as it chooses.

I have the sense that the world of testing has grown up a firmer set of
conventions since the days when tcltest built up its feature set. Even
a question posed earlier "Is there a better way to do unit tests for
module compilation?" suggests that there's an expected set of vocabulary
and functionality in the testing world that was not as well established
in the days of tcltest's origin. A renovation or reboot schooled in
modern expectations might be very useful in capturing and serving this
set of expectations. I am not immersed in the testing world nearly
enough to be the best person to attempt that.