testLibraryError

[Ralf Hemmecke]

Hmmm, it seems that the unittest.spad is not really something that works
nicely if there are testLibraryError calls to be tested.

I just figured out that if I have foo.input (see attachment), then the
output will be (see below), i.e. the B tests are never executed.

Well, that's somehow understandible, but I would like the collect the
common tests in one macro TEST. Now if one (expeced) error occurs, then
all remaining tests in TEST are skipped.

That's, of course, not what I want.

Is there a way around this problem?
The only way I can imagine at the moment is to create a file
TESTS.input that contains all the tests, but not in a macro and
instead of calling the macro, I simply )read TEST.input.
Better suggestions are appreciated.


====================================================================

)clear complete

All user variables and function definitions have been cleared.
All )browse facility databases have been cleared.
Internally cached functions and constructors have been cleared.
)clear completely is finished.
)set break resume

)expose UnittestCount UnittestAux Unittest

UnittestCount is now explicitly exposed in frame frame1
UnittestAux is now explicitly exposed in frame frame1
Unittest is now explicitly exposed in frame frame1
-------------------------------------------------------------------
testsuiteNoClear "elt-UniversalSegment"

WARNING: string for testsuite should have less than 15 characters!

Type: Void
-------------------------------------------------------------------
A := "A"


(2) "A"
Type:
String
B := "B"


(3) "B"
Type:
String
TESTS ==>
testLibraryError("error(A NAME)")
testLibraryError("error(B NAME)")


Type: Void

testcaseNoClear "List"


Type: Void
NAME := "List"


(6) "List"
Type:
String
TESTS


Error signalled from user code:
AList

Continuing to read the file...


testcaseNoClear "Vector"


Type: Void
NAME := "Vector"


(8) "Vector"
Type:
String
TESTS


Error signalled from user code:
AVector

Continuing to read the file...


statistics()


=============================================================================
General WARNINGS:
* do not use ')clear completely' before having used 'statistics()'
It clears the statistics without warning!
* do not forget to pass the arguments of the testXxxx functions as
Strings!
Otherwise, the test will fail and statistics() will not notice!
* testLibraryError does not prevent FriCAS from aborting the current
block.
Thus, if a block contains other test functions, they will not be
executed
and statistics() will not notice!


=============================================================================
Testsuite: elt-UniversalSegment
failed (total): 0 (2)


=============================================================================
testsuite | testcases: failed (total) | tests: failed (total)
elt-UniversalSegment 0 (2) 0 (2)

=============================================================================
File summary.
unexpected failures: 0
expected failures: 0
unexpected passes: 0
total tests: 2

Type: Void

[Waldek Hebisch]

On Sat, Jun 11, 2022 at 04:30:42PM +0200, Ralf Hemmecke wrote:
> Hmmm, it seems that the unittest.spad is not really something that works
> nicely if there are testLibraryError calls to be tested.
>
> I just figured out that if I have foo.input (see attachment), then the
> output will be (see below), i.e. the B tests are never executed.

Yes. AFAICS the unittest code really does not catch error, instead
depends on toplevel intepreter loop to restart test script at next
toplevel construct. Probaby similar problem appears when there are
fatal tests. We probably should implement this part of unittest
differently, to catch errors. More preciesly, we need function
'catch_all_errors' (or maybe 'ignore_errors') that executes code
(function) catching all errors.

--
Waldek Hebisch

[Waldek Hebisch]

We actually have special macro 'trappedSpadEval' to catch errors
during evaluation. It missed few errors, so I had to modify our
error handling. Now it looks that it catches all errors.

I also modified 'unittest.spad' to use 'trappedSpadEval'.
AFAICS now errors do not propagate past test constructs.
In particular 'testLibraryError' returns to relevant block.

--
Waldek Hebisch