new feature: test results now include fail-condition description

[Anton Vodonosov]

Hello.

Some lisps libraries depend on foreign libraries.
When we run tests on different machines, results
for such libraries may be different depending on
presence of the foreign libraries.

This complicates comparison of test results form
different machines.

For example, suppose we tested Quicklisp 2013-02-17
on MachineA and Quicklisp 2013-03-12 on MachineB.
If MachineB does not have some foreign libraries,
the corresponding lisp tests fail and it appears as if
Quicklisp 2013-03-12 introduces some new bugs
in the lisp libraries; while in reality it is just a
difference in environment external to Lisp/Quicklisp.

To solve this I've for a long time wanted to be able
to distinguish FFI related failures from other failures.

Finally I did a required step in this direction.
When unhandled SERIOUS-CONDITION is signaled during
testing (ql:quickload of an ASDF system or testsuite run),
the condition class its textual message are recorded
in test results.

To hold his information two new attributes of test results are introduced:

:fail-condition-type
NIL or string specifying type of unhandled condition signaled during testing.
Includes package name, for example:
"COMMON-LISP:SIMPLE-ERROR", "CFFI:LOAD-FOREIGN-LIBRARY-ERROR".

:fail-condition-text
NIL or a string, result of princ-to-string for the condition.
"Unable to load foreign library (LIBBRLAPI).
Error opening shared library libbrlapi.dll"

This information will be enough to recognize the FFI errors
and exclude them from comparison when generation reports.

Also, this feature will be useful to collect the complete
list of foreign libraries used by Quicklisp distributed
code.

I have already collected the failure information,
using CCL on my Windows machine. The following
simple Lisp code allows me to perform a test run
and print all distinct fail conditions:

(pushnew "C:/path/to/cl-test-grid/" asdf:*central-registry* :test #'equal)
(ql:quickload :test-grid-agent)
(ql:quickload :test-grid-reporting)

;; Perform test run and store results in a directory:
(defparameter *ccl-1.8-x86-64* (make-instance 'lisp-exe:ccl
:exe-path "C:\\Users\\anton\\unpacked\\ccl\\ccl-1.8-windows\\wx86cl64.exe"))

(tg-agent::complete-test-run2 '(:lib-world "quicklisp 2013-03-12"
:contact-email "avodo...@yandex.ru")
(merge-pathnames "projects/cl-test-grid/work-dir/custom-test-run"
(user-homedir-pathname))
(merge-pathnames "projects/cl-test-grid/work-dir/agent1/quicklisp/"
(user-homedir-pathname))
*ccl-1.8-x86-64*)

;; Convert the results into a plain list of TEST-GRID-REPORTING::RESULT objects
(defparameter *results*
(tg-rep::list-results
(tg-data:add-test-run (tg-data:make-db)
(tg-utils::safe-read-file
(merge-pathnames "projects/cl-test-grid/work-dir/agent1/test-runs/abcl-rerun/test-run-info.lisp"
(user-homedir-pathname))))))

;; Print distinct fail conditions:
(format t "~{~A~%~}"
(sort (remove-duplicates (mapcar (lambda (r)
(format nil "~A ~A"
(tg-rep::fail-condition-type r)
(tg-rep::fail-condition-text r)))
*results*)
:test #'string=)
#'string<))


The output: http://paste.lisp.org/display/136820
As you can see, most of the unhandled failure conditions are FFI related.
Some are CFFI:LOAD-FOREIGN-LIBRARY-ERROR, some are cffi-grovel failures,
there is a "COMMON-LISP:SIMPLE-ERROR Foreign variable "WNOHANG" not found",
there an "ASDF:LOAD-SYSTEM-DEFINITION-ERROR [...] Unable to load any of the alternatives:
("libffi-6.dll" "libffi-5.dll" "libffi.dll")"

Next step in this direction would be to learn to recognize all the FFI caused
conditions and exclude them from comparison.

Below is a rough FFI libraries I quickly extracted manually from the above output.
It doesn't include cff-grovel failures, as I don't know how to extract library
name from the conditions.

libffi
libglfw
libao
libevent_core
libportaudio
libLLVM
libsqlite3
alut.dll
fbclient.dll
ftgl.dll
libglib-2.0-0.dll
freeglut.dll
DevIL.dll
libWand.dll
libblas.dll
libblas.dll
libbrlapi.dll
libcairo-2.dll
libcurses.dll
libev.dll
libfreetype.dll
libgfortran.so.3
libgfortran.so.3
libkyotocabinet.dll
libmpg123.dll
libmysql.dll
libplplotd.dll
libRmath.dll
librrd.dll
libsnappy.dll
libtokyocabinet.dll
libxml2.dll
net.dll
SDL.dll
sybdb.dll
libz.dll (zlib)

It must be noted that the foreign dependency list for Quicklisp
collected that way can not be 100% precise and may need several
passes to improve, because:
1. I do have some foreign libraries installed, so they are not signalled.
2. If a lisp library depends on several foreign libraries, it fails
on the first absent foreign library, and the second foreign library
is not signalled. Similary, if the lisp library fails due to some other
error before foreign library is even tried, the foreign library is not signalled.
Although, assuming that SBCL/linux libraries do not have errors, as Xach
only releases libraries which build successfully on his server where
all the needed foreign libraries are installed, collecting failures
from SBCL/linux is going to produce the most precise list.

Meantime, cl-test-grid is collecting new test results
and fail-conditions for quicklisp 2013-04-20.

Best regards,
- Anton