Plans for integrating a unit test runner?

[gutzofter]

Are there any plans to integrate a unit test runner to use with CUSP?

[Timothy Jasko]

(I'm assuming you mean a unit test runner for Lips code being
developed with Cusp, not unit testing for Cusp itself.)

Currently, there are no such plans. This is not to say that we're
against such a thing if someone else were to write up a patch. It's
just not a feature I've felt a need for. Really, I don't even know how
such a thing should look, so I'd be poorly qualified to create it
anyway.

If some enterprising individual wished to add it, on the other hand,
I'd be happy to commit the patch. How would you envision the system
working, particularly the user interface portions? Will this also
involve adding a dependency on a particular lisp unit testing library?

[O'Rorke Paul]

if someone does integrate a unit testing framework with cusp, I vote
for FiveAM.
(none of the common lisp unit testing frameworks out there seemed
super great but FiveAM is the one I chose when I did my evaluation a
little while back.)
I have been using it for a while and it would be great if any unit
testing integration with cusp used it.

One thing that would be handy would be a nice graphical indicator of
whether and which tests passed.
FiveAM prints periods, f's and X's but they sometimes get buried with
logging to console type output.
It would be nice if there were a way to click on something next to the
failed tests to go to the code for the relevant test.
Also would be nice if there were a menu or menus that let you run
suites and perhaps individual tests sans typing.
(currently rerun with history button but sometimes there's a lot of
other stuff in there.)
---Paul O

[gutzofter]

Sorry for the delay, I've been under the weather lately.

So far I envision a user interface of:

Output: Red/Green Indicator
Output: Output message from test framework.

Input: Run WorkSpace/Project/File/Individual Test Selection.

-OR-

Input: Run All Packages/Package/File/Individual Test Selection.

The only way I could see to make this unit test agnostic is if the
output is a service of some sort, connected to SBCL.

The input would have to have some generic API.

I really don't like the idea of the agnostic version. I would rather
stick with a unit tester and grow it to meet the user needs.

The unit tester I'm partial to is lisp-unit.

I will need to research the source for CUSP, to see if I can find some
hooks.

These are just vague ideas, but I think it is workable.

On Jul 20, 8:01 pm, "Timothy Jasko" <tja...@bitfauna.com> wrote:
> (I'm assuming you mean a unit test runner for Lips code being
> developed with Cusp, not unit testing for Cusp itself.)
>
> Currently, there are no such plans. This is not to say that we're
> against such a thing if someone else were to write up a patch. It's
> just not a feature I've felt a need for. Really, I don't even know how
> such a thing should look, so I'd be poorly qualified to create it
> anyway.
>
> If some enterprising individual wished to add it, on the other hand,
> I'd be happy to commit the patch. How would you envision the system
> working, particularly the user interface portions? Will this also
> involve adding a dependency on a particular lisp unit testing library?
>

[Sergey Kolos]

I took a look at both FiveAM and lisp-unit and I would say that lisp-
unit looks much easier to integrate with cusp that FiveAM. I also
don't like test agnostic framework.

I am going to play with lisp-unit some more and will see what is the
best (or easiest) way to integrate.

[gutzofter]

Some ideas I've had is to not integrate a unit testing framework, but
for cusp to provide an API that would allow a unit test framework to
send messages to cusp.

The view in cusp for displaying unit test results, at a minimum should
show red/green, with a glance. Maybe on the status bar. The ideal
would be to have a unit test console with some type of indicator.

Right now I've been evolving lisp-unit. Here is my work in progress
(Download My Projects):

http://uglylispcode.wordpress.com/

At the moment I'm working on trying to get sb-cover to work.

[Sergey Kolos]

I've just committed patch that adds TestsView, which also has Run
Tests button. It doesn't do much yet: it just calls (run-all-tests
pkg), where pkg is chosen from a dialog box. All output from test runs
is dumped to TestView (so it is not pretty). I intended to play with
it to see what kind of output I get before I start making TestView
interface fancier. However I noticed that lisp-unit just prints
results of tests to *standard-output*. To interface it with cusp it
would be much better to have structured output - i.e. list with names
of test run, successes, failures, errors and status (i.e. test was not
found, error description or OK).

I looked at the code and saw that although it is possible to modify it
to get this structured output, it is not trivial and would require
some work. If during your work on evolving lisp-unit you could add
functions run-tests-structured and run-all-tests-structured, it would
help a lot. Also it doesn't report line numbers where tests are
located and I could not find simple way to accomplish that. If we want
interface which allows us jumping to failed tests, then we need line
numbers.

[gutzofter]

Yep gotcha.

This will be the lisp-unit API:

(verbosity) {this will allow sending status to *standard-output* when
run-tests or run-test is executed}
(run-tests package-list) {default = all}
(run-test test-list) {default = none}
(remove-tests) {see run-tests}
(remove-test) {see run-test}
(status-tests) {this will be the structure of the status from run-
tests}
status-test {this will be the structure of the status from run-test}

I would like to time-box this for about one-week. Friday 22 Aug 08.

Does this sound do-able?

[Sergey Kolos]

Sounds good for me. I am already using my unit test interface - cool
feature is that when I remove or rename tests, I don't need to call
remove-test functions - cusp figure out what to remove automatically.

However the interface is not ready for general consumption ...

[Scott Michel]

For what it's worth, I did have to edit lisp-unit.lisp in order to
resolve (i.e., beat down to a dull roar) all of SBCL's style warnings
about functions used before they're defined.

I know that it doesn't really have much impact, but it's the kind of
thing that gives n00bies all kinds of reasons to ask questions and
point fingers.

Might be a good idea to commit the diff back to the original author
(I'm happy to do that if someone can point me in a direction.)


-scooter

[Timothy Jasko]

I don't know that I have a lot to contribute on this one, but I did
want to cheer you guys on. You're breaking entirely new ground in the
Lisp IDE realm.

Regarding jumping to tests, there's probably a good chance that you
can hook into the SLIME function to get the source location where the
test's name is defined. It's never been perfect, but it's usually
close enough.

[markhaniford]

Timothy Jasko wrote:
> I don't know that I have a lot to contribute on this one, but I did
> want to cheer you guys on. You're breaking entirely new ground in the
> Lisp IDE realm.
>
>

Yeah, great work guys. I'm sure there's lots of people out there that
would like to use the power of CL, but aren't happy with Emacs.

[gutzofter]

If you rename a test does the old test name stay as one of the tests?
The way lisp unit works is that it puts all the test code into a hash
when you use the define-test macro. When you run-all-tests, it will
then access the hash and run all the blocks. If you add code to the
define-test it will re-store the new code. If you then change the name
of the test, it will then just add the rename test as a new test.

[gutzofter]

Can you describe what causes this 'dull roar'. I don't think I've seen
this. The original author is Chris Riesbech.

See my 'Download Lisp' on my blog.

Or: http://www.cs.northwestern.edu/academics/courses/325/readings/lisp-unit.html

[Sergey Kolos]

When you rename test in the source Cusp detects that it needs to
recompile that part of the source code, also it detects that the name
of a top level form has changed and that the top level form is define-
test. Using this information it compiles the form, thus creating new
test in hash table, but also it calls remove-test with old name,
removing old test from hash. It worked very well for me.

[Sergey Kolos]

I tried that - no luck. Probably has something to do with the fact
that define-test is a macro, so the macro's first argument is not
recognized by swank as a name worth remembering the line number. It
might be possible to fix, but right now I am busy with other stuff, so
if there is an adventurous soul who can figure out how to deal with
that it would be great.

On Aug 18, 4:01 am, "Timothy Jasko" <tja...@bitfauna.com> wrote:
> I don't know that I have a lot to contribute on this one, but I did
> want to cheer you guys on. You're breaking entirely new ground in the
> Lisp IDE realm.
>
> Regarding jumping to tests, there's probably a good chance that you
> can hook into the SLIME function to get the source location where the
> test's name is defined. It's never been perfect, but it's usually
> close enough.
>

[Scott Michel]

Output from SBCL 0.9.19 before the patch (notice all of the style
warnings, basically about things used before they're defined):

This is SBCL 1.0.19.29, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http://www.sbcl.org/>.

SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses. See the CREDITS and COPYING files in the
distribution for more information.

; in: LAMBDA NIL
; (LISP-UNIT::EXPAND-ASSERT :EQUAL LISP-UNIT::FORM LISP-UNIT::FORM
; LISP-UNIT::EXPECTED LISP-UNIT::EXTRAS :TEST #'EQ)
;
; caught STYLE-WARNING:
; undefined function: EXPAND-ASSERT

;
; caught STYLE-WARNING:
; This function is undefined:
; EXPAND-ASSERT
;
; compilation unit finished
; caught 2 STYLE-WARNING conditions

; in: LAMBDA NIL
; (LISP-UNIT::EXPAND-ASSERT :EQUAL LISP-UNIT::FORM LISP-UNIT::FORM
; LISP-UNIT::EXPECTED LISP-UNIT::EXTRAS :TEST #'EQL)
;
; caught STYLE-WARNING:
; undefined function: EXPAND-ASSERT

;
; caught STYLE-WARNING:
; This function is undefined:
; EXPAND-ASSERT
;
; compilation unit finished
; caught 2 STYLE-WARNING conditions

; in: LAMBDA NIL
; (LISP-UNIT::EXPAND-ASSERT :EQUAL LISP-UNIT::FORM LISP-UNIT::FORM
; LISP-UNIT::EXPECTED LISP-UNIT::EXTRAS :TEST #'EQUAL)
;
; caught STYLE-WARNING:
; undefined function: EXPAND-ASSERT

;
; caught STYLE-WARNING:
; This function is undefined:
; EXPAND-ASSERT
;
; compilation unit finished
; caught 2 STYLE-WARNING conditions

; in: LAMBDA NIL
; (LISP-UNIT::EXPAND-ASSERT :EQUAL LISP-UNIT::FORM LISP-UNIT::FORM
; LISP-UNIT::EXPECTED LISP-UNIT::EXTRAS :TEST #'EQUALP)
;
; caught STYLE-WARNING:
; undefined function: EXPAND-ASSERT

;
; caught STYLE-WARNING:
; This function is undefined:
; EXPAND-ASSERT
;
; compilation unit finished
; caught 2 STYLE-WARNING conditions

; in: LAMBDA NIL
; (LISP-UNIT::EXPAND-ASSERT :ERROR LISP-UNIT::FORM
; (LISP-UNIT::EXPAND-ERROR-FORM LISP-UNIT::FORM) CONDITION LISP-
UNIT::EXTRAS)
;
; caught STYLE-WARNING:
; undefined function: EXPAND-ASSERT

; (LISP-UNIT::EXPAND-ERROR-FORM LISP-UNIT::FORM)
;
; caught STYLE-WARNING:
; undefined function: EXPAND-ERROR-FORM

;
; caught STYLE-WARNING:
; These functions are undefined:
; EXPAND-ASSERT EXPAND-ERROR-FORM
;
; compilation unit finished
; caught 3 STYLE-WARNING conditions

; in: LAMBDA NIL
; (LISP-UNIT::EXPAND-ASSERT :MACRO LISP-UNIT::FORM
; (LISP-UNIT::EXPAND-MACRO-FORM LISP-UNIT::FORM LISP-UNIT::ENV)
; LISP-UNIT::EXPANSION LISP-UNIT::EXTRAS)
;
; caught STYLE-WARNING:
; undefined function: EXPAND-ASSERT

; (LISP-UNIT::EXPAND-MACRO-FORM LISP-UNIT::FORM LISP-UNIT::ENV)
;
; caught STYLE-WARNING:
; undefined function: EXPAND-MACRO-FORM

;
; caught STYLE-WARNING:
; These functions are undefined:
; EXPAND-ASSERT EXPAND-MACRO-FORM
;
; compilation unit finished
; caught 3 STYLE-WARNING conditions

; in: LAMBDA NIL
; (LISP-UNIT::EXPAND-ASSERT :RESULT LISP-UNIT::FORM LISP-
UNIT::FORM NIL
; LISP-UNIT::EXTRAS)
;
; caught STYLE-WARNING:
; undefined function: EXPAND-ASSERT

;
; caught STYLE-WARNING:
; This function is undefined:
; EXPAND-ASSERT
;
; compilation unit finished
; caught 2 STYLE-WARNING conditions

; in: LAMBDA NIL
; (LISP-UNIT::EXPAND-ASSERT :EQUAL LISP-UNIT::FORM LISP-UNIT::FORM
; LISP-UNIT::EXPECTED LISP-UNIT::EXTRAS :TEST LISP-UNIT::TEST)
;
; caught STYLE-WARNING:
; undefined function: EXPAND-ASSERT

;
; caught STYLE-WARNING:
; This function is undefined:
; EXPAND-ASSERT
;
; compilation unit finished
; caught 2 STYLE-WARNING conditions

; in: LAMBDA NIL
; (LISP-UNIT::EXPAND-ASSERT :OUTPUT LISP-UNIT::FORM
; (LISP-UNIT::EXPAND-OUTPUT-FORM LISP-UNIT::FORM) LISP-
UNIT::OUTPUT
; LISP-UNIT::EXTRAS)
;
; caught STYLE-WARNING:
; undefined function: EXPAND-ASSERT

; (LISP-UNIT::EXPAND-OUTPUT-FORM LISP-UNIT::FORM)
;
; caught STYLE-WARNING:
; undefined function: EXPAND-OUTPUT-FORM

;
; caught STYLE-WARNING:
; These functions are undefined:
; EXPAND-ASSERT EXPAND-OUTPUT-FORM
;
; compilation unit finished
; caught 3 STYLE-WARNING conditions

; in: LAMBDA NIL
; (LISP-UNIT::EXPAND-ASSERT :RESULT LISP-UNIT::FORM LISP-
UNIT::FORM T
; LISP-UNIT::EXTRAS)
;
; caught STYLE-WARNING:
; undefined function: EXPAND-ASSERT

;
; caught STYLE-WARNING:
; This function is undefined:
; EXPAND-ASSERT
;
; compilation unit finished
; caught 2 STYLE-WARNING conditions

; in: LAMBDA NIL
; (LISP-UNIT::EXPAND-EXTRAS LISP-UNIT::EXTRAS)
;
; caught STYLE-WARNING:
; undefined function: EXPAND-EXTRAS

;
; caught STYLE-WARNING:
; This function is undefined:
; EXPAND-EXTRAS
;
; compilation unit finished
; caught 2 STYLE-WARNING conditions

; in: LAMBDA NIL
; (LISP-UNIT::GET-TEST-THUNK LISP-UNIT::NAME PACKAGE)
;
; caught STYLE-WARNING:
; undefined function: GET-TEST-THUNK

;
; caught STYLE-WARNING:
; This function is undefined:
; GET-TEST-THUNK
;
; compilation unit finished
; caught 2 STYLE-WARNING conditions

; in: LAMBDA NIL
; (LISP-UNIT:GET-TEST-CODE LISP-UNIT::NAME PACKAGE)
;
; caught STYLE-WARNING:
; undefined function: GET-TEST-CODE

;
; caught STYLE-WARNING:
; This function is undefined:
; GET-TEST-CODE
;
; compilation unit finished
; caught 2 STYLE-WARNING conditions

; in: LAMBDA NIL
; (LISP-UNIT::GET-PACKAGE-TABLE PACKAGE)
;
; caught STYLE-WARNING:
; undefined function: GET-PACKAGE-TABLE

;
; caught STYLE-WARNING:
; This function is undefined:
; GET-PACKAGE-TABLE
;
; compilation unit finished
; caught 2 STYLE-WARNING conditions

; in: LAMBDA NIL
; (LISP-UNIT::GET-PACKAGE-TABLE PACKAGE)
;
; caught STYLE-WARNING:
; undefined function: GET-PACKAGE-TABLE

;
; caught STYLE-WARNING:
; This function is undefined:
; GET-PACKAGE-TABLE
;
; compilation unit finished
; caught 2 STYLE-WARNING conditions

; in: LAMBDA NIL
; (LISP-UNIT::GET-PACKAGE-TABLE PACKAGE)
;
; caught STYLE-WARNING:
; undefined function: GET-PACKAGE-TABLE

;
; caught STYLE-WARNING:
; This function is undefined:
; GET-PACKAGE-TABLE
;
; compilation unit finished
; caught 2 STYLE-WARNING conditions

; in: LAMBDA NIL
; (LISP-UNIT::RECORD-RESULT LISP-UNIT::PASSED TYPE LISP-UNIT::FORM
; LISP-UNIT::EXPECTED LISP-UNIT::ACTUAL LISP-UNIT::EXTRAS)
;
; caught STYLE-WARNING:
; undefined function: RECORD-RESULT

; (LISP-UNIT::TEST-PASSED-P TYPE LISP-UNIT::EXPECTED LISP-
UNIT::ACTUAL
; LISP-UNIT::TEST)
;
; caught STYLE-WARNING:
; undefined function: TEST-PASSED-P

;
; caught STYLE-WARNING:
; These functions are undefined:
; RECORD-RESULT TEST-PASSED-P
;
; compilation unit finished
; caught 3 STYLE-WARNING conditions

; in: LAMBDA NIL
; (LISP-UNIT::GET-FAILURE-MESSAGE TYPE)
;
; caught STYLE-WARNING:
; undefined function: GET-FAILURE-MESSAGE

; (LISP-UNIT::SHOW-FAILURE TYPE (LISP-UNIT::GET-FAILURE-MESSAGE
TYPE)
; LISP-UNIT::NAME LISP-UNIT::FORM LISP-UNIT::EXPECTED LISP-
UNIT::ACTUAL
; LISP-UNIT::EXTRAS)
;
; caught STYLE-WARNING:
; undefined function: SHOW-FAILURE

;
; caught STYLE-WARNING:
; These functions are undefined:
; GET-FAILURE-MESSAGE SHOW-FAILURE
;
; compilation unit finished
; caught 3 STYLE-WARNING conditions

; in: LAMBDA NIL
; (LISP-UNIT:LOGICALLY-EQUAL (CAR LISP-UNIT::ACTUAL) (CAR LISP-
UNIT::EXPECTED))
;
; caught STYLE-WARNING:
; undefined function: LOGICALLY-EQUAL

;
; caught STYLE-WARNING:
; This function is undefined:
; LOGICALLY-EQUAL
;
; compilation unit finished
; caught 2 STYLE-WARNING conditions

; in: LAMBDA NIL
; (LISP-UNIT::RUN-TEST-THUNK (CAR LISP-UNIT::TEST-THUNK)
; (CADR LISP-UNIT::TEST-THUNK))
;
; caught STYLE-WARNING:
; undefined function: RUN-TEST-THUNK

; (LISP-UNIT::SHOW-SUMMARY 'LISP-UNIT::TOTAL LISP-UNIT::TOTAL-TEST-
COUNT
; LISP-UNIT::TOTAL-PASS-COUNT LISP-UNIT::TOTAL-ERROR-COUNT)
;
; caught STYLE-WARNING:
; undefined function: SHOW-SUMMARY

;
; caught STYLE-WARNING:
; These functions are undefined:
; RUN-TEST-THUNK SHOW-SUMMARY
;
; compilation unit finished
; caught 3 STYLE-WARNING conditions
; in: LAMBDA NIL
; (LISP-UNIT::SHOW-SUMMARY LISP-UNIT::*TEST-NAME* LISP-UNIT::*TEST-
COUNT*
; LISP-UNIT::*PASS-COUNT*)
;
; caught STYLE-WARNING:
; undefined function: SHOW-SUMMARY

; (LISP-UNIT::USE-DEBUGGER-P LISP-UNIT::E)
;
; caught STYLE-WARNING:
; undefined function: USE-DEBUGGER-P

;
; caught STYLE-WARNING:
; These functions are undefined:
; SHOW-SUMMARY USE-DEBUGGER-P
;
; compilation unit finished
; caught 3 STYLE-WARNING conditions

After my patch:
----------
This is SBCL 1.0.19.29, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http://www.sbcl.org/>.

SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses. See the CREDITS and COPYING files in the
distribution for more information.
----------

Es ist klar?


-scooter

[Sergey Kolos]

Actually I think it is possible to call load file in sbcl with style
warning suppressed.
That would be more permanent solution than fixing files each time.

[Scott Michel]

Yeah, I've reached an impasse with Chris re: defun-before-use. He
clearly disagrees, although, notably, swank abides by this particular
style "preference". Numerous threads abound where newbies are often
tripped up by this sbcl warning (and for darned good reason too --
they really did leave out a function that should have been defined.)

OTOH, it might just be simpler to add a lisp-unit-loader.lisp file
similar to swank-loader.lisp that manages the compilation and loading.
That way, we get the best of both worlds, compiled code loaded at
startup (mostly) and don't have to deal with patching the source.


-scooter

> ...
>
> read more »

[gutzofter]

Since my platform is windows I'm using SBCL 1.0.13.

This is my output when I load my ASD:


; compiling file "C:\\_Holder\\WorkSpace\\gWeb\\lisp-unit\
\defpackage.lisp" (written 17 AUG 2008 04:51:36 PM):
; compiling (IN-PACKAGE :COMMON-LISP-USER)

; compiling (DEFPACKAGE :LISP-UNIT ...)


; C:\_Holder\WorkSpace\gWeb\lisp-unit\defpackage.fasl written
; compilation finished in 0:00:00
; compiling file "C:\\_Holder\\WorkSpace\\gWeb\\lisp-unit\
\status.lisp" (written 14 AUG 2008 08:07:30 PM):
; compiling (IN-PACKAGE :LISP-UNIT)

; compiling (DEFPARAMETER EMPTY-TOKEN ...)

; compiling (DEFPARAMETER PASS-TOKEN ...)

; compiling (DEFPARAMETER FAIL-TOKEN ...)

; compiling (DEFPARAMETER ERROR-TOKEN ...)

; compiling (DEFMACRO CONC-STRING ...)

; compiling (DEFMACRO APPEND-STATUS ...)

; compiling (DEFUN CREATE-STATUS ...)

; compiling (DEFUN CLEAR ...)

; compiling (DEFUN ADD-ERROR ...)

; compiling (DEFUN ADD-FAIL ...)

; compiling (DEFUN ADD-PASS ...)

; compiling (DEFUN GET-STATUS ...)


; C:\_Holder\WorkSpace\gWeb\lisp-unit\status.fasl written
; compilation finished in 0:00:00
; compiling file "C:\\_Holder\\WorkSpace\\gWeb\\lisp-unit\\lisp-
unit.lisp" (written 18 AUG 2008 06:40:35 PM):
; compiling (IN-PACKAGE #:LISP-UNIT)

; compiling (DEFPARAMETER *TEST-LISTENER* ...)

; compiling (DEFPARAMETER *TESTS* ...)

; compiling (DEFVAR *ASSERT-COUNT* ...)

; compiling (DEFVAR *PASS-COUNT* ...)

; compiling (DEFVAR *ERROR-COUNT* ...)

; compiling (DEFVAR *TEST-NAME* ...)

; compiling (DEFPARAMETER *USE-DEBUGGER* ...)

; compiling (DEFVAR *STATUS* ...)

; compiling (DEFUN INC-ERROR ...)

; compiling (DEFUN INC-PASS ...)

; compiling (DEFUN INC-FAIL ...)

; compiling (DEFMACRO DEFINE-TEST ...)

; compiling (DEFMACRO RUN-ALL-TESTS ...)

; compiling (DEFMACRO RUN-TESTS ...)

; compiling (DEFUN GET-TEST-THUNKS ...)

; compiling (DEFUN GET-TEST-THUNK ...)

; compiling (DEFUN USE-DEBUGGER ...)

; compiling (DEFMACRO WITH-TEST-LISTENER ...)

; compiling (DEFUN GET-TEST-CODE ...)

; compiling (DEFUN GET-TESTS ...)

; compiling (DEFUN REMOVE-TESTS ...)

; compiling (DEFUN REMOVE-ALL-TESTS ...)

; compiling (DEFUN GET-PACKAGE-TABLE ...)

; compiling (DEFUN STORE-TEST-CODE ...)

; compiling (DEFUN GET-TEST-NAME ...)

; compiling (DEFUN RUN-TEST-THUNKS ...)

; compiling (DEFUN RUN-TEST-THUNK ...)

; compiling (DEFUN USE-DEBUGGER-P ...)

; compiling (PROVIDE "lisp-unit")


; C:\_Holder\WorkSpace\gWeb\lisp-unit\lisp-unit.fasl written
; compilation finished in 0:00:01
; compiling file "C:\\_Holder\\WorkSpace\\gWeb\\lisp-unit\
\asserts.lisp" (written 14 AUG 2008 08:07:30 PM):
; compiling (IN-PACKAGE :LISP-UNIT)

; compiling (DEFMACRO ASSERT-EQ ...)

; compiling (DEFMACRO ASSERT-EQL ...)

; compiling (DEFMACRO ASSERT-EQUAL ...)

; compiling (DEFMACRO ASSERT-EQUALP ...)

; compiling (DEFUN GENERIC-ASSERT-EQUAL ...)

; compiling (DEFMACRO ASSERT-EXPANDS ...)

; compiling (DEFMACRO ASSERT-EQUALITY ...)

; compiling (DEFMACRO ASSERT-PRINTS ...)

; compiling (DEFMACRO ASSERT-TRUE ...)

; compiling (DEFMACRO ASSERT-FALSE ...)

; compiling (DEFMACRO ASSERT-ERROR ...)

; compiling (DEFUN EXPAND-ASSERT ...)

; compiling (DEFUN EXPAND-ERROR-FORM ...)

; compiling (DEFUN EXPAND-OUTPUT-FORM ...)

; compiling (DEFUN EXPAND-MACRO-FORM ...)

; compiling (DEFUN EXPAND-EXTRAS ...)

; compiling (DEFUN INTERNAL-ASSERT ...)

; compiling (DEFUN RECORD-RESULT ...)

; compiling (DEFUN DEFAULT-LISTENER ...)

; compiling (DEFUN TEST-PASSED-P ...)


; C:\_Holder\WorkSpace\gWeb\lisp-unit\asserts.fasl written
; compilation finished in 0:00:01
; compiling file "C:\\_Holder\\WorkSpace\\gWeb\\lisp-unit\\equality-
predicates.lisp" (written 14 AUG 2008 08:07:30 PM):
; compiling (IN-PACKAGE #:LISP-UNIT)

; compiling (DEFUN LOGICALLY-EQUAL ...)

; compiling (DEFUN SET-EQUAL ...)


; C:\_Holder\WorkSpace\gWeb\lisp-unit\equality-predicates.fasl written
; compilation finished in 0:00:00
; compiling file "C:\\_Holder\\WorkSpace\\gWeb\\lisp-unit\
\output.lisp" (written 16 AUG 2008 11:01:32 AM):
; compiling (IN-PACKAGE #:LISP-UNIT)

; compiling (DEFUN GET-FAILURE-MESSAGE ...)

; compiling (DEFUN SHOW-FAILURE ...)

; compiling (DEFUN SHOW-STATUS ...)

; compiling (DEFUN SHOW-SUMMARY ...)

; compiling (DEFUN FORMAT-SUMMARY ...)

; compiling (DEFUN FORMAT-DELETED ...)

; compiling (DEFUN UPDATE-STATUS ...)

; compiling (DEFUN IS-ERROR-COUNT? ...)

; compiling (DEFUN FAIL-COUNT ...)

; compiling (DEFUN COLLECT-FORM-VALUES ...)


; C:\_Holder\WorkSpace\gWeb\lisp-unit\output.fasl written
; compilation finished in 0:00:00
; compiling file "C:\\_Holder\\WorkSpace\\gWeb\\lisp-unit\
\fixture.lisp" (written 17 AUG 2008 04:51:02 PM):
; compiling (IN-PACKAGE #:LISP-UNIT)

; compiling (DEFUN EXTRACT-SETUP ...)

; compiling (DEFUN CREATE-DEFAULT-SETUP ...)

; compiling (DEFMACRO CREATE-TEST ...)

; compiling (DEFMACRO CREATE-TESTS ...)

; compiling (DEFMACRO TEST-FIXTURE ...)


; C:\_Holder\WorkSpace\gWeb\lisp-unit\fixture.fasl written
; compilation finished in 0:00:00
; compiling file "C:\\_Holder\\WorkSpace\\gWeb\\lisp-unit\\lisp-unit-
tests.lisp" (written 17 AUG 2008 05:01:08 PM):
; compiling (IN-PACKAGE #:LISP-UNIT)

; compiling (DEFUN EXPECTED-OUTPUT-ALL ...)

; compiling (DEFUN EXPECTED-OUTPUT ...)

; compiling (REMOVE-ALL-TESTS :LISP-UNIT)

; compiling (DEFINE-TEST FORMAT-SUMMARY ...)

; compiling (DEFINE-TEST IS-ERROR-COUNT? ...)

; compiling (DEFINE-TEST FAIL-COUNT ...)

; compiling (DEFINE-TEST TESTING-DEFINE-TEST ...)

; compiling (DEFINE-TEST STATUS ...)

; compiling (DEFINE-TEST STATUS-PASS ...)

; compiling (DEFINE-TEST STATUS-FAIL ...)

; compiling (DEFINE-TEST STATUS-ERROR ...)

; compiling (DEFINE-TEST STATUS-MULTIPLE ...)

; compiling (DEFINE-TEST STATUS-CLEAR ...)

; compiling (DEFINE-TEST CONCATENATE-STRING ...)

; compiling (DEFINE-TEST APPEND-STATUS ...)

; compiling (RUN-ALL-TESTS :LISP-UNIT)


; C:\_Holder\WorkSpace\gWeb\lisp-unit\lisp-unit-tests.fasl written
; compilation finished in 0:00:01
; compiling file "C:\\_Holder\\WorkSpace\\gWeb\\lisp-unit\\lisp-unit-
single-test.lisp" (written 14 AUG 2008 08:07:30 PM):
; compiling (IN-PACKAGE #:LISP-UNIT)

; compiling (REMOVE-ALL-TESTS :LISP-UNIT)

; compiling (DEFINE-TEST SINGLE-TEST ...)

; compiling (RUN-ALL-TESTS :LISP-UNIT)


; C:\_Holder\WorkSpace\gWeb\lisp-unit\lisp-unit-single-test.fasl
written
; compilation finished in 0:00:00
; compiling file "C:\\_Holder\\WorkSpace\\gWeb\\lisp-unit\\assertion-
tests.lisp" (written 14 AUG 2008 08:07:30 PM):
; compiling (IN-PACKAGE :LISP-UNIT)

; compiling (REMOVE-ALL-TESTS :LISP-UNIT)

; compiling (DEFINE-TEST TEST-PASSED-P-EQUAL ...)

; compiling (DEFINE-TEST TEST-PASSED-P-RESULT ...)

; compiling (DEFINE-TEST TEST-PASSED-P-ERROR ...)

; compiling (DEFINE-TEST TEST-PASSED-P-ASSERT-ERROR ...)

; compiling (DEFINE-TEST ERROR-GEN ...)

; compiling (RUN-ALL-TESTS :LISP-UNIT)


; C:\_Holder\WorkSpace\gWeb\lisp-unit\assertion-tests.fasl written
; compilation finished in 0:00:00
; compiling file "C:\\_Holder\\WorkSpace\\gWeb\\lisp-unit\\coverage-
tests.lisp" (written 14 AUG 2008 08:07:30 PM):
; compiling (IN-PACKAGE #:LISP-UNIT)

; compiling (REMOVE-ALL-TESTS :LISP-UNIT)

; compiling (DEFINE-TEST COVERAGE-TESTS-FAIL ...)

; compiling (DEFINE-TEST COVERAGE-TESTS-PASS ...)

; compiling (DEFINE-TEST COVERAGE-TESTS-FORM-ERROR ...)

; compiling (RUN-ALL-TESTS :LISP-UNIT)


; C:\_Holder\WorkSpace\gWeb\lisp-unit\coverage-tests.fasl written
; compilation finished in 0:00:00
; compiling file "C:\\_Holder\\WorkSpace\\gWeb\\lisp-unit\\tests-
status-tests.lisp" (written 18 AUG 2008 10:14:13 AM):
; compiling (IN-PACKAGE :LISP-UNIT)

; compiling (DEFSTRUCT TEST-STATUS ...)

; compiling (DEFUN TEST-STATUS-FAILURES ...)

; compiling (DEFMACRO INC-STATUS ...)

; compiling (DEFUN INCREMENT ...)

; compiling (DEFUN MAKE-STATUS-LIST ...)


; file: C:\_Holder\WorkSpace\gWeb\lisp-unit\tests-status-tests.lisp
; in: DEFUN MAKE-STATUS-LIST
; (LAMBDA (LISP-UNIT::NAME) NIL)
; ==>
; #'(LAMBDA (LISP-UNIT::NAME) NIL)
;
; caught STYLE-WARNING:
; The variable NAME is defined but never used.

; compiling (DEFUN STATUS-COUNT ...)

; compiling (DEFUN ADD-STATUS ...)

; compiling (DEFUN RETRIEVE-STATUS ...)

; compiling (REMOVE-ALL-TESTS :LISP-UNIT)

; compiling (TEST-FIXTURE :TESTS-STATUS ...)

; compiling (TEST-FIXTURE :TEST-STATUS-DATA ...)

; compiling (RUN-ALL-TESTS :LISP-UNIT)

; compiling (DEFINE-TEST TWO-HELLO ...)


; C:\_Holder\WorkSpace\gWeb\lisp-unit\tests-status-tests.fasl written
; compilation finished in 0:00:00
WARNING:
COMPILE-FILE warned while performing
on

.

Test Status: (12)
[............]

;TOTAL TESTS: 12
15 Assertions
15 Passed
0 Failed
0 Errors
Evaluation took:
0.078 seconds of real time
0.078125 seconds of user run time
0.0 seconds of system run time
0 calls to %EVAL
0 page faults and
1,828,752 bytes consed.

Test Status: (1)
[.]

;TOTAL TESTS: 1
1 Assertions
1 Passed
0 Failed
0 Errors
Evaluation took:
0.0 seconds of real time
0.0 seconds of user run time
0.0 seconds of system run time
0 calls to %EVAL
0 page faults and
134,776 bytes consed.

Test Status: (5)
[.....]

;TOTAL TESTS: 5
8 Assertions
8 Passed
0 Failed
0 Errors
Evaluation took:
0.063 seconds of real time
0.046875 seconds of user run time
0.015625 seconds of system run time
0 calls to %EVAL
0 page faults and
1,145,592 bytes consed.
COVERAGE-TESTS-FAIL: NIL failed:
Expected: T Actual: NIL
COVERAGE-TESTS-FORM-ERROR:
The function FORM-ERROR-FUN is undefined.

Test Status: (3)
[FE.]

TOTAL TESTS: 3
2 Assertions
1 Passed
1 Failed
1 Errors
Evaluation took:
0.063 seconds of real time
0.046875 seconds of user run time
0.015625 seconds of system run time
[Run times include 0.047 seconds GC run time.]
0 calls to %EVAL
0 page faults and
400,928 bytes consed.
; in: LAMBDA NIL
; (LET ((LISP-UNIT::DATA (LISP-UNIT::MAKE-TEST-
STATUS :NAME :TEST)))
; (LET ((LISP-UNIT::DATA
; (LISP-UNIT::MAKE-TEST-STATUS :NAME :TEST :ASSERTS 1 :PASS-
ASSERTS 1
; :ERRORS 0)))
; (LISP-UNIT:ASSERT-EQUAL :TEST
; (LISP-UNIT::TEST-STATUS-NAME LISP-
UNIT::DATA))
; (LISP-UNIT:ASSERT-EQUAL 1
; (LISP-UNIT::TEST-STATUS-ASSERTS LISP-
UNIT::DATA))
; (LISP-UNIT:ASSERT-EQUAL 1
; (LISP-UNIT::TEST-STATUS-PASS-ASSERTS
; LISP-UNIT::DATA))
; (LISP-UNIT:ASSERT-EQUAL 0
; (LISP-UNIT::TEST-STATUS-FAILURES
; LISP-UNIT::DATA))
; (LISP-UNIT:ASSERT-EQUAL 0
; (LISP-UNIT::TEST-STATUS-ERRORS LISP-
UNIT::DATA))))
;
; caught STYLE-WARNING:
; The variable DATA is defined but never used.

Test Status: (7)
[.......]

;TOTAL TESTS: 7
21 Assertions
21 Passed
0 Failed
0 Errors
Evaluation took:
0.078 seconds of real time
0.078125 seconds of user run time
0.0 seconds of system run time
0 calls to %EVAL
0 page faults and
2,053,288 bytes consed.
; in: LAMBDA NIL
; (LISP-UNIT::FORM-ERROR-FUN)
;
; caught STYLE-WARNING:
; undefined function: LISP-UNIT::FORM-ERROR-FUN

;
; caught STYLE-WARNING:
; This function is undefined:

; LISP-UNIT::FORM-ERROR-FUN
;
; compilation unit finished
; caught 4 STYLE-WARNING conditions


Something must be very wrong with me ;)

[gutzofter]

Sergey ??????????????

Are you using lisp-unit or your own grown unit-tester?

[Timothy Jasko]

In the spirit of fostering best practices in new Lisp programmers,
perhaps we should give some thought to adding some of this to new lisp
projects. For example, each new project could have "Unit Tests"
subfolder with a test.lisp inside, possibly containing a skeleton
example of how to use the framework.

There are, of course, some major issues to think about if we do that.
Do we add the unit tests to the .asd file? We'd almost have to, but
then we are also required to load in lisp-unit.lisp before we do
anything. We could try being really smart and only load the test
directory and lisp-unit on demand, but that may not be doable either,
since there's not really a concept of the "current project."

Ideas?

[Sergey Kolos]

There is an option in preferences, which allows to turn using lisp-
unit on or off.
If it is on, lisp-unit is loaded at start (similar to asdf-extensions
with manage packages).
I plan to make new project wizard sensitive to that option, so that
when option is on, the option of generating new project with unit
tests will be grayed out (and selected by default).

[Sergey Kolos]

On Aug 19, 2:55 am, gutzofter <gutzof...@yahoo.com> wrote:
> Sergey ??????????????
>
> Are you using lisp-unit or your own grown unit-tester?

Yep. Although right now it is not pretty:
http://www.sergeykolos.com/cusp/intro/files/tests-example.png

[Sergey Kolos]

Early morning - didn't read your question carefully - it is lisp-unit.

On Aug 19, 2:55 am, gutzofter <gutzof...@yahoo.com> wrote:

[gutzofter]

There is some gotchas in making the tests part of the same package as
the code your trying to test. You need to be wary when writing macros
and interning symbols.

I'm always a fan of convention over configuration though. Maybe, by
default it would put the test in the same package.

[O'Rorke Paul]

an advantage of putting the tests in the package being tested is that
you can more easily test everything in the package
including "internal" things that would normally not be visible to the
tests in the usual way since they have not been exported.
---Paul O

[Timothy Jasko]

I would suggest making an asd file for lisp-unit and telling sbcl
where it is. That way, we can just add it to the dependencies of
projects, and not need to worry about loading it up ourselves.

[Sergey Kolos]

Agreed. But sometimes having tests outside the package is also good -
i.e. for release packages. So I plan to put an options in new project
wizard to create both.

[Timothy Jasko]

Sergey, do you have test project with various unit tests that you
could zip and send?

[gutzofter]

I'm almost finished with the changes to lisp-unit. The changes all
have unit tests. I should be finished sometime tonight or tomorrow
night (Pacific time). There are also some coverage tests for lisp-unit
that I've created.

[Sergey Kolos]

Tim,

I just committed change to new lisp project wizard which allows
creation of tests.lisp and generation of example function with its
test. Just run this wizard and after the project is generated hit run
tests button on repl.

[Sergey Kolos]

Any progress on this?

[gutzofter]

I've ran into some scheduling conflicts (back to school nights) during
this week. The pace has slowed due to the asserts being macros and
trying to push the status object into the asserts. Thank god I have
some unit tests, so I should be able to refactor it to use the status
as a global. I'm going to be on this all tonight until finsihed. Where
do you want it? e-mail?

[Sergey Kolos]

There is no rush, since I am busy with another feature at the moment.
But I'd like to finish this unit testing feature while I remember what
is going on in the code (with so many new features and bugs I worked
on recently I won't remember what my plans were about unit-testing in
a week or two :) ).

e-mail would be perfect.

Thanks.

[Sergey Kolos]

Any progress on this? I am ready to get back to finishing unit testing
interface.

Thanks

[Sergey Kolos]

I've just checked the archive you posted - nice job. However, one of
things I liked about original lisp-unit is that it was just as single
file, which is easy to load by calling load function. Loading package
on lisp startup is an issue in Cusp since this operation is not
consistent on all platforms + requires significant runnable magic. Can
you repackage your changes into a single file which can be loaded with
simple (load) call?

Thanks

[gutzofter]

There is an update to lisp-unit in the file section. If you load the
run-all-unit-tests.lisp file in the project it will run, except for
the specific file paths. These will need to be changed. I could not
find any command for finding the current project directory. If you
know it please let me know.

[Sergey Kolos]

It looks like this message was missed, so reposting:

I've just checked the archive you posted - nice job. However, one of
things I liked about original lisp-unit is that it was just as single
file, which is easy to load by calling load function. Loading package
on lisp startup is an issue in Cusp since this operation is not
consistent on all platforms + requires significant runnable magic. Can
you repackage your changes into a single file which can be loaded with
simple (load) call?

Thanks

[gutzofter]

Yes, I can repack into a single file. The only reason I broke it out
into several files, is to split responsibilities.

[Sergey Kolos]

Last thing that is left in my todo list for release 1.0 is updating
test facility with new lisp-unit.
Will you have time to repackage it into single file during this
weekend? If not, release 1.0 will go out without updated testing.

Sergey

[gutzofter]

I was busy doing other work this week-end.

As I write this it is being repackaged with some more refactoring.

[Sergey Kolos]

Unit test is usable right now, although without very nice results
display. We'll add results later - you can try to do this yourself, or
I'll get back to this when I have time (not very soon though). If you
decide to do this yourself and feel like need help navigating Cusp
code - ask questions in this group (it is called Cusp Development for
the reason).