rackunit and logging

[Shriram Krishnamurthi]

I'm trying to understand the design of the logging portion of rackunit:

https://docs.racket-lang.org/rackunit/Testing_Utilities.html#%28part._.Logging_.Test_.Results%29

1. The "log" seems to only be a count, not the actual rackunit output (as the term "log" would suggest). Since I want to show a summary of tests — including output — on a different medium, after the test suite has run, it looks like I need to essentially create my own logging support? (Perhaps the "check-info stack" is useful here, but I don't think so.)
2. Why do the check-… procedures not return any value? It would seem natural for them to return, say, false in case of passing and a failure information structure in case of failing (or a structure in both cases). But they seem to only return void, so I'm not entirely sure how else to extract the information for the log without rewriting the check's.
3. As an aside, I'm not entirely sure what `test-log!` is there for. Presumably it's to record in the log "tests" run by operations that are not part of rackunit? I'm curious how people have used it.

TL;DR: If I want to record what happened on all the checks for post-execution processing, do I need to (a) create my own log and, to do so, (b) rewrite all the checking predicates to provide the information that the detailed log needs?

Thanks,

Shriram

[Alex Harsanyi]

You can use foldts-test-suite (part of the rackunit package) to write your own test runner which collects and reports on the tests according to your needs.  This will only work for the rackunit tests which are organized in test suites and test cases, not with simple `check-*` written inside test modules.

I used it myself to write a test runner which runs rackunit tests and outputs the results as JUnit XML test files.  This is a widely supported test result format that can be imported into test reporting applications.

Alex.

[David Storrs]

Hi Shriram,

I have a module, handy/test-more (https://pkgs.racket-lang.org/package/handy), that I think does everything you want; the downside is that the documentation is thorough but it's in the form of essay-style comment sections instead of Scribble.  Breaking that out into actual Scribble is on my todo list but hasn't happened yet.  You can read it on GitHub here https://github.com/dstorrs/racket-dstorrs-libs/blob/master/test-more.rkt 

If you think you would find this useful, I'll make a point to do the Scribble this weekend.

Here's the synopsis statement:

;;======================================================================                      
;;    The racket testing module has a few things I wish it did differently:                    
;;                                                                                            
;; 1) The test function names are verbose and redundant.  check-this, check-that, etc          
;;                                                                                            
;; 2) The test functions display nothing on success.  There's no way                          
;; to tell the difference between "no tests ran" and "all tests                                
;; succeeded"                                                                                  
;;                                                                                            
;; 3) The tests return nothing.  You can't do conditional tests like:                          
;;        (unless (is os 'windows) (ok test-that-won't-pass-on-windows))                      

Some quick examples; there's quite a lot more that can be done with it if this looks useful.   The output is shown after the code.

#lang racket

(require handy/test-more)

(expect-n-tests 4)
(test-suite
 "examples"

 (struct person (name age) #:transparent)
 (define people (hash 'alice (person "alice" 19)  'bob (person "bob" 17)))

 ; basic equality testing
 (is       (* 1 2) 2 "(* 1 2) = 2")
 (is-false (hash-has-key? people 'charlie) "as expected, don't know charlie")
 (isnt     (hash-has-key? people 'charlie) "same as above")

 ; Tests return values so you can chain them or nest them
 (lives
  (thunk
   (and (ok (hash-has-key? people 'alice) "we have an entry for alice")
        (let ([alice (hash-ref people 'alice)])
          (is (person-age alice) 19 "alice is 19")
          (like (person-name alice) #px"^alice$" "alice's name is string, lowercase, trimmed"))))
  "alice-related tests did not raise exception")

 ; exceptions
 (define thnk (thunk (+ 1 "foo")))
 (dies   thnk "it died, that's all I care about")
 (throws thnk exn:fail:contract? "match against an arbitrary predicate")
 (throws thnk #px"expected: number.+\"foo\"" "match against a regex")
 (is (throws thnk (lambda (e) 'ok) "throws can pass exn to arbitrary one-arg proc for testing")
     'ok
     #:op (lambda (got expected)
            (if (exn? got) 'ok 'nope))
     "tested 'throws' with hand-rolled predicate, then chained into `is` using hand-rolled equality testing (the equality test is illustrative but not sensible)")
 )

When you run the above code, the following will be sent to STDOUT.  Note that it tracks the number of tests it runs, numbers them, and warns you that you didn't run the expected number. 

######## (START test-suite:  examples)
ok 1 - (* 1 2) = 2
ok 2 - as expected, don't know charlie
ok 3
ok 4 - we have an entry for alice
ok 5 - alice is 19
ok 6 - alice's name is string, lowercase, trimmed
ok 7 - alice-related tests did not raise exception
ok 8 - it died, that's all I care about
ok 9 - match against an arbitrary predicate
ok 10 - match against a regex
ok 11 - throws can pass exn to arbitrary one-arg proc for testing
ok 12 - tested 'throws' with hand-rolled predicate, then chained into `is` using hand-rolled e$
ok 13 - test-suite completed without throwing uncaught exception

Total tests passed so far: 13
Total tests failed so far: 0
######## (END test-suite:  examples)

        !!ERROR!!:  Expected 4 tests, actually saw 13

--
You received this message because you are subscribed to the Google Groups "Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to racket-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/e7cafb9d-793d-4f3b-9eb4-308e43df73f2%40googlegroups.com.

[Alexis King]

On May 22, 2020, at 18:47, Shriram Krishnamurthi <shr...@gmail.com> wrote:

As an aside, I'm not entirely sure what `test-log!` is there for. Presumably it's to record in the log "tests" run by operations that are not part of rackunit? I'm curious how people have used it.

Other people have answered other parts of your question, but I don’t think anyone has answered this point. The answer is that test-log! isn’t really part of RackUnit at all—it comes from the module rackunit/log, but note that it comes from the package testing-util-lib, not rackunit-lib. (I can’t say for certain, but my guess is that the module is named rackunit/log for historical reasons more than anything else.)

test-log! is really mostly an API for cooperating with `raco test`. When you run `raco test`, you’ll find that it reports how many tests were run and how many of them passed, even if you test multiple modules at once. Somehow, test frameworks need to communicate this information to `raco test`, and test-log! is the mechanism through which they do that.

Aside from that, `raco test` mostly just instantiates modules in the usual way. All of the other success or failure reporting is just ordinary side-effectful printing performed by test frameworks themselves. All of RackUnit’s “check info stack” machinery is specific to RackUnit; `raco test` doesn’t know or care about that at all. RackUnit calls test-log! internally during execution of each check, so if you use define-check, you don’t have to call test-log! yourself (and indeed, you shouldn’t, or else your tests will be counted twice).

TL;DR: If I want to record what happened on all the checks for post-execution processing, do I need to (a) create my own log and, to do so, (b) rewrite all the checking predicates to provide the information that the detailed log needs?

As the above explanation implies, if you don’t want to use RackUnit, you can do whatever you want. You can print your own (arbitrary) messages upon failure, and if you want to integrate with `raco test`, you should just make sure to call test-log! once for each executed test case.

If you do want to use RackUnit, then yes, you have to cooperate with all of RackUnit’s machinery for implementing checks. Personally, I find RackUnit’s design frustrating here. As you have discovered, checks aren’t very compositional, so it’s not easy to implement a new check in terms of an existing one (while still providing good error messages on failure).

If what you’re doing is difficult to express in RackUnit, consider just ditching it and doing your own thing. Test suites in Racket are just simple, side-effectful programs, and RackUnit doesn’t do that much all things considered.

Alexis

[Shriram Krishnamurthi]

Thank you all!

Alexis, thanks for the explanation.

Alex, thanks for that information. I'm going to go investigate that next.

Dave, the documentation style is fine, it's sometimes easier to read the doc right next to the implementation. (-:

However, I'm not quite sure how even your example works. Maybe someone can check my logic? For instance, you say you want to write tests like

    (unless (is os 'windows) (ok test-that-won't-pass-on-windows))

However, `is` seems to return the same value no matter whether the test passed or failed: it returns the first argument, *irrespective* of the outcome of the test. So in the above test, the returned value is going to be that of `os`, which is presumably some non-false value. That means the guarded test will *never* be run, on any OS.

[Separately, I'm not sure why one would use a testing utility in that conditional, rather than just a standard conditional, but that's a different matter.]

In general, this seems to be a property of your underlying function, `test-more-check`: it returns either the return value sent in through #:return or the value in the checked position (#:got). But in either case, this is independent of the success of the test. The only difference is in the *message*, which is printed as output. I suppose I could parameterize where it's printed and capture it — but then I have to parse all the information back out. I'm just not seeing how to compositionally use your testing primitives?

As an aside, when trying to install the package in a Docker container running Ubuntu 18.04 with Racket 7.7 installed, I got this error:

raco setup: docs failure: query-exec: unable to open the database file
  error code: 14
  SQL: "ATTACH $1 AS other"
  database: #<path:/root/.racket/7.7/doc/docindex.sqlite>
  mode: 'read-only
  file permissions: (write read)

which I didn't get on macOS Catalina. The package certainly has a … lot of stuff! Even links to EDGAR filings. (-:

Thanks,
Shriram

[Alexis King]

On May 23, 2020, at 08:53, Shriram Krishnamurthi <s...@cs.brown.edu> wrote:

Alex, thanks for that information. I'm going to go investigate that next.

Related to that, I just remembered the existence of rackunit/text-ui and rackunit/gui, which implement two different reporters for RackUnit test cases/suites. Looking at their source might be informative, since neither is particularly complicated. The text reporter uses fold-test-results, and the GUI reporter uses foldts-test-suite.

[Shriram Krishnamurthi]

Thank you, Alex, this seems to have just the right set of information. I'm curious why you use `foldts-test-suite` instead of  `fold-test-results`, whose documentation says "Hence it should be used in preference to foldts-test-suite." (And while David's points about the verbosity are well taken, my TAs are likely to be used to the rackunit primitives, though I think I'll consider borrowing some of his aliases for shorter testing primitive names.)

In the success case, the result seems to be `void` (rather than, say, a value corresponding to the type of test). Are there cases where it's not void?

In the failure case, is there a procedure to convert the `exn:test:check` value into a human-readable string? I'm guessing you have some serialization process for the XML test report.

Sadly, errors during test execution aren't sandboxed, so if there's an error in a test, the whole suite breaks down. I think this means I'm going to have to write custom testing procedures anyway.

Shriram

[Shriram Krishnamurthi]

For those reading this later: there's a bunch of useful information in Alex Harsanyi's blog post and corresponding code:

https://alex-hhh.github.io/2019/11/custom-rackunit-test-runner.html

https://github.com/alex-hhh/ActivityLog2/blob/master/test/custom-test-runner.rkt

Shriram

[Shriram Krishnamurthi]

Sorry to be thinking out loud here…

I thought the reason Alex might be using `foldts-test-suite` instead of `fold-test-results` is because the latter automatically runs each test but the former leaves that in programmatic control. I thought this would enable me to catch exceptions, thus (using Alex's really good names for the pieces). E.g.:

(define-test-suite hw
  (check-equal? 1 1)
  (check-equal? 1 (/ 1 0))
  (check-equal? 1 2))

(foldts-test-suite
 (λ (suite name before after seed) (before) seed)
 (λ (suite name before after seed kid-seed) (after) (append seed kid-seed))
 (λ (case name action seed)
   (let ([outcome (with-handlers ([exn:fail?
                                   (λ (e)
                                     (println "got here")
                                     (cons 'test-failed seed))])
                    (run-test-case name action))])
     (cons outcome seed)))
 empty
 hw)

Unfortunately, the `with-handers` does not seem to be taking effect at all: the runner still halts with an error. Can anyone see what I'm missing?

Shriram

[Shriram Krishnamurthi]

Thanks to a helpful reply from Matthias I came across this posting

https://github.com/racket/rackunit/pull/107#issuecomment-480808330

which pointed out "The test forms are the things that wrap evaluation, catch errors and continue, etc."

If I rewrite the above as

(define-test-suite hw
  (test-equal? "1" 1 1)
  (test-equal? "2" 1 (/ 1 0))
  (test-equal? "3" 1 (error "raised an error"))
  (test-equal? "4" 1 2))

(foldts-test-suite
 (λ (suite name before after seed) (before) seed)
 (λ (suite name before after seed kid-seed) (after) (append seed kid-seed))

 (λ (case name action seed) (cons (run-test-case name action) seed))
 empty
 hw)

then I don't need the exception handler at all (it appears…). So that may be most of what I need? Not sure if I'm missing something else, I'll report back if I am. (-:

Shriram

[Alex Harsanyi]

On Saturday, May 23, 2020 at 10:25:07 PM UTC+8, sk wrote:

Thank you, Alex, this seems to have just the right set of information. I'm curious why you use `foldts-test-suite` instead of  `fold-test-results`, whose documentation says "Hence it should be used in preference to foldts-test-suite."

I no longer remember why I choose `foldts-test-suite`, I remember experimenting with a few approaches before settling on the current one, but looking at the code now, I think `fold-test-results` would have worked equally well in my case.

 

(And while David's points about the verbosity are well taken, my TAs are likely to be used to the rackunit primitives, though I think I'll consider borrowing some of his aliases for shorter testing primitive names.)

In the success case, the result seems to be `void` (rather than, say, a value corresponding to the type of test). Are there cases where it's not void?

I am not sure I understand this question: whose result is "void" and what would "A value corresponding to the type of the test" be?

 

In the failure case, is there a procedure to convert the `exn:test:check` value into a human-readable string? I'm guessing you have some serialization process for the XML test report.

Unfortunately, I had to import some functions from the "private" section of rackunit. in particular, the display-test-failure/error function.

 

Sadly, errors during test execution aren't sandboxed, so if there's an error in a test, the whole suite breaks down. I think this means I'm going to have to write custom testing procedures anyway.

I am not sure what "errors during test execution" means for you, but a check failure in a test case will not prevent other test cases from running, and neither will any other failure that raises an exception.  I'm sure a creative programmer can write a test case which will prevent other test cases from running, but the issue does not seem important enough to prevent them.

The "unit" is however the test case, not the individual check, so if a check fails inside a test case, other checks in the same test case will not run.

Alex.

[Alex Harsanyi]

As I mentioned in my previous post, the "unit" that can succeed or fail is the "test case", and this will work:

#lang racket
(require rackunit)

(define-test-suite hw
  (test-case "one" (check-equal? 1 1))
  (test-case "two" (check-equal? 1 (/ 1 0)))
  (test-case "three" (check-equal? 1 2)))

(foldts-test-suite
 (λ (suite name before after seed) (before) seed)
 (λ (suite name before after seed kid-seed) (after) (append seed kid-seed))
 (λ (case name action seed)

   (let ([outcome (run-test-case name action)])

     (cons outcome seed)))
 empty
 hw)

;; Produces:
;; '(#<test-failure> #<test-error> #<test-success>)

You won't be able to catch errors from the test case in a handler -- this is handled by rackunit, so the `with-handlers` is useless there.

Alex.

[David Storrs]

On Sat, May 23, 2020 at 9:54 AM Shriram Krishnamurthi <s...@cs.brown.edu> wrote:

Thank you all!

Dave, the documentation style is fine, it's sometimes easier to read the doc right next to the implementation. (-:

However, I'm not quite sure how even your example works. Maybe someone can check my logic? For instance, you say you want to write tests like

    (unless (is os 'windows) (ok test-that-won't-pass-on-windows))

However, `is` seems to return the same value no matter whether the test passed or failed: it returns the first argument, *irrespective* of the outcome of the test. So in the above test, the returned value is going to be that of `os`, which is presumably some non-false value. That means the guarded test will *never* be run, on any OS.

You're absolutely right -- that should have been an `ok`, not an `is`.

[Separately, I'm not sure why one would use a testing utility in that conditional, rather than just a standard conditional, but that's a different matter.]

A standard conditional would say "If we are being run on Windows...", where the `ok` is saying "I expect that this test file is being run on Windows".

In general, this seems to be a property of your underlying function, `test-more-check`: it returns either the return value sent in through #:return or the value in the checked position (#:got). But in either case, this is independent of the success of the test. The only difference is in the *message*, which is printed as output. I suppose I could parameterize where it's printed and capture it — but then I have to parse all the information back out. I'm just not seeing how to compositionally use your testing primitives?

Number of success and failures is available through (tests-passed) and (tests-failed), so that's one option. (current-test-num), (inc-test-num!), and (next-test-num) allow you to determine and modify the number of tests that will be reported, so you can conditionally run a test and then pretend it didn't happen if you don't like the outcome.   `ok` returns a boolean so it can be used to conditionally run groups of tests.  The other functions return their argument so you can chain it through a series of tests.      

I'd be delighted to add more options if I knew that other people were using the package -- just let me know.

As an aside, when trying to install the package in a Docker container running Ubuntu 18.04 with Racket 7.7 installed, I got this error:

raco setup: docs failure: query-exec: unable to open the database file
  error code: 14
  SQL: "ATTACH $1 AS other"
  database: #<path:/root/.racket/7.7/doc/docindex.sqlite>
  mode: 'read-only
  file permissions: (write read)

which I didn't get on macOS Catalina. The package certainly has a … lot of stuff! Even links to EDGAR filings. (-:

Yeah, it's an absolute junkpile.  When I was learning Racket I wrote all these things but didn't have the sense to put them in separate modules.  Now that I'm older and hopefully wiser, in my Copious Free Time I'm working on splitting it all into stand-alone modules and documenting everything, but that's going slowly.  If test-more looks like something you might use then I'll prioritize it so that you aren't stuck with the rest of the kitchen sink.

Thanks,
Shriram

--
You received this message because you are subscribed to the Google Groups "Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to racket-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/CAJUf2yS7%2B2OSq899U5%2BQrgZwFNwdowQNdByU4AkxxwK%3Dby5wOQ%40mail.gmail.com.