Testing Servlets: exceptions not raised/passed through; how to pass extra arguments to tested servlet

[Marc Kaufmann]

Hi all,

I have gone through https://docs.racket-lang.org/web-server/test.html to finally stop manually checking whether I didn't introduce new bugs in one of my servlets. I think that I have figured out most of the wrinkles for this, except one.

I run my tests via `raco test server.rkt`, and server.rkt has the following structure:

(require web-server/test web-server/servlet ...)

(define (start request)
  (define-values (top-dispatch top-urls)
    (dispatch-rules
      [("") home]
      [("login") #:method (or "post" "get") log-in]
      [("mp") #:method (or "post" "get") mp]
      ...
      ))

  (top-dispatch request))

(module+ main
  (serve/servlet start ...))

(module+ test
 
  (define (test-start-servlet)

    (define experiment-servlet-tester
      (make-servlet-tester start))

    ; Test all the routes. This only tests that they don't raise exceptions
    (define routes
      (list
        '()                   ; Tests "/"
        '("/login")
        '("/mp")
        ))

    (for ([route routes])
      (check-not-exn (λ () (apply experiment-servlet-tester route))))

    (define matrix-servlet-tester
      (make-servlet-tester (λ (req)
                              (matrix req #:h (hash 'mturk-id "TESTER")))))

    (matrix-servlet-tester)

    (displayln "All pages render without exception"))

  (test-start-servlet))

The meat of what is going on is in the test module at the end. I create a servlet-tester for the start servlet, which uses a dispatcher, and since I have a bunch of routes I go through them in a for-loop. However, even though the "/login" route throws an exception, the (test-start-servlet) code is happily marching on, checks the "/mp" route, and then prints "All pages render without exception". And raco test tells me that all the tests are passing!

How do I know that there is an exn? Because it prints the following right before the test results:

Servlet (@ /372d34a3-df04-478f-8d84-94fb212725c2) exception:
>: contract violation
  expected: real?
  given: #<sql-null>
  argument position: 1st
  other arguments...:
   0

So, what is going on here? Clearly test-servet does some exception catching in between, yet it still prints it to the screen? How can I test for this, since I want to know how many and which pages blow up?

Also, when I want to test matrix at the end, I have to pass it an extra argument. Is it the right way of passing in the argument by defining a new servlet where I set the argument to some fixed value, as I do above via:

(define matrix-servlet-tester
      (make-servlet-tester (λ (req)
                              (matrix req #:h (hash 'mturk-id "TESTER")))))

It seems to work, but I am wondering if I can pass arguments directly to `make-servlet-tester`, but I couldn't figure out how.

Thanks,
Marc

[Jay McCarthy]

The output of `make-servlet-tester` returns the HTTP response object that the servlet returns. `make-servlet-tester` really makes a HTTP connection to your servlet and is checking the actual result that you sent back. If the servlet throws an exception, then probably you have the default exception handler which turns that into an HTML display. That HTML display would be returned and thus, there was no exception. In other words, it is not a sufficient test of what you want to check that no exception is thrown, you want to make sure a desirable page is returned.

Jay

--
Jay McCarthy
Associate Professor @ CS @ UMass Lowell
http://jeapostrophe.github.io
Vincit qui se vincit.

--
You received this message because you are subscribed to the Google Groups "Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to racket-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/a7ae5893-5bdd-4b3f-924a-42e4936d2a68%40googlegroups.com.

[Marc Kaufmann]

Thanks for clearing this up. If I understand correctly, the following happens:

- the servlet raises an exception

- this is caught by the default exception handler

- it prints the traceback and exception message to standard output which I see (and which made me realize an exception had been raised), and it passes the response to the servlet-tester. However, the exception didn't bubble up to check-not-exn

Since I can't turn off the exception handling of the servlets, I have to look at the output to find that an exception was thrown. Could I pass a different exception-handler to the test-servlet, which would return something easier (and more robust) to check whether an exception was thrown or not? Or should I bundle the servlet into a lambda for this, and have an exception-handler in that lambda that simply outputs "exception-raised" or some such? Otherwise I can of course parse the output, but that seems a little more error-prone.

Thanks

[Jay McCarthy]

On Thu, Sep 26, 2019 at 5:46 AM Marc Kaufmann <marc.ka...@gmail.com> wrote:
>
> Thanks for clearing this up. If I understand correctly, the following happens:
>
> - the servlet raises an exception
> - this is caught by the default exception handler
> - it prints the traceback and exception message to standard output which I see (and which made me realize an exception had been raised), and it passes the response to the servlet-tester. However, the exception didn't bubble up to check-not-exn
>
> Since I can't turn off the exception handling of the servlets, I have to look at the output to find that an exception was thrown. Could I pass a different exception-handler to the test-servlet, which would return something easier (and more robust) to check whether an exception was thrown or not? Or should I bundle the servlet into a lambda for this, and have an exception-handler in that lambda that simply outputs "exception-raised" or some such? Otherwise I can of course parse the output, but that seems a little more error-prone.

I think you're thinking about this wrong. It is not that can't "turn
off the exception handling" or that "the exception didn't bubble up".
If you go to pkgs.racket-lang.org and there's an exception in the code
on the server, do you expect the Racket exception to be thrown to
Google Chrome where a `with-handlers` can catch it? Of course not.
`make-servlet-tester` is literally a network client. It starts up a
hidden server and connects to it.

So, you need to ask what it is that you are trying to test...

If you want to know if a certain Racket function throws an exception,
then just test that function directly and use `check-not-exn`. That's
what Rackunit, chk, and other libraries are for. In this case, you are
testing a Racket function for its behavior: so you ask "Racket"
questions, like "Are exceptions thrown?".

If you want to know if your servlet returns certain pages, then use
`make-servlet-tester` to inspect the pages that get generated. In this
case, you are testing a Web app for its behavior: so you ask "Web"
questions, like "Is the background of the page purple?".

Jay

[Marc Kaufmann]

Ah yes, that's right. I did somehow (without really thinking about it, admittedly) think that the exception would get returned with the response. I did not realize that test-servlet sets up a server.

So for testing the functions, I just create the right requests and do a check-* test. That should also work to see if dispatch-rules calls the correct routes, right? Or does that require something from a running server that I don't realize?

I'll probably need some time before I figure out which type of testing belongs where, hopefully I'll get to it this weekend.

[George Neuner]

On 9/26/2019 5:45 AM, Marc Kaufmann wrote:
> Thanks for clearing this up. If I understand correctly, the following
> happens:
>
> - the servlet raises an exception
> - this is caught by the default exception handler
> - it prints the traceback and exception message to standard output
> which I see (and which made me realize an exception had been raised),
> and it passes the response to the servlet-tester. However, the
> exception didn't bubble up to check-not-exn
>
> Since I can't turn off the exception handling of the servlets, I have
> to look at the output to find that an exception was thrown. Could I
> pass a different exception-handler to the test-servlet, which would
> return something easier (and more robust) to check whether an
> exception was thrown or not? Or should I bundle the servlet into a
> lambda for this, and have an exception-handler in that lambda that
> simply outputs "exception-raised" or some such? Otherwise I can of
> course parse the output, but that seems a little more error-prone.

I vector every URL through a common dispatch routine that catches
exceptions from the handlers and sends proper errors back to the
client.   Of course this means I (you) can't use the ready-made dispatch
machinery provided by web-server.

I wrote my own dispatcher routine, but there may be a way to do it
working with web-server.  I was under some time constraints when I did
this so I didn't have the opportunity to delve into it too deeply.  And
once it worked, inertia took over and I reused the code. <shrug>

YMMV,
George