Exception throwing in web-server

[Norman Gray]

Greetings.

I define a custom exception:

(define-struct (my-exception exn:fail) ())

(define (my-error fmt . args)
(let ((msg (apply format (cons fmt args))))
(raise (my-exception msg (current-continuation-marks)))))

The plan is that I can throw this within a servlet, and then catch it,
and produce a 5xx response, within a with-handlers clause wrapping the
dispatcher passed to serve/servlet. However this doesn't always work as
I expect it to. The exception is thrown inside the 'output' procedure
that's provided as the last argument to the 'response' constructor (I
belatedly realise this is probably a bad idea).

Sometimes, when the custom exception is thrown, that exception appears,
not prettyprinted via the handler that I have defined, but printed on
stderr via the default exception output formatter:

My exception-message ...
context...:
[...blah...]
/Data/LocalApplications/racket/7.7/share/pkgs/web-server-lib/web-server/http/response.rkt:141:12

Looking at that place in the file, I see

(define (output-response-body/chunked conn bresp)
[....blah...]
(with-handlers ([exn:fail? (lambda (e)
(kill-thread to-chunker-t))])

...which causes the servlet to (effectively) hang, so that the client
sits around waiting for output that isn't going to come.

Exactly the same thing happens if I avoid wrapping the dispatcher, and
instead provide a #:servlet-responder procedure as a keyword argument to
serve/servlet.

Now, in tracking this down I can see that I have a wrong design here:
the servlet has started producing output before the exception is thrown,
so it's at this point really too late for me to be throwing errors and
producing custom 5xx error pages.

But (a) what should I be doing? And (b) since that exception is caught
in this with-handlers clause, what is it that's producing the (default)
exception output message? And (c) should I expect the client just to
hang here?

I'm guessing that an answer to (a) is 'avoid throwing exceptions inside
'output', but given that that will sometimes happen, is the
output-response-body/chunked procedure doing the right thing here? Am I
missing something?

Best wishes,

Norman


--
Norman Gray : https://nxg.me.uk
SUPA School of Physics and Astronomy, University of Glasgow, UK

[Brian Adkins]

On Saturday, May 16, 2020 at 3:16:18 PM UTC-4, Norman Gray wrote:

[...]

The exception is thrown inside the 'output' procedure
that's provided as the last argument to the 'response' constructor (I
belatedly realise this is probably a bad idea).

[...]

But (a) what should I be doing? And (b) since that exception is caught
in this with-handlers clause, what is it that's producing the (default)
exception output message?  And (c) should I expect the client just to
hang here?

I'm guessing that an answer to (a) is 'avoid throwing exceptions inside
'output', but given that that will sometimes happen, is the
output-response-body/chunked procedure doing the right thing here?  Am I
missing something?

I think you need to decide when to stream, and when not to stream. In my web framework, most requests involve computing the entire response string prior to calling (response ...), so if an error is encountered, I can send an error response instead of a success response.

Currently, the only time I stream a response is when the client is downloading CSV output. In that case, there is little chance of an error occurring once the streaming begins. I just briefly reviewed some HTTP protocol information, and I couldn't find anything that would allow sending a 500 response after the 200 response has already be sent i.e. no way to switch from success to error mid-stream.

[Jesse Alama]

Hi Norman,

On Saturday, May 16, 2020 at 9:16:18 PM UTC+2, Norman Gray wrote:

But (a) what should I be doing? And (b) since that exception is caught
in this with-handlers clause, what is it that's producing the (default)
exception output message?  And (c) should I expect the client just to
hang here?

I'm guessing that an answer to (a) is 'avoid throwing exceptions inside
'output', but given that that will sometimes happen, is the
output-response-body/chunked procedure doing the right thing here?  Am I
missing something?

I suggest thinking of a servlet as a response builder, and, if possible, to delegate serialization of the response (output-response & friends) till after a response? value has been created. You can happily throw exceptions during the construction of the response? value; the exceptions will be handled by whatever scaffolding you're using for your web server (that is, directly handled using with-handlers of your own making that wraps the construction of the response?, or by the handler you gave as the value of #:servlet-responder in `serve/servlet`, and so on). Does that help?

Jesse

[Norman Gray]

Thank you, Brian and Jesse, for your thoughts on this. There may still
be an exception problem here, though.

(and sorry for being sluggish to respond)

On 16 May 2020, at 20:16, Norman Gray wrote:

> Now, in tracking this down I can see that I have a wrong design here:
> the servlet has started producing output before the exception is
> thrown, so it's at this point really too late for me to be throwing
> errors and producing custom 5xx error pages.

Brian said:

> I think you need to decide when to stream, and when not to stream. In
> my
> web framework, most requests involve computing the entire response
> string
> prior to calling (response ...), so if an error is encountered, I can
> send
> an error response instead of a success response.

and Jesse:

> I suggest thinking of a servlet as a response builder, and, if
> possible, to
> delegate serialization of the response (output-response & friends)
> till
> after a response? value has been created.

I agree this is the right way of thinking about things here, and it's
reassuring to have that confirmed. Part of what was confusing me was
that it's not particularly clear from the documentation what
serve/servlet's #:servlet-responder is there for. It appears to be just
an odd spelling of 'exception handler', as far as I can tell from the
code.

Indeed it's true that, once the HTTP status code has hit the wire,
there's no provision in the protocol to change one's mind and come up
with a different status (it's possible that forthcoming HTTP/3, with its
concern to multiplex content on the wire, will come up with something
here, but I haven't examined HTTP/3 in detail, and I'd be surprised if
this was one of its concerns).

However, a problem comes when the serialiser _does_ produce a 'real'
exception -- meaning an exception that isn't one that I expected it to
produce. In that case, the response.rkt code just hangs.

Consider:

#lang racket/base
(require web-server/servlet
web-server/servlet-env)

(define (make-response/output writer)
(λ (req)
(response 200 #"OK" (current-seconds) #"text/plain" '()
writer)))

(define my-app/simple
(make-response/output
(λ (op)
(display "hello" op))))
(define my-app/error
(make-response/output
(λ (op)
(error "Oooops")
(display "Hello" op))))
(define my-app/handlers
(make-response/output
(λ (op)
(with-handlers ((exn:fail? (λ (ex) (display "Eeek!" op))))
(error "Oooops")
(display "Hello" op)))))

(serve/servlet my-app/error
#:servlet-regexp #rx""
#:command-line? #t)

If we run this server, and dereference <http://localhost:8000/>, for
example with curl, then the retrieval simply hangs.

It appears that the handler in
web-server-lib/web-server/response.rkt:148 is supposed to handle this
case, but it appears not to. I think it's possible the handler should
be in to-chunker-t instead or as well.

This means that a further possibility is to have an exception handler
within the serialiser, and handle exceptions appropriately there (as in
my-app/handlers above). However all this means that a carefully-written
servlet _must_ have such handlers, if an inadvertent exception in the
serialiser procedure isn't to stall a client.

[Ryan Culpepper]

As I understand the HTTP protocol (that is, some but not lots), the most reasonable thing for the server to do if it discovers an error after the status code has been sent seems to be to just hang up and let the client realize that *something* went wrong. I don't mean just truncate the output; I mean the server should say "here comes another chunk" and then close the TCP connection, so it cannot be mistaken for a valid response. (The servlet should probably make a note so the next request doesn't just fail in exactly the same way.)

Is there are reason not to change the web server to do this automatically?

Ryan

--
You received this message because you are subscribed to the Google Groups "Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to racket-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/AB2357DF-5A41-429F-A7BB-7B4321EEDBE3%40glasgow.ac.uk.

[Matthew Butterick]

AFAICT this is the intended behavior. To me it is consistent with the usual policy: an uncaught error stops the program. If you want the program to keep running, then you have to catch the error and make other arrangements. All my servlet routes are surrounded by a top-level `with-handlers` block that catches `exn:fail?`. If I get an error, I usually a) log it to the remote system, b) send an email to myself, and c) send a status 400 response to the browser with the error string. But the web server keeps running as usual.

[Norman Gray]

Ryan and Matthew, hello.

On 25 May 2020, at 19:43, Ryan Culpepper wrote:

> As I understand the HTTP protocol (that is, some but not lots), the
> most
> reasonable thing for the server to do if it discovers an error after
> the
> status code has been sent seems to be to just hang up and let the
> client
> realize that *something* went wrong. I don't mean just truncate the
> output;
> I mean the server should say "here comes another chunk" and then close
> the
> TCP connection, so it cannot be mistaken for a valid response. (The
> servlet
> should probably make a note so the next request doesn't just fail in
> exactly the same way.)

I have spent a fair amount of quality time with the HTTP RFCs, and I'm
surprised I can't think of an answer to this off the top of my head.

Looking through RFC 7230, however (RFCs 7230--5 replace RFC 2616), we
find in Sect.3.3.3 'Message Body Length',

Since there is no way to distinguish a successfully completed,
close-delimited message from a partially received message
interrupted
by network failure, a server SHOULD generate encoding or
length-delimited messages whenever possible. The close-delimiting
feature exists primarily for backwards compatibility with HTTP/1.0.

If a response includes a Content-Length header, then truncation would be
detectable, if not, not.

This passage is talking about network failure, but I think the
server-failure we're talking about here is morally similar. RFC 7230
Sect 6.3.2, though it's talking about a slightly different thing, also
conceives of the notion of 'partial failure conditions' whilst being
vague about what these are or what a client should do (the implication
is that the client should... do the right thing).

HTTP is generally deliberately rather vague about the payload -- the
representation of the named resource -- and RFC 7231 Sect.3.3 'Payload
Semantics' is a mere four paragraphs long. It includes text

For example, the payload of a
200 (OK) response to GET (Section 4.3.1) represents the current
state
of the target resource, as observed at the time of the message
origination date

There's quite a lot that doesn't say, -- it's even prefaced by 'for
example'. It doesn't even say that the payload _accurately_ represents
the state of the resource. That sounds like quibbling, but it fits in
with a general idea of 'the client gets what it's given, and it'll like
it'.

However vague this is, I think this would not be consistent with a
server deliberately causing a TCP error, in a protocol at a lower layer
than HTTP. Apart from anything else (a) the HTTP transaction might not,
in principle, be running over TCP, and (b) it would be a lie, since the
problem wasn't a TCP problem.

In other words, truncating the output isn't desirable, obviously, but
the alternatives of a deliberate lower-layer error, or stalling, seem
both to be against the spirit of the spec.

Matthew said:

> AFAICT this is the intended behavior. To me it is consistent with the
> usual policy: an uncaught error stops the program. If you want the
> program to keep running, then you have to catch the error and make
> other arrangements.

But what happens in this case (the my-app/error case in my example) is
that the (server) program keeps going but the client stalls. The
unexpected error in the response-output procedure is caught, and (as far
as I can see) handled by killing the producer thread _without_ closing
the connection. To be clear, I think that the handler should do both.

> All my servlet routes are surrounded by a top-level `with-handlers`
> block that catches `exn:fail?`. If I get an error, I usually a) log it
> to the remote system, b) send an email to myself, and c) send a status
> 400 response to the browser with the error string. But the web server
> keeps running as usual.

I'm not positive where you mean by 'servlet routes'. If you mean
creating a handler to wrap my-app/foo in my example, then yes, that's
what I do, too, and/or create a handler within my-app/foo before I
create the response object. But it looks as if I must _additionally_
create a handler inside the response-output procedure (on the occasions
when I do that 'by hand'), to cope with any exceptions thrown in there.

Of course, I should design that response-output procedure so that it
won't throw exceptions, but... never say never.

Best wishes,

Norman


--
Norman Gray : http://www.astro.gla.ac.uk/users/norman/it/
Research IT Coordinator : School of Physics and Astronomy
// My current template week for IT tasks is: Monday, Tuesday, and Friday

[Ryan Culpepper]

I'm assuming chunked Transfer-Encoding, which is IIRC what the Racket web server uses for unknown-length payloads. If the server hasn't committed to chunked encoding (by sending the header), then it probably hasn't sent the status code either. So truncation is definitely detectable.

RFC 7230 Section 6.3.1 and 6.3.2 give more specific guidance than "do the right thing", as I read it. I would summarize it as: Only automatically retry if you know that the operation is idempotent (6.3.1) or hadn't been processed yet (6.3.2), and only automatically retry once; otherwise, kick the error up a level and let them decide.

RFC 7230 Section 9.6 (Message Integrity) mentions using "length or chunk-delimited framing to detect completeness", so I think using a chunk framing violation to signal incompleteness is fair game.

Re semantics: I haven't read that RFC yet, but generally, *semantics* only apply to *syntactically* well-formed things. So introducing a syntax error (the chunk framing violation) is a good way of saying "nevermind, don't even *try* to interpret this payload", given that there are no other options.

Re TCP: It's not a TCP error, and the client shouldn't assume it was. Section 6.3.1 starts with "Connections can be closed at any time, with or without intention." And the idea generalizes to any connection-oriented transport layer. (Hanging up the phone isn't a phone error, and if you're, say, a telemarketer, you probably wouldn't even consider the possibility that it was a network failure.)

So I still think closing the connection after making the response syntactically valid is a good default for the Racket web server.

BTW, if you control the client, you could also use "trailing headers" (see Section 4.1), but the internet tells me that browsers don't support them.

Ryan

[Bogdan Popa]

Norman Gray writes:

> But what happens in this case (the my-app/error case in my example) is
> that the (server) program keeps going but the client stalls. The
> unexpected error in the response-output procedure is caught, and (as
> far as I can see) handled by killing the producer thread _without_
> closing the connection. To be clear, I think that the handler should
> do both.

The handler you pointed to in your other email is intended to catch
network errors and abrupt hangups from the client, but will not catch
exceptions raised by the response's output function. For that, there
needs to be an exception handler in the chunker thread.

I've opened a PR to add such a handler here:

https://github.com/racket/web-server/pull/93