Akka HTTP - Responding to requests early

[Joe Edwards]

I have an application in which clients can upload large request entities to the server. In some situations the request will be rejected early (before the request body is totally consumed, as this may be hundreds of MB).

Is there a standard way of handling this situation? Since the rejections may happen asynchronously, I can't guarantee how much (if any) of the request body has been read.

As far as I can tell from RFC 7230 section 6.5

• If the request is still in flight, we should respond early with a 'Connection: Close' (and the request stream will die as the connection closes).
  
• If the request has been totally received, we can respond normally and reuse the connection.

Is there any easy way to determine when a response is being sent early?

More generally, is it even legal to send a response before the entire request is received without closing the connection? I can't really tell from the spec, but if not it would make much more sense for akka-http to close the connection for us.

Thanks, Joe

[Daniel Armak]

I can't comment on akka-http, but here's my understanding of the HTTP spec.

It's certainly legal to send a response before receiving the request entity, but the client isn't obligated to start reading the response before it finishes sending the request. And if the server closes the connection without reading the whole request, the client may not see the response, depending on whether it fit into its receive buffer and on how it's written. If you just close the connection, the client might think it a network error and retry the request. So if you want to guarantee the client sees the rejection response, you have to read and discard the whole request. 

If you control the client as well as the server, and the server can reject the request based on its headers, the standard HTTP solution is to send Expect: 100-continue in the request, and then either 100 Continue or a 4xx rejection from the server. But if the server can only reject the request based on some prefix of the entity, HTTP provides no way to abort without closing the connection. You could cheat though: use chunked encoding for the request entity, and send an empty chunk ending the request entity as soon as you read the server's rejection headers.

[Joe Edwards]

Thanks for the explanation.

Unfortunately I don't control the clients so it sounds like we have to send a 'Connection: Close' header and hope the client aborts the request for us, otherwise we have to read (and discard) the whole request to ensure the client will read the response. That's a bit of a pain, but I suppose it's really the client's problem if they insist on ignoring the response until they've uploaded their whole 2GB message.

Re: akka-http Is it true once the stream processing the request is cancelled we still need to attach a (blackhole) sink to the request stream, otherwise the back-pressure may simply stall the client indefinitely?

[Björn Antonsson]

Hi Joe,

If you properly cancel the request stream I don't think that you need to consume it with a black hole sink.

B/

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

-- 

Björn Antonsson

Typesafe – Reactive Apps on the JVM

twitter: @bantonsson

[Richard Bradley]

We are now running up against this issue in practice, so we need a workaround.

To recap: if a client makes a large upload request to an Akka HTTP ("akka-stream-and-http-experimental-1.0-M2") and the server responds (i.e. returns an HttpResponse to the "handlerFlow" function) before reading the full request body, then the TCP stream stalls.

The client receives the response, but thinks that the stream is still valid. It will then send the next HTTP request on the same TCP stream, but Akka will never read the request, as it’s still waiting for the (now aborted) request stream to finish reading in the responded-to request.

So, for our Akka HTTP server to be well behaved, we must either always read in the full request and/or half-close the TCP upload stream.

How do I do those two things?

I'd like to do this in my my routing layer, in case there is an exception in the processing layer.

How can I detect whether the request stream needs cancelling (i.e. whether it has not been fully read)? Or is it safe to always cancel?

How can I "cancel" the stream? Given an instance of HttpRequest, the entity has “dataBytes”, but that’s not a materialized stream, but a Source. How do I get a reference to the materialized stream to cancel it? I tried "request.entity.dataBytes.runWith(Sink.cancelled)", but that seems wrong.

Does cancelling the stream half-close the TCP upload stream? If not, how do I do that? (Send Akka a "connection: close" header on the response? Again -- how do I detect when that's needed, i.e. when the request has not been fully read?)

(see also https://github.com/akka/akka/issues/16510 )

Many thanks,

Rich

[Roland Kuhn]

Hi Richard,

11 feb 2015 kl. 11:30 skrev Richard Bradley <richard.brad...@gmail.com>:

We are now running up against this issue in practice, so we need a workaround.

To recap: if a client makes a large upload request to an Akka HTTP ("akka-stream-and-http-experimental-1.0-M2") and the server responds (i.e. returns an HttpResponse to the "handlerFlow" function) before reading the full request body, then the TCP stream stalls.

The client receives the response, but thinks that the stream is still valid. It will then send the next HTTP request on the same TCP stream, but Akka will never read the request, as it’s still waiting for the (now aborted) request stream to finish reading in the responded-to request.

So, for our Akka HTTP server to be well behaved, we must either always read in the full request and/or half-close the TCP upload stream.

As has been discussed earlier, the only reliable way to treat this case is to fully close the connection, I’m not sure that half-closing will improve the situation. You may or may not try to emit a response before closing, but as the standard does not mandate that the client read the response before the request body has been delivered this cannot really be guaranteed to work.

Another way to tackle this problem would be to require the use of an `Expect: 100-continue` header, which would allow exactly the early response that you want to give.

How do I do those two things?

I'd like to do this in my my routing layer, in case there is an exception in the processing layer.

How can I detect whether the request stream needs cancelling (i.e. whether it has not been fully read)? Or is it safe to always cancel?

Yes, canceling is the way of saying “I am no longer interested”.

How can I "cancel" the stream? Given an instance of HttpRequest, the entity has “dataBytes”, but that’s not a materialized stream, but a Source. How do I get a reference to the materialized stream to cancel it? I tried "request.entity.dataBytes.runWith(Sink.cancelled)", but that seems wrong.

No, that is exactly right.

Does cancelling the stream half-close the TCP upload stream? If not, how do I do that? (Send Akka a "connection: close" header on the response? Again -- how do I detect when that's needed, i.e. when the request has not been fully read?)

Currently canceling of the kind of stream-of-stream setup that HTTP uses does not work correctly, ticket is here. I think if cancelation is properly propagated upstream then all the right things will happen.

Regards,

Roland

Dr. Roland Kuhn
Akka Tech Lead
Typesafe – Reactive apps on the JVM.
twitter: @rolandkuhn

[Richard Bradley]

On Thursday, February 12, 2015 at 7:50:50 PM UTC, rkuhn wrote:

Hi Richard,

11 feb 2015 kl. 11:30 skrev Richard Bradley <richard.brad...@gmail.com>:

We are now running up against this issue in practice, so we need a workaround.

To recap: if a client makes a large upload request to an Akka HTTP ("akka-stream-and-http-experimental-1.0-M2") and the server responds (i.e. returns an HttpResponse to the "handlerFlow" function) before reading the full request body, then the TCP stream stalls.

The client receives the response, but thinks that the stream is still valid. It will then send the next HTTP request on the same TCP stream, but Akka will never read the request, as it’s still waiting for the (now aborted) request stream to finish reading in the responded-to request.

So, for our Akka HTTP server to be well behaved, we must either always read in the full request and/or half-close the TCP upload stream.

As has been discussed earlier, the only reliable way to treat this case is to fully close the connection, I’m not sure that half-closing will improve the situation. You may or may not try to emit a response before closing, but as the standard does not mandate that the client read the response before the request body has been delivered this cannot really be guaranteed to work.

I don't think that's true.

The HTTP specs seem clear -- the server is allowed to reply early and close the upload stream while the client is doing a large upload, and the client should listen for such a close:

http://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html#sec8.2.2

"An HTTP/1.1 (or later) client sending a message-body SHOULD monitor the network connection for an error status while it is transmitting the request."

In the case of a large upload where the server finds out half-way-through that the upload is invalid, the correct behaviour for the server is to:

 1. Respond on the "down" stream with the error details (HTTP 400 bad request, etc. etc.)

 2. Close the "up" stream (i.e. half-close the connection) so that the client stops sending the second half of the large request.

 (3. Don't close the "down" stream, as we want the client to read our error message.)

As well as the specs, we can look to pre-existing servers. The Apache AXIS SOAP server (Tomcat based) does this, for example if you have a very large XML upload with a syntax error half-way through it, Apache AXIS will follow steps 1-3 above when it reaches that point in the stream.

I think that a well-behaved server needs to always either read in the full request and/or half-close the TCP upload stream.

My questions from before still stand: 

 1) how do I know, in the akka.http.server.ExceptionHandler interface (or nearby), whether I need to half-close a part-read request body stream, or if the request body has been read?

 2) in the akka.http.server.ExceptionHandler interface (or nearby) how do I half-close a part-read materialized request body stream, when I don't have a reference to it? I only have a reference to the unmaterialized Source.

 

Another way to tackle this problem would be to require the use of an `Expect: 100-continue` header, which would allow exactly the early response that you want to give.

That's not relevant here -- I'm talking about an error part-way through the request body, not an error in the request headers.

 

How can I "cancel" the stream? Given an instance of HttpRequest, the entity has “dataBytes”, but that’s not a materialized stream, but a Source. How do I get a reference to the materialized stream to cancel it? I tried "request.entity.dataBytes.runWith(Sink.cancelled)", but that seems wrong.

No, that is exactly right.

As I feared, this doesn't work.

I have written an isolated test case which does the following:

 1. The client makes a large upload

 2. The routing code in the Akka server dispatches the request to a worker layer

 3. The worker layer reads half of the upload body, then returns an exception to the router

 4. The routing layer runs "request.entity.dataBytes.runWith(Sink.cancelled)"

  a. ... but this doesn't do anything, as the newly materialized stream isn't actually connected to the TCP -- see https://github.com/akka/akka/issues/15835

 5. The server responds with an error code

 6. Now the TCP upload stream is still neither fully read nor closed. The client is blocked trying to upload the second half of the request. The Akka server code thinks the stream is still in use somewhere, as it doesn't know that the worker code has stopped reading the request.

 7. That TCP connection is stalled, but the client doesn't know it. The next client request on that connection fails with a client timeout.

This test-case is tied to some of my app framework code. I'll try to minimise it into a test case and post it here or raise an issue.

Do you see what I mean?

I think we need some support for this case from the Akka framework -- it seems very difficult to deal with in userland code only.

Thanks very much,

Rich

[Richard Bradley]

> The next client request on that connection fails with a client timeout.

When I see it in isolation like this, this is rather dubious behaviour on the part of the client. The client shouldn't put the TCP connection back in the pool for reuse when the previous upload is still incomplete.

Unfortunately, this is Java's HttpUrlConnection which is exhibiting this behaviour, so we probably ought to support it.

We're currently using that client (backed by sun.net.www.protocol.http.HttpURLConnection), as the Akka client isn't working for us yet (it doesn't have connection pooling - see https://github.com/akka/akka/issues/16856 and it has some other bugs -- see https://github.com/akka/akka/issues/16865 )

I'll create an isolated repro case and raise a ticket. Give me a day or two to pull the code out of our app framework.

[Richard Bradley]

I have raised https://github.com/akka/akka/issues/16893

It has an isolated repro case.

[Roland Kuhn]

Hi Richard,

in my previous reply I forgot to add a link to the relevant ticket: https://github.com/akka/akka/issues/16870 With this cancellation should do the right thing and everything should work as you describe. I agree that this is what we want, thanks for raising it!

Regards,

Roland

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.