Pulsar socket close on 401 HTTP code

[Stanislas Plum]

Hello,

In situation of authentication, if Pulsar raises an HttpException 401, It sets the connection header to "close", indicating that the connection will be closed and the client should not reuse it.

Is there a way to make Pulsar read the whole stream sent by the client before closing its socket? Because this causes bugs and failures in case of multiple clients.

For the moment we found this workaround but it is pretty dirty and not optimal:

                    stream = environ['wsgi.input'].read()

                    if isfuture(stream):

                        yield stream

                    raise HttpException(status=401, headers=headers)

Would there be another way?

Thanks a lot

[lsbardel]

Hi,

Is there a way to make Pulsar read the whole stream sent by the client before closing its socket? Because this causes bugs and failures in case of multiple clients.

The best way is to use pulsar wait_for_body_middleware.

This middleware is usually used in conjunction with web framework such as django and flask.
For an example check the django pulse application or the flask example.

[Stanislas Plum]

Hi,

Thank you for your answer.

The problem is that when pulsar raises an HTTP error, he resets the connection, causing the client to receive RST packets.

Here is an extract of the RFC 7230 explaining the problem and solution:

"If a server performs an immediate close of a TCP connection, there is

   a significant risk that the client will not be able to read the last

   HTTP response.  If the server receives additional data from the

   client on a fully closed connection, such as another request that was

   sent by the client before receiving the server's response, the

   server's TCP stack will send a reset packet to the client;

   unfortunately, the reset packet might erase the client's

   unacknowledged input buffers before they can be read and interpreted

   by the client's HTTP parser.

   To avoid the TCP reset problem, servers typically close a connection

   in stages.  First, the server performs a half-close by closing only

   the write side of the read/write connection.  The server then

   continues to read from the connection until it receives a

   corresponding close by the client, or until the server is reasonably

   certain that its own TCP stack has received the client's

   acknowledgement of the packet(s) containing the server's last

   response.  Finally, the server fully closes the connection."

Is there a workaround in pulsar or is there a way to configure pulsar to never close the socket in case of error and wait for client close?

Thanks a lot.

[lsbardel]

On Wednesday, September 9, 2015 at 9:33:49 AM UTC+1, Stanislas Plum wrote:

Hi,

Thank you for your answer.

The problem is that when pulsar raises an HTTP error, he resets the connection, causing the client to receive RST packets.

 

Which HTTP client do you use? How do you send requests? Do you pipeline them?

 

Here is an extract of the RFC 7230 explaining the problem and solution:

   To avoid the TCP reset problem, servers typically close a connection

   in stages.  First, the server performs a half-close by closing only

   the write side of the read/write connection.  The server then

   continues to read from the connection until it receives a

   corresponding close by the client, or until the server is reasonably

   certain that its own TCP stack has received the client's

   acknowledgement of the packet(s) containing the server's last

   response.  Finally, the server fully closes the connection."

pulsar uses asyncio implementation to close the connection.

The closing is done in three stages:

1) stop reading from the connection (by removing the reader handler)

2) write any outstanding buffer

3) finally close the connection

Implementation here

https://github.com/python/cpython/blob/master/Lib/asyncio/selector_events.py#L559

 

Is there a workaround in pulsar or is there a way to configure pulsar to never close the socket in case of error and wait for client close?

Your problem occurs if you pipeline requests and don't check for errors (https://tools.ietf.org/html/rfc7230#section-6.3.2).

If you are not pipelining, can you come up with an mvce and I'll take a look?

Regards

[Stanislas Plum]

The http client is a standerd brownser.

It is actually not about pipelining this happens with a single request-response exchange.

imagine a client making a PUT request to upload a huge file.

The client sends its header and then start sending the packects of the file. 

As soon as pulsar gets the header of the client's request and sees he is not authenticated, he sends backs an HTTP error 401.

So far so good.

But he also closes his socket immediatly sending a RST packet whereas the normal HTTP way would be to send a FIN packet, receive FINACK, and wait faor the client's FIN packet to close the socket.

It is actually not about waiting for the whole body of the request, which could be not very optimal if it's a really big one, but about closing the socket after the client said he had been told about it.

We could imagine a simple pulsar app sending back 401 error to any request received. If you would use curl to send a big request you would see that it is cut in the middle of it in an inappropriate way.

The wait-for-body-middleware is a functionnal workaround since it waits for the whole request's body, but it would be not optimal in case of huge requests.

This is why I ask if there was a possibility to configure pulsar to never close its sockets until client does.

[Stanislas Plum]

I don't know if you see what I mean, and if there is a possible fix or wordaround?

[lsbardel]

Stanislas,

On Wednesday, September 9, 2015 at 12:47:07 PM UTC+1, Stanislas Plum wrote:

The http client is a standerd brownser.

It is actually not about pipelining this happens with a single request-response exchange.

imagine a client making a PUT request to upload a huge file.

The client sends its header and then start sending the packects of the file. 

As soon as pulsar gets the header of the client's request and sees he is not authenticated, he sends backs an HTTP error 401.

So far so good.

But he also closes his socket immediatly sending a RST packet whereas the normal HTTP way would be to send a FIN packet, receive FINACK, and wait faor the client's FIN packet to close the socket.

It is actually not about waiting for the whole body of the request, which could be not very optimal if it's a really big one, but about closing the socket after the client said he had been told about it.

We could imagine a simple pulsar app sending back 401 error to any request received. If you would use curl to send a big request you would see that it is cut in the middle of it in an inappropriate way.

Are you imagining or do you have a real use case here?

If you have a real use case, can you strip it down to the bare minimum and post the code somewhere (gist maybe) so I can take a look?

As I've asked you in my previous response, an mcwe will go a long way about solving your problem.

I don't have time to try to replicate your issue from scratch and I never had this issue myself.

Regards

[Stanislas Plum]

Hi,

Here is a real example:

Run the following server code:

from pulsar.apps import wsgi
from pulsar.utils.exceptions import HttpException

def hello(environ, start_response=None):
    raise HttpException(status=401)


if __name__ == '__main__':
    wsgi.WSGIServer(callable=hello).start()

$ python3 server.py --bind X.X.X.X

Note that you have to use an IP address corresponding to a real ethernet card, not the loopback interface (The MTU needs to be small enough, lo usually has a pretty big MTU, 65k on my system vs 1500 for an ethernet card).

 

Create a big file to upload (we have to use a pretty big file because the RTT between the client and the server is really low in this configuration, in real life, we have experienced the problem with 5 Mo files).

$ dd if=/dev/urandom of=bigfile.bin bs=1K count=…

 

Upload the file to the server, which most of the time makes the bug appear:

$ curl –v --upload-file …

curl output:

* Send failure: Connection reset by peer
* Closing connection 0
curl: (55) Send failure: Connection reset by peer

As described before, this bug is due to the server immediately closing the connection on error, instead of first performing a half close and then waiting for the close from the client (“TCP reset problem” described here: http://tools.ietf.org/html/rfc7230#section-6.6).

Instead of closing the socket directly with socket.close(), the framework should follow those steps:

·         Call socket.shutdown(socket.SHUT_WR), to indicate that we want to close the connection

·         Continue reading what the client is sending

·         Once the client has closed the connection, call socket.close().

[lsbardel]

As described before, this bug is due to the server immediately closing the connection on error, instead of first performing a half close and then waiting for the close from the client (“TCP reset problem” described here: http://tools.ietf.org/html/rfc7230#section-6.6).

Instead of closing the socket directly with socket.close(), the framework should follow those steps:

·         Call socket.shutdown(socket.SHUT_WR), to indicate that we want to close the connection

·         Continue reading what the client is sending

·         Once the client has closed the connection, call socket.close().

OK, can you check if this commit in master fixes your problem?

https://github.com/quantmind/pulsar/commit/6f33f9501460bc1b01cf5631f00d07eb3402635c

I cannot replicate your issue on my local machine, even with a 10GB file and the card ethernet address.

I don't think I want the connection to continue reading if the server has decided to shut it down.

Regards

[lsbardel]

OK, can you check if this commit in master fixes your problem?

https://github.com/quantmind/pulsar/commit/6f33f9501460bc1b01cf5631f00d07eb3402635c

I cannot replicate your issue on my local machine, even with a 10GB file and the card ethernet address.

Is the change working for you? 

[Mathieu Sornay]

I think it is a race condition really similar to this : https://github.com/benoitc/gunicorn/issues/872

I have not yet being able to consistently reproduce the issue, but the RST packet is definitely there early. I'll look into other WSGI implementations to see how they are dealing with it.