response before wsgi.input is empty cause flaky network responses

[Andrew Dalke]

I just figured out a problem in our paste-based server which took a
LONG time for me (a decidedly non-network developer) to resolve.

One of my test cases checks to see that sending a POST request to a
GET service returns a correct 405 error, and that the message-body
contains an XML error document.

My regressions would sometimes fail with a

socket.error: [Errno 54] Connection reset by peer

It was more repeatable by doing 100 requests. Each request took 2
seconds. Doing the non-error version completes 100 requests in under a
second.

Eventually (so much happened in the 'eventually') I found a solution.

There was still wsgi.input data sitting there, untouched and unloved.
If I read one byte from it then everything was fine.

def wsgi_application(self, environ, start_response):
# There's some sort of problem if the application
# hasn't read any data. This can occur, for example,
# when sending a POST to a GET service, returning
# a 405 message.
wsgi_input = environ["wsgi.input"]
try:
return self._wsgi_application(environ, start_response)
finally:
# This seems to work even if there's 10K of input.
wsgi_input.read(1)
# If I really want to read all of the data ...
#while wsgi_input.read(100000):
# pass

def _wsgi_application(self, environ, start_response):


I haven't tracked this down inside of paste.htttpserver to figure out
what's going on.

Any comments?

Andrew
da...@dalkescientific.com

[Andrew Dalke]

On Dec 15, 6:45 am, Andrew Dalke <andrewda...@gmail.com> wrote:
> I just figured out a problem in our paste-based server which took a
> LONG time for me (a decidedly non-network developer) to resolve.

Running my fix through my test suite I see it doesn't work in the face
of bad data.

There is a bug in paste.httpserver . It does not validate that the
input content-length is non-negative.

curl -H "Content-Length: -100" http://localhost:8880/

My POST handlers check that case and raise an exception.

If I then read(1) from that file, which is a LimitedLengthFile wrapper
with self.length<0 then it hits

def read(self, length=None):
left = self.length - self._consumed
if length is None:
length = left
else:
length = min(length, left)
# next two lines are hnecessary only if read(0) blocks
if not left:
return ''
data = self.file.read(length)


left = -100 - 0 # == -100
length = min(-100, -100) # == -100
data = self.file.read(-100)

That causes Python to read until EOF, so the server is blocking until
the input is done.

If you're curious, try something like this, pointing it to your Paste
server and a service which accepts a POST

curl -H "Content-Length: -100" --data-binary A http://localhost:8080/

I personally would expect this to return an HTTP 400 error at the
Paste level.

        Andrew
        da...@dalkescientific.com

[Andrew Dalke]

On Dec 15, 8:09 am, Andrew Dalke <andrewda...@gmail.com> wrote:
> > LONG time for me (a decidedly non-network developer) to resolve.

Oh good, I wrote that escape clause.

> There was still wsgi.input data sitting there, untouched and unloved.
> If I read one byte from it then everything was fine.

After cleaning up my various kludges to get to the point where I was
able to show there was a problem and a solution, I managed to make the
failure case disappear.

I now suspect there was a misconfigured piece of code that was doing a
network request in just the wrong place to cause the last 8 hours of
work to be pointed in the wrong direction.

It's still a bug that Paste accepts negative Content-Lengths. ;)

Andrew Dalke <da...@dalkescientific.com>

[Andrew Dalke]

My apologies for treating this like a chat room. I have been working
on this for entirely too long and should have been to bed hours ago.

On Dec 15, 8:42 am, Andrew Dalke <andrewda...@gmail.com> wrote:
> After cleaning up my various kludges to get to the point where I was
> able to show there was a problem and a solution, I managed to make the
> failure case disappear.

It helps if I manage to install the change before doing new tests.

Here's my final workaround code

def wsgi_application(self, environ, start_response):
# There's some sort of problem if the application
# hasn't read any data. This can occur, for example,
# when sending a POST to a GET service, returning
# a 405 message.
wsgi_input = environ["wsgi.input"]
try:
return self._wsgi_application(environ, start_response)
finally:

# change the "if 1" to "if 0" and run
# test_server.test_405_error_message_mega
# You should get socket.error "Connection reset by peer"
errors.
if 1 and isinstance(wsgi_input,
httpserver.LimitedLengthFile):
# Consume something if nothing was consumed *and* work
# around a bug where paste.httpserver allows negative
lengths
if (wsgi_input._consumed == 0 and wsgi_input.length >
0):

# This seems to work even if there's 10K of input.
wsgi_input.read(1)

def _wsgi_application(self, environ, start_response):

With that "if 1" in place my tests work. Replace "1" with "0" and I
get

socket.error: [Errno 54] Connection reset by peer

That's it. Patch works. All tests pass. Time for sleep. ;)

- Andrew Dalke <da...@dalkescientific.com>