Recovering from broken pipe

[Larry Martell]

I have a python server that has this in the main:

from gevent import pywsgi

try:
    httpd = pywsgi.WSGIServer(('0.0.0.0', 8000), app)
    httpd.serve_forever()
except KeyboardInterrupt:
    pass

Recently we began getting HTTPError: 504 Server Error: Gateway
Time-out on requests to the server. Looking in the logs I saw this:

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/gevent/pywsgi.py", line
508, in handle_one_response
    self.run_application()
  File "/usr/local/lib/python2.7/dist-packages/gevent/pywsgi.py", line
495, in run_application
    self.process_result()
  File "/usr/local/lib/python2.7/dist-packages/gevent/pywsgi.py", line
486, in process_result
    self.write(data)
  File "/usr/local/lib/python2.7/dist-packages/gevent/pywsgi.py", line
380, in write
    self._write_with_headers(data)
  File "/usr/local/lib/python2.7/dist-packages/gevent/pywsgi.py", line
400, in _write_with_headers
    self._sendall(towrite)
  File "/usr/local/lib/python2.7/dist-packages/gevent/pywsgi.py", line
355, in _sendall
    self.socket.sendall(data)
  File "/usr/local/lib/python2.7/dist-packages/gevent/socket.py", line
460, in sendall
    data_sent += self.send(_get_memory(data, data_sent), flags)
  File "/usr/local/lib/python2.7/dist-packages/gevent/socket.py", line
437, in send
    return sock.send(data, flags)
error: [Errno 32] Broken pipe
{'GATEWAY_INTERFACE': 'CGI/1.1',
 'HTTP_ACCEPT': '*/*',
 'HTTP_ACCEPT_ENCODING': 'gzip, deflate',
 'HTTP_CONNECTION': 'close',
 'HTTP_HOST': 'foo.bar.com',
 'HTTP_USER_AGENT': 'python-requests/2.9.1',
 'HTTP_X_FORWARDED_FOR': '192.168.10.5',
 'HTTP_X_REAL_IP': '192.168.10.5',
 'PATH_INFO': '/readings',
 'QUERY_STRING': 'target=2225217193085335062&tool=HM',
 'REMOTE_ADDR': '172.17.42.1',
 'REMOTE_PORT': '41708',
 'REQUEST_METHOD': 'GET',
 'SCRIPT_NAME': '',
 'SERVER_NAME': 'fdffcd119506',
 'SERVER_PORT': '8000',
 'SERVER_PROTOCOL': 'HTTP/1.0',
 'SERVER_SOFTWARE': 'gevent/1.0 Python/2.7',
 'wsgi.errors': <open file '<stderr>', mode 'w' at 0x7f4c299bc1e0>,
 'wsgi.input': <gevent.pywsgi.Input object at 0x7f4c19e55210>,
 'wsgi.multiprocess': False,
 'wsgi.multithread': False,
 'wsgi.run_once': False,
 'wsgi.url_scheme': 'http',
 'wsgi.version': (1, 0)} failed with error

Once this occurs it seems that no further requests are served. I
googled this and found many places where people said this was
happening, and many places where people explained why it was
happening, but no place did I find a way to fix or workaround this.

Does anyone know how to get it to continue processing after this occurs?

[Jason Madden]

First, you seem to be using gevent 1.0. The latest version is 1.1.1 and it contains many fixes and improvements to pywsgi. I highly encourage you to upgrade.

Second, on this particular issue, a broken pipe from this location means that the client socket disconnected while the request was in progress, and pywsgi is unable to send the application response. This *should not* cause any problems with future requests. As far as pywsgi is concerned, each client socket is distinct. One of the changes in gevent 1.1.1 is to adjust the logging of events like this to more correctly reflect that it's a client problem and not all that uncommon.

Off the top of my head, the only reasons I can think of for not being able to serve future requests are

(1) some application state got corrupted and the application simply stops running the callable;
(2) the upstream proxy decided that the pywsgi server was "down" because it took so long to handle the request, and so it's stopped passing them on; or
(3) the actual pywsgi listening socket that accepts the client connections in the first place went down somehow (but that's not this error and is quite unlikely).

> --
> You received this message because you are subscribed to the Google Groups "gevent: coroutine-based Python network library" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to gevent+un...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.