Intermittent 502 error with Apache fronting Waitress

[Yap Sok Ann]

I have Apache configuration like this:

    <Proxy balancer://production>

    BalancerMember http://127.0.0.1:6543

    BalancerMember http://127.0.0.1:6544

    </Proxy>

    ProxyPass / balancer://production/

    ProxyPassReverse / balancer://production/

Ports 6543 and 6544 are pserve processes using Waitress.

From time to time, requests to pages will fail with 502 error (roughly 1 out of 3000 requests). When that happens, there will be logs like this from Waitress:

2012-08-07 18:06:28,607 ERROR [waitress][Dummy-3] Exception when serving /cases/new

Traceback (most recent call last):

  File "c:\virtualenv\production\lib\site-packages\waitress-0.8.1-py2.7.egg\waitress\channel.py", line 329, in service

    task.service()

  File "c:\virtualenv\production\lib\site-packages\waitress-0.8.1-py2.7.egg\waitress\task.py", line 173, in service

    self.execute()

  File "c:\virtualenv\production\lib\site-packages\waitress-0.8.1-py2.7.egg\waitress\task.py", line 412, in execute

    self.write(chunk)

  File "c:\virtualenv\production\lib\site-packages\waitress-0.8.1-py2.7.egg\waitress\task.py", line 285, in write

    channel.write_soon(rh)

  File "c:\virtualenv\production\lib\site-packages\waitress-0.8.1-py2.7.egg\waitress\channel.py", line 311, in write_soon

    self.outbufs[-1].append(data)

  File "c:\virtualenv\production\lib\site-packages\waitress-0.8.1-py2.7.egg\waitress\buffers.py", line 246, in append

    buf.append(s)

  File "c:\virtualenv\production\lib\site-packages\waitress-0.8.1-py2.7.egg\waitress\buffers.py", line 54, in append

    read_pos = file.tell()

ValueError: I/O operation on closed file.

2012-08-07 18:06:28,608 ERROR [waitress][Dummy-3] Exception when servicing <waitress.channel.HTTPChannel 127.0.0.1:61326 at 0x7883198>

Traceback (most recent call last):

  File "c:\virtualenv\production\lib\site-packages\waitress-0.8.1-py2.7.egg\waitress\task.py", line 76, in handler_thread

    task.service()

  File "c:\virtualenv\production\lib\site-packages\waitress-0.8.1-py2.7.egg\waitress\channel.py", line 342, in service

    task.service() # must not fail

  File "c:\virtualenv\production\lib\site-packages\waitress-0.8.1-py2.7.egg\waitress\task.py", line 173, in service

    self.execute()

  File "c:\virtualenv\production\lib\site-packages\waitress-0.8.1-py2.7.egg\waitress\task.py", line 320, in execute

    self.write(tobytes(body))

  File "c:\virtualenv\production\lib\site-packages\waitress-0.8.1-py2.7.egg\waitress\task.py", line 285, in write

    channel.write_soon(rh)

  File "c:\virtualenv\production\lib\site-packages\waitress-0.8.1-py2.7.egg\waitress\channel.py", line 311, in write_soon

    self.outbufs[-1].append(data)

  File "c:\virtualenv\production\lib\site-packages\waitress-0.8.1-py2.7.egg\waitress\buffers.py", line 246, in append

    buf.append(s)

  File "c:\virtualenv\production\lib\site-packages\waitress-0.8.1-py2.7.egg\waitress\buffers.py", line 54, in append

    read_pos = file.tell()

ValueError: I/O operation on closed file.

We didn't encounter this error with Apache fronting CherryPy. It only started happening after switching to Waitress.

[Yap Sok Ann]

Correction: intermittent 502 error also happens with Apache fronting CherryPy, although much rarer (1 out of 20000 requests).

I have lost confidence in mod_proxy_http, and will go forward with Apache + mod_proxy_ajp + flup#ajp, which I think is a decent and underrated combo.