"Application did not call start_response"
536 views
Skip to first unread message
Phillip B Oldham
Jun 30, 2010, 11:39:26 AM6/30/10
to gevent: coroutine-based Python network library
We're using gevent's WSGI server for our webapp. During some requests
we connect to a remote (thrift[1]) service via a socket. We're using
gevent.monkey.patch_all(), so socket calls should be happening using
gevent's model.
We're seeing some errors when the request-rate increases on the WSGI
app. Specifically, the request to the remote service isn't completing
correctly and the WSGI server returns an "Internal Server Error". In
the commandline output we're seeing the following errors:
AssertionError: Application did not call start_response
AssertionError: start_response was already called
We're pretty sure this isn't a socket timeout issue - the socket is
being set to timeout after 30s but the error comes back within 5s.
Can someone explain why these are being raised? Shouldn't gevent block
the greenlet when the socket call occurs?
we connect to a remote (thrift[1]) service via a socket. We're using
gevent.monkey.patch_all(), so socket calls should be happening using
gevent's model.
We're seeing some errors when the request-rate increases on the WSGI
app. Specifically, the request to the remote service isn't completing
correctly and the WSGI server returns an "Internal Server Error". In
the commandline output we're seeing the following errors:
AssertionError: Application did not call start_response
AssertionError: start_response was already called
We're pretty sure this isn't a socket timeout issue - the socket is
being set to timeout after 30s but the error comes back within 5s.
Can someone explain why these are being raised? Shouldn't gevent block
the greenlet when the socket call occurs?
Denis Bilenko
Jun 30, 2010, 11:55:18 AM6/30/10
to gev...@googlegroups.com
Hi Phillip,
Basically, the messages describe break of the WSGI protocol by wsgi
application - it must call start_response and do it exactly once. Are
you sure that this is always the case?
If yes, then can you try the other wsgi server (gevent.pywsgi) and see
if the error persists or disappears?
I don't think that sockets and monkey patching has anything to do with
this - it seems that this is the consequence of the logic error in
your WSGI app.
As usual, minimal example that reproduces the error would help. If
that's not possible post your application function here.
Cheers,
Denis.
Phillip B Oldham
Jul 1, 2010, 4:12:49 AM7/1/10
to gevent: coroutine-based Python network library
On Jun 30, 4:55 pm, Denis Bilenko <denis.bile...@gmail.com> wrote:
> Basically, the messages describe break of the WSGI protocol by wsgi
> application - it must call start_response and do it exactly once. Are
> you sure that this is always the case?
Pretty sure. Investigating now, however I wrote the initial wsgi app
> Basically, the messages describe break of the WSGI protocol by wsgi
> application - it must call start_response and do it exactly once. Are
> you sure that this is always the case?
myself and there's only one start_reponse call that I can see in the
code. We're using Beaker's WSGI middleware, however... that may be
causing the problem.
> If yes, then can you try the other wsgi server (gevent.pywsgi) and see
> if the error persists or disappears?
File "build/bdist.linux-i686/egg/gevent/pywsgi.py", line 263, in
handle_one_response
write(''.join(towrite))
File "build/bdist.linux-i686/egg/gevent/pywsgi.py", line 194, in
write
raise AssertionError("write() before start_response()")
AssertionError: write() before start_response()
> As usual, minimal example that reproduces the error would help.
Phillip B Oldham
Jul 1, 2010, 5:26:08 AM7/1/10
to gevent: coroutine-based Python network library
On Jul 1, 9:12 am, Phillip B Oldham <phillip.old...@gmail.com> wrote:
> On Jun 30, 4:55 pm, Denis Bilenko <denis.bile...@gmail.com> wrote:
>
> > Basically, the messages describe break of the WSGI protocol by wsgi
> > application - it must call start_response and do it exactly once. Are
> > you sure that this is always the case?
>
> Pretty sure. Investigating now, however I wrote the initial wsgi app
> myself and there's only one start_reponse call that I can see in the
> code. We're using Beaker's WSGI middleware, however... that may be
> causing the problem.
Further information: start_response errors don't occur when using the
> On Jun 30, 4:55 pm, Denis Bilenko <denis.bile...@gmail.com> wrote:
>
> > Basically, the messages describe break of the WSGI protocol by wsgi
> > application - it must call start_response and do it exactly once. Are
> > you sure that this is always the case?
>
> Pretty sure. Investigating now, however I wrote the initial wsgi app
> myself and there's only one start_reponse call that I can see in the
> code. We're using Beaker's WSGI middleware, however... that may be
> causing the problem.
standard Python WSGI server. In fact, no errors occur at all. Every
request is successful, although slower than gevent.
Phillip B Oldham
Jul 1, 2010, 6:25:35 AM7/1/10
to gevent: coroutine-based Python network library
On Jun 30, 4:55 pm, Denis Bilenko <denis.bile...@gmail.com> wrote:
> I don't think that sockets and monkey patching has anything to do with
> this - it seems that this is the consequence of the logic error in
> your WSGI app.
After a little more investigation I'm leaning towards the problem
> this - it seems that this is the consequence of the logic error in
> your WSGI app.
being with the monkey-patched sockets.
On the "server" side we're seeing "Broken Pipe" errors. When those are
raised the "client" raises the above "start_response" errors.
If I don't monkey-patch the socket the process blocks for the socket
request but it returns as expected and there are no errors.
Denis Bilenko
Jul 1, 2010, 7:11:20 AM7/1/10
to gev...@googlegroups.com
On Thu, Jul 1, 2010 at 5:25 PM, Phillip B Oldham
<phillip...@gmail.com> wrote:
> On Jun 30, 4:55 pm, Denis Bilenko <denis.bile...@gmail.com> wrote:
>> I don't think that sockets and monkey patching has anything to do with
>> this - it seems that this is the consequence of the logic error in
>> your WSGI app.
>
> After a little more investigation I'm leaning towards the problem
> being with the monkey-patched sockets.
<phillip...@gmail.com> wrote:
> On Jun 30, 4:55 pm, Denis Bilenko <denis.bile...@gmail.com> wrote:
>> I don't think that sockets and monkey patching has anything to do with
>> this - it seems that this is the consequence of the logic error in
>> your WSGI app.
>
> After a little more investigation I'm leaning towards the problem
> being with the monkey-patched sockets.
That wouldn't explain why the problem exists both in gevent.wsgi and
gevent.pywsgi. The former does not use Python sockets at all.
Maybe you could add this line:
import traceback; traceback.print_stack() to the beginning of
start_response() in gevent/wsgi.py?
this would help tracing the situation when start_response is called twice.
Just in case, please try this with the latest tip:
http://bitbucket.org/denis/gevent
Phillip B Oldham
Jul 1, 2010, 8:39:16 AM7/1/10
to gevent: coroutine-based Python network library
On Jul 1, 12:11 pm, Denis Bilenko <denis.bile...@gmail.com> wrote:
> On Thu, Jul 1, 2010 at 5:25 PM, Phillip B Oldham
>
> On Thu, Jul 1, 2010 at 5:25 PM, Phillip B Oldham
>
> <phillip.old...@gmail.com> wrote:
> > On Jun 30, 4:55 pm, Denis Bilenko <denis.bile...@gmail.com> wrote:
> >> I don't think that sockets and monkey patching has anything to do with
> >> this - it seems that this is the consequence of the logic error in
> >> your WSGI app.
>
> > After a little more investigation I'm leaning towards the problem
> > being with the monkey-patched sockets.
>
> That wouldn't explain why the problem exists both in gevent.wsgi and
> gevent.pywsgi. The former does not use Python sockets at all.
To explain my thought process: The webapp uses sockets to communicate
> > On Jun 30, 4:55 pm, Denis Bilenko <denis.bile...@gmail.com> wrote:
> >> I don't think that sockets and monkey patching has anything to do with
> >> this - it seems that this is the consequence of the logic error in
> >> your WSGI app.
>
> > After a little more investigation I'm leaning towards the problem
> > being with the monkey-patched sockets.
>
> That wouldn't explain why the problem exists both in gevent.wsgi and
> gevent.pywsgi. The former does not use Python sockets at all.
with another service. We're patching the sockets module with the
gevent.socket module, so that the webapp can communicate to the
service without blocking the entire process. When we make a number of
simultaneous requests to the webapp it opens a number of communication
channels to the service. Occasionally something is happening during
this communication which results in the webapp raising the errors
above and often causes the service to raise a "Broken Pipe" error.
This "something" doesn't happen when using the standard socket module
& gevent running multiple processes (to account for the blocking
socket), but does happen when running a single "patched" process.
> Maybe you could add this line:
> import traceback; traceback.print_stack() to the beginning of
> start_response() in gevent/wsgi.py?
greenlet.py", line 403, in run
result = self._run(*self.args, **self.kwargs)
File "/var/.venvs/webapp/lib/python2.6/site-packages/gevent/
wsgi.py", line 187, in handle
handler.handle()
File "/var/.venvs/webapp/lib/python2.6/site-packages/gevent/
wsgi.py", line 109, in handle
result = self.server.application(env, self.start_response)
File "/var/.venvs/webapp/lib/python2.6/site-packages/Beaker-1.5.4-
py2.6.egg/beaker/middleware.py", line 152, in __call__
return self.wrap_app(environ, session_start_response)
File "/var/.venvs/webapp/lib/python2.6/site-packages/framework-0.0.1-
py2.6.egg/framework/__init__.py", line 806, in __call__
return self.__do_request()
File "/var/.venvs/webapp/lib/python2.6/site-packages/framework-0.0.1-
py2.6.egg/framework/__init__.py", line 879, in __send_response
self._start_response(response.code, response.headers.list())
File "/var/.venvs/webapp/lib/python2.6/site-packages/Beaker-1.5.4-
py2.6.egg/beaker/middleware.py", line 151, in session_start_response
return start_response(status, headers, exc_info)
File "/var/.venvs/webapp/lib/python2.6/site-packages/gevent/
wsgi.py", line 29, in start_response
import traceback; traceback.print_stack()
Traceback (most recent call last):
File "/var/.venvs/webapp/lib/python2.6/site-packages/gevent/
wsgi.py", line 109, in handle
result = self.server.application(env, self.start_response)
File "/var/.venvs/webapp/lib/python2.6/site-packages/Beaker-1.5.4-
py2.6.egg/beaker/middleware.py", line 152, in __call__
return self.wrap_app(environ, session_start_response)
File "/var/.venvs/webapp/lib/python2.6/site-packages/framework-0.0.1-
py2.6.egg/framework/__init__.py", line 806, in __call__
return self.__do_request()
File "/var/.venvs/webapp/lib/python2.6/site-packages/framework-0.0.1-
py2.6.egg/framework/__init__.py", line 879, in __send_response
self._start_response(response.code, response.headers.list())
File "/var/.venvs/webapp/lib/python2.6/site-packages/Beaker-1.5.4-
py2.6.egg/beaker/middleware.py", line 151, in session_start_response
return start_response(status, headers, exc_info)
File "/var/.venvs/webapp/lib/python2.6/site-packages/gevent/
wsgi.py", line 30, in start_response
assert self.reason is None, 'start_response was already called'
AssertionError: start_response was already called
I'm statically building at the moment, direct from the tip I believe.
Here's how I'm doing it:
git clone http://github.com/schmir/gevent.git
cd gevent
git checkout origin/wip-all
python setup.py build_libevent build install
If I don't do this I get a lot of errors from gevent/core.c before a
"gcc failed" message. I'm working on CentOS5.2 on both 32 and 64bit
platforms.
The above gets my install to version 0.13.0dev.
Denis Bilenko
Jul 1, 2010, 8:58:20 AM7/1/10
to gev...@googlegroups.com
On Thu, Jul 1, 2010 at 7:39 PM, Phillip B Oldham
<phillip...@gmail.com> wrote:
> To explain my thought process: The webapp uses sockets to communicate
> with another service. We're patching the sockets module with the
> gevent.socket module, so that the webapp can communicate to the
> service without blocking the entire process. When we make a number of
> simultaneous requests to the webapp it opens a number of communication
> channels to the service. Occasionally something is happening during
> this communication which results in the webapp raising the errors
> above and often causes the service to raise a "Broken Pipe" error.
> This "something" doesn't happen when using the standard socket module
> & gevent running multiple processes (to account for the blocking
> socket), but does happen when running a single "patched" process.
<phillip...@gmail.com> wrote:
> To explain my thought process: The webapp uses sockets to communicate
> with another service. We're patching the sockets module with the
> gevent.socket module, so that the webapp can communicate to the
> service without blocking the entire process. When we make a number of
> simultaneous requests to the webapp it opens a number of communication
> channels to the service. Occasionally something is happening during
> this communication which results in the webapp raising the errors
> above and often causes the service to raise a "Broken Pipe" error.
> This "something" doesn't happen when using the standard socket module
> & gevent running multiple processes (to account for the blocking
> socket), but does happen when running a single "patched" process.
I see.
I don't know what causes "Broken Pipe" but even in case of the error like that
your application should not call start_response() again. The better
strategy is don't
do anything at all and let the error propagate. If the error is not
caught by the app or by
the middleware then the server will generate 500 error.
>
>> Just in case, please try this with the latest tip:http://bitbucket.org/denis/gevent
>
> I'm statically building at the moment, direct from the tip I believe.
> Here's how I'm doing it:
>
> git clone http://github.com/schmir/gevent.git
> cd gevent
> git checkout origin/wip-all
This is not the same as http://bitbucket.org/denis/gevent
> python setup.py build_libevent build install
>
> If I don't do this I get a lot of errors from gevent/core.c before a
> "gcc failed" message. I'm working on CentOS5.2 on both 32 and 64bit
> platforms.
What versions of libevent does this OS provides?
$ yum info libevent
If it's a recent 1.4 then install libevent and libevent-devel packages
and you should be able to build it without downloading anything
manually.
Ralf Schmitt
Jul 1, 2010, 8:58:34 AM7/1/10
to gev...@googlegroups.com
Phillip B Oldham <phillip...@gmail.com> writes:
>
> I'm statically building at the moment, direct from the tip I believe.
> Here's how I'm doing it:
>
> git clone http://github.com/schmir/gevent.git
> cd gevent
> git checkout origin/wip-all
> python setup.py build_libevent build install
>
> If I don't do this I get a lot of errors from gevent/core.c before a
> "gcc failed" message. I'm working on CentOS5.2 on both 32 and 64bit
> platforms.
>
> The above gets my install to version 0.13.0dev.
The above is not quite current. Please instead use the wip-download
branch to get a current version and build with
,----
| GEVENT_DOWNLOAD_LIBEVENT=1 python setup.py build install
`----
Sorry for the inconvenience. I'm still hoping to get those changes into
the mainline repo before the next release...
Phillip B Oldham
Jul 1, 2010, 9:18:06 AM7/1/10
to gevent: coroutine-based Python network library
On Jul 1, 1:58 pm, Denis Bilenko <denis.bile...@gmail.com> wrote:
> This is not the same ashttp://bitbucket.org/denis/gevent
>
> This is not the same ashttp://bitbucket.org/denis/gevent
>
> What versions of libevent does this OS provides?
> $ yum info libevent
>
> If it's a recent 1.4 then install libevent and libevent-devel packages
> and you should be able to build it without downloading anything
> manually.
Seems as though CentOS has recently had an update of libevent, since
> $ yum info libevent
>
> If it's a recent 1.4 then install libevent and libevent-devel packages
> and you should be able to build it without downloading anything
> manually.
up to last week only 1.1a was available. 1.4.13 s now in the repo.
I've updated libevent, cloned your bitbucket repo, removed and
reinstalled gevent, and the issue is still there:
File "/var/.venvs/webapp/lib/python2.6/site-packages/gevent/
greenlet.py", line 403, in run
result = self._run(*self.args, **self.kwargs)
File "/var/.venvs/webapp/lib/python2.6/site-packages/gevent/
handler.handle()
File "/var/.venvs/webapp/lib/python2.6/site-packages/gevent/
wsgi.py", line 108, in handle
result = self.server.application(env, self.start_response)
File "/var/.venvs/webapp/lib/python2.6/site-packages/Beaker-1.5.4-
py2.6.egg/beaker/middleware.py", line 152, in __call__
return self.wrap_app(environ, session_start_response)
File "/var/.venvs/webapp/lib/python2.6/site-packages/framework-0.0.1-
py2.6.egg/framework/__init__.py", line 806, in __call__
return self.__handle_request()
File "/var/.venvs/webapp/lib/python2.6/site-packages/Beaker-1.5.4-
py2.6.egg/beaker/middleware.py", line 152, in __call__
return self.wrap_app(environ, session_start_response)
File "/var/.venvs/webapp/lib/python2.6/site-packages/framework-0.0.1-
py2.6.egg/framework/__init__.py", line 806, in __call__
File "/var/.venvs/webapp/lib/python2.6/site-packages/framework-0.0.1-
py2.6.egg/framework/__init__.py", line 859, in __handle_request
return self.__send_response()
File "/var/.venvs/webapp/lib/python2.6/site-packages/framework-0.0.1-
py2.6.egg/framework/__init__.py", line 879, in __send_response
self._start_response(response.code, response.headers.list())
File "/var/.venvs/webapp/lib/python2.6/site-packages/Beaker-1.5.4-
py2.6.egg/beaker/middleware.py", line 151, in session_start_response
return start_response(status, headers, exc_info)
File "/var/.venvs/webapp/lib/python2.6/site-packages/gevent/
wsgi.py", line 29, in start_response
py2.6.egg/framework/__init__.py", line 879, in __send_response
self._start_response(response.code, response.headers.list())
File "/var/.venvs/webapp/lib/python2.6/site-packages/Beaker-1.5.4-
py2.6.egg/beaker/middleware.py", line 151, in session_start_response
return start_response(status, headers, exc_info)
File "/var/.venvs/webapp/lib/python2.6/site-packages/gevent/
wsgi.py", line 29, in start_response
traceback.print_stack()
Traceback (most recent call last):
File "/var/.venvs/webapp/lib/python2.6/site-packages/gevent/
wsgi.py", line 108, in handle
Traceback (most recent call last):
File "/var/.venvs/webapp/lib/python2.6/site-packages/gevent/
result = self.server.application(env, self.start_response)
File "/var/.venvs/webapp/lib/python2.6/site-packages/Beaker-1.5.4-
py2.6.egg/beaker/middleware.py", line 152, in __call__
return self.wrap_app(environ, session_start_response)
File "/var/.venvs/webapp/lib/python2.6/site-packages/framework-0.0.1-
py2.6.egg/framework/__init__.py", line 806, in __call__
return self.__handle_request()
File "/var/.venvs/webapp/lib/python2.6/site-packages/Beaker-1.5.4-
py2.6.egg/beaker/middleware.py", line 152, in __call__
return self.wrap_app(environ, session_start_response)
File "/var/.venvs/webapp/lib/python2.6/site-packages/framework-0.0.1-
py2.6.egg/framework/__init__.py", line 806, in __call__
File "/var/.venvs/webapp/lib/python2.6/site-packages/framework-0.0.1-
py2.6.egg/framework/__init__.py", line 859, in __handle_request
return self.__send_response()
Denis Bilenko
Jul 1, 2010, 9:40:31 AM7/1/10
to gev...@googlegroups.com
On Thu, Jul 1, 2010 at 8:18 PM, Phillip B Oldham
<phillip...@gmail.com> wrote:
> On Jul 1, 1:58 pm, Denis Bilenko <denis.bile...@gmail.com> wrote:
>> This is not the same ashttp://bitbucket.org/denis/gevent
>>
>> What versions of libevent does this OS provides?
>> $ yum info libevent
>>
>> If it's a recent 1.4 then install libevent and libevent-devel packages
>> and you should be able to build it without downloading anything
>> manually.
>
> Seems as though CentOS has recently had an update of libevent, since
> up to last week only 1.1a was available. 1.4.13 s now in the repo.
>
> I've updated libevent, cloned your bitbucket repo, removed and
> reinstalled gevent, and the issue is still there:
<phillip...@gmail.com> wrote:
> On Jul 1, 1:58 pm, Denis Bilenko <denis.bile...@gmail.com> wrote:
>> This is not the same ashttp://bitbucket.org/denis/gevent
>>
>> What versions of libevent does this OS provides?
>> $ yum info libevent
>>
>> If it's a recent 1.4 then install libevent and libevent-devel packages
>> and you should be able to build it without downloading anything
>> manually.
>
> Seems as though CentOS has recently had an update of libevent, since
> up to last week only 1.1a was available. 1.4.13 s now in the repo.
>
> I've updated libevent, cloned your bitbucket repo, removed and
> reinstalled gevent, and the issue is still there:
Regardless of presence of the socket error, I'd try to figure out why
it is possible
to call start_response() twice.
After that, I'd look into "bad pipe" issue - could you send me that
traceback too?
Phillip B Oldham
Jul 1, 2010, 10:04:45 AM7/1/10
to gevent: coroutine-based Python network library
On Jul 1, 2:40 pm, Denis Bilenko <denis.bile...@gmail.com> wrote:
> Regardless of presence of the socket error, I'd try to figure out why
> it is possible
> to call start_response() twice.
Could you suggest how I may investigate further? The traceback doesn't
> Regardless of presence of the socket error, I'd try to figure out why
> it is possible
> to call start_response() twice.
suggest (to me) that it is in fact calling send_response twice. The
framework I threw together only has one "start_response" reference
wrapped in another method which can be called from other places, but I
really can't see how it could be called more than once. If that were
possible, shouldn't it be throwing a similar error with other WSGI
server implementations?
> After that, I'd look into "bad pipe" issue - could you send me that
> traceback too?
Traceback (most recent call last):
File "/var/.venvs/service/lib/python2.6/site-packages/thrift/server/
TServer.py", line 105, in handle
File "/var/services/service/remote/stock/Stock.py", line 85, in
process
self._processMap[name](self, seqid, iprot, oprot)
File "/var/services/service/remote/stock/Stock.py", line 98, in
process_checkStock
result.write(oprot)
File "/var/services/service/remote/stock/Stock.py", line 230, in
write
iter13.write(oprot)
File "/var/services/service/remote/stock/ttypes.py", line 366, in
write
self.venue.write(oprot)
File "/var/services/service/remote/stock/ttypes.py", line 76, in
write
oprot.writeFieldBegin('unid', TType.STRING, 1)
File "/var/.venvs/service/lib/python2.6/site-packages/thrift/
protocol/TBinaryProtocol.py", line 52, in writeFieldBegin
File "/var/.venvs/service/lib/python2.6/site-packages/thrift/
protocol/TBinaryProtocol.py", line 94, in writeI16
File "/var/.venvs/service/lib/python2.6/site-packages/thrift/
transport/TSocket.py", line 88, in write
error: [Errno 32] Broken pipe
Not sure if that will be of any use - we use Apache Thrift for
communication, rather than raw sockets. I believe "Broken pipe" is a
standard socket exception, however.
Equand
Dec 11, 2012, 3:01:01 AM12/11/12
to gev...@googlegroups.com
Hi Denis!
I've encountered the same problem
Traceback (most recent call last):
File "/usr/local/lib/python2.6/site-packages/gevent/pywsgi.py", line 438, in handle_one_response
self.run_application()
File "/usr/local/lib/python2.6/site-packages/gevent/pywsgi.py", line 425, in run_application
self.process_result()
File "/usr/local/lib/python2.6/site-packages/gevent/pywsgi.py", line 416, in process_result
self.write(data)
File "/usr/local/lib/python2.6/site-packages/gevent/pywsgi.py", line 333, in write
raise AssertionError("The application did not call start_response()")
AssertionError: The application did not call start_response()
<WSGIServer fileno=4 address=91.219.236.4:8080>: Failed to handle request:
request = GET /abuse HTTP/1.1 from ('94.21.252.129', 57614)
application = {'loggedin': 0, 'user': u'root'}
After some time it started working correctly, I don't know how it is related and to what...
Reply all
Reply to author
Forward
0 new messages