Using tornado's timeout feature
Mazdak Rezvani
We're using in an extremely high-throughput and performance sensitive
environment. We're really pleased with the performance as we're
getting roughly 1500 requests/second on each of our machines in a non-
trivial application.
However, our app has the unique requirement of responding to the
client with a specific HTTP response within 80 milliseconds. Currently
this feature is implemented with a thread that monitors a function and
terminates it if it doesn't return within the specified internal. This
only works well 80% of the time.
I am wondering if I can use Tornado's timeouts to achieve the same at
a lower level and get better performance.
Thank you
Ben Darnell
then you can use tornado's timeouts instead of a separate thread. If
the time is spent in synchronous work, however (i.e. most database
drivers or simply cpu-intensive work), then IOLoop timeouts can't help
you and you'll have to use a separate thread (or use timeouts built
into the database driver where available).
In the case of synchronous work that can take upwards of 80ms, you'll
probably want to move that work out to a separate thread or process
pool. Then the main IOLoop thread remains unblocked and can use a
timeout to send an error response if the other thread/process doesn't
return in time.
-Ben
Mazdak Rezvani
My last attempt to use a thread pool ended in disaster. I don't
believe Python and threading mix that well :)
I tried a process pool; however that failed as well... Now, the
problem there could be that my program had errors in it and not
necessarily Tornado or Python related. I have to consider a refactor
of my code there.
However, having a timeout at the IOLoop layer is still very useful. So
if I simply implement a timeout handler, it should do the trick? How
do I tell my worker process/thread to give up because IOLoop has timed
out?
Thanks
Ben Darnell
> Thanks very much for the advice Ben.
>
> My last attempt to use a thread pool ended in disaster. I don't
> believe Python and threading mix that well :)
It's unfortunately true that threading is not as well supported in
python as in some other languages, although I have had success with
threading in tornado apps.
>
> I tried a process pool; however that failed as well... Now, the
> problem there could be that my program had errors in it and not
> necessarily Tornado or Python related. I have to consider a refactor
> of my code there.
>
> However, having a timeout at the IOLoop layer is still very useful. So
> if I simply implement a timeout handler, it should do the trick? How
> do I tell my worker process/thread to give up because IOLoop has timed
> out?
If it's a process you can simply kill it. If it's a thread things are
trickier, and depending on what it's doing there may not be a reliable
and safe way to stop it. The safest approach is to have the main
thread set a variable that the worker thread can manually check at
appropriate times (e.g. after completing a database lookup). A
java-centric discussion of the complexities of interrupting a working
thread can be found here:
http://download.oracle.com/javase/1.4.2/docs/guide/misc/threadPrimitiveDeprecation.html
-Ben
Mazdak Rezvani
You mention that you've had good success with threads. I'd like to
possibly get a link or something to your preferred implementation of
this pattern.
As an FYI, here are the requirements for the application I am working
on:
At load time, the app loads hundreds of megabytes of data in to memory
(it has to do this because of the sub 50ms response time required).
This data is mostly dicts.
The application uses an extremely large keepalive timeout. The clients
connecting to the server are other servers and not web browsers.
They server receives around 2000 requests per second.
The application does talk to memcached (pylibmc) and logs to disk on
every request.
Thanks
On Dec 16, 7:16 pm, Ben Darnell <b...@bendarnell.com> wrote:
> > Thanks very much for the advice Ben.
>
> > My last attempt to use a thread pool ended in disaster. I don't
> > believe Python and threading mix that well :)
>
> It's unfortunately true that threading is not as well supported in
> python as in some other languages, although I have had success with
> threading in tornado apps.
>
>
>
> > I tried a process pool; however that failed as well... Now, the
> > problem there could be that my program had errors in it and not
> > necessarily Tornado or Python related. I have to consider a refactor
> > of my code there.
>
> > However, having a timeout at the IOLoop layer is still very useful. So
> > if I simply implement a timeout handler, it should do the trick? How
> > do I tell my worker process/thread to give up because IOLoop has timed
> > out?
>
> If it's a process you can simply kill it. If it's a thread things are
> trickier, and depending on what it's doing there may not be a reliable
> and safe way to stop it. The safest approach is to have the main
> thread set a variable that the worker thread can manually check at
> appropriate times (e.g. after completing a database lookup). A
> java-centric discussion of the complexities of interrupting a working
Christopher Becker
I had trouble reconciling ioloop with other operations by using threads for work happening outside the ioloop interface. It could be my inexperience with threads.
But it seems that, in general, using a separate thread (or threads) can be tricky when working with data structures that the ioloop's thread of control may interact with any time. I would suggest trying to build it with the ioloop interface alone if you can, avoiding blocking calls.
If you must do blocking calls, a proper thread pool should handle it. You may want to use something (RLock comes to mind) to avoid concurrent manipulation of data structures by your thread pool vs the ioloop.
Cheers,
Chris
Ben Darnell
> Hi again Ben
>
> You mention that you've had good success with threads. I'd like to
> possibly get a link or something to your preferred implementation of
> this pattern.
When I've used threading with tornado it hasn't been for concurrency
per se, but as a part of a transitional strategy for bridging legacy
synchronous django code with tornado-based async code. I had django
running in a WSGIContainer and when it needed to call out to async
code it would deadlock if the async code tried to use the same IOLoop
as the WSGIContainer. I had a second thread running its own IOLoop
and AsyncHTTPClient. To make a fetch, I'd use add_callback to
transfer control to the second thread, call the http client's fetch
method from that thread, and then use a threading.Event to let the
django code wait for the async fetch to finish. (If it was a single
fetch I could have just used a synchronous HTTPClient; actual use
often involved multiple fetches in parallel before reaching
event.set()).
-Ben