Tornado performance profiling and tips ?

[stef....@gmail.com]

Hello folks !

I write a TCP syslog server using Tornado. Code is available here : https://github.com/stephane-martin/pyloggr/blob/master/pyloggr/main/syslog_server.py (Pyloggr is a prototype to centralize logs, parse them, and store them in a Postgres DB. Kind of logstash clone, but using RabbitMQ for durability and resilience. Project is not usable yet).

The syslog server was not too complicated to implement using Tornado. My head just exploded twice before i started to understand coroutines and futures.

The server does a pretty fine job to handle many syslog clients. But it eats logs from a single syslog client one order of magnitude slower than Rsyslog (that's not so bad).

So, is there a prefered method to profile a Tornado service ? The async thingie makes traditional profiling somewhat cumbersome.
Do you people have generic performance tips or good practices to speed up the service ?

Kind regards,
Stephane

PS: i also wrote some wrapper for pika rabbitmq client, with coroutines. See https://github.com/stephane-martin/pyloggr/tree/master/pyloggr/rabbitmq

[Ben Darnell]

I use the same tools for profiling Tornado services as I use for anything else; I haven't seen anything in this area that is specialized for asynchronous code. It just takes some practice to read the profiles that were generated without the benefits of full stack traces. In general, the stuff that is worth paying attention to will stand out in asynchronous profiles just as much as it would in a synchronous profile.

I use cProfile for tests and controlled benchmarks, and plop (https://github.com/bdarnell/plop) for collecting data from running services (a contributor just added flamegraph support to plop, so there are now some more UI options). It's also important to use IOLoop.set_blocking_log_threshold and watch the logs, to detect inadvertent blocking.

-Ben

--
You received this message because you are subscribed to the Google Groups "Tornado Web Server" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python-tornad...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.