Performance from 1.1 to 1.2.1 dropped :(

[mgale]

Hello,

We are using the ioloop and iostream modules from Tornado in an
application level proxy for JMS messages. Short version is old
software we can not upgrade currently and need the benefits of a
proxy:
- External auth modules
- Smart load balancing keeping consumers and producers together

So we wrapped ioloop, iostream and stack_context in a daemon process
and created a JMS handler which could speak the JMS protocol. In
trying to upgrade the modules because of a few SSL bugs, we ran into a
performance issue.

In benchmarking the ioloop and iostream from the 1.1 release we found
that we could do:
- about 725 jms messages per second and with about 30ms of latency

When comparing that with the changes in 1.2.1 we found:
- about 625 jms messages per second and with about 130ms of latency

This seemed odd, so I run Iperf through over a tunnel handler we had
written, it just takes in data from one socket and sends it out over
another, build on top of ioloop and iostream.
The results from release 1.1 are:
- 293 Mbits/sec

The results from release 1.2.1 are:
- 30.1 Mbits/sec

In looking at the input vs output I noticed the following:

1.1 release - read and write calls to the socket would appear to
always be 4096 bytes
1.2.1 release - read calls would always be 4096 bytes but write calls
would bounce between about 2000 bytes to almost 4000 bytes, never of
4000.

It looks like the 1.1 release was more efficient in sending network
data, I think changes in 1.2.1 would result in more packets being
required and more iterations of the ioloop to send the same amount of
data.

So my questions for the list, is there a known performance drop
between the 1.1 and 1.2.1 release, if so is this something that is
going to be addressed in a future release?

Also thanks again for making Tornado open source, that fact we can use
these modules is awesome and have saved us a ton of work. If there is
anything I can do to help track down and make changes to just me know.

Thanks again
Michael

[mgale]

Hello,

Update - We removed stack_context from iostream, by doing the
opposite of the the changes made here
https://github.com/facebook/tornado/commit/b19953441d88c43b8ba6efd3ed40526c5049d2e3
https://github.com/facebook/tornado/commit/3205b3682de94f907053b643aaf465560c52b6a4

This appears to have resolved most if not all of the performance
issues, with the above commits we would get 30Mbits/sec and removing
them we are getting over 200Mbits/sec.

Let me know if there is anything I can do to help in this area.

Thanks again
Michael

[Ben Darnell]

Wow, that's a pretty big drop in throughput.  Would you mind either posting your test code or re-running the tests with the current master branch on github?  (preferably the former, so I can run it under a profiler)  There have been some changes that I think will improve things for this use case (although not as much as removing stack_context entirely).  

-Ben

[mgale]

Hello,

Will see what I can do, make take a day or two ... so swamped :(

Michael

> >https://github.com/facebook/tornado/commit/b19953441d88c43b8ba6efd3ed...
>
> >https://github.com/facebook/tornado/commit/3205b3682de94f907053b643aa...