Brubeck vs. Tornado on m1.large

Showing 1-34 of 34 messages
Brubeck vs. Tornado on m1.large James Dennis 3/28/12 11:45 AM
I just ran some tests comparing the speed of Brubeck to that of Tornado. It's a simple hello world test, but now we have some actual numbers to show people.


It's the same link as before, but a new version. These numbers were typical of what I saw while running it but represent just two tests. Brubeck is also backed by gevent. The hardware is an AWS m1.large.
Re: Brubeck vs. Tornado on m1.large BenBeecher 3/28/12 11:48 AM
What does the concurrency number mean?
Re: Brubeck vs. Tornado on m1.large James Dennis 3/28/12 11:50 AM
That's how many requests siege keeps open for the whole test.

You can either say, "Show me how quickly you handle 10,000 connections" or, "show me how many connections you can make if you always have 500 open for 10 seconds."

I chose the latter there.
Re: Brubeck vs. Tornado on m1.large BenBeecher 3/28/12 11:57 AM
so "Concurrency: 15.72"

means you could have ~ 16 connections open above the 500?
Re: Brubeck vs. Tornado on m1.large James Dennis 3/28/12 12:02 PM
That number represents how many connections are open simultaneously. A lower number means it is completing the tasks fast enough to not have many of them being handled simultaneously. In the context of web requests low = good as long as num of transactions is high.
Re: Brubeck vs. Tornado on m1.large John Krauss 3/28/12 12:08 PM
Cool info.  Seems like the ballbuster is Tornado's 3.09 second
"longest transaction" (vs. Brubeck's 0.43 sec.)

Any way to pull a stddev/mean/median on transaction lengths?  That
would seem to be the money shot.

-john

--
John Krauss
http://blog.accursedware.com/

Re: Brubeck vs. Tornado on m1.large BenBeecher 3/28/12 12:17 PM
Yeah what exactly does that mean? That tornado's io loop was so bogged down that it took 3 seconds to resume a connection? 
Re: Brubeck vs. Tornado on m1.large BenBeecher 3/28/12 12:21 PM
Also was there any io/async in this? 
Re: Brubeck vs. Tornado on m1.large James Dennis 3/28/12 12:23 PM
Siege provides that output but I don't have it anymore.

Next time!

Re: Brubeck vs. Tornado on m1.large James Dennis 3/28/12 12:23 PM
Yup!
Re: Brubeck vs. Tornado on m1.large James Dennis 3/28/12 12:24 PM
None. Just a hello world. The difference would be bigger if blocking calls were involved. :D
Re: Brubeck vs. Tornado on m1.large Malcolm 3/28/12 1:00 PM
I'm surprised Brubeck didn't do way better than Tornado, I thought one of your tests before showed Brubeck at Node.js speeds (which annihilates Tornado in every hello-world test I've seen)?  Is this because the test just wasn't pushing the edge that far?

Can you pull out per-transaction information too btw?  Would be nice to see if the longest transaction was just noise (in which case you can't infer much from the number) or a lot of transactions were much longer.  I'm guessing, in general, transactions were longer in Tornado since the Concurrency is about 4x that of Brubeck.  It'd also be interesting to see how well each of them do on the edge of working.  Do Brubeck and Tornado handle dropping connections the same? Does one of them explode when it gets overwhelmed?

Another thing I'd like to see is longer runs.  Is 10s enough to see the affects of the GC?  What does the memory profile look like throughout it?  Is memory being fragmented?  What does performance look like after a day of running (I'd be willing to donate to a Testing Fund so you don't have to shoulder the cost).  My guess is Tornado and Brubeck are going to be pretty similar (AFAIK, CPython is a fairly simple ref counting GC, in which case you wouldn't see weird pauses on something steady-state like this but you could see memory fragmentation which limits viability for long running services), but would be interesting if you see latency oscillate on a very long runs.  I've been doing a lot of performance testing lately and have found that runs that last less than an hour are near useless.  The testing I've been doing is generally ~4 hours.  I'd happily do several day tests if my schedule allowed it, though.  Might consider looking into basho_bench too (http://wiki.basho.com/Benchmarking.html).  It's designed more for datastores but you can use it for any operation.  You just have to write a fairly trivial driver in Erlang and then setup a config, run, and then it has some R scripts to generate graphs.  Although doing really heavy testing on hello-world is more of testurbation than useful work, I guess.

<rant>
In the end though, performance numbers for a web framework are likely mostly irrelevant.  People still use Rails.  The bottleneck is probably closer to all the other stuff that is happening in a web request, not the shuffling of bytes.  For most, that's probably the database.  So are getting these numbers useful?  Somewhat, it's nice to just know how it stacks up against other frameworks so you at least know it's not totally slow.  But at the same time the real issue in choosing a framework is going to likely come down to what you are doing more than what framework you choose.  If you're doing a lot of very heavy computations in your web requests, then a Java framework is probably going to perform much better than a Python framework.  If latencies inside the web request are really important (such as adserving) then Python might be a poor choice too.  If the bottleneck is the DB then Python is great.  So maybe a good way to talk about these performance tests isn't in terms of performance (who's faster/slower) but overhead.  These Overhead Tests tell us that Brubeck has roughly the same overhead as Tornado (given the test results in this thread).
</rant>

/M

Re: Brubeck vs. Tornado on m1.large James Dennis 3/28/12 1:40 PM
I'm surprised Brubeck didn't do way better than Tornado, I thought one of your tests before showed Brubeck at Node.js speeds (which annihilates Tornado in every hello-world test I've seen)?  Is this because the test just wasn't pushing the edge that far?

Correct. I have performed tests that show it running at Node.js speeds but this was a quick one-off. Not meant to be comprehensive, just an update to the old test on brubeck.io
 
Can you pull out per-transaction information too btw?

As I mentioned before, yes. But I didn't keep all that data. Siege prints it all out though.
 
Would be nice to see if the longest transaction was just noise (in which case you can't infer much from the number) or a lot of transactions were much longer.

I ran the tests a few times and the output in the gist is typical of what I saw, but it's a simple test.
 
 I'm guessing, in general, transactions were longer in Tornado since the Concurrency is about 4x that of Brubeck.  It'd also be interesting to see how well each of them do on the edge of working.  Do Brubeck and Tornado handle dropping connections the same? Does one of them explode when it gets overwhelmed?

Agreed. 
 
Another thing I'd like to see is longer runs.  Is 10s enough to see the affects of the GC?  What does the memory profile look like throughout it?  Is memory being fragmented?  What does performance look like after a day of running (I'd be willing to donate to a Testing Fund so you don't have to shoulder the cost).

As I mentioned, it was a simple test. Not meant to be comprehensive. I would probably model a more comprehensive test after how Nick uses httpperf here: http://nichol.as/benchmark-of-python-web-servers
 
 My guess is Tornado and Brubeck are going to be pretty similar (AFAIK, CPython is a fairly simple ref counting GC, in which case you wouldn't see weird pauses on something steady-state like this but you could see memory fragmentation which limits viability for long running services), but would be interesting if you see latency oscillate on a very long runs.

This is not a good assumption. The memory systems are different because Brubeck uses greenlets which have their own stack management. The link above from Nick suggests that gevent is a lot better at memory management than Tornado, but I have not confirmed this myself with a test. 

I have been coordinating some friends as we develop a bunch of tests in different languages & frameworks for something much more comprehensive, but we're not there yet.

The test will include node.js, webmachine (erlang), go (which includes framework components), python and sinatra (ruby). If anyone would like to offer other systems, I'd be happy to include them.
 
I've been doing a lot of performance testing lately and have found that runs that last less than an hour are near useless.  The testing I've been doing is generally ~4 hours.  I'd happily do several day tests if my schedule allowed it, though.  

I think that'd be awesome. My biggest issue with testing is that AWS itself is extremely volatile, but I can't get such simple pricing otherwise. I might spend $2 running all these tests but the hardware will perform all over the place. I'd prefer a different place where I know I get dedicated & predictable performance from the hardware.
 
Might consider looking into basho_bench too (http://wiki.basho.com/Benchmarking.html).  It's designed more for datastores but you can use it for any operation.  You just have to write a fairly trivial driver in Erlang and then setup a config, run, and then it has some R scripts to generate graphs.  Although doing really heavy testing on hello-world is more of testurbation than useful work, I guess.

I will read that link soon. Thanks for the tip.
 
<rant>
In the end though, performance numbers for a web framework are likely mostly irrelevant.  People still use Rails.  The bottleneck is probably closer to all the other stuff that is happening in a web request, not the shuffling of bytes.

Of course. This is exactly one of the things that I like gevent for. The blocking pain of Tornado is mitigated substantially into high-performance alternatives. There is also a C++ mysql driver that is compatible with Gevent for great nonblocking performance on Mysql, which Tornado users are most likely blocking. I include things like that in "all the other stuff that is happening in a web request".
 
 For most, that's probably the database.  So are getting these numbers useful?

Certainly! They're in the comprehensive test that is coming.
 
Somewhat, it's nice to just know how it stacks up against other frameworks so you at least know it's not totally slow.  But at the same time the real issue in choosing a framework is going to likely come down to what you are doing more than what framework you choose.  If you're doing a lot of very heavy computations in your web requests, then a Java framework is probably going to perform much better than a Python framework.

Not necessarily. Zmq makes it easy to reach out to external services written in other languages. Python is just the web glue if you use Brubeck.
 
 If latencies inside the web request are really important (such as adserving) then Python might be a poor choice too.  If the bottleneck is the DB then Python is great.  So maybe a good way to talk about these performance tests isn't in terms of performance (who's faster/slower) but overhead.  These Overhead Tests tell us that Brubeck has roughly the same overhead as Tornado (given the test results in this thread).
</rant>

That's a fair consideration. What would you suggest for getting down into the guts of the overhead? It's hard to hold those constant. Web frameworks can be different from the moment the request is recieved to when a response is sent out.


Also worth mentioning, Tornado did not have any nginx heads in front, as is typical in deployments, but Brubeck did have Mongrel2 in front. Even with this additional overhead, Brubeck beats Tornado.

I hope further tests show a bigger difference, but like I said, I whipped this one up quickly to update the other completely unscientific test that was on brubeck.io.

James
Re: Brubeck vs. Tornado on m1.large Andrew Gwozdziewycz 3/28/12 2:52 PM
On Wed, Mar 28, 2012 at 4:40 PM, James Dennis <jde...@gmail.com> wrote:
>> Somewhat, it's nice to just know how it stacks up against other frameworks
>> so you at least know it's not totally slow.  But at the same time the real
>> issue in choosing a framework is going to likely come down to what you are
>> doing more than what framework you choose.  If you're doing a lot of very
>> heavy computations in your web requests, then a Java framework is probably
>> going to perform much better than a Python framework.
>
>
> Not necessarily. Zmq makes it easy to reach out to external services written
> in other languages. Python is just the web glue if you use Brubeck.

You can use Tornado as web glue too, and just consume other APIs that
speak HTTP or some other easy to implement RPC system (perhaps maybe
messagepack).

<rant>
I'm flabbergasted that most arguments against tornado come down to,
"using mysql, or other databases block," and therefore tornado sucks.
Sure, tornado doesn't fit everywhere, but if one is shortsighted
enough to think that every web application has to speak to a database
*directly*, then one is probably shortsighted enough to believe shitty
benchmarks as well.
</rant>

--
http://www.apgwoz.com

Re: Brubeck vs. Tornado on m1.large Malcolm 3/28/12 3:02 PM
Inline

On Mar 28, 2012, at 10:40 PM, James Dennis wrote:

> I'm surprised Brubeck didn't do way better than Tornado, I thought one of your tests before showed Brubeck at Node.js speeds (which annihilates Tornado in every hello-world test I've seen)?  Is this because the test just wasn't pushing the edge that far?
>
> Correct. I have performed tests that show it running at Node.js speeds but this was a quick one-off. Not meant to be comprehensive, just an update to the old test on brubeck.io

These two facts contradict each other in my head.  This benchmark (http://www.ostinelli.net/a-comparison-between-misultin-mochiweb-cowboy-nodejs-and-tornadoweb/) shows NodeJS beating tornado by 3x.  Hopefully the next round of testing will clear this up.


>  
> Can you pull out per-transaction information too btw?
>
> As I mentioned before, yes. But I didn't keep all that data. Siege prints it all out though.

Awesome.

>  
> Would be nice to see if the longest transaction was just noise (in which case you can't infer much from the number) or a lot of transactions were much longer.
>
> I ran the tests a few times and the output in the gist is typical of what I saw, but it's a simple test.
>  
>  I'm guessing, in general, transactions were longer in Tornado since the Concurrency is about 4x that of Brubeck.  It'd also be interesting to see how well each of them do on the edge of working.  Do Brubeck and Tornado handle dropping connections the same? Does one of them explode when it gets overwhelmed?
>
> Agreed.
>  
> Another thing I'd like to see is longer runs.  Is 10s enough to see the affects of the GC?  What does the memory profile look like throughout it?  Is memory being fragmented?  What does performance look like after a day of running (I'd be willing to donate to a Testing Fund so you don't have to shoulder the cost).
>
> As I mentioned, it was a simple test. Not meant to be comprehensive. I would probably model a more comprehensive test after how Nick uses httpperf here: http://nichol.as/benchmark-of-python-web-servers
>  

I didn't mean to imply this test was comprehensive, that's why I offered to donate to a Test Fund so you could do a more robust test.

>  My guess is Tornado and Brubeck are going to be pretty similar (AFAIK, CPython is a fairly simple ref counting GC, in which case you wouldn't see weird pauses on something steady-state like this but you could see memory fragmentation which limits viability for long running services), but would be interesting if you see latency oscillate on a very long runs.
>
> This is not a good assumption. The memory systems are different because Brubeck uses greenlets which have their own stack management. The link above from Nick suggests that gevent is a lot better at memory management than Tornado, but I have not confirmed this myself with a test.

I would expect most of the memory to be on the Python side, still, which is handled by the Python GC.  The link above shows gevent using much less memory than Tornado, but the actual performance profile could be similar.  I'm not sure how Nick gathered memory information but it's a bit trickier than looking at the ps table.  But the next round of testing should have interesting information, one way or the other.

>
> I have been coordinating some friends as we develop a bunch of tests in different languages & frameworks for something much more comprehensive, but we're not there yet.
>
> The test will include node.js, webmachine (erlang), go (which includes framework components), python and sinatra (ruby). If anyone would like to offer other systems, I'd be happy to include them.
>  
> I've been doing a lot of performance testing lately and have found that runs that last less than an hour are near useless.  The testing I've been doing is generally ~4 hours.  I'd happily do several day tests if my schedule allowed it, though.  
>
> I think that'd be awesome. My biggest issue with testing is that AWS itself is extremely volatile, but I can't get such simple pricing otherwise. I might spend $2 running all these tests but the hardware will perform all over the place. I'd prefer a different place where I know I get dedicated & predictable performance from the hardware.
>  
> Might consider looking into basho_bench too (http://wiki.basho.com/Benchmarking.html).  It's designed more for datastores but you can use it for any operation.  You just have to write a fairly trivial driver in Erlang and then setup a config, run, and then it has some R scripts to generate graphs.  Although doing really heavy testing on hello-world is more of testurbation than useful work, I guess.
>
> I will read that link soon. Thanks for the tip.
>  
> <rant>
> In the end though, performance numbers for a web framework are likely mostly irrelevant.  People still use Rails.  The bottleneck is probably closer to all the other stuff that is happening in a web request, not the shuffling of bytes.
>
> Of course. This is exactly one of the things that I like gevent for. The blocking pain of Tornado is mitigated substantially into high-performance alternatives. There is also a C++ mysql driver that is compatible with Gevent for great nonblocking performance on Mysql, which Tornado users are most likely blocking. I include things like that in "all the other stuff that is happening in a web request".
>  

Sure, but I don't mean this in the context of Gevent vs Tornado, but really Anything vs Anything.  Even the examples you gave aren't much more than shuffling bytes.  There are plenty of situations out there where Gevent is as weak a solution as Tornado.

>  For most, that's probably the database.  So are getting these numbers useful?
>
> Certainly! They're in the comprehensive test that is coming.
>  
> Somewhat, it's nice to just know how it stacks up against other frameworks so you at least know it's not totally slow.  But at the same time the real issue in choosing a framework is going to likely come down to what you are doing more than what framework you choose.  If you're doing a lot of very heavy computations in your web requests, then a Java framework is probably going to perform much better than a Python framework.
>
> Not necessarily. Zmq makes it easy to reach out to external services written in other languages. Python is just the web glue if you use Brubeck.
>  

Yes there are lots of possible solutions here, but the point is that the world is complex so a hello-world benchmark needs to be viewed in the proper context.  Some SLAs may require handling more complicated work in the handler.  Work that some language, no matter what framework, is not sufficient for doing.

>  If latencies inside the web request are really important (such as adserving) then Python might be a poor choice too.  If the bottleneck is the DB then Python is great.  So maybe a good way to talk about these performance tests isn't in terms of performance (who's faster/slower) but overhead.  These Overhead Tests tell us that Brubeck has roughly the same overhead as Tornado (given the test results in this thread).
> </rant>
>
> That's a fair consideration. What would you suggest for getting down into the guts of the overhead? It's hard to hold those constant. Web frameworks can be different from the moment the request is recieved to when a response is sent out.
>

What do you mean here?  My point is that these tests involve such trivial work in the handler that what we're really testing the minimum amount of overhead the framework places on a solution.  It's a high-level comparison so the actual differences in what's going on under the hood are largely irrelevant, you just know that if I solve a problem in a particular framework I won't be able to beat X requests/sec.

>
> Also worth mentioning, Tornado did not have any nginx heads in front, as is typical in deployments, but Brubeck did have Mongrel2 in front. Even with this additional overhead, Brubeck beats Tornado.
>

I suspect you're barely scratching what Mongrel2 can do so I'm not surprised.

> I hope further tests show a bigger difference, but like I said, I whipped this one up quickly to update the other completely unscientific test that was on brubeck.io.
>
> James

Re: Brubeck vs. Tornado on m1.large James Dennis 3/28/12 3:22 PM
You can use Tornado as web glue too, and just consume other APIs that
speak HTTP or some other easy to implement RPC system (perhaps maybe
messagepack).

Of course you can. It's easy to use almost anything for glue if you have a full SOA. You just have to build the SOA first.
 
<rant>
I'm flabbergasted that most arguments against tornado come down to,
"using mysql, or other databases block," and therefore tornado sucks.
Sure, tornado doesn't fit everywhere, but if one is shortsighted
enough to think that every web application has to speak to a database
*directly*, then one is probably shortsighted enough to believe shitty
benchmarks as well.
</rant>

When we discuss Tornado vs. Brubeck it comes down to more than just issues with blocking:

1) treatment of blocking calls
2) style of interacting with context switches
3) raw performance

And being blocked on database calls really sucks, so you're flabbergasted over stuff that is in fact a serious issue for performance.
Re: Brubeck vs. Tornado on m1.large James Dennis 3/28/12 3:26 PM
These two facts contradict each other in my head.  This benchmark (http://www.ostinelli.net/a-comparison-between-misultin-mochiweb-cowboy-nodejs-and-tornadoweb/) shows NodeJS beating tornado by 3x.  Hopefully the next round of testing will clear this up.

Yes, the one-off from today doesn't show the same results so further testing is necessary. 
 
I didn't mean to imply this test was comprehensive, that's why I offered to donate to a Test Fund so you could do a more robust test.

I am happy to use more expensive hardware. Where would you guys do this? Rackspace? Linode?
 
I would expect most of the memory to be on the Python side, still, which is handled by the Python GC.  The link above shows gevent using much less memory than Tornado, but the actual performance profile could be similar.  I'm not sure how Nick gathered memory information but it's a bit trickier than looking at the ps table.  But the next round of testing should have interesting information, one way or the other.

We'll defer this point until further testing.
 
Sure, but I don't mean this in the context of Gevent vs Tornado, but really Anything vs Anything.  Even the examples you gave aren't much more than shuffling bytes.  There are plenty of situations out there where Gevent is as weak a solution as Tornado.

Please share some of these. If it's Erlang vs. Python, then we know the answer. If it's Python choices all against each other, that's different.

What do you mean here?  My point is that these tests involve such trivial work in the handler that what we're really testing the minimum amount of overhead the framework places on a solution.  It's a high-level comparison so the actual differences in what's going on under the hood are largely irrelevant, you just know that if I solve a problem in a particular framework I won't be able to beat X requests/sec.

My point is that the overhead IS the framework. It's part of what's being tested.

After we have the baseline for what to expect from very basic hello worlds, we have some idea of how to measure overhead as we adjust the tests. When they do no work, they perform like <this>. When they do more interesting work, they perform like <that>. etc.
 

Re: Brubeck vs. Tornado on m1.large James Dennis 3/28/12 4:24 PM
Malcolm -

Bummer the link you shared doesn't cover WebMachine too.

It's my impression that WebMachine is regarded as *the* way to do web in Erlang at the moment. Do you agree?

James

Re: Brubeck vs. Tornado on m1.large Andrew Gwozdziewycz 3/28/12 4:33 PM

sent from a mobile phone.


On Mar 28, 2012 6:22 PM, "James Dennis" <jde...@gmail.com> wrote:
>>
>> You can use Tornado as web glue too, and just consume other APIs that
>> speak HTTP or some other easy to implement RPC system (perhaps maybe
>> messagepack).
>
>
> Of course you can. It's easy to use almost anything for glue if you have a full SOA. You just have to build the SOA first.

What it comes down to is that tornado is inappropriate as the only layer in your app. That doesn't fit in most simple architectures, but fine.

>> <rant>
>> I'm flabbergasted that most arguments against tornado come down to,
>> "using mysql, or other databases block," and therefore tornado sucks.
>> Sure, tornado doesn't fit everywhere, but if one is shortsighted
>> enough to think that every web application has to speak to a database
>> *directly*, then one is probably shortsighted enough to believe shitty
>> benchmarks as well.
>> </rant>
>
> When we discuss Tornado vs. Brubeck it comes down to more than just issues with blocking:
>
> 1) treatment of blocking calls
> 2) style of interacting with context switches
> 3) raw performance

A benchmark like this speaks nothing of point 2 (yes, I know it was not your intention).

Point 1 is interesting. If you make a blocking call in a framework/language that's not designed for them, then of course they are going to starve other resources. But, blocking isn't always the end of the world. See for instance: http://wingolog.org/archives/2012/03/08/an-in-depth-look-at-the-performance-of-guiles-web-server

which achieves 10k reqs/second in a single process, and single threaded, (on different hardware, so it's certainly not a apples to apples) with *blocking* I/O.

> And being blocked on database calls really sucks, so you're flabbergasted over stuff that is in fact a serious issue for performance.

Well of course! This is why I'm advocating that you don't do it. Naively blocking on resources in an event driven world is bad news. Good thing there are lots of ways to avoid this!

Re: Brubeck vs. Tornado on m1.large Andrew Gwozdziewycz 3/28/12 4:35 PM


On Wed, Mar 28, 2012 at 7:33 PM, Andrew Gwozdziewycz <apg...@gmail.com> wrote:

sent from a mobile phone.


On Mar 28, 2012 6:22 PM, "James Dennis" <jde...@gmail.com> wrote:
>>
>> You can use Tornado as web glue too, and just consume other APIs that
>> speak HTTP or some other easy to implement RPC system (perhaps maybe
>> messagepack).
>
>
> Of course you can. It's easy to use almost anything for glue if you have a full SOA. You just have to build the SOA first.

What it comes down to is that tornado is inappropriate as the only layer in your app. That doesn't fit in most simple architectures, but fine.

>> <rant>
>> I'm flabbergasted that most arguments against tornado come down to,
>> "using mysql, or other databases block," and therefore tornado sucks.
>> Sure, tornado doesn't fit everywhere, but if one is shortsighted
>> enough to think that every web application has to speak to a database
>> *directly*, then one is probably shortsighted enough to believe shitty
>> benchmarks as well.
>> </rant>
>
> When we discuss Tornado vs. Brubeck it comes down to more than just issues with blocking:
>
> 1) treatment of blocking calls
> 2) style of interacting with context switches
> 3) raw performance

A benchmark like this speaks nothing of point 2 (yes, I know it was not your intention).

Point 1 is interesting. If you make a blocking call in a framework/language that's not designed for them, then of course they are going to starve other resources. But, blocking isn't always the end of the world. See for instance: http://wingolog.org/archives/2012/03/08/an-in-depth-look-at-the-performance-of-guiles-web-server

which achieves 10k reqs/second in a single process, and single threaded, (on different hardware, so it's certainly not a apples to apples) with *blocking* I/O.

And, yes, I know this *blocking* is different from the blocking you're talking about *in theory*, but it's really the same.

> And being blocked on database calls really sucks, so you're flabbergasted over stuff that is in fact a serious issue for performance.

Well of course! This is why I'm advocating that you don't do it. Naively blocking on resources in an event driven world is bad news. Good thing there are lots of ways to avoid this!




--
http://www.apgwoz.com
Re: Brubeck vs. Tornado on m1.large James Dennis 3/28/12 5:34 PM

What it comes down to is that tornado is inappropriate as the only layer in your app. That doesn't fit in most simple architectures, but fine.


Sure. I wouldn't disagree with that.

Brubeck differentiates itself here by attempting to be a tool that will perform reliably for you from the early days to the point where it's just glue on a full SOA system. Perhaps Mongrel2 can talk to different services itself, rather than requiring Brubeck.

Having ZeroMQ as the communication system makes it easy to build lots of different components, communicating in multiple patterns that are easily accessible.

Of course, this can be done with HTTP. I am aware of a few places using Tornado very successfully with this model. 

I actually think Brubeck's autoapi could make a somewhat easy way to build services. If the hacker could use a language faster than Python, I'd recommend it, but if not then I think Brubeck could be a good option.
 

>> <rant>
>> I'm flabbergasted that most arguments against tornado come down to,
>> "using mysql, or other databases block," and therefore tornado sucks.
>> Sure, tornado doesn't fit everywhere, but if one is shortsighted
>> enough to think that every web application has to speak to a database
>> *directly*, then one is probably shortsighted enough to believe shitty
>> benchmarks as well.
>> </rant>
>
> When we discuss Tornado vs. Brubeck it comes down to more than just issues with blocking:
>
> 1) treatment of blocking calls
> 2) style of interacting with context switches
> 3) raw performance

A benchmark like this speaks nothing of point 2 (yes, I know it was not your intention).

Point 1 is interesting. If you make a blocking call in a framework/language that's not designed for them, then of course they are going to starve other resources. But, blocking isn't always the end of the world. See for instance: http://wingolog.org/archives/2012/03/08/an-in-depth-look-at-the-performance-of-guiles-web-server

which achieves 10k reqs/second in a single process, and single threaded, (on different hardware, so it's certainly not a apples to apples) with *blocking* I/O.

Interesting. 

If I'm reading this correctly, the output suggests some of the requests took as long as 9ms to fulfill. The link you offer suggested a GC takes about 5ms, so I gotta assume that it accounts for some of that time.
 

> And being blocked on database calls really sucks, so you're flabbergasted over stuff that is in fact a serious issue for performance.

Well of course! This is why I'm advocating that you don't do it. Naively blocking on resources in an event driven world is bad news. Good thing there are lots of ways to avoid this!

Future tests will show access to things that are commonly blocking.

I think I will probably turn on something beefy on AWS and turn on one process per core to see how it does with something of a startup-worthy setup.
Re: Brubeck vs. Tornado on m1.large Malcolm 3/28/12 11:25 PM

No, mochiweb is quite popular as well AFAIK.

Re: Brubeck vs. Tornado on m1.large Malcolm 3/28/12 11:40 PM


On Mar 29, 2012 12:26 AM, "James Dennis" <jde...@gmail.com> wrote:
>>
>> These two facts contradict each other in my head.  This benchmark (http://www.ostinelli.net/a-comparison-between-misultin-mochiweb-cowboy-nodejs-and-tornadoweb/) shows NodeJS beating tornado by 3x.  Hopefully the next round of testing will clear this up.
>
>
> Yes, the one-off from today doesn't show the same results so further testing is necessary. 
>  
>>
>> I didn't mean to imply this test was comprehensive, that's why I offered to donate to a Test Fund so you could do a more robust test.
>
>
> I am happy to use more expensive hardware. Where would you guys do this? Rackspace? Linode?
>  

>>
>> I would expect most of the memory to be on the Python side, still, which is handled by the Python GC.  The link above shows gevent using much less memory than Tornado, but the actual performance profile could be similar.  I'm not sure how Nick gathered memory information but it's a bit trickier than looking at the ps table.  But the next round of testing should have interesting information, one way or the other.
>
>
> We'll defer this point until further testing.
>  
>>
>> Sure, but I don't mean this in the context of Gevent vs Tornado, but really Anything vs Anything.  Even the examples you gave aren't much more than shuffling bytes.  There are plenty of situations out there where Gevent is as weak a solution as Tornado.
>
>
> Please share some of these. If it's Erlang vs. Python, then we know the answer. If it's Python choices all against each other, that's different.
>

Again, I'm being more general than specific framework vs specific framework. If you look St how gevent operates you can come up with examples it won't perform well in. Any situation where IO is free relative to computation Brubeck will effectively handle web requests in serial. This is what is going on in your hello-world test. Everything between getting the request from mongrel to giving it back to mongrel is 1 atomic operation.  Now keep on adding things to the handler that don't do IO and things become worse. A concrete example of this is LMAX. As I understand it their pipe is fat and IO is basically free. Each handler does no IO. Executing code fast there is very important because that is the bottleneck. Lucky the situations Brubeck are good at are the most common (for now).

>> What do you mean here?  My point is that these tests involve such trivial work in the handler that what we're really testing the minimum amount of overhead the framework places on a solution.  It's a high-level comparison so the actual differences in what's going on under the hood are largely irrelevant, you just know that if I solve a problem in a particular framework I won't be able to beat X requests/sec.
>
>
> My point is that the overhead IS the framework. It's part of what's being tested.
>
> After we have the baseline for what to expect from very basic hello worlds, we have some idea of how to measure overhead as we adjust the tests. When they do no work, they perform like <this>. When they do more interesting work, they perform like <that>. etc.
>  
>

We are saying the same thing.

Re: Brubeck vs. Tornado on m1.large Fabio Dive 3/29/12 2:30 AM
On Wed, Mar 28, 2012 at 20:00,  <mmat...@gmail.com> wrote:
> I'm surprised Brubeck didn't do way better than Tornado, I thought one of your tests before showed Brubeck at Node.js speeds (which annihilates Tornado in every hello-world test I've seen)?  Is this because the test just wasn't pushing the edge that far?
>
> Can you pull out per-transaction information too btw?  Would be nice to see if the longest transaction was just noise (in which case you can't infer much from the number) or a lot of transactions were much longer.  I'm guessing, in general, transactions were longer in Tornado since the Concurrency is about 4x that of Brubeck.  It'd also be interesting to see how well each of them do on the edge of working.  Do Brubeck and Tornado handle dropping connections the same? Does one of them explode when it gets overwhelmed?
>
> Another thing I'd like to see is longer runs.  Is 10s enough to see the affects of the GC?  What does the memory profile look like throughout it?  Is memory being fragmented?  What does performance look like after a day of running (I'd be willing to donate to a Testing Fund so you don't have to shoulder the cost).  My guess is Tornado and Brubeck are going to be pretty similar (AFAIK, CPython is a fairly simple ref counting GC, in which case you wouldn't see weird pauses on something steady-state like this but you could see memory fragmentation which limits viability for long running services), but would be interesting if you see latency oscillate on a very long runs.  I've been doing a lot of performance testing lately and have found that runs that last less than an hour are near useless.  The testing I've been doing is generally ~4 hours.  I'd happily do several day tests if my schedule allowed it, though.  Might consider looking into basho_bench too (http://wiki.basho.com/Benchmarking.html).  It's designed more for datastores but you can use it for any operation.  You just have to write a fairly trivial driver in Erlang and then setup a config, run, and then it has some R scripts to generate graphs.  Although doing really heavy testing on hello-world is more of testurbation than useful work, I guess.
>
> <rant>
> In the end though, performance numbers for a web framework are likely mostly irrelevant.  People still use Rails.  The bottleneck is probably closer to all the other stuff that is happening in a web request, not the shuffling of bytes.  For most, that's probably the database.  So are getting these numbers useful?  Somewhat, it's nice to just know how it stacks up against other frameworks so you at least know it's not totally slow.  But at the same time the real issue in choosing a framework is going to likely come down to what you are doing more than what framework you choose.  If you're doing a lot of very heavy computations in your web requests, then a Java framework is probably going to perform much better than a Python framework.  If latencies inside the web request are really important (such as adserving) then Python might be a poor choice too.

Here I am not agree..let's talk about pyramid framework, it can
achieve the speed of 5000 req/s and now let's talk about something
similar but completely greenlet based, with a logic similar to twisted
python. I think there is no framework capable of that speed

Re: Brubeck vs. Tornado on m1.large Malcolm 3/29/12 3:32 AM
There are many non-python frameworks capable of doing orders of
magnitude more req/s than any Python framework I've seen.
Re: Brubeck vs. Tornado on m1.large Brian Knox 3/29/12 3:42 AM

I think that'd be awesome. My biggest issue with testing is that AWS itself is extremely volatile, but I can't get such simple pricing otherwise. I might spend $2 running all these tests but the hardware will perform all over the place. I'd prefer a different place where I know I get dedicated & predictable performance from the hardware.

James - at my new gig we're putting in an order for our lab hardware this week.  It'll be a few weeks before it's all in place, but when it is we should talk.  We're going to do some performance testing of a few things, and since we've been experimenting with brubeck anyway, we could probably slip in some tests on dedicated hardware. 

Brian (that taotetek guy from twitter ;) )
Re: Brubeck vs. Tornado on m1.large James Dennis 3/29/12 5:17 AM
Brian - that'd be awesome!

(glad to see you on the list too!)
Re: Brubeck vs. Tornado on m1.large Kr Ace Kumar Ramaraju 3/30/12 7:57 AM
Any test on pypy ?

On Thursday, March 29, 2012 12:53:28 AM UTC+5:30, James Dennis wrote:
Siege provides that output but I don't have it anymore.

Next time!

On Mar 28, 2012, at 3:08 PM, John Krauss <irving...@gmail.com> wrote:

> Cool info.  Seems like the ballbuster is Tornado's 3.09 second
> "longest transaction" (vs. Brubeck's 0.43 sec.)
>
> Any way to pull a stddev/mean/median on transaction lengths?  That
> would seem to be the money shot.
>
> -john
>
> On Wed, Mar 28, 2012 at 3:02 PM, James Dennis <jde...@gmail.com> wrote:
>> That number represents how many connections are open simultaneously. A lower
>> number means it is completing the tasks fast enough to not have many of them
>> being handled simultaneously. In the context of web requests low = good as
>> long as num of transactions is high.
>>
>>
>> On Mar 28, 2012, at 2:57 PM, Ben Beecher <benbe...@gmail.com> wrote:
>>
>> so "Concurrency: 15.72"
>>
>> means you could have ~ 16 connections open above the 500?
>>
>>
>> On Wed, Mar 28, 2012 at 2:50 PM, James Dennis <jde...@gmail.com> wrote:
>>>
>>> That's how many requests siege keeps open for the whole test.
>>>
>>> You can either say, "Show me how quickly you handle 10,000 connections"
>>> or, "show me how many connections you can make if you always have 500 open
>>> for 10 seconds."
>>>
>>> I chose the latter there.
>>>
>>>
>>> On Wed, Mar 28, 2012 at 2:48 PM, Ben Beecher <benbe...@gmail.com> wrote:
>>>>
>>>> What does the concurrency number mean?
>>>>
>>>>
>>>> On Wed, Mar 28, 2012 at 2:45 PM, James Dennis <jde...@gmail.com> wrote:
>>>>>
>>>>> I just ran some tests comparing the speed of Brubeck to that of Tornado.
>>>>> It's a simple hello world test, but now we have some actual numbers to show
>>>>> people.
>>>>>
>>>>> https://gist.github.com/882555
>>>>>
>>>>> It's the same link as before, but a new version. These numbers were
>>>>> typical of what I saw while running it but represent just two tests. Brubeck
>>>>> is also backed by gevent. The hardware is an AWS m1.large.
>>>>
>>>>
>>>
>>
>
>
>
> --
> John Krauss
> http://blog.accursedware.com/

Re: Brubeck vs. Tornado on m1.large James Dennis 3/30/12 8:08 AM
Last I checked gevent or eventlet didn't have support in pypy.

I am interested to primarily see eventlet in pypy because so much if it is pure python. Greenlets, obviously, are still C but I believe they work with pypy already.
Re: Brubeck vs. Tornado on m1.large Brian Knox 3/30/12 8:11 AM
pyzmq as well does not work in pypy.  There are experimental bindings for zeromq for pypy but they don't implement the full api and I wouldn't use them in production.
Re: Brubeck vs. Tornado on m1.large Kr Ace Kumar Ramaraju 3/30/12 8:11 AM
Yes, gevent doesn't run on pypy, but greenlet should, here is the link https://bitbucket.org/pypy/pypy/src/default/lib_pypy/greenlet.py

I don't know how much functionality is covered.
-- 
"Talk is cheap, show me the code" -- Linus Torvalds
Regards
Kracekumar.R
www.kracekumar.com
Re: Brubeck vs. Tornado on m1.large Kr Ace Kumar Ramaraju 3/30/12 8:13 AM
Inline replies!


pyzmq as well does not work in pypy.  There are experimental bindings for zeromq for pypy but they don't implement the full api and I wouldn't use them in production.

Because pyzmq is written in Cython, it would be super awesome if pypy supports zmq. This is interesting to me.
-- 
"Talk is cheap, show me the code" -- Linus Torvalds
Regards
Kracekumar.R
www.kracekumar.com
Re: Brubeck vs. Tornado on m1.large James Dennis 3/30/12 8:44 AM
pyzmq as well does not work in pypy.  There are experimental bindings for zeromq for pypy but they don't implement the full api and I wouldn't use them in production.

I am really curious to see how pypy eventually does against gevent. It will speak towards the benefits of implementing things in C after the existence of pypy.

Because pyzmq is written in Cython, it would be super awesome if pypy supports zmq. This is interesting to me.

 I agree. I think pypy looks very promising.

I am quite happy with the performance of gevent at the moment, but would welcome gains at the level of python. Somewhat expensive tasks, like instantiating and parsing objects, could probably be made cheaper.

I have found that the demos that don't use classes perform better than the class system. I am interested in exploring the system's performance, now that the general architecture is pretty stable. It would be interesting to see if pypy made the difference have less effect. 

Re: Brubeck vs. Tornado on m1.large Malcolm 3/30/12 9:09 AM

Does tornado run in pypy?

More topics »