Round 4 is now available

[Brian Hauer]

Hi all,

We've posted Round 4!  Here are the URLs.  The results are here:

http://www.techempower.com/benchmarks/#section=data-r4

And our blog has some observations and notes:

http://www.techempower.com/blog/2013/05/02/frameworks-round-4/

As always, we look forward to your feedback.  Questions, suggestions, critiques, or anything else.  Thanks again to everyone who has helped and contributed!

[Brian Hauer]

Good call.  Done!

[Ray Shan]

Hi, thanks for all your hard work. Would you please consider AngularJS? It's a popular MVC framework for single-page-applications. Hope it's appropriate for the benchmark.

http://angularjs.org/

On Thursday, May 2, 2013 9:45:46 AM UTC-5, Brian Hauer wrote:

Good call.  Done!

[Brian Hauer]

Hi Ray!  Thanks for the note.  We love AngularJS, but our project is focused on server application frameworks, and AngularJS is a client-side framework.  :)

[Ray Shan]

Thanks Brian, I totally missed that distinction. But great work nonetheless. Will follow your project closely.

[Rhomel Chinsio]

First off I appreciate the breadth of coverage in terms of languages and frameworks in addition to server configurations (virtual vs physical).

For the perl frameworks I can't find much information on configurations. In particular, the perl frameworks (kelp, dancer) seem to do miserably on the JSON serialization benchmark which could be because the frameworks are using the pure-perl JSON implementation. If this is true, I'd like to see another perl configuration where the JSON pure perl is run along side the JSON::XS configuration. JSON::XS is a drop-in replacement for JSON (in fact JSON will use JSON::XS automatically if it is installed) but written is C/perl-xs code in order to achieve significant performance increases (see https://metacpan.org/module/JSON::XS#SPEED ).

[Steven Luu]

If I recall correctly, JRuby is a lot more performant after a number of requests/executions. The compiled version is used after that. Does the test do any warm-up for JRuby and or JVM platforms in general?

Reference: https://github.com/jruby/jruby/wiki/JRubyCompiler#just-in-time-jit-compilation

[Brian Hauer]

Hi Rhomel,

I'm not a Perl developer, so I can't formulate an opinion between the pure-perl JSON implementation and JSON::XS.  Could I ask you to please chat with the folks who contributed the Perl tests ( https://github.com/TechEmpower/FrameworkBenchmarks/pull/156 ) to see if there is general consensus among Perl developers about this change?

Like all tests, we want the Perl test to perform as well as possible, but also represent real-world/production-grade code.  So if, for the sake of argument, JSON::XS were not production-grade, we'd be hesitant to include it.

[Brian Hauer]

Hi Steven,

Yes, we run a 30-second warm-up cycle prior to capturing any performance data.

[Steven Luu]

Brian,

I did basically the same benchmark on my computer (Macbook Air, i5 4GB RAM) with this combination:

- Sinatra, JRuby, Mizuno as the server, MySQL (using DataMapper instead of ActiveRecord) and on average I get about 1000 reqs/s

Just simple benchmark using ab tool: `ab -n 1000 -c 50 http://localhost:9292`

How would I go about a pull-request as such? All of the mentioned components can be included in the Gemfile.

[Skamander]

Hey guys, thanks for the benchmark. :)

Had you time to play with the connections for play-mongo? As you know on my local machine 1 connection was surprisingly the fastest, so I would be interested if it was the same for you.

Keep up the good work.

Cheers

[Josh Stevenson]

Why isn't Zend Framework included in these benchmarks?

[Brian Hauer]

Hi Steven,

There are some instructions about preparing a test for contribution as a pull request at the root of the Github page.  Is that what you are looking for?

https://github.com/TechEmpower/FrameworkBenchmarks

If, however, your specific pull request is fixing a configuration or coding glitch in the Sinatra test, the question we'd like to ask is: does that make more sense as a pull request to modify the existing test?

We understand that in some cases, it's desirable to have multiple configuration permutations for some frameworks, especially when there is lively debate about the framework's canonical high-performance production-grade configuration.

But if possible, we'd prefer that the single test for any given framework represents the community's "best practice."

Does all of that make sense?

[Brian Hauer]

Hi Josh,

I believe the short answer is that we simply have not received a pull request with a Zend Framework implementation of our tests.  Might I be able to entice you to create one?

[Brian Hauer]

Hi Skamander!

Thanks again for all of your contributions!  Thanks also for helping us answer so many questions.  :)

I'm not sure if Pat fine-tuned the connection configuration for play-mongo.  I'll ask him and we'll follow up.

[Rob Olmos]

Can you please include the PHP related configurations for review such as FPM and APC?

Thanks

[Jordan L]

Could you provide me a link to the implementation details of the PHP environment. I can't seem to dig it up via github.

Two settings to consider for all the PHP environments is making sure APC buffers are not overlowed. APC's default apc.shm_size (32M) may need to be increased from standard cache size to ensure it's not being overflowed. 64m/96m should do it. 

Additionally on PHP PSR-0 frameworks like Symphony you'll need to make sure path_size_realcache() is bumped up from 16k to something like 128k-256k, to avoid hitting disk when looking up files. PSR-0 frameworks require a large increase in this buffer, as their paths are longer and they have many files.

These two caches are very important in ensuring that PHP is running at optimal speed and usually need to be increased from their defaults.

Curious if other developers have other optimizations for their platforms of choice as well.

Thanks for the benchmarks, they're useful.

[Brian Hauer]

Hi Rob and Jordan,

The PHP configuration files are at the following locations:

https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/config/php.ini
https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/config/php-fpm.conf

These are referenced by the setup.py files in each framework's directory.  For example, here is Cake's:

https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/cake/setup.py

[er...@gochat.us]

Hi Brian,

It seems that your filter for MongoDB-only is not working when combined with the PHP-only filter (unchecking all other languages & DBs), there yields no results. I believe that Li3/Lithium supports MongoDB, although not sure if any of the other PHP-frameworks listed do.

Also, could you kindly add the Vork Enterprise Framework to your testing (http://vork.us) - it supports MongoDB, Couchbase and every other imaginable DB and has been built specifically for scaling high-performance applications. 

Considering all the work that you have put into this fascinating resource, perhaps you would find the framework comparison page interesting (http://vork.us/it-roi) - the results can be replicated/verified by anyone quite easily since it was done on commodity Rackspace cloud-servers and uses a remote load-testing service from BlazeMeter (an impartial 3rd-party.)

Thank you,

Eric

[Pat Falls]

Hi Skamander,

I haven't been able to do much testing with the connections yet, it's on my list for Round 5 though.

[Brian Hauer]

Hi Eric,

Thanks for the note!

The filters narrow the view of the tests we have implemented or received as community contributions (more than half of the tests you see now have been contributed by the community).  The filters do not indicate which databases are supported by each framework--just the pairings for which we have test coverage.

When you narrow in on PHP and then select only MongoDB, the reason nothing is illuminated is simply that we have no tests for PHP frameworks paired with MongoDB.  We'd be happy to include such a test if we got a pull request, of course!  :)

We'd also be happy to include Vork in a future round.  We have a long backlog of requests, so the quickest way to get a new framework added would be for you (or whomever) to implement the test cases according to Vork's best practices and submit a Github pull request.  We can answer any questions you have about the specifications of the test cases or how to set up the framework meta-data for our benchmark tool.

Thanks again for your feedback!

[Johannes Rudolph]

Hi,

the results for onion in the json benchmark on dedicated hardware are strange. Could it be that there is some kind of ceiling where there's not enough traffic generated to saturate the server? Do you measure the load on the load-generator and server to rule that out? E.g. a latency of 1 ms between both machines at a concurrency level of 256 would already cap the maximum number of requests possible to 256k/s.

1s / 1 ms * 256 parallel requests = 256k requests

Do you have any other explanation?

Cheers,

Johannes

[Johannes Rudolph]

On Wednesday, May 8, 2013 6:27:22 PM UTC+2, Johannes Rudolph wrote:

E.g. a latency of 1 ms between both machines at a concurrency level of 256 would already cap the maximum number of requests possible to 256k/s.

1s / 1 ms * 256 parallel requests = 256k requests

Thinking more about it, the same calculation would rule out latency as the cap here: if you look at the maximum numbers achieved at concurrency 8 (66k requests / s) the latency cannot be much bigger than 0.1 ms.

Still, it seems strange why onion's lead by a factor of 1.3 on EC2 just disappears running on faster hardware.

Johannes

[Brian Hauer]

Hi Johannes,

Yes, in fact we have a theory about frameworks that achieve 200,000+ responses per second in the i7 JSON test.  Put briefly, we are saturating our gigabit Ethernet.  They probably have even higher performance that we cannot properly measure.

The very small payload of the JSON test allows our gigabit Ethernet to transmit a great deal of requests per second, but with such a small payload, the response headers end up playing a bigger role in network saturation than the payload itself.

As a result, the frameworks that render fewer response headers end up slightly higher on the results for i7 JSON.  We have a Github issue to normalize and specify a minimum set of required response headers.

On EC2, the JSON test is CPU limited.  On i7, for the fastest frameworks, the JSON test is network limited.  In spot tests, we observed some frameworks achieving 300,000+ rps when we ran the load tool local to the application server (meaning cutting into the CPU power of the server but eliminating the network bottleneck).

I have pointed this out elsewhere, but didn't capture it in the FAQ on the site.  I'll do so for the next round.

Also, if anyone is willing to sponsor the project with some 10 Gigabit hardware, I'm all ears.  :)

[Johannes Rudolph]

Hi Brian,

thanks for the explanation. A simple benchmark as simple as the json
one can definitely bring the infrastructure to its limits.

On Wed, May 8, 2013 at 7:29 PM, Brian Hauer <teona...@gmail.com> wrote:
> As a result, the frameworks that render fewer response headers end up
> slightly higher on the results for i7 JSON. We have a Github issue to
> normalize and specify a minimum set of required response headers.

Slightly related, wouldn't it make sense to add a reasonable set of
request headers as well (wrk `-H` option) because a) there's a
realistic set of headers which each real world request will contain
(like `User-Agent`, `Accept-Language`, `Accept`, and maybe `Cookies`)
and b) since the json benchmark should benchmark the baseline of the
platform it would make sense to put some pressure on the basic
functionality of the HTTP stacks in the benchmark.

From our perspective of building the spray http stack (for which we
will hopefully provide a benchmark soon as well) we've seen that
speedy processing of headers can be a vital ingredient to good
performance of the HTTP implementation.

--
Johannes

-----------------------------------------------
Johannes Rudolph
http://virtual-void.net

[Brian Hauer]

Hi Johannes,

That is a fantastic point about the request headers.  I will capture that as a Github issue.  It's unlikely we'll be able to revise the configuration for Round 5, but perhaps soon afterward.

Also, we're looking forward to seeing the Spray tests.

[Mathias Doenitz]

Brian,

I'd like to second the request for adding a set of realistic request headers, especially for the JSON test.

Apart from improving validity of the benchmark results for real-world applications it might also help with increasing the discriminatory "resolution" of the benchmark in the top (> 200K req/s) region, because requests/sec will go down with load being shifted a bit more onto the CPU and away from the network stack latency on your gigabit NICs. (I assume your NICs are being saturated with regard to packets sent per second and not yet with regard to bytes sent per second, so increasing packet size resulting in less packets sent per second somewhat eases the pressure.)

> It's unlikely we'll be able to revise the configuration for Round 5, but perhaps soon afterward.

Bummer that you won't be able to bring in this fix for Round 5.

Is there something we can do to help make it happen?

Cheers,

Mathias

[Brian Hauer]

Hi Mathias,

That's precisely right, the saturation appears to be related to packets and not bytes per second.  With larger payloads, higher throughput in bytes per second is observed, but no more requests per second can be processed.

Since there is significant interest for the request headers to be provided, we'll discuss this internally tomorrow and attempt to make it a priority.  The total work involved for this should be small since it should be isolated to the client side.

Normalizing the server's responses (a separate issue we have on the to-do list) will require review and changes to many test implementations.

[Andy]

Also, if anyone is willing to sponsor the project with some 10 Gigabit hardware, I'm all ears.  :)

If 10G NIC is not available, would it make sense to run the benchmark on localhost? That way you can remove the network bottleneck and measure the true upper limit of the frameworks' performance. 

[Skamander]

Weird, my original message get's not displayed.

@bhauer - have you considered link aggregation? I'm not a sysadmin, so I have no idea what I'm talkin about, but I've overheard serveral disuccions from my colleagues that it can give you around 3.5 times the throughput if you link 4 channels, if I recall corecctly.

[Brian Hauer]

Hi Andy,

Indeed, we've done spot testing running Wrk on the same host as the application server ("localhost").  That is what I was referring to earlier in my reply to Johannes:

https://groups.google.com/d/msg/framework-benchmarks/aYhXRD74zo0/zWreu6pJeA0J

The highest performance frameworks demonstrated a capacity to reach above even 300,000 requests per second when tested this way, but we have not posted that data.  Even though the numbers are higher, splitting CPU capacity between Wrk and the application server gives me pause.  It trades an easily-explained wall (gigabit Ethernet) for a more difficult to explain wall (we're splitting CPU resource).  I'd rather be able to show realistic 10 GBE tests.

Skamander has an interesting idea about link aggregation (multiple gigabit Ethernet connections that work in parallel).  Although that would be cheaper to set up than 10 GBE, we also don't have the hardware necessary to pull that off.  The i7 2600K machines that we are using for these tests are just our workstations that were purchased in early 2011.  These have plain magnetic platter disks and plain gigabit Ethernet cards.  They are not proper servers.

For the time being, I'd prefer to allocate energy elsewhere such as deeper tests.  That said, since I know this network saturation matter is interesting for readers, to-date I have (a) suggested that anyone who has access to superior hardware could run the tests on that hardware and send us the results.json file and (b) mostly in jest, I've fished for hardware, network, or hosting firms to sponsor and contribute some 10 GBE servers.  :)

[Mathias]

Brian,

> Since there is significant interest for the request headers to be provided,
> we'll discuss this internally tomorrow and attempt to make it a priority.
> The total work involved for this should be small since it should be
> isolated to the client side.

Excellent! We were assuming that this improvement shouldn't be too hard to put in.
We'd be happy to set up a pull-request with the patch if it's any help.

Also, just to provide a starting point, these are some of the headers we typically add to test requests comparing framework perf:

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*; q=0.8
Accept-Language: en-US,en; q=0.8
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu/12.10 Chromium/27.0.1453.6 Chrome/27.0.1453.6 Safari/537.36
Cookie: JSESSIONID.5202fd0b=aa670e931ae087dff6e88bd10e834a72; org.cups.sid=55621155fa7d08deaeff9ef8111ec82d

They reflect quite nicely what a user of a typical web application would be sending along with every browser request.
Obviously some of the headers vary wildly in real life (like the cookie header) but something along these lines probably makes sense. Even though we quite often see much larger cookie values it likely makes sense to not go overboard here.

Also, it's probably a good idea to not add things that might actually affect response creation significantly, like an `Accept-Encoding: gzip, deflate` header, which might cause some of the "benchmarkees" to actually compress the response whereas other might simply ignore the header, depending on configuration.

Thanks a lot for all your work on this benchmark!
It sure it becoming an increasingly helpful reference for a wide range of technology decisions surrounding HTTP stuff…

Cheers,
Mathias

---
mat...@spray.io
http://spray.io

> --
> You received this message because you are subscribed to the Google Groups "framework-benchmarks" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to framework-benchm...@googlegroups.com.
> Visit this group at http://groups.google.com/group/framework-benchmarks?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Brian Hauer]

Hi again Mathias,

An unrelated erroneous labeling of Scalatra as Laravel has motivated me to push the latest results web site to the server.  The latest version includes a few things, but related to this discussion, we added requirements for each test type:

http://www.techempower.com/benchmarks/#section=code

For Round 5, where necessary, we will configure Wrk to use the request headers we have specified in those requirements.  I didn't include Cookie, and we'll debate that a bit internally.  Presently, none of our tests simulate requests with associated user sessions.  We expect that the applications/frameworks are configured to support sessions if they were being used (otherwise we mark them as "stripped"), but no tests yet actually use sessions.

[Mathias]

Brian,

> For Round 5, where necessary, we will configure Wrk to use the request
> headers we have specified in those requirements.

Ok, thanks for implementing this improvement!

One question: Why are you using such a short User-Agent header?
As you can see from [this list][1] all popular (und unpopular) browsers produce *much* longer User-Agent header values.
Since every web server has to somehow deal with this part of the request IMHO it'd be good to have the benchmark use some more or less realistic value. (I think the only thing that matters is the length of the value, probably most servers do not actually look at the internal structure of this header (spray does and is probably an exception here)).

> I didn't include Cookie,
> and we'll debate that a bit internally. Presently, none of our tests
> simulate requests with associated user sessions. We expect that the

> applications/frameworks are configured to *support* sessions if they were

> being used (otherwise we mark them as "stripped"), but no tests yet
> actually use sessions.

Yes, I understand.
The point of adding a Cookie header is not so much to exercise user sessions. Since we are only testing single-request performance no one would expect session mgmt. to be a part of the benchmark.
The interesting thing about Cookie headers is that their value *is* usually parsed structurally by the server (as opposed to the User-Agent header, whose value will mostly be treated as an opaque string). The Cookie header value contains a list of key-value-pairs that servers usually parse into some kind of map structure. The efficiency of this internal parsing is something that is interesting for the JSON benchmark. In the top region of this benchmark (where benchmarkees reach several hundred thousand requests/second) these differences become important.
Since most non-trivial real-world web-applications do rely on cookies in one way or another one could argue that a good benchmark should cover server-side cookie parsing performance as well… no?

Cheers,
Mathias

[1]: http://www.useragentstring.com/pages/Browserlist/

---
mat...@spray.io
http://spray.io

On 09.05.2013, at 20:46, Brian Hauer <teona...@gmail.com> wrote:

> Hi again Mathias,
>
> An unrelated erroneous labeling of Scalatra as Laravel has motivated me to
> push the latest results web site to the server. The latest version
> includes a few things, but related to this discussion, we added
> requirements for each test type:
>
> http://www.techempower.com/benchmarks/#section=code
>
> For Round 5, where necessary, we will configure Wrk to use the request
> headers we have specified in those requirements. I didn't include Cookie,
> and we'll debate that a bit internally. Presently, none of our tests
> simulate requests with associated user sessions. We expect that the

> applications/frameworks are configured to *support* sessions if they were

>> mat...@spray.io <javascript:>
>> http://spray.io
>>
>> On 09.05.2013, at 02:09, Brian Hauer <teona...@gmail.com <javascript:>>

[Brian Hauer]

Hi Mathias,

I'll check, but off hand, I'm not certain if Wrk allows us to specify the User-Agent header.  When I put Example into the example requests, I literally meant that as a stand-in for whatever Wrk sends by default.  If it does allow us to configure a custom User-Agent, I have no problem with specifying something much longer.

You bring up a good point about Cookie parsing.  We'll discuss it over here and decide how to proceed.  I am of mixed opinion myself.  On one hand, I would like to craft a test that expressly uses a Cookie so that we can know that Cookies are being parsed and used explicitly.  On the other hand, I do like adding Cookies even when not used to demonstrate possible lazy-evaluation performance tuning frameworks might offer.  (Not using the Cookies?  Well, then, I won't bother parsing them.)

[Mathias]

Brian,

by default Wrk (v2.0.0 compiled under OS/X) doesn't send *any* User-Agent header.
If you don't specify one or more additional headers on the command line it will render only a single request header: the `Host` header.
That is why we think the modeling of more realistic test requests with explicit header additions is somewhat important.

> If it does allow us to configure a custom User-Agent, I have no problem with specifying something much longer.

Yes, apparently wrk has no special treatment for User-Agent headers at all. Whatever you tell it to render on the command line it will happily render.
You can even override the automatically added Host header value with the `-H` switch if you want.

> You bring up a good point about Cookie parsing. We'll discuss it over here
> and decide how to proceed. I am of mixed opinion myself. On one hand, I
> would like to craft a test that expressly uses a Cookie so that we can know
> that Cookies are being parsed and used explicitly. On the other hand, I do
> like adding Cookies even when not used to demonstrate possible
> lazy-evaluation performance tuning frameworks might offer. (Not using the
> Cookies? Well, then, I won't bother parsing them.)

Yes, right. Seeing how the server deals with cookies, even when the application is *not* using them, is definitely interesting.
All too often users carry around huge cookie values for ages. By how much users like these end up slowing down a site is something that a benchmark can rather easily cover.
The effect is likely to be negligible for all "slow" frameworks, but probably allows for better separation between the high-throughput servers, especially in the JSON test.

Cheers,
Mathias

---
mat...@spray.io
http://spray.io

[Mathias]

Andy,

the choice to *not* test on localhost is indeed a good one.
Servers do differ in how they treat "free CPU" on their side for optimizing performance, i.e. some will behave differently depending on whether there is some CPU utilization "breathing room" left or not. Putting the load-generating client onto the same CPU allows for all kinds of really difficult to analyze and understand mutual influencing effects.

It is also quite likely that the server-side steals CPU time from the client and prevents it from generating test requests fast enough, or measure response times properly.
Even when you confine the server and client side of things to dedicated cores (using OS process affinity) you still have interdependent effects occur on the kernel layer.

So, the best choice is indeed to generate load purely from the outside, which is also what happens in the real-world case.

Cheers,
Mathias

---
mat...@spray.io
http://spray.io

[Brian Hauer]

Hi Mathias,

We are presently planning to add a Cookie (or two or three) to the request headers but it will not use any platform's session identifier.  We do not want to deal with sessions yet.  The addition will add some bytes to the request payload and also the opportunity to illustrate the different behavior in frameworks' parsing of Cookies that are not used.

[Mathias]

Brian,

great!
Maybe my example was a bit misleading in that it appeared to be geared towards actual session usage (with its "JSESSIONID" cookie), even though we were just too lazy to construct something obviously "fake".
As you say, some random values like "screen-resolution=1900x1200" or the like will do just fine…

Already looking forward to the Round 5 results!

Cheers,
Mathias

---
mat...@spray.io
http://spray.io

On 10.05.2013, at 16:14, Brian Hauer <teona...@gmail.com> wrote:

> Hi Mathias,
>
> We are presently planning to add a Cookie (or two or three) to the request

> headers but it will *not* use any platform's session identifier. We do not

> want to deal with sessions yet. The addition will add some bytes to the
> request payload and also the opportunity to illustrate the different
> behavior in frameworks' parsing of Cookies that are not used.
>