Benchmarking Performance
72 views
Skip to first unread message
KevBurnsJr
Aug 9, 2009, 6:04:07 AM8/9/09
to Recess PHP Framework
So Josh Davey (creator of Madeam) did some "hello world" benchmarking
on a range of PHP frameworks the other day.
http://spreadsheets.google.com/pub?key=t30KFl5whumjYn25Sr2ElKQ&oid=1&output=image
Naturally I figured, "gee, why isn't Recess! on here?"
So I asked him and he said he had to omit Recess! from the test
because it was scoring less than 2 requests per second. Which is
weird. So I forked his test suite and started playing around.
http://github.com/joshdavey/framark/tree/master
I did a bunch of fiddling, putting the app in Production mode and
making sure Apc and Memcached were working properly, but the request
times didn't get any better. In fact, they're absurd. On my server,
every request from ApacheBench is taking 15 seconds to process.
Because the requests are being issued and processed on my server's
localhost, I haven't had a chance to take a good look at the requests
being issued by ApacheBench, but my server's access logs do look a
little suspicious.
Long story short, Recess! is choking on ApacheBench and it's making
the framework look slow.
http://httpd.apache.org/docs/2.0/programs/ab.html
Kris and Josh, what do you use for benchmarking?
on a range of PHP frameworks the other day.
http://spreadsheets.google.com/pub?key=t30KFl5whumjYn25Sr2ElKQ&oid=1&output=image
Naturally I figured, "gee, why isn't Recess! on here?"
So I asked him and he said he had to omit Recess! from the test
because it was scoring less than 2 requests per second. Which is
weird. So I forked his test suite and started playing around.
http://github.com/joshdavey/framark/tree/master
I did a bunch of fiddling, putting the app in Production mode and
making sure Apc and Memcached were working properly, but the request
times didn't get any better. In fact, they're absurd. On my server,
every request from ApacheBench is taking 15 seconds to process.
Because the requests are being issued and processed on my server's
localhost, I haven't had a chance to take a good look at the requests
being issued by ApacheBench, but my server's access logs do look a
little suspicious.
Long story short, Recess! is choking on ApacheBench and it's making
the framework look slow.
http://httpd.apache.org/docs/2.0/programs/ab.html
Kris and Josh, what do you use for benchmarking?
Kris Jordan
Aug 9, 2009, 12:44:38 PM8/9/09
to recess-f...@googlegroups.com
Still haven't been able to identify the source of the slowness with Apache's ab tool. I've seen the numbers you're seeing Kev. When using httperf with production & APC the benchmark numbers are indicative of how they actually feel when using an application from the browser. I haven't installed this benchmark suite on the slice, may try that soon, so I don't have a frame of reference but I am suspecting somewhere between Zend/Cake and Kohana/CI.
For a 'hello world' benchmark the motivation is that you're supposed to get a sense of the critical path of a framework. "How long does it take for a request to an app with one route to a pass-thru controller and a view that prints hello world?" Rasmus first popularized this benchmark a couple years back. The numbers were most interesting then because frameworks hadn't yet been built specifically to kill 'hello world'.
A Digression on How Doo Wins the 'Hello World' Benchmark
====================================
I don't believe this benchmark is irrelevant, but do understand what this benchmark is measuring and how a framework like Doo achieves 8x performance over Zend. I'm going to pick on one aspect of Doo's implementation real quick because it demonstrates why optimizing for (and winning) a 'hello world' benchmark can be naive and dangerous.
Let's focus on routing. Three quick references:
* Hello World app routes.php: http://github.com/joshdavey/framark/blob/a6c9033fc2273abab4195729c2d8be7d3910e298/projects/doo-1.0/app/protected/config/routes.conf.php
* Routing Documentation: http://doophp.com/doc/guide/basic/routes
* Router Implementation: http://github.com/joshdavey/framark/blob/a6c9033fc2273abab4195729c2d8be7d3910e298/projects/doo-1.0/dooframework/uri/DooUriRouter.php
Routes in Doo are a multidimensional PHP array of arrays. The first dimension is 'http method', the second is 'uri', i.e. $route['*']['/'] means all methods for URI /. Similarly $route['get']['/person/:id'] means GETs to URI /person/$id where ID is variable. When programming in Doo you are authoring these routes in one PHP file in one big array.
Now, looking at lines 231 through 320 of the DooUriRouter we can see how a route is chosen. If there is a static match on the URI, meaning no ':dynamic/:parts', a match is immediately found and returned in lines 231 through 240. This is the path the hello world benchmark takes, simple, fast, immediate. In a real application, where there are paths with dynamic parts, every single route in the app (even static ones) is iterated through, exploded, and compared. If more than one match in the exploded array size is found (i.e., if you used a convention like: '/person/:id', '/car/:id', these would both be initial hits for a request on '/foo/bar') then all matches are iterated through again and regex/string comparisons occur on their parts. This leads to a runtime of O(# routes) + O(# matching routes with/same/part/count).
What's the big deal? As your Doo app grows in complexity your performance degrades linear to the number of routes you have (and 2x linear if your routes are mostly of/similar/depth). For a trivial app with 5 routes this means the routing code is at least 5x-10x slower than routing in hello world. For a non-trivial app with 50 routes Doo's routing code becomes 50-100x slower.
Compare this to Recess where routes are specified in annotations on controllers. In production routes wind up cached in a serialized tree. Recess takes a performance hit by having to retrieve routes from cache (hopefully APC/memcache/memory) and deserialize. From there Recess does a recursive descent matching each part of the route. So the runtime complexity is based on the depth of the request url, i.e. this/route/request will be resolved in 3 steps (or less, if there isn't a match). Notice the ammount of PHP run to decide a route does not depend on the number of routes, but rather the depth of a URL.
So in a Hello World benchmark Recess' routing will always be slower than Doo's with the additional overhead of uncaching/unserializing. However, as the # of routes in your application's grows Recess' routing performance will remain roughly the same while Doo's degrades by 1-2x with every additional route.
Wow, that was a bigger digression than I was hoping for. Point being: techniques to be the fastest framework with 'hello world' don't imply techniques for being a fastest framework in the 'real world' :)
Back to Kev's Email
============
I will make another push to look into the horrible AB performance with this next iteration of Recess for 5.3. Hello world perf is looking competitive (not on Doo's scale) with httperf and there are some things in the critical path (like loading a PDO object for your databases) that should be lazy and could improve it even further. If anyone else loves perf and would like to help profile/optimize shoot me an email.
At the end of the day, though, it should be clear that the #1 'benchmark' Recess is focused on is the programmability model and internal architecture. So far we've done a great job on them both without sacrificing performance. In fact, we're incredibly well with performance given the annotations/meta-programming model we've pushed forward.
I wouldn't lose sleep at night because we aren't winning the 'Hello World' benchmark race. I *would* lose sleep at night if our framework required central configuration in large PHP arrays or if the performance of Recess applications degraded significantly as they became more complex. What I am losing sleep over right now is ensuring we've got the best foundation possible for PHP 5.3 and beyond. The promise of PHP 5.3, with features like namespaces, late static binding, and closures, is that Recess 5.3 will be way faster as namespaces replace the Library and have a better programmability model.
That's all, for now. Hope everyone has had a great weekend!
-Kris
For a 'hello world' benchmark the motivation is that you're supposed to get a sense of the critical path of a framework. "How long does it take for a request to an app with one route to a pass-thru controller and a view that prints hello world?" Rasmus first popularized this benchmark a couple years back. The numbers were most interesting then because frameworks hadn't yet been built specifically to kill 'hello world'.
A Digression on How Doo Wins the 'Hello World' Benchmark
====================================
I don't believe this benchmark is irrelevant, but do understand what this benchmark is measuring and how a framework like Doo achieves 8x performance over Zend. I'm going to pick on one aspect of Doo's implementation real quick because it demonstrates why optimizing for (and winning) a 'hello world' benchmark can be naive and dangerous.
Let's focus on routing. Three quick references:
* Hello World app routes.php: http://github.com/joshdavey/framark/blob/a6c9033fc2273abab4195729c2d8be7d3910e298/projects/doo-1.0/app/protected/config/routes.conf.php
* Routing Documentation: http://doophp.com/doc/guide/basic/routes
* Router Implementation: http://github.com/joshdavey/framark/blob/a6c9033fc2273abab4195729c2d8be7d3910e298/projects/doo-1.0/dooframework/uri/DooUriRouter.php
Routes in Doo are a multidimensional PHP array of arrays. The first dimension is 'http method', the second is 'uri', i.e. $route['*']['/'] means all methods for URI /. Similarly $route['get']['/person/:id'] means GETs to URI /person/$id where ID is variable. When programming in Doo you are authoring these routes in one PHP file in one big array.
Now, looking at lines 231 through 320 of the DooUriRouter we can see how a route is chosen. If there is a static match on the URI, meaning no ':dynamic/:parts', a match is immediately found and returned in lines 231 through 240. This is the path the hello world benchmark takes, simple, fast, immediate. In a real application, where there are paths with dynamic parts, every single route in the app (even static ones) is iterated through, exploded, and compared. If more than one match in the exploded array size is found (i.e., if you used a convention like: '/person/:id', '/car/:id', these would both be initial hits for a request on '/foo/bar') then all matches are iterated through again and regex/string comparisons occur on their parts. This leads to a runtime of O(# routes) + O(# matching routes with/same/part/count).
What's the big deal? As your Doo app grows in complexity your performance degrades linear to the number of routes you have (and 2x linear if your routes are mostly of/similar/depth). For a trivial app with 5 routes this means the routing code is at least 5x-10x slower than routing in hello world. For a non-trivial app with 50 routes Doo's routing code becomes 50-100x slower.
Compare this to Recess where routes are specified in annotations on controllers. In production routes wind up cached in a serialized tree. Recess takes a performance hit by having to retrieve routes from cache (hopefully APC/memcache/memory) and deserialize. From there Recess does a recursive descent matching each part of the route. So the runtime complexity is based on the depth of the request url, i.e. this/route/request will be resolved in 3 steps (or less, if there isn't a match). Notice the ammount of PHP run to decide a route does not depend on the number of routes, but rather the depth of a URL.
So in a Hello World benchmark Recess' routing will always be slower than Doo's with the additional overhead of uncaching/unserializing. However, as the # of routes in your application's grows Recess' routing performance will remain roughly the same while Doo's degrades by 1-2x with every additional route.
Wow, that was a bigger digression than I was hoping for. Point being: techniques to be the fastest framework with 'hello world' don't imply techniques for being a fastest framework in the 'real world' :)
Back to Kev's Email
============
I will make another push to look into the horrible AB performance with this next iteration of Recess for 5.3. Hello world perf is looking competitive (not on Doo's scale) with httperf and there are some things in the critical path (like loading a PDO object for your databases) that should be lazy and could improve it even further. If anyone else loves perf and would like to help profile/optimize shoot me an email.
At the end of the day, though, it should be clear that the #1 'benchmark' Recess is focused on is the programmability model and internal architecture. So far we've done a great job on them both without sacrificing performance. In fact, we're incredibly well with performance given the annotations/meta-programming model we've pushed forward.
I wouldn't lose sleep at night because we aren't winning the 'Hello World' benchmark race. I *would* lose sleep at night if our framework required central configuration in large PHP arrays or if the performance of Recess applications degraded significantly as they became more complex. What I am losing sleep over right now is ensuring we've got the best foundation possible for PHP 5.3 and beyond. The promise of PHP 5.3, with features like namespaces, late static binding, and closures, is that Recess 5.3 will be way faster as namespaces replace the Library and have a better programmability model.
That's all, for now. Hope everyone has had a great weekend!
-Kris
KevBurnsJr
Aug 9, 2009, 4:16:55 PM8/9/09
to Recess PHP Framework
Looks like ApacheBench is a wrapper for httperf, and httperf gives
more sane results (60-130 req/s).
I've forked Josh's test suite, updated it to use Recess 0.2 (was using
0.12) in PRODUCTION mode with Apc and Memcache enabled, and added a
script to run benchmarks using httperf rather than ApacheBench.
http://github.com/KevBurnsJr/framark
New benchmark on my wimpy slice (running PHP 5.2.4) is reporting
25 req/s in DEVELOPMENT mode
60 req/s in PRODUCTION mode
110 req/s in PRODUCTION mode with Memcache enabled
130 req/s in PRODUCTION mode with Memcache and Apc enabled
Still considerably less than Doo's 568 req/s but at least it's a more
fair comparison.
- --- -
Hints on ApacheBench caveats:
"The way apache bench works seems fishy to me - it uses HTTP/1.0 but
with a Connection header which doesn't seem correct to me [...] Plus
it doesn't appear to support chunked transfer encoding, so your test
has to ensure that Content-Length is used on the response."
http://fandev.org/sidewalk/topic/663
Cheers,
- Kev
On Aug 9, 9:44 am, Kris Jordan <krisjor...@gmail.com> wrote:
> Still haven't been able to identify the source of the slowness with Apache's
> ab tool. I've seen the numbers you're seeing Kev. When using httperf with
> production & APC the benchmark numbers are indicative of how they actually
> feel when using an application from the browser. I haven't installed this
> benchmark suite on the slice, may try that soon, so I don't have a frame of
> reference but I am suspecting somewhere between Zend/Cake and Kohana/CI.
>
> For a 'hello world' benchmark the motivation is that you're supposed to get
> a sense of the critical path of a framework. "How long does it take for a
> request to an app with one route to a pass-thru controller and a view that
> prints hello world?" Rasmus first popularized this benchmark a couple years
> back. The numbers were most interesting then because frameworks hadn't yet
> been built specifically to kill 'hello world'.
>
> A Digression on How Doo Wins the 'Hello World' Benchmark
> ====================================
> I don't believe this benchmark is irrelevant, but do understand what this
> benchmark is measuring *and* how a framework like Doo achieves 8x
> way faster as namespaces replace the Library *and* have a better
more sane results (60-130 req/s).
I've forked Josh's test suite, updated it to use Recess 0.2 (was using
0.12) in PRODUCTION mode with Apc and Memcache enabled, and added a
script to run benchmarks using httperf rather than ApacheBench.
http://github.com/KevBurnsJr/framark
New benchmark on my wimpy slice (running PHP 5.2.4) is reporting
25 req/s in DEVELOPMENT mode
60 req/s in PRODUCTION mode
110 req/s in PRODUCTION mode with Memcache enabled
130 req/s in PRODUCTION mode with Memcache and Apc enabled
Still considerably less than Doo's 568 req/s but at least it's a more
fair comparison.
- --- -
Hints on ApacheBench caveats:
"The way apache bench works seems fishy to me - it uses HTTP/1.0 but
with a Connection header which doesn't seem correct to me [...] Plus
it doesn't appear to support chunked transfer encoding, so your test
has to ensure that Content-Length is used on the response."
http://fandev.org/sidewalk/topic/663
Cheers,
- Kev
On Aug 9, 9:44 am, Kris Jordan <krisjor...@gmail.com> wrote:
> Still haven't been able to identify the source of the slowness with Apache's
> ab tool. I've seen the numbers you're seeing Kev. When using httperf with
> production & APC the benchmark numbers are indicative of how they actually
> feel when using an application from the browser. I haven't installed this
> benchmark suite on the slice, may try that soon, so I don't have a frame of
> reference but I am suspecting somewhere between Zend/Cake and Kohana/CI.
>
> For a 'hello world' benchmark the motivation is that you're supposed to get
> a sense of the critical path of a framework. "How long does it take for a
> request to an app with one route to a pass-thru controller and a view that
> prints hello world?" Rasmus first popularized this benchmark a couple years
> back. The numbers were most interesting then because frameworks hadn't yet
> been built specifically to kill 'hello world'.
>
> A Digression on How Doo Wins the 'Hello World' Benchmark
> ====================================
> I don't believe this benchmark is irrelevant, but do understand what this
> performance over Zend. I'm going to pick on one aspect of Doo's
> implementation real quick because it demonstrates why optimizing for (and
> winning) a 'hello world' benchmark can be naive and dangerous.
>
> Let's focus on routing. Three quick references:
> * Hello World app routes.php:http://github.com/joshdavey/framark/blob/a6c9033fc2273abab4195729c2d8...
> implementation real quick because it demonstrates why optimizing for (and
> winning) a 'hello world' benchmark can be naive and dangerous.
>
> Let's focus on routing. Three quick references:
> * Routing Documentation:http://doophp.com/doc/guide/basic/routes
> * Router Implementation:http://github.com/joshdavey/framark/blob/a6c9033fc2273abab4195729c2d8...
> programmability model.
>
> That's all, for now. Hope everyone has had a great weekend!
>
> -Kris
>
>
>
> On Sun, Aug 9, 2009 at 6:04 AM, KevBurnsJr <kevburn...@gmail.com> wrote:
>
> > So Josh Davey (creator of Madeam) did some "hello world" benchmarking
> > on a range of PHP frameworks the other day.
>
> >http://spreadsheets.google.com/pub?key=t30KFl5whumjYn25Sr2ElKQ&oid=1&...
>
> That's all, for now. Hope everyone has had a great weekend!
>
> -Kris
>
>
>
> On Sun, Aug 9, 2009 at 6:04 AM, KevBurnsJr <kevburn...@gmail.com> wrote:
>
> > So Josh Davey (creator of Madeam) did some "hello world" benchmarking
> > on a range of PHP frameworks the other day.
>
Kris Jordan
Aug 9, 2009, 10:13:04 PM8/9/09
to recess-f...@googlegroups.com
Kev,
Thanks for doing the legwork to get this running and some feedback on the numbers. You could do a little better turning the view provider to 'native' view instead of 'layouts' on the welcome controller. Additionally, if you were to name the route something like 'hello-world' you could reference 'http://host/recess-tests-dir/hello-world.html' to bypass the parsing of whatever httperf sends as its Accept header.
I'll try to push a change to make loading PDO lazy soon and turn-off content-negotiation by default (meant to do this anyway because of WebKit's poor accept header which prioritizes XML over HTML). Those two switches should provide another improvement.
-Kris
Thanks for doing the legwork to get this running and some feedback on the numbers. You could do a little better turning the view provider to 'native' view instead of 'layouts' on the welcome controller. Additionally, if you were to name the route something like 'hello-world' you could reference 'http://host/recess-tests-dir/hello-world.html' to bypass the parsing of whatever httperf sends as its Accept header.
I'll try to push a change to make loading PDO lazy soon and turn-off content-negotiation by default (meant to do this anyway because of WebKit's poor accept header which prioritizes XML over HTML). Those two switches should provide another improvement.
-Kris
joshdavey
Aug 9, 2009, 10:29:21 PM8/9/09
to Recess PHP Framework
Hey guys,
Thanks for contributing back to the framark. Glad we got some better
results showing for Recess. I didn't want to publish them before
because I knew they couldn't be right but I made sure to include
Recess in the suite anyway hoping someone would fix it. (And you
did!). I'm excited to see some of the discussion about performance
starting up around this topic. Already there is some talk in the
Kohana forums: http://forum.kohanaphp.com/comments.php?DiscussionID=3073&page=2.
I hope some new perspectives and overall improvements to all the
frameworks comes out of it.
Kris, those are some great discussion points regarding why DooPHP is
as fast as it is. All things considered speed is great (very very
important). But at one point in every project its not just about
writing fast code but writing clean and organized code. I think as
framework developers this is the thing we try to balance every day as
we design our APIs.
I really enjoy a lot of the ideas coming out of your project and look
forward to seeing it grow.
Cheers,
Joshua Davey
Author of Madeam PHP Framework
On Aug 9, 10:13 pm, Kris Jordan <krisjor...@gmail.com> wrote:
> Kev,
>
> Thanks for doing the legwork to get this running and some feedback on the
> numbers. You could do a little better turning the view provider to 'native'
> view instead of 'layouts' on the welcome controller. Additionally, if you
> were to name the route something like 'hello-world' you could reference 'http://host/recess-tests-dir/hello-world.html'to bypass the parsing of
> whatever httperf sends as its Accept header.
>
> I'll try to push a change to make loading PDO lazy soon and turn-off
> content-negotiation by default (meant to do this anyway because of WebKit's
> poor accept header which prioritizes XML over HTML). Those two switches
> should provide another improvement.
>
> -Kris
>
Thanks for contributing back to the framark. Glad we got some better
results showing for Recess. I didn't want to publish them before
because I knew they couldn't be right but I made sure to include
Recess in the suite anyway hoping someone would fix it. (And you
did!). I'm excited to see some of the discussion about performance
starting up around this topic. Already there is some talk in the
Kohana forums: http://forum.kohanaphp.com/comments.php?DiscussionID=3073&page=2.
I hope some new perspectives and overall improvements to all the
frameworks comes out of it.
Kris, those are some great discussion points regarding why DooPHP is
as fast as it is. All things considered speed is great (very very
important). But at one point in every project its not just about
writing fast code but writing clean and organized code. I think as
framework developers this is the thing we try to balance every day as
we design our APIs.
I really enjoy a lot of the ideas coming out of your project and look
forward to seeing it grow.
Cheers,
Joshua Davey
Author of Madeam PHP Framework
On Aug 9, 10:13 pm, Kris Jordan <krisjor...@gmail.com> wrote:
> Kev,
>
> Thanks for doing the legwork to get this running and some feedback on the
> numbers. You could do a little better turning the view provider to 'native'
> view instead of 'layouts' on the welcome controller. Additionally, if you
> were to name the route something like 'hello-world' you could reference 'http://host/recess-tests-dir/hello-world.html'to bypass the parsing of
> whatever httperf sends as its Accept header.
>
> I'll try to push a change to make loading PDO lazy soon and turn-off
> content-negotiation by default (meant to do this anyway because of WebKit's
> poor accept header which prioritizes XML over HTML). Those two switches
> should provide another improvement.
>
> -Kris
>
Kris Jordan
Aug 9, 2009, 11:06:03 PM8/9/09
to recess-f...@googlegroups.com
Josh,
The benchmark suite is great and a step in the right direction for being able to have quantitative comparison of frameworks.
What I would love to see as another benchmark is one that pushes on the routing capabilities of each framework. Let's choose a real-world application API (say, Basecamp's http://developer.37signals.com/basecamp/index.shtml) and recreate the application/configuration structure needed to accept the Basecamp API URIs/routes. Nothing real happens behind the scenes, vanilla controllers with one method per route leading to simple views returned like in Hello World.
Possible benchmarks:
* Hit every route in the application N times
* Hit the longest route (by number/of/url/parts) N times
* Hit a common length dynamic route N times
* Hit a DELETE route N times
* Hit the last route specified N times
This would give us a solid sample of routes from a real world app and should more accurately represent the real 'bootstrap' time in each framework. I'm willing to bet beer that this shakes up the playing field in interesting ways. Josh, I can help put together Basecamp's route list & a Recess app implementation if you're interested. Without having run your framark I'm not sure if there's an obvious way to introduce new benchmarks & apps?
Kris
The benchmark suite is great and a step in the right direction for being able to have quantitative comparison of frameworks.
What I would love to see as another benchmark is one that pushes on the routing capabilities of each framework. Let's choose a real-world application API (say, Basecamp's http://developer.37signals.com/basecamp/index.shtml) and recreate the application/configuration structure needed to accept the Basecamp API URIs/routes. Nothing real happens behind the scenes, vanilla controllers with one method per route leading to simple views returned like in Hello World.
Possible benchmarks:
* Hit every route in the application N times
* Hit the longest route (by number/of/url/parts) N times
* Hit a common length dynamic route N times
* Hit a DELETE route N times
* Hit the last route specified N times
This would give us a solid sample of routes from a real world app and should more accurately represent the real 'bootstrap' time in each framework. I'm willing to bet beer that this shakes up the playing field in interesting ways. Josh, I can help put together Basecamp's route list & a Recess app implementation if you're interested. Without having run your framark I'm not sure if there's an obvious way to introduce new benchmarks & apps?
Kris
Reply all
Reply to author
Forward
0 new messages