Framework Benchmarks

[Patrick Leahy]

This post is in response to the Framework Benchmarks posted three weeks ago*. I have taken the time to download the code, set up a local copy, and profile the tests performed. The results I will characterize shortly, but first I would like to discuss the validity of the benchmarks themselves.

The benchmark code involved is not the best-performing code that can be written within the CakePHP framework. No tricks are employed; it is a simple and by-the-book example of JSON View. It is important to note that performance is not necessarily the highest asset to a given framework, nor for this one in particular: we like to imagine that flexibility and speed of development are of greater primacy. It should be obvious to state that in a production environment where performance is a concern, caching is a necessity, no matter the technology employed.

That said, the benchmark results are simply embarrassing.

So, I fired up xdebug, and a few things became apparent. The average request spends the largest amount of time in App::load(). Obviously reading files from disk is going to be slow under any circumstances, and the solution is equally obvious: load fewer files from disk. The devil is assuredly in the details. Other pain points include bootstrap, the Hash class, and event dispatching.

App::load()

Both Controllers and Models take a long time to load. Related: these are deeply inherited classes. JSON View and FileLog are also up on that list. Why FileLog is loaded at all is, to me, a good question. It would be nice to be able to reduce the amount of files that are loaded and processed, but most of these don't add significantly to the time taken. The inheritance issue is much more problematic.

Hash class

This is relatively straightforward and should not require any major changes to Cake. Hash::insert and Hash::_simpleOp are slow, and I suspect that the entire class is a bit slower than it could be.

Bootstrap

The average request spends 1/3 of its time in this file. The largest chunk of that time is spent in Configure::bootstrap(). I have no idea what that is doing, but it should probably stop doing that.

Event Dispatching

This to me is the most tragic component on this list. The Events system is so very useful, and so slow! The only thing that I can think of is to not dispatch events unless it is absolutely necessary -- somehow we need to default to not doing this.

The profiler files may be viewed with WinCacheGrind on Windows, KCacheGrind on Linux, and with Webgrind on any platform with a web server installed. I think that while these benchmarks are quite unpleasant, there must surely be a lot of low-hanging fruit. It would also be nice to develop some performance tests, so that things like this won't catch us completely by surprise.

--PL

* The benchmarks have been updated twice but are still just as ugly for us.

[mark_story]

Thanks for spending more time on this, I've looked a Configure::bootstrap() a few times, and have never really figured out why it was so slow, it is certainly worth another look.

FileLog is loaded because the default logging is connected. In 2.x loggers are loaded and constructed when config() is called, much like Cache engines. One option is to make them lazily load on the first log message written. I've already made this change for 2.x, and I don't *think* it would break existing compatibility, but it is hard to know without trying.

It might be worth looking at the event system and seeing if there is anything slow that can be made faster. José and I knew that having a standalone event system would be less efficient than the system it replaced, but the flexibility it affords seemed worth it to us.

-Mark

[Simon East]

Thanks for your work Patrick.  I saw those benchmarks a week or two ago and was also a bit disappointed.  I'm glad you took the time to investigate it.

Were you/they using Apache and mod_php?  Do you reckon the results would improve much if nginx, PHP-FPM and APC were used?  Or FastCGI rather than mod_php?  Caching the compiled scripts in memory (ie. APC) should surely help with the large amount of file-reads, yeah?

Simon East  |  Technical Lead  |  surfacedigital

Level 1, 60 Hardware Lane, Melbourne VIC 3000 Australia
Ph (03) 9670 7062   Skype simoneast_oz   Tw @surfacedigital   www.surfacedigital.com.au

--
You received this message because you are subscribed to the Google Groups "cakephp-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cakephp-core...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Patrick Leahy]

Yes, the benchmarks were performed using nginx and php5-fpm. I don't believe that this is relevant, and doubt that it represents a substantial improvement over apache/mod_php, since the tests were run one request at a time on an idle server. I don't think that mitigating the problem is an improvement over attempting to solve it.

[José Lorenzo]

Not sure if you have seen the updated bechmarks, they fixed a couple problems in CakePHP deployment and now it looks a bit better. In general major PHP frameworks are in the bottom of the list but in at this time Cake beats a few of them, including nodejs + mysql!

So I'll try to share my own experience doing profiling in CakePHP:

App::load()

The bottleneck is actually disk I/O, file parsing and interpreting for each of the loaded classes. The function itself can hardly be faster in production environments, as the location for each class is cached so it is always a single operation to fetch the path and include it. Solution: Install APC and set apc.stat = 0 in the production config. Files will stay in memory already compiled to opcodes. This can dramatically reduce response time. Another solution not involving APC might be tray to reduce even more the amount of files required by default.

Hash class and Event Dispatching
When you look at profiling data, you realize that both functions' problem is not that they are slow per tse, but that they are called too often. The solution would be to figure out a way to call those routines less. Hash is called over and over by Configure, maybe encourage people to just have a giant configure array in core.php and bootstrap.php?. As for Event dispatching the solution could be to no attach listeners if they are actually not interested in listening, a good example of this are Helpers, I have rarely made a helper using the callbacks, yet they are fired for each one of the helpers you have loaded.

Bootstrap

This one is a frustrating point for me, as Mark said, we have looked several times at the code and with the current architecture still cannot figure out a way how to make it faster. Maybe we need more eyes to help?

Thanks for bringing this up!

[mark_story]

I once had a crazy idea of attempting to figure out if a helper had actually declared any of its callback methods using reflection and figuring out where the method was defined. I never went through with it as I figured the reflection would cost as much as just firing the events. I could be wrong though.

Alternatively Helper + Component could omit the callback methods entirely. The collections could conditionally bind events based on the presence of methods. This would help reduce the number of callback methods without the potentially cost of reflection.

-Mark

[Andy Hobbs]

Could you also just register the callbacks using the event manager in the helper/component constructor or during initialization?

[mark_story]

You could, but that would require developers to either manually register the events they are interested in or use a bit of reflection to register callbacks. If I have time I might see how the convention/reflection based approach would work and if there is any performance benefit.  Anyone else is more than welcome to try this as well, since we have a good discussion started :)

-Mark

[Andy Hobbs]

Having to manually register callbacks would suit me down to a T, but I realise that makes it more complicated especially for developers new to the framework.

One other thought I've had is...

The problem has been highlighted mainly by benchmarking or single requests in an unoptimised hosting environment. The really good thing about cakephp is that it is really easy to get going with. I started using it as a perl developer with little php experience (i.e. before using cake I last touched php when php3 was just appearing). It is also really good from the point of view of an advanced user because it has a lot of very nice features and it is relatively easy to add to or override all these features. 

So, for a basic user like someone getting to grips with the framework or someone writing a small scale website where performance is not super critical the current setup is really just fine. For more advanced users what's needed is the ability to tune the framework to perform well in potentially different ways based on requirements. 

Personally I'd rather define event handlers that are actually required to save on any possible overhead of the framework trying to work out what I want to do.  So if the developer friendly features like using reflection to find callbacks could be disabled in the config that would be great. I would have thought that the expense of checking a config option is relatively small compared to using reflection to check for something that isn't there. I will try and make time this week to do some testing around this.

Also. I've been using APC to improve response times with good results and I am now looking into hiphop to see if facebook's latest incarnation has any further improvement to offer. These solutions obviously help massively with the app start up time but do nothing to improve performance around events/callbacks.

I think my point is that time could be well spent helping more advanced users of the framework tune it for performance gains as the out of the box speed is good enough for a lot of users. But maybe I'm just being a bit selfish there too!

Andy

[mark_story]

If reflection was used to register callbacks was used it would be a separate method that could easily be overridden in a subclass. It might make sense to make Helpers direct event listeners, instead of the HelperCollection. This has the drawback of making helpers unlike other implementations of ObjectCollection.  But might be the shortest route to improving performance.

I think we need to balance ease of use, advanced user tuning and backwards compatibility when discussing these kinds of things. All 3 are very important aspects of a framework and focusing only on one can make the other aspects more unpleasant.

If a break in compatibility would gain large performance gains it might be worth looking at how big of a break is required and how large it will impact end developers.

-Mark

[ADmad]

In earlier cake versions all callbacks dispatching was done through Object::dispatchMethod() to avoid using call_user_func* for upto 5 params. Perhaps having the EventManager do the same might improve things a bit.