Benchmark testing should be done with non-trivial code, and given these frameworks abstract databases, and that abstraction can affect performance, it should probably include tests that include database access.
> The difference I did was putting the output in view instead of in
> controller. And the tool I used is siege with following command:
> siege -c 100 -r 3
> With the 'Transaction rate' result, I also make a bar chart with
> OpenOffice.org.
> The good news is CakePHP 1.2 is faster than Symfony 1.2.1 on my
> computer. ;)
> ps. I didn't use APC, just comparing the execution time in one simple
> request.
> Benchmark testing should be done with non-trivial code, and given these
> frameworks abstract databases, and that abstraction can affect
> performance, it should probably include tests that include database access.
> > The difference I did was putting the output in view instead of in
> > controller. And the tool I used is siege with following command:
> > siege -c 100 -r 3
> > With the 'Transaction rate' result, I also make a bar chart with
> > OpenOffice.org.
> > The good news is CakePHP 1.2 is faster than Symfony 1.2.1 on my
> > computer. ;)
> > ps. I didn't use APC, just comparing the execution time in one simple
> > request.
Your benchmarks are still "Hello World" examples, from what I can see there.
Great as a benchmark for those using static only pages with their
frameworks, but I suggest that the majority of developers out there are
doing something more complex.
kiang wrote:
> You could have a look at the result I did. ;)
> ---
> kiang
> On 1月2日, 上午6時37分, Graham Weldon <gra...@grahamweldon.com> wrote:
>> I won't say I have looked extensively through the code provided for all
>> frameworks, but the CakePHP 1.2 BenchmarkController is far from thorough.
>> Its bypassing view parsing, by putting a die() call in the controller
>> action:
>> Benchmark testing should be done with non-trivial code, and given these
>> frameworks abstract databases, and that abstraction can affect
>> performance, it should probably include tests that include database access.
>>> The difference I did was putting the output in view instead of in
>>> controller. And the tool I used is siege with following command:
>>> siege -c 100 -r 3
>>> With the 'Transaction rate' result, I also make a bar chart with
>>> OpenOffice.org.
>>> The good news is CakePHP 1.2 is faster than Symfony 1.2.1 on my
>>> computer. ;)
>>> ps. I didn't use APC, just comparing the execution time in one simple
>>> request.
> Your benchmarks are still "Hello World" examples, from what I can see there.
> Great as a benchmark for those using static only pages with their
> frameworks, but I suggest that the majority of developers out there are
> doing something more complex.
> Cheers,
> Graham Weldon
> kiang wrote:
> > You could have a look at the result I did. ;)
> > ---
> > kiang
> > On 1月2日, 上午6時37分, Graham Weldon <gra...@grahamweldon.com> wrote:
> >> I won't say I have looked extensively through the code provided for all
> >> frameworks, but the CakePHP 1.2 BenchmarkController is far from thorough.
> >> Its bypassing view parsing, by putting a die() call in the controller
> >> action:
> >> Benchmark testing should be done with non-trivial code, and given these
> >> frameworks abstract databases, and that abstraction can affect
> >> performance, it should probably include tests that include database access.
> >>> The difference I did was putting the output in view instead of in
> >>> controller. And the tool I used is siege with following command:
> >>> siege -c 100 -r 3
> >>> With the 'Transaction rate' result, I also make a bar chart with
> >>> OpenOffice.org.
> >>> The good news is CakePHP 1.2 is faster than Symfony 1.2.1 on my
> >>> computer. ;)
> >>> ps. I didn't use APC, just comparing the execution time in one simple
> >>> request.
> Benchmark testing should be done with non-trivial code, and given these
> frameworks abstract databases, and that abstraction can affect
> performance, it should probably include tests that include database access.
I agree.
We don't know if the tester tried a business application, then saw his
framework performed poorly and then changed the terms of the test so
that his framework would perform best.
A business application such as a blog would best represent the
performance people are looking for.
Also, with the cost of hardware dropping everyday. A framework
designed purely for performance will become outdated as other more
complex "slower" frameworks offer better features and return on
development costs.
A well designed framework no matter how slow its internals can be
improved without affecting the functionality of the application the
framework is driving. Something that the Cake team can address with
maintenance releases to address performance.
With all that said, I do believe Cake is one of the slower frameworks.
The extensive use of for/loops during dispatching and rendering of
views has to impact performance, but the level of performance impacted
depends on the implementation of the website.
Cake has a lot of things going for it, and I mean A LOT that the other
PHP frameworks don't have. However, I disagree that performance
doesn't matter much. A lot of us (especially in this economy) are
downsizing our servers which is one of the main reason to even look at
a PHP frameworks over something like Rails. Setting up a Mongrel
cloud and the CPU/Memory to back it up on a host can be an expensive
proposition whereas PHP interacts with Apache directly and PHP in
general is widely supported among all hosting companies. Coming from
a Ruby on Rails background, this was my main draw to developing in the
CakePHP framework; speed and savings due to decreased need for
hardware plus added benefits of not having to re-learn techniques I've
learned from years of ROR developing.
Don't get me wrong, I'm very impressed with the speed of CakePHP and
the framework itself, but it is a little scary to see the performance
nearly cut in half going from 1.1 to 1.2 in each of those performance
tests. I hope this will not be a trend with every release.
On Jan 2, 7:43 am, Mathew <nfoscar...@yahoo.com> wrote:
> Also, with the cost of hardware dropping everyday. A framework
> designed purely for performance will become outdated as other more
> complex "slower" frameworks offer better features and return on
> development costs.
> A well designed framework no matter how slow its internals can be
> improved without affecting the functionality of the application the
> framework is driving. Something that the Cake team can address with
> maintenance releases to address performance.
> With all that said, I do believe Cake is one of the slower frameworks.
> The extensive use of for/loops during dispatching and rendering of
> views has to impact performance, but the level of performance impacted
> depends on the implementation of the website.
If your primary goal is to have a FAST webpage serving images and
multimedia content, you build static HTML with maybe some PHP using
lighttpd.
If you are building a social network/heavy interaction app (think
friendfeed/facebook) you either build the framework yourself or spend
equally lots of time tweaking a framweork like cake or zend. Or do it
like google, where the google.com frontend/search dispatchers are in
low-level C++.
I think Cake is a fast, stable and mature framework compared to a lot
of the other offerings. However, the main reason for most people for
chosing cake is fast coding and deployment. Many of the cake apps are
business applications where modeling of business logic, forms and
views is important, and thus the speed of doing this is much more
important than the 0.2 or 0.6 processing time of the finished app.
(This is guessing, but I belive many cake apps are internal business
apps of which few are shown to the public or on the "who's using cake"
page on cakephp.org).
For most people making such apps the speed of the dispatcher, template
engine etc is unimportant. The parts slowing the app would be heavy
SQL, SOAP/XML-RPC-request to other apps or huge batch ETLs. For most
people (including most public websites) the most important thing would
be faster SQL/active records and a better implementation of HABTM
joins, not the ability to serve x or y pages per sec.
> Cake has a lot of things going for it, and I mean A LOT that the other
> PHP frameworks don't have. However, I disagree that performance
> doesn't matter much. A lot of us (especially in this economy) are
> downsizing our servers which is one of the main reason to even look at
> a PHP frameworks over something like Rails. Setting up a Mongrel
> cloud and the CPU/Memory to back it up on a host can be an expensive
> proposition whereas PHP interacts with Apache directly and PHP in
> general is widely supported among all hosting companies. Coming from
> a Ruby on Rails background, this was my main draw to developing in the
> CakePHP framework; speed and savings due to decreased need for
> hardware plus added benefits of not having to re-learn techniques I've
> learned from years of ROR developing.
> Don't get me wrong, I'm very impressed with the speed of CakePHP and
> the framework itself, but it is a little scary to see the performance
> nearly cut in half going from 1.1 to 1.2 in each of those performance
> tests. I hope this will not be a trend with every release.
> On Jan 2, 7:43 am, Mathew <nfoscar...@yahoo.com> wrote:
>> Also, with the cost of hardware dropping everyday. A framework
>> designed purely for performance will become outdated as other more
>> complex "slower" frameworks offer better features and return on
>> development costs.
>> A well designed framework no matter how slow its internals can be
>> improved without affecting the functionality of the application the
>> framework is driving. Something that the Cake team can address with
>> maintenance releases to address performance.
>> With all that said, I do believe Cake is one of the slower frameworks.
>> The extensive use of for/loops during dispatching and rendering of
>> views has to impact performance, but the level of performance impacted
>> depends on the implementation of the website.
> For most people making such apps the speed of the dispatcher, template
> engine etc is unimportant. The parts slowing the app would be heavy
> SQL, SOAP/XML-RPC-request to other apps or huge batch ETLs. For most
> people (including most public websites) the most important thing would
> be faster SQL/active records and a better implementation of HABTM
> joins, not the ability to serve x or y pages per sec.
Absolutely true, CakePHP is most definitely a developers framework,
and developers are easy to please in regards to performance. Users
and clients on the other hand, are not. This is true in any framework
(or lack there of). Performance matters, why develop in such a
wonderful framework if its frustrating for the end user? (Twitter
comes to mind with ROR).
CakePHP is a developers framework, true, its inherently going to be
slower than pure HTML etc; indeed. What has me shaking my head is
looking at all the other frameworks going from 1.x to 2.x increasing
their performance with each major release whereas we've gone backwards
at a 1:2 (1:4 in linked Yii Benchmark) rate with something as simple
as pages per second of static (or near static) content.
I admit I'm new to CakePHP, and I will reiterate how impressed I am
with the framework as a whole. I was seduced by Cake by its ease of
use and console/bake feature which I've grown to love. I'm moving
away from ROR because of performance issues and cost.
I just don't want that to be a trend is all, because I love to bake
and eat cake oh so much. *smile*
On Jan 2, 11:21 am, Mathew <nfoscar...@yahoo.com> wrote:
> > For most people making such apps the speed of the dispatcher, template
> > engine etc is unimportant. The parts slowing the app would be heavy
> > SQL, SOAP/XML-RPC-request to other apps or huge batch ETLs. For most
> > people (including most public websites) the most important thing would
> > be faster SQL/active records and a better implementation of HABTM
> > joins, not the ability to serve x or y pages per sec.
> CakePHP is a developers framework, true, its inherently going to be
> slower than pure HTML etc; indeed. What has me shaking my head is
> looking at all the other frameworks going from 1.x to 2.x increasing
> their performance with each major release whereas we've gone backwards
> at a 1:2 (1:4 in linked Yii Benchmark) rate with something as simple
> as pages per second of static (or near static) content.
Yeah but 1.2 has a million new features over 1.1. And I'm happy with
the speed of Cake, there is not much tom complain about. On slow
pages, there are tons of tweaks you can can do to get it up to snuff.
Like you said, it's a developers framework.
On Fri, Jan 2, 2009 at 7:43 PM, rtconner <rtcon...@gmail.com> wrote:
> > CakePHP is a developers framework, true, its inherently going to be
> > slower than pure HTML etc; indeed. What has me shaking my head is
> > looking at all the other frameworks going from 1.x to 2.x increasing
> > their performance with each major release whereas we've gone backwards
> > at a 1:2 (1:4 in linked Yii Benchmark) rate with something as simple
> > as pages per second of static (or near static) content.
> Yeah but 1.2 has a million new features over 1.1. And I'm happy with
> the speed of Cake, there is not much tom complain about. On slow
> pages, there are tons of tweaks you can can do to get it up to snuff.
> Like you said, it's a developers framework.
Do a benchmark using a full fledged system (perhaps a whole social
community) and see how the work. Simply hello world pages are nothing
because no frameworks are used for that.
I have done a lot bit of profiling on my own apps. I have also
compared Cake 1.1 with 1.2 when migrating an allication, but not
compared another framework to Cake.
What I can generalize from these tests is that using the common simple
test-app probably 1.1 is faster than 1.2. Probably these benchmarks
are correct when they do similar comparisons and find that Cake 1.2 is
"slow" compared to other frameworks.
What I have also discovered is that when I do the same benchmarks
using "heavy data" things look very different. When I say heavy data I
am talking about a filemanager dealing with meta-data form 3'000 files
and not just 20 odd files on my development system. I am talking about
a reporting system loaded with 1.2 million transactions, not 120
transactions. I am talking about 400'000 user accounts and not 40.
It can be a real pain getting that much data into your app. I have
been lucky to have had access to large datasets I can copy and do
tests on. I wouldn't like to try making 400'000 unique usernames
myself :)
In these cases I have seen that CakePHP 1.2 has been extremely good at
handling the load. Many things that slow down the small application
speeds up the "big" application. If my 40 users were processed in 0.4
instead of 0.2 seconds, that must be compared to how my 400'000 uses
were processed in 4sec instead of running out of memory after 2
minutes of execution. (no real figures, btw)
The developers of Cake have been very good at identifying and building
tools that help an applications as it grows. Sure I can manually write
all the same optimized queries and do a lot of things to make 1.1 (or
some other framework) process in similar times, but in Cake 1.2 all
these things are there, built-in and tested, ready to go. I remember
one of these memory problems from this summer. When the problem
occurred online, on a live server (OMG!) I could very quickly identify
and correct a "sloppy find" in a matter of minutes and then all was
well again. I am not suggesting anyone wait for real users to discover
their bottle-necks, but it is a good indicator of how easy it is to
"optimize" a Cake application.
This has been a big anecdotal, I know. I don't have any real bench-
results to share unfortunately. But I wanted to try to make the point
that Cake 1.2 may not be the fastest from the outset but when you load
your app down with more and more stuff Cakes keeps its speed really
well.
That would make an interesting benchmark. Making a few associations
between a few Models and benching with a small 20KB database. Then
testing with the database weighing in at 200MB or so. It would be cool
to see the curve of each framework as the data started piling up.
@Mathew
The most important tool for optimization has in my experience been
Containable (and "recursive = -1") which can really limit the amount
of data loaded with each request. My second tip is to try to use few
"foreach($found_data as $one)". For lopps can really kill you when
there are a few hundred thousand items in the array :)
/Martin
On Jan 3, 4:42 am, gearvOsh <mileswjohn...@gmail.com> wrote:
> Do a benchmark using a full fledged system (perhaps a whole social
> community) and see how the work. Simply hello world pages are nothing
> because no frameworks are used for that.
It's been said before, but I'll say it again....
index.html is the fastest framework of them all when it comes to
outputting "hello world".
If you've optimized the SQL, enabled caching on all tiers of your
application and made sure you've followed the rules (i.e. no
requestAction() or aggressive model loading)... Attempted lazy loading
on the models, tried both Containable and Linkable behviors...
and now you are finding that Dispatcher is slowing things down. You
really have 2 options:
1. Make sure your app is profitable so you can add some hardware (or
cloud-ware)
2. Post a realistic question with specific problem you are having here
or IRC and someone will help you out.
I firmly believe if cake 1.1 can more or less painlessly serve mozilla
add-ons it can certainly do the same for the rest of us.
Don't waste time on benchmarking requests per seconds, it would be
much better spent on fine-tuning your app.
On Jan 7, 8:52 am, Martin Westin <martin.westin...@gmail.com> wrote:
> I have done a lot bit of profiling on my own apps. I have also
> compared Cake 1.1 with 1.2 when migrating an allication, but not
> compared another framework to Cake.
> What I can generalize from these tests is that using the common simple
> test-app probably 1.1 is faster than 1.2. Probably these benchmarks
> are correct when they do similar comparisons and find that Cake 1.2 is
> "slow" compared to other frameworks.
> What I have also discovered is that when I do the same benchmarks
> using "heavy data" things look very different. When I say heavy data I
> am talking about a filemanager dealing with meta-data form 3'000 files
> and not just 20 odd files on my development system. I am talking about
> a reporting system loaded with 1.2 million transactions, not 120
> transactions. I am talking about 400'000 user accounts and not 40.
> It can be a real pain getting that much data into your app. I have
> been lucky to have had access to large datasets I can copy and do
> tests on. I wouldn't like to try making 400'000 unique usernames
> myself :)
> In these cases I have seen that CakePHP 1.2 has been extremely good at
> handling the load. Many things that slow down the small application
> speeds up the "big" application. If my 40 users were processed in 0.4
> instead of 0.2 seconds, that must be compared to how my 400'000 uses
> were processed in 4sec instead of running out of memory after 2
> minutes of execution. (no real figures, btw)
> The developers of Cake have been very good at identifying and building
> tools that help an applications as it grows. Sure I can manually write
> all the same optimized queries and do a lot of things to make 1.1 (or
> some other framework) process in similar times, but in Cake 1.2 all
> these things are there, built-in and tested, ready to go. I remember
> one of these memory problems from this summer. When the problem
> occurred online, on a live server (OMG!) I could very quickly identify
> and correct a "sloppy find" in a matter of minutes and then all was
> well again. I am not suggesting anyone wait for real users to discover
> their bottle-necks, but it is a good indicator of how easy it is to
> "optimize" a Cake application.
> This has been a big anecdotal, I know. I don't have any real bench-
> results to share unfortunately. But I wanted to try to make the point
> that Cake 1.2 may not be the fastest from the outset but when you load
> your app down with more and more stuff Cakes keeps its speed really
> well.
> That would make an interesting benchmark. Making a few associations
> between a few Models and benching with a small 20KB database. Then
> testing with the database weighing in at 200MB or so. It would be cool
> to see the curve of each framework as the data started piling up.
> @Mathew
> The most important tool for optimization has in my experience been
> Containable (and "recursive = -1") which can really limit the amount
> of data loaded with each request. My second tip is to try to use few
> "foreach($found_data as $one)". For lopps can really kill you when
> there are a few hundred thousand items in the array :)
> /Martin
> On Jan 3, 4:42 am, gearvOsh <mileswjohn...@gmail.com> wrote:
> > These benchmarks dont mean anything.
> > Do a benchmark using a full fledged system (perhaps a whole social
> > community) and see how the work. Simply hello world pages are nothing
> > because no frameworks are used for that.
On Wed, Jan 7, 2009 at 10:09 AM, teknoid <teknoid.cake...@gmail.com> wrote:
> It's been said before, but I'll say it again.... > index.html is the fastest framework of them all when it comes to > outputting "hello world".
> If you've optimized the SQL, enabled caching on all tiers of your > application and made sure you've followed the rules (i.e. no > requestAction() or aggressive model loading)... Attempted lazy loading > on the models, tried both Containable and Linkable behviors...
> and now you are finding that Dispatcher is slowing things down. You > really have 2 options:
> 1. Make sure your app is profitable so you can add some hardware (or > cloud-ware) > 2. Post a realistic question with specific problem you are having here > or IRC and someone will help you out.
> I firmly believe if cake 1.1 can more or less painlessly serve mozilla > add-ons it can certainly do the same for the rest of us.
> Don't waste time on benchmarking requests per seconds, it would be > much better spent on fine-tuning your app.
> On Jan 7, 8:52 am, Martin Westin <martin.westin...@gmail.com> wrote: >> My 2¢ on this subject:
>> I have done a lot bit of profiling on my own apps. I have also >> compared Cake 1.1 with 1.2 when migrating an allication, but not >> compared another framework to Cake. >> What I can generalize from these tests is that using the common simple >> test-app probably 1.1 is faster than 1.2. Probably these benchmarks >> are correct when they do similar comparisons and find that Cake 1.2 is >> "slow" compared to other frameworks.
>> What I have also discovered is that when I do the same benchmarks >> using "heavy data" things look very different. When I say heavy data I >> am talking about a filemanager dealing with meta-data form 3'000 files >> and not just 20 odd files on my development system. I am talking about >> a reporting system loaded with 1.2 million transactions, not 120 >> transactions. I am talking about 400'000 user accounts and not 40.
>> It can be a real pain getting that much data into your app. I have >> been lucky to have had access to large datasets I can copy and do >> tests on. I wouldn't like to try making 400'000 unique usernames >> myself :)
>> In these cases I have seen that CakePHP 1.2 has been extremely good at >> handling the load. Many things that slow down the small application >> speeds up the "big" application. If my 40 users were processed in 0.4 >> instead of 0.2 seconds, that must be compared to how my 400'000 uses >> were processed in 4sec instead of running out of memory after 2 >> minutes of execution. (no real figures, btw)
>> The developers of Cake have been very good at identifying and building >> tools that help an applications as it grows. Sure I can manually write >> all the same optimized queries and do a lot of things to make 1.1 (or >> some other framework) process in similar times, but in Cake 1.2 all >> these things are there, built-in and tested, ready to go. I remember >> one of these memory problems from this summer. When the problem >> occurred online, on a live server (OMG!) I could very quickly identify >> and correct a "sloppy find" in a matter of minutes and then all was >> well again. I am not suggesting anyone wait for real users to discover >> their bottle-necks, but it is a good indicator of how easy it is to >> "optimize" a Cake application.
>> This has been a big anecdotal, I know. I don't have any real bench- >> results to share unfortunately. But I wanted to try to make the point >> that Cake 1.2 may not be the fastest from the outset but when you load >> your app down with more and more stuff Cakes keeps its speed really >> well.
>> That would make an interesting benchmark. Making a few associations >> between a few Models and benching with a small 20KB database. Then >> testing with the database weighing in at 200MB or so. It would be cool >> to see the curve of each framework as the data started piling up.
>> @Mathew >> The most important tool for optimization has in my experience been >> Containable (and "recursive = -1") which can really limit the amount >> of data loaded with each request. My second tip is to try to use few >> "foreach($found_data as $one)". For lopps can really kill you when >> there are a few hundred thousand items in the array :)
>> /Martin
>> On Jan 3, 4:42 am, gearvOsh <mileswjohn...@gmail.com> wrote:
>> > These benchmarks dont mean anything.
>> > Do a benchmark using a full fledged system (perhaps a whole social >> > community) and see how the work. Simply hello world pages are nothing >> > because no frameworks are used for that.
|
