SS3: Anybody keen to compare framework performance?

[Ingo Schommer]

I've just discovered a github project to compare PHP frameworks: https://github.com/fabpot/framework-benchs

Created by Fabien Potencier, core dev for Symfony 2: http://symfony-reloaded.org/fast

I don't expect us to come out too well in this comparison,

but it'll help us focus our performance efforts for SilverStripe 3.

Anybody keen to extend this with a SilverStripe example,

book an EC2 instance for a couple of hours, run the tests, and publish the results?

Ingo

---

Ingo Schommer | Senior Developer

SilverStripe

http://silverstripe.com

in...@silverstripe.com

Phone: +64 4 978 7330 x 42

Skype: chillu23

[Mirko Sassetti]

I can do that, if you give me some days.
ok?

Mirko.

On 24 Feb, 22:18, Ingo Schommer <i...@silverstripe.com> wrote:
> I've just discovered a github project to compare PHP frameworks:https://github.com/fabpot/framework-benchs
> Created by Fabien Potencier, core dev for Symfony 2:http://symfony-reloaded.org/fast
>
> I don't expect us to come out too well in this comparison,
> but it'll help us focus our performance efforts for SilverStripe 3.
>
> Anybody keen to extend this with a SilverStripe example,
> book an EC2 instance for a couple of hours, run the tests, and publish the results?
>
> Ingo
>
> ---
> Ingo Schommer | Senior Developer
> SilverStripehttp://silverstripe.com

> i...@silverstripe.com

[Hamish Campbell]

Not to be crass, but did anyone else take a double take on that
username?

I'm curious - where is the documentation on how the benchmark is
performed and what exactly is being tested. Benchmarks written for a
particular framework tend to focus on that framework's strengths. I
know it doesn't really matter since I think we're probably more
interested in the improvements between 2.4.x and 3.x and into the
future, rather than a direct comparison, but I'd like to know.

Regards

Hamish

On Feb 25, 10:18 am, Ingo Schommer <i...@silverstripe.com> wrote:
> I've just discovered a github project to compare PHP frameworks:https://github.com/fabpot/framework-benchs
> Created by Fabien Potencier, core dev for Symfony 2:http://symfony-reloaded.org/fast
>
> I don't expect us to come out too well in this comparison,
> but it'll help us focus our performance efforts for SilverStripe 3.
>
> Anybody keen to extend this with a SilverStripe example,
> book an EC2 instance for a couple of hours, run the tests, and publish the results?
>
> Ingo
>
> ---
> Ingo Schommer | Senior Developer
> SilverStripehttp://silverstripe.com

> i...@silverstripe.com

[Ingo Schommer]

Mirko: Awesome, let us know if you need any help with

implementing the specific app examples in SilverStripe.

Hamish: As with any benchmark, you have to take it with a grain of salt.

If you read through the links provided, they're going through great

lengths to make the tests objective and reproducible.

SilverStripe 2.4 vs. 3.0 is a different kettle of fish, but worth testing as well.

--
You received this message because you are subscribed to the Google Groups "SilverStripe Core Development" group.
To post to this group, send email to silverst...@googlegroups.com.
To unsubscribe from this group, send email to silverstripe-d...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/silverstripe-dev?hl=en.

---

Ingo Schommer | Senior Developer

SilverStripe

http://silverstripe.com

in...@silverstripe.com

[xeraa]

1) I'd question the use of the published benchmark results. <<The
"Product" Application>> test for <<real world>> applications was done
with:
# No page/template cache
# No database access
# No session
# No output escaping
That doesn't sound like my world...

2) I was thinking about doing something similar after finishing my
current "project", however I'd focus on SilverStripe only. I'd think a
fair comparison is hardly possible, both because of personal favorites
and because very few people will know half a dozen frameworks / CMS
equally well.
My approach would rather be, how much speed can SilverStripe offer -
based on different use cases (10 pages vs 1000,...) and technologies
(does it make a difference to use PostgreSQL rather than MySQL,...).
But more on that hopefully in a month or so...

> > For more options, visit this group athttp://groups.google.com/group/silverstripe-dev?hl=en.

>
> ---
> Ingo Schommer | Senior Developer

[Hamish Campbell]

On Mar 1, 4:31 am, xeraa <p.xe...@gmail.com> wrote:
> 1) I'd question the use of the published benchmark results. <<The
> "Product" Application>> test for <<real world>> applications was done
> with:
> # No page/template cache
> # No database access
> # No session
> # No output escaping
> That doesn't sound like my world...

I was thinking that as well. It looks like a routing and control test
more than anything related to real world bottlenecks. The criteria
appear to most of the important areas for potential optimization that
a framework can apply. I mean, who cares if frame X has half the no-
template build time vs. Y if Y is 10x faster with the template cache.
Y will win every time in a real world installation.

And no database access?

I think it would be more useful to set some benchmarks against:

1. Common database operation IO times
2. Template cache execution
3. Controller routing
4. Template build times
...

A benchmarking module would be a useful addition, although I don't
know if Dawn already provides some of this functionality.

[Michael Gall]

You could write a selenium test case that just measures performance. Load up the admin, login, fill in 10 or (100) pages and count how long it took. This could then become the baseline for improvements to the framework, and ensure that there are no performance regressions.


Cheers,

MIchael

Checkout my new website: http://myachinghead.net
http://wakeless.net

[Sam Minnée]

Hrm, CMS performance and framework performance are two very different things. I'd prefer to see a benchmark of a simple UI implemented against a simple data model.

For example, create this model

StaffMember
- FirstName
- Surname
- Email
- Phone

Generate 100 items

Create a view+controller that lists all of those 100 items in a single HTML page.

Then benchmark the loadtime of that page.

The disadvantage is that it would require a wee bit of custom dev in each framework. It seems like an interesting thing to tackle as a standalone project, rather than something we're doing just for SilverStripe.

-------
Sam Minnée | Chief Technology Officer
SilverStripe
http://silverstripe.com

Phone: +64 4 978 7334
Skype: sam.minnee

[Michael Gall]

I suppose using the word "framework" is the wrong term. But I'm talking about the entire CMS. There's a huge amount of JS changes and backend/frontend changes planned and having a baseline for performance of these would be great. Keeps eyes on the fact that just cause the technology is cooler/better it still needs to be fast.


Cheers,

Michael

[Sam Minnée]

> But I'm talking about the entire CMS. There's a huge amount of JS changes and backend/frontend changes planned and having a baseline for performance of these would be great. Keeps eyes on the fact that just cause the technology is cooler/better it still needs to be fast.

yeah I guess, I guess there are two completely separate things to benchmark, but they're both important. Benchmarking CMS load time against other CMSes doesn't really seem very useful, though; just against older versions of SilverStripe.

[Michael Gall]

yeah I guess, I guess there are two completely separate things to benchmark, but they're both important.  Benchmarking CMS load time against other CMSes doesn't really seem very useful, though; just against older versions of SilverStripe.

I think loading and working times are mildly interesting (and shouldn't be discounted) when compared to other CMSes. Though as you say, much more interesting when compared to other SS versions and perhaps on different browsers as well.

Cheers,

Michael

[Sigurd Magnusson]

Note that we've done manual performance testing of various CMS functions in IE7, IE8, FireFox on SilverStripe 2.4. This study was done primarily based on us having customers asking which browser was best for content authoring, whereby users were typically either on IE or Firefox. (i.e. if we had done this test for website developers, we might have also included other browsers like Chrome and Safari.)

http://silverstripe.org/testing-reveals-ie8-and-firefox-as-recommended-browsers-for-doing-content-entry/

(Yes, this is only part of the equation in terms of benchmarks, but I assume several people in the list aren't aware of the report at the URL above.)

Cheers,

Sigurd.

--

[Mirko Sassetti]

Ok, i admit to have some problems doing the test.
I made a SilverStripe install on local and configured to made the
tests like the other framework. Tried on local and all seem to work,
so i include it on the forked repository. Next i try to follow the
instruction in the readme to setup the server without problems and
cloned the repository on the web server root, but SilverStripe don't
work as if he could not find some php files. The strange thing is that
also other frameworks do not seem to work properly.
In the coming days, if anything, will try to do other tests.

Sorry for now :)

> > For more options, visit this group athttp://groups.google.com/group/silverstripe-dev?hl=en.

>
> ---
> Ingo Schommer | Senior Developer