CMS speed in 2.4 some core issues....

244 views
Skip to first unread message

Nicolaas Thiemen Francken - Sunny Side Up

unread,
Apr 17, 2012, 12:25:18 AM4/17/12
to silverstripe-dev
Hi Everyone

We have some pretty slow websites and so we decided to investigate using

http://xdebug.org/ - record PHP computing

We found two core culprits:

1. i18n::include_by_locale: called 500 times -> 37,000 calls to file_exist function (this only happens when there is no language file for the locale used).
    add $lang["en_NZ"] = array() to your config file to fix this.
    You can also fix it in the core, like this:

+++ sapphire/core/i18n.php      (working copy)
@@ -1886,6 +1886,10 @@
 
                // Finally, load any translations from registered plugins
                if ($load_plugins) self::plugins_load($locale);
+
+               if(!isset($lang[$locale])){
+                       $lang[$locale] = array();
+               }
        }


2. MysqlQuery::nextRecord: 22,000 calls to Mysql::field_name, you need to patch core for this AFAIK... This can be done by using associative arrays in fetching mysql rather than looking up every single field one by one.  QUESTION: should one this or is this suicide by datamashup?

+++ sapphire/core/model/MySQLDatabase.php       (working copy)
@@ -1076,9 +1076,8 @@
 
        public function nextRecord() {
                // Coalesce rather than replace common fields.
-               if(is_resource($this->handle) && $data = mysql_fetch_row($this->handle)) {
-                       foreach($data as $columnIdx => $value) {
-                               $columnName = mysql_field_name($this->handle, $columnIdx);
+               if(is_resource($this->handle) && $data = mysql_fetch_assoc($this->handle)) {
+                       foreach($data as $columnName => $value) {
                                // $value || !$ouput[$columnName] means that the *last* occurring value is shown
                                // !$ouput[$columnName] means that the *first* occurring value is shown
                                if(isset($value) || !isset($output[$columnName])) {


3. Next ones to look at:

Object::combine_static 7% of loading time
DataObject::buildSQL 3.5% of loading time

By fixing / patching these two methods, we were able to save up to 20% of the loading time of the CMS. 

Hope that helps.

Is there anyone who has other ideas on how to speed up CMS?

Cheers

Nicolaas









Marcus Nyeholt

unread,
Apr 17, 2012, 12:39:16 AM4/17/12
to silverst...@googlegroups.com
Thanks for these - I'll be applying to a large system that's soon to go into production! 

I'm currently using XHProf for tracing performance of certain bits and pieces; given the system isn't live yet, I don't have anything worth feeding into the current discussion just yet, but will look this thread up again in a month or so when I should be finding out. 


Nicolaas









--
You received this message because you are subscribed to the Google Groups "SilverStripe Core Development" group.
To post to this group, send email to silverst...@googlegroups.com.
To unsubscribe from this group, send email to silverstripe-d...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/silverstripe-dev?hl=en.

Sam Minnée

unread,
Apr 17, 2012, 12:45:46 AM4/17/12
to silverst...@googlegroups.com
Ingo's reworking the i18n system to be based on Zend_Translate at the moment; hopefully it's in-memory caching is more efficient than this?

37000 calls to file_exists is definitely something that should be avoided.

Nicolaas Thiemen Francken - Sunny Side Up

unread,
Apr 17, 2012, 3:45:42 AM4/17/12
to silverst...@googlegroups.com
Thank you both for your replies. 

What about the 22,000 mysql requests...That also seems a bit excessive. Any chance we can make some changes to 2.4 branches to reduce the number of mysql calls?


Nicolaas

Ingo Schommer

unread,
Apr 17, 2012, 4:02:59 AM4/17/12
to silverst...@googlegroups.com
Hey Nicolaas, thanks for looking into this! XHProf is really a life saver for this kind of stuff.

I've halved the number of _t() calls by caching DataObject::fieldLabels() in master, see
That should be easily backportable to 2.4. I've added a similar fix to your 
empty array assignment patch to master as well. Wanna send a pull request?

Ingo

Steve Chan

unread,
Apr 17, 2012, 4:05:31 AM4/17/12
to silverst...@googlegroups.com
On Mon, Apr 16, 2012 at 9:25 PM, Nicolaas Thiemen Francken - Sunny
Side Up <ma...@sunnysideup.co.nz> wrote:
> Is there anyone who has other ideas on how to speed up CMS?

One thing we noticed is that performance for CMS content editing
between administrators and non-administrators was almost twice as fast
- for example, bringing up a new page in the editor interface may take
3 seconds for an admin, but 6 seconds for a conventional editor.
Buried in the code there is a check for if a user has admin privs, and
if so, it bypasses all the permissions checks - for non-admins the
permissions checks take relatively huge amounts of time.

The difference is much smaller for small sites - our site has over
a thousand pages. We haven't looked into how to tweak the code for
non-admins - it seemed a lost cause since 3.0 was coming out.

Steve

--
"Sow a thought, reap an action. Sow an action, reap a habit. Sow a
habit, reap a character. Sow a character, reap a destiny." - Samuel
Smiles

Nicolaas Thiemen Francken - Sunny Side Up

unread,
Apr 30, 2012, 7:32:31 PM4/30/12
to silverst...@googlegroups.com
Maybe one of the core devs or someone else can have a look at this ticket I just added:


I dont have time to investiage this right now... For starters, it would be great if someone can confirm my finding.  I am  sure once it is a confirmed bug it will be relatively easy to fix it in the core and I am also sure this will make a big difference in terms of loading times, as some of the thirdparty folders are rather large.

In advance, my apologies if I made a mistake here, but from what I can tell, it is not working correctly.

Cheers

Nicolaas

Nicolaas Thiemen Francken - Sunny Side Up

unread,
Jun 7, 2012, 5:01:39 AM6/7/12
to silverst...@googlegroups.com
I just looked at Member:

public function getCMSFields() {
.....
$mainFields->replaceField('Locale', new DropdownField(
"Locale", 
_t('Member.INTERFACELANG', "Interface Language", PR_MEDIUM, 'Language of the CMS'), 
i18n::get_existing_translations()
));
....
}

Then I checked i18n::get_existing_translations():

static function get_existing_translations() {
$locales = array();
$baseDir = Director::baseFolder();
$modules = scandir($baseDir);
foreach($modules as $module) {
if($module[0] == '.') continue;
$moduleDir = $baseDir . DIRECTORY_SEPARATOR . $module;
$langDir = $moduleDir . DIRECTORY_SEPARATOR . "lang";
if(is_dir($moduleDir) && is_file($moduleDir . DIRECTORY_SEPARATOR . "_config.php") && is_dir($langDir)) {
$moduleLocales = scandir($langDir);
foreach($moduleLocales as $moduleLocale) {
if(preg_match('/(.*)\.php$/',$moduleLocale, $matches)) {
if(isset($matches[1]) && isset(self::$all_locales[$matches[1]])) {
$locales[$matches[1]] = self::$all_locales[$matches[1]];
}
}
}
}

// sort by title (not locale)
asort($locales);
return $locales;
}

In the Sapphire "master" this is improved to:

static function get_existing_translations() {
$locales = array();

// TODO Inspect themes
$modules = SS_ClassLoader::instance()->getManifest()->getModules();

foreach($modules as $module) {
if(!file_exists("{$module}/lang/")) continue;

$moduleLocales = scandir("{$module}/lang/");
foreach($moduleLocales as $moduleLocale) {
preg_match('/(.*)\.[\w\d]+$/',$moduleLocale, $matches);
if($locale = @$matches[1]) {
// Normalize locale to include likely region tag.
// TODO Replace with CLDR list of actually available languages/regions
$locale = str_replace('-', '_', self::get_locale_from_lang($locale));
$locales[$locale] = (@self::$all_locales[$locale]) ? self::$all_locales[$locale] : $locale;
}
}
}

// sort by title (not locale)
asort($locales);

return $locales;
}

I am wondering if we could have some sort of setting where we can just set the locales available as this would add some speed. For example, for a fash and chaps store only one locale is ever relevant (AFAIK).

I am posting this here, because in the previous post for this "topic", we identified the "language file searches" as one of the culprits for making Sapphire 2.4 run slow.

I dont have time to investigate this in further detail, but I hope it can help some people (i.e. if you run 2.4 then you can replace i18n::get_existing_translations() with: (UNTESTED - CHECK EXACT FORMAT OF ARRAY)
static function get_existing_translations() {
       return array("en_NZ" => "en_NZ");
}

Nicolaas


Jeremy Thomerson

unread,
Jun 7, 2012, 11:40:07 AM6/7/12
to silverst...@googlegroups.com, n...@sunnysideup.co.nz
If I were to submit pull requests against a 2.4.x branch or tag for the following three enhancements, could they be accepted and spawn a 2.4.8 release?

  1. The many file_exists calls mentioned in the original post
  2. The many mysql_field_name calls mentioned in the original post
  3. Improving get_existing_translations (from this most recent post)
Thanks,
Jeremy Thomerson

Ingo Schommer

unread,
Jun 7, 2012, 12:41:47 PM6/7/12
to silverst...@googlegroups.com
We'll only accept security-critical patches for the 2.4.x line,
but are generally happy to put (well tested) code into "post-2.4".
At the moment we don't plan a 2.5 release, so this is
mainly an attempt to bridge the gap in terms of upgrades.

Our release process is documented here BTW:

--
You received this message because you are subscribed to the Google Groups "SilverStripe Core Development" group.
To view this discussion on the web visit https://groups.google.com/d/msg/silverstripe-dev/-/suysHfZB5rsJ.

Nicolaas Thiemen Francken - Sunny Side Up

unread,
Jun 7, 2012, 9:46:50 PM6/7/12
to silverst...@googlegroups.com
THANK YOU Jeremy and Ingo for your replies.

I wonder if we can make an exception or if we can setup our own 2.4.8 version to do this, because there are "many-many" sites running on 2.4* and there will be for a long time, the changes are relatively simple - especially the file_exists ones  and it will make Silverstripe a lot faster.  

Are there any other ways to get this available for everyone? 

Right now, a lot of our clients are complaining about speed and I would dearly like to improve this. If this is the case for us, then I am sure a lot of other people will have the same problem especially on setups with a separate file server where file_exists * many-many can slow things down. 

Thank you again

Nicolaas

Ingo Schommer

unread,
Jun 8, 2012, 3:51:38 AM6/8/12
to silverst...@googlegroups.com
C'mon, this is just a guess, not even a measured phenomenon, right?
With your arguments ("relatively simple change") we could probably
merge hundreds of fixes into 2.4, but that's besides the point,
and what we created post-2.4 for.
> --
> You received this message because you are subscribed to the Google Groups "SilverStripe Core Development" group.

Sigurd Magnusson

unread,
Jun 8, 2012, 4:14:52 AM6/8/12
to silverst...@googlegroups.com
On 8 June 2012 19:51, Ingo Schommer <in...@silverstripe.com> wrote:
C'mon, this is just a guess, not even a measured phenomenon, right?

I think Nicolaas pointed to a 20% speed improvement.

One question from me is whether this is relevant to SS3, potentially post 3.0 stable, though.

Ingo Schommer

unread,
Jun 8, 2012, 4:24:56 AM6/8/12
to silverst...@googlegroups.com
I read Nicolaas' email out of context sorry. That's what happens when you interact with mailinglists via email I guess (didn't see his original post from April referencing XHProf etc).

@Nicolaas: Are you running a bytecode cache like APC or XCache, and if yes, with which settings? Would be interesting to see the difference in speed improvement with/without these caches.
Also, 37k calls to file_exists() might sound scary, but they don't result in the same number of stat calls on the filesystem, as they're cached by PHP (even without bytecode caches): http://de3.php.net/file_exists

A 20% speed improvement across the board would be very tempting to get into 2.4.x,
but it'd be good to have more solid numbers on it first (I find XHProf run diffs quite helpful for that).

Mat Weir

unread,
Jun 8, 2012, 4:42:11 AM6/8/12
to silverst...@googlegroups.com
Hi Ingo,

Is there Silverstripe documentation on APC/XCache recommended settings?

Mat Weir 


--
You received this message because you are subscribed to the Google Groups "SilverStripe Core Development" group.
To view this discussion on the web visit https://groups.google.com/d/msg/silverstripe-dev/-/ZMn0fgyvJOcJ.

Nicolaas Thiemen Francken - Sunny Side Up

unread,
Jun 8, 2012, 4:47:32 AM6/8/12
to silverst...@googlegroups.com
@Jeremy: do you already have some solutions or were you thinking about writing them?

@Ingo: in any case, I am wondering if in future SS versions there is an option to fix the locale to one as this would speed things up.  I dont really know the answers to the caching questions right now - but I think we should fix rather than relying on caching.  Caching might speed it up, but it would be even better if we just changed the code a wee bit. Not everyone knows how to setup these fancy caching mechanisms and it seems to me that we could apply some "plasters" that would make a big difference.  

@Marcus: what sort of fixes did you make? You mentioned you were looking into this. 

Gordon Anderson

unread,
Jun 16, 2012, 4:02:39 AM6/16/12
to silverst...@googlegroups.com
hi

Although this is not core silverstripe, DataObjectManager is a dependency of many modules in 2.4 (and thus many websites), and has a major unpatched memory bug.  For details see https://github.com/unclecheese/DataObjectManager/issues/11 - it effectively limits the size of your website to how much RAM the PHP process can get.

Secondly, if anyone requires a large site for testing purposes I can provide a copy of the database and code for my own site (SS 2.4), http://www.tripodtravel.co.nz/, where I have imported my Flickr pictures and sets as Flickr pages within Flickr folders.  I realize that it would make more sense to use DataObjects instead of Pages, but I wanted to see how Silverstripe coped with a large amount of data.  Currently there are around 37000 Pages on the site - it was during the importing of these that I discovered the memory bug in DataObjectManager

Cheers

Gordon
Reply all
Reply to author
Forward
0 new messages