Multilanguage support massively slows down app

[Vahrokh Vain]

Hello,

I am developing a large Angular JS application fetching database data through PHP calls. These standards are customer company imposed.

While I don't need any visual output (Angular JS does that), I need speed and compact-ness, so I have choosen Fat Free Framework.
The whole software is being hosted on a good quality VPS, where I have other VPSes (other customers, each got his own) delivering 100k+ hit a day websites with no problem.

So far I've been quite pleased with F3, but I have bumped into an issue. I need to produce multilanguage messages, F3 does it but it's totally killing my software performance.

As I add more messages, the PHP scripts are taking 420ms as of now and it's only going to grow as I add more translated messages.

The attached PNG file is a WebCacheGrind screenshot showing the issue.

Basically, for what I understand, Base->set() is a quite time expensive function and for what I understand, it's being massively invoked at startup to load the translations.
So far I am just at the beginning of the translations, 2 languages with few words each. Example:

/*
 * SPF PHP interface to AngularJS - English translation
 * (c)2016 [Removed] - all rights reserved
 *
 * @author Dario Fumagalli <removed>
 */

return array(
    'STR_RESULT_NO_MATCHING_DESC' => 'No description available',
    'STR_RESULT_SUCCESS' => 'Operation success',
    'STR_RESULT_FAILURE' => 'Operation failure',
    'STR_RESULT_OUT_OF_BOUNDS' => 'Input parameter value out of bounds',
    'STR_RESULT_NOT_FOUND' => 'Value not found',
    'STR_RESULT_INVALID_VALUE' => 'Entered value is not allowed',
    'STR_RESULT_DATABASE_ERROR' => 'Database error during operation',
    'STR_RESULT_KEY_NOT_FOUND' => 'Key does not match with any value',
    'STR_ERROR_GT0' => 'Integer value greater than zero required',
    'STR_ERROR_OUT_OF_BOUNDS' => 'Value out of bounds',
);

As you can see it's very small stuff, however it's killing my performance.

You might wonder where do the other Base->set() calls come from. They come from config.ini:

[globals]
DEBUG=3
CACHE="memcache=localhost:11211"
UI="ui/;src/[Removed]/View/"
LOCALES="src/[Removed]/Language/"
PREFIX="DICT."
FALLBACK="en"
SELF_URL=[Removed]
RSPP_PATH=src/[Removed]/

; Development server MySQL dsn
spf.db.dsn="mysql:host=127.0.0.1;dbname=[Removed]"
spf.db.username="[Removed]"
spf.db.password="[Removed]"

[routes]
GET|POST /controller-base-json-request-dump=\[Removed]\Controller\ControllerBase->dumpJSONRequest
GET|POST /controller-spf-base-json-request-dump=\[Removed]\Controller\ControllerSPFBase->dumpJSONRequest
GET|POST /spf-session-session-create/@key/@value=\[Removed]\Helper\DebugClass->sessionCreate
GET|POST /spf-session-session-dump=\[Removed]\Helper\DebugClass->sessionDump
GET|POST /spf-session-session-destroy=\[Removed]\Helper\DebugClass->sessionDestroy

The routes section is not present in the production branch.



Conclusion: it looks like translation support is calling an expensive function for every translated sentence, this makes serious multilanguage app development impervious at best.

What could I do to improve the app performance? Any way to cache the translated strings loading?

[bcosca]

Grab the current code from fatfree-core and change this setting:

[globals]

LOCALES="src/[Removed]/Language/",60

This will cache translations for 60 secs.

[Vahrokh Vain]

This is why I love F3 and I love you guys. I can have any kind of issue, but in F3 there's always some hidden nook or cranny where a solution can be found!

I have another little question, then: is it possible to cache some (or the whole) config.ini entries as well? I don't mean caching routes or similar, I know I can do that. I mean, caching the config.ini reading in some way similar to how you just suggested I can do for translations? I really want to avoid as many Base->set() calls as possible, my code must work within 200ms tops.

[bcosca]

You can do it selectively:

spf.db.dsn="mysql:host=127.0.0.1;dbname=[Removed]",86400
spf.db.username="[Removed]",86400
spf.db.password="[Removed]",86400

[Vahrokh Vain]

Thank you again, so much!!!!

[Vahrokh Vain]

Hello again,

I don't know if I am doing something wrong but the solution does not work for me.

I've downloaded the 3.6-dev files from the core distribution and made sure to upload them to my Ubuntu server.

If I use the suggested syntax as follows:

[globals]
LOCALES="src/RSPPItalia/Language/",3600

and save config.ini and try to use my software, then I get this:

Internal Server Error

preg_split() expects parameter 2 to be string, array given

[php/lib/base.php:2068] Base->error()
[php/lib/base.php:610] preg_split()
[php/lib/base.php:989] Base->split()
[php/lib/base.php:368] Base->lexicon()
[php/index.php:25] Base->config()

This happens in index.php when reading config.ini, well before it even gets to my routes.

If I use XDebug I see this happens: set() incorrectly gets an array with the key value + ttl instead of a "normal" value and ttl as 3rd parameter.

See the screenshot at the end of the post, a picture tells more than a thousand words.


I have tried putting the ttl inside the "" but it does not work either. $ttl is always set to 0.

[globals]
LOCALES="src/RSPPItalia/Language/,3600"

[bcosca]

My bad. I totally forgot that:

xyz=1,"abc",3

will assign array(1,'abc',3) to xyz.

Hang on. Let me think of something.

[xfra35]

Hi Vahrokh,

You're not the first to complain about this. See these topics:

https://groups.google.com/forum/#!topic/f3-framework/eDuCGUR1dBA

https://github.com/bcosca/fatfree-core/issues/92

https://github.com/bcosca/fatfree/issues/845

[bcosca]

Try base.php in the latest core commit. Cached config variables are implemented like so:-

LOCALES="src/[Removed]/Language/" | 3600

spf.db.dsn="mysql:host=127.0.0.1;dbname=[Removed]" | 86400
spf.db.username="[Removed]" | 86400
spf.db.password="[Removed]" | 86400

xyz=1,"abc",3 | 60

Let me know if you run into any issues.

[ikkez]

yes the point is to drop the mset for locales completly.. (https://github.com/bcosca/fatfree-core/issues/92)

I think this gives most impact and is easy to solve, and should be relatively safe, since we do not expect magic variables in the locale files.

[Vahrokh Vain]

It works awesomely!

It shaved off about 95ms out of 420ms, caching lexicon and the other settings discussed in this thread.

As a proof of functionality, the Memcached attached screenshot shows the lexicon memcached keys being created in both languages (I started the app twice, with different locale settings) and showing their contents below.

Since I believe this could be good food for thought, I have also attached a WebCacheGrind profiler screenshot showing 4 points, indicated in red:

1) Total execution time with (mem)caching: 325ms (was 420ms until your last update).

2) However, and I mention Ikkez here, mset() is still a major slowdown factor. Notice the execution time: 48ms just for that. Therefore, removing it could be a good next objective.

3) Config parsing is still quite slow: 101ms, 46 of which spent in mset().

4) Since absolute milliseconds times could give a vague idea of the scale of the slowdown, point (4) shows how an uncached PDO database query took 20ms. It took just 20ms despite it being a session record seek with a lot of serialized data.
This means, despite caching, mset() + config.ini parsing take 5 times as much as connecting to the database server and fetching the uncached data through F3's database session manager.

[bcosca]

I don't think mset (at least for the lexicons) can be dispensed with because it is responsible for adding the dictionary entries to the hive. Having said that, the latest core commit applies the TTL setting of LOCALES also to mset. This should shave off a few more clock ticks.

[ikkez]

ohhh... I think this makes it even worse, as you would hit the cache for every single language key now

mset should be replaced with the portion that is resonsible for adding the keys to the hive, so:

        foreach ($lex as $key=>$val) {
            $ref=&$this->ref($this->hive['PREFIX'].$key);
            $ref=$val;
            unset($ref);
        }   

as suggested in https://github.com/bcosca/fatfree-core/issues/92

[bcosca]

@ikkez is right. I updated the repo per recommendation.