JLanguage::_load()

[Ian MacLennan]

So...

I've got a branch in which I'm looking at some changes to the implementation of this method, as has been discussed previously.  Basically the idea is to adopt a more standardized ini file format so that we can take advantage of PHP's built-in INI parser.  We have previously talked about the implication of this being that our keys would have to change.  In fact, there are a few other things that will also be affected.

But first, let me present some of my profiling findings.  I think I mentioned in another thread that the method JLanguage::_load took a good deal of time during a page load.  I don't remember the exact figures that I mentioned at the time.

In any event, in my branch, I have made the appropriate changes to the JLanguage class and have done some profiling.

I did two runs of the current trunk.  In both of those runs, the profile results indicated that JLanguage::_load took up approximately 25% of the processing cycles during a page load.

After I made my changes, JLanguage::_load took up less than 1% of the processing cycles during the page load.

This seems to me to be a fairly significant win in terms of performance.  A reduction of almost 25% is quite substantial.

So, the implications on ini file format:
1. As previously mentioned, our keys will have to change.  Spaces are not allowed.  The general practice will be to use underscores for spacing.  Also, the plan is to name space strings, so that you would end up with something like COM_EXAMPLE_MYKEY for your key.  According to the php manual, The characters {}|&~![()^" cannot be used in keys.  There are also reserved words for keys: null, yes, no, true, false, on, off, none.

2. The syntax for the value will also change.  According to the PHP manual, if a value in the ini file contains any non-alphanumeric characters it needs to be enclosed in double-quotes (").  That will basically mean all of our values will have to be enclosed in quotes, as a space is non alpha numeric.  From what I gather, if you want to include a double-quote in your string, you use two double-quotes.  Finally, according to the PHP manual, the {}|&~![()^" characters have a special meaning in the value.  Unfortunately, they don't really unpack what that special meaning is - anybody have experience in this area?

3. Comment syntax will change, as we are typically using pound signs (#) and the standard parser requires semi colons (;).

To merely change the syntax of these language strings is easy.  The tedious part is adjusting the language strings so that they follow our namespacing changes.

I should note that Mark made the suggestion of using INI sections to namespace the strings.  That would mean that you would just put [COM_EXAMPLE] at the top of your INI file and the strings would be properly namespaced that way.  This would require an API change to JText::_ to accomodate stuffing in the namespace as well.  It could either use a dot notation and we can split it or we can add another parameter.  The first option would probably be better if we were going to go down the road of namespacing the strings use INI sections.

So...  I think that is enough for this first post.  Would be interested in hearing comments or suggestions.

Ian

[Hannes Papenberg]

Hi Ian,
the performance gain sounds absolutely phenomenal and at least in my
book it would be worth the work. Making Joomla 25% faster would be
absolutely amazing. I am all for the namespacing and dot-notation sounds
good. The question would be, if we have some way to support old
extensions here. Would a switch be possible? Or a seperate legacy plugin?

Hannes

Ian MacLennan schrieb:

> to the php manual, /The characters {}|&~![()^"* */cannot be used in

> keys. There are also reserved words for keys: null, yes, no, true,
> false, on, off, none.
>
> 2. The syntax for the value will also change. According to the PHP
> manual, if a value in the ini file contains any non-alphanumeric
> characters it needs to be enclosed in double-quotes ("). That will
> basically mean all of our values will have to be enclosed in quotes,
> as a space is non alpha numeric. From what I gather, if you want to
> include a double-quote in your string, you use two double-quotes.

> Finally, according to the PHP manual, /the {}|&~![()^" /characters

> have a special meaning in the value. Unfortunately, they don't really
> unpack what that special meaning is - anybody have experience in this
> area?
>
> 3. Comment syntax will change, as we are typically using pound signs
> (#) and the standard parser requires semi colons (;).
>
> To merely change the syntax of these language strings is easy. The
> tedious part is adjusting the language strings so that they follow our
> namespacing changes.
>
> I should note that Mark made the suggestion of using INI sections to
> namespace the strings. That would mean that you would just put
> [COM_EXAMPLE] at the top of your INI file and the strings would be
> properly namespaced that way. This would require an API change to
> JText::_ to accomodate stuffing in the namespace as well. It could
> either use a dot notation and we can split it or we can add another
> parameter. The first option would probably be better if we were going
> to go down the road of namespacing the strings use INI sections.
>
> So... I think that is enough for this first post. Would be
> interested in hearing comments or suggestions.
>
> Ian
>

> --
>
> You received this message because you are subscribed to the Google
> Groups "Joomla! Framework Development" group.
> To post to this group, send an email to
> joomla-dev...@googlegroups.com.
> To unsubscribe from this group, send email to
> joomla-dev-frame...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/joomla-dev-framework?hl=en-GB.

[Christophe Demko]

25% is fantastic!!!

6 months ago, I was working on a com_localise Joomla!1.6 component
(see joomlacode project). We were thinking about the use of sections
inside an extension for organizing keys.

So if we use sections, it will be a good think to allow several
sections in one component

[com_mycomponent]
...
[com_mycomponent.menus]
...
[com_mycomponent.labels]
...

The _load functions would have to load all com_mycomponent sections
when loading a component.
Using sections inside a component would help translation of Joomla in
foreign languages.

PS: for com_localise, we are waiting for a beta release of Joomla!1.6
before releasing it

[infograf768]

Thanks for the research, Ian.

A few remarks at reading available doc.
1.It looks like this parser is specifically designed to handle
configuration files and that parse_ini_file (if that is the function
you are using) has been updated since php 5.0, thus has not go the
same functionnality in varous versions of php 5.
See: http://php.net/manual/en/function.parse-ini-file.php

2. The main reason to use the double quote would not only be the space
but the utf8 content of the value

3. What would become a string of this kind?
PARAMPAGINATIONRESULTS=Montrer/Masquer l'ordre de pagination des
résultats (ex : 1-4 de 4 )
or one that contains html tags as <br /> or others ?

As you may guess, we have to balance speed and readability/convenience
for Translators.

> *The characters {}|&~![()^" *cannot be used in keys.  There are also

> reserved words for keys: null, yes, no, true, false, on, off, none.
>
> 2. The syntax for the value will also change.  According to the PHP manual, if
> a value in the ini file contains any non-alphanumeric characters it needs to
> be enclosed in double-quotes (").  That will basically mean all of our
> values will have to be enclosed in quotes, as a space is non alpha numeric.
> From what I gather, if you want to include a double-quote in your string,

> you use two double-quotes.  Finally, according to the PHP manual, *the
> {}|&~![()^" *characters have a special meaning in the value.  Unfortunately,

[infograf768]

Found this that could help overpass the value limitations:
http://mach13.com/loose-and-multiline-parse_ini_file-function-in-php

The above strings are OK (after adding the usual ENT_COMPAT, 'UTF-8'
in their test file.

[Hannes Papenberg]

The question is, if this will provide the same performance gain like the
raw PHP function.

Hannes

infograf768 schrieb:

> Found this that could help overpass the value limitations:
> http://mach13.com/loose-and-multiline-parse_ini_file-function-in-php
>
> The above strings are OK (after adding the usual ENT_COMPAT, 'UTF-8'
> in their test file.
>
> On 13 Dec, 17:08, infograf768 <infograf...@gmail.com> wrote:
>
>> Thanks for the research, Ian.
>>
>> A few remarks at reading available doc.
>> 1.It looks like this parser is specifically designed to handle
>> configuration files and that parse_ini_file (if that is the function
>> you are using) has been updated since php 5.0, thus has not go the
>> same functionnality in varous versions of php 5.
>> See:http://php.net/manual/en/function.parse-ini-file.php
>>
>> 2. The main reason to use the double quote would not only be the space
>> but the utf8 content of the value
>>
>> 3. What would become a string of this kind?
>> PARAMPAGINATIONRESULTS=Montrer/Masquer l'ordre de pagination des

>> r�sultats (ex&nbsp;: 1-4 de 4 )

[Andrew Eddie]

Ok, the bad news is PHP 5.2 and PHP 5.3 behave differently. PHP 5.2
doesn't honour the double quote within a literal quoted string (PHP
5.3 does). So that really sucks. That really sinks any chance of
using the native method *sigh*

If you look in the comments, a guy did some tepid profiling of use the
JSON format and it yields comparable performance.
http://www.php.net/manual/en/function.parse-ini-file.php#93807

Upside is we could use any keys (meaning converting keys is not as
high a priority). Downside is both the keys and values need to be
quoted but the format is not radically dissimilar to what we'd need.

Could also profile JM's other suggestion using preg matching
http://mach13.com/loose-and-multiline-parse_ini_file-function-in-php

Bummer.

Regards,
Andrew Eddie
http://www.theartofjoomla.com - the art of becoming a Joomla developer




2009/12/13 infograf768 <infog...@gmail.com>:

[Ian MacLennan]

On Sun, Dec 13, 2009 at 1:34 PM, Andrew Eddie <mamb...@gmail.com> wrote:

Ok, the bad news is PHP 5.2 and PHP 5.3 behave differently.  PHP 5.2
doesn't honour the double quote within a literal quoted string (PHP
5.3 does).  So that really sucks.  That really sinks any chance of
using the native method *sigh*

Hmmm...  that is unfortunate news.  I did do some experimentation, and using a constant would be an option, though admittedly less readable.  e.g.:

In PHP:

define("QUOTE", '"');

and then in our INI files something like: test="This is a test"QUOTE"and I am wondering if a quote got inserted"

 

If you look in the comments, a guy did some tepid profiling of use the
JSON format and it yields comparable performance.
http://www.php.net/manual/en/function.parse-ini-file.php#93807

Upside is we could use any keys (meaning converting keys is not as
high a priority).  Downside is both the keys and values need to be
quoted but the format is not radically dissimilar to what we'd need.

JSON does look appealing as well, and the format is also fairly clean.  I think this is certainly a workable option.

 

Could also profile JM's other suggestion using preg matching

http://mach13.com/loose-and-multiline-parse_ini_file-function-in-php

We could...  if somebody wants to rebuild the _load function with that code then I can certainly give it a run through the profiler.

 

Bummer.

I think we have good options - the conversion to JSON might be marginally more work but IMO it is worth it for the performance gain that it delivers.  The constant option with INI files is awkward but would also work.  I think one of these options is worth pursuing.

[Andrew Eddie]

2009/12/13 Ian MacLennan <ian.ma...@joomla.org>:

>
> I think we have good options - the conversion to JSON might be marginally
> more work but IMO it is worth it for the performance gain that it delivers.
>  The constant option with INI files is awkward but would also work.  I think
> one of these options is worth pursuing.
>

JM, could you ask your Translation Teams to give this some thought,
particularly moving to a JSON format.

The difference would be something like:

JDISABLED=Disabled
JENABLED=Enabled
JNO=No
JYES=Yes

to

{
'JDISABLED': 'Disabled',
'JENABLED': 'Enabled',
'JNO': 'No',
'JYES': 'Yes'
}

Still human readable but keys don't have to be as strict.

[JM Simonet]

Andrew, here are my first thoughts:

A. Concerning TTs, looks like it would not be a major issue -at this stage- using traditional tools (UTF8 Text Editors or com_localize) for 2 reasons:

1. Unhappily no one has proposed a script to port 1.5 lang files to 1.6 format, whichever it is, so TTs will anyway have to redo most of the work, string per string. As the tokens/keys will all be non-explicit as to the value given to them, it will be a tedious one.

2. I guess we can modify com_localise to fit (Correct me Christophe if I mistake).

Side note: we would anyway face the same issue with the function I quoted above, as the only practical difference is the single quote vs the double quote in the value.

B. This new JSON format creates more work to correct 1.5 Extensions language files. I know some which have been already using the new format for token/keys introduced in 1.6 to be able to port them easily to 1.6 when ready.

Some macros could be used to mass change Return to 'Return' and = to '=', but the { and } would have to be done manually.

C. Also, we have been recently testing some collaborative tools (using XLIFF) for example (yep, there si a joomla component translating inis to xliff and back), and we have to test if these would work with this new format. Format of a XLIFF file is this type:

-----------

<?xml version='1.0' encoding='utf-8'?>

<xliff xmlns="urn:oasis:names:tc:xliff:document:1.2" version="1.2">
<file original="jolomeaFrontOfficeIniJoomla_com_banners" source-language="en-GB" target-language="fr-FR" datatype="htmlbody">
<body>
<trans-unit id="BNR_CLIENT_NAME">
<source><![CDATA[You must select a name for the Client.]]></source>
 <target><![CDATA[Vous devez sélectionner un nom pour l'annonceur.]]></target>

</trans-unit>

-------------

I wonder if adding the single quotes in the tokens/keys and the { } would not break all.

JM

JM, could you ask your Translation Teams to give this some thought,
particularly moving to a JSON format.

The difference would be something like:

JDISABLED=Disabled
JENABLED=Enabled
JNO=No
JYES=Yes

to

{
'JDISABLED': 'Disabled',
'JENABLED': 'Enabled',
'JNO': 'No',
'JYES': 'Yes'
}

Still human readable but keys don't have to be as strict.

Regards,
Andrew Eddie
http://www.theartofjoomla.com - the art of becoming a Joomla developer

--

You received this message because you are subscribed to the Google Groups "Joomla! Framework Development" group.
To post to this group, send an email to joomla-dev...@googlegroups.com.
To unsubscribe from this group, send email to joomla-dev-frame...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/joomla-dev-framework?hl=en-GB.

-- 

Please keep the Subject wording in your answers

This e-mail and any attachments may be confidential. You must not disclose or use the information contained in this e-mail if you are not the
intended recipient. If you have received this e-mail in error, please notify us immediately and delete the e-mail and all copies.

-----------------------------------------------------------

Jean-Marie Simonet  /  infograf768

Joomla! Translation Coordination Team 

[infograf768]

On 14 Dec, 04:58, Ian MacLennan <ian.maclen...@joomla.org> wrote:
> > Could also profile JM's other suggestion using preg matching
> >http://mach13.com/loose-and-multiline-parse_ini_file-function-in-php

> We could...  if somebody wants to rebuild the _load function with that code

Could you share the changes you introduced to use parse_ini_file in he
hope that a volunteers shows up?

>
> > You received this message because you are subscribed to the Google Groups
> > "Joomla! Framework Development" group.
> > To post to this group, send an email to
> > joomla-dev...@googlegroups.com.
> > To unsubscribe from this group, send email to

> > joomla-dev-frame...@googlegroups.com<joomla-dev-framework%2Bunsu...@googlegroups.com>
> > .

[Christophe Demko]

I think we could propose a way to solve this problem:

We could create a compiled version (in a .php file) of a .ini file
when it is loaded for the first time or when it has been modified.

So in the administrator/language/en-GB/ directory there would be for
example a en-GB.ini file and a en-GB.php file which contains a helper
class:

en-GB.php (automatically generated by Joomla framework):

<?php
defined('_JEXEC') or die();

abstract class en_GBLanguageHelper
{
public static function load()
{
$obj = new JObject();
$obj->{'JDISABLED'}='Disabled';
$obj->{'JYES'}='Yes';
....
return $obj;
}
}


When the JLanguage;;_load function try to load a language file:

it looks at the existence of its .php version
if it does not exists
it load the .ini file (using JRegistry::loadINI ) and it generates
its .php version
else
it compares modification date between the .ini and the .php one
if .ini is more recent than the .php one
it load the .ini file (using JRegistry::loadINI ) and it
regenerates its .php version
else
it load the .php one and call en_GBLanguageHelper::load();
endif
endif

For language files stored in extension directory (1.6 facility),
the .php version would be stored in the Joomla directory (JPATH_BASE/
administrator/language or JPATH_BASE/language)

Ch.D

[Christophe Demko]

> For language files stored in extension directory (1.6 facility),
> the .php version would be stored in the Joomla directory (JPATH_BASE/
> administrator/language or JPATH_BASE/language)

After studying the code, it will be easier to store the .php file in
the .ini directory (in these special cases)
Ch.D

[infograf768]

Christophe, would that mean that each ini file would have his php
counterpart?

[Christophe Demko]

Yep but automatically generated. I'm currently trying to write the
patch for J!1.6

Ch.D

[Gergo Erdosi]

What's exactly the problem with the native INI format? I read the manual and the comments but couldn't find any issues which would make this format impossible to use.

---

Gergo Erdosi

[Christophe Demko]

Hi Gergo,

Because of the notes of the ttp://www.php.net/manual/en/function.parse-ini-file.php
function

Note: This function has nothing to do with the php.ini file. It is
already processed by the time you run your script. This function can
be used to read in your own application's configuration files.

Note: If a value in the ini file contains any non-alphanumeric

characters it needs to be enclosed in double-quotes (").

Note: There are reserved words which must not be used as keys for
ini files. These include: null, yes, no, true, false, on, off, none.
Values null, no and false results in "", yes and true results in "1".
Characters {}|&~![()^" must not be used anywhere in the key and have a

special meaning in the value.

> >http://www.theartofjoomla.com- the art of becoming a Joomla developer
>
> > 2009/12/13 infograf768 <infograf...@gmail.com>:

> > > For more options, visit this group athttp://groups.google.com/group/joomla-dev-framework?hl=en-GB.

>
> > --
>
> > You received this message because you are subscribed to the Google Groups "Joomla! Framework Development" group.
> > To post to this group, send an email to joomla-dev...@googlegroups.com.
> > To unsubscribe from this group, send email to joomla-dev-frame...@googlegroups.com.

> > For more options, visit this group athttp://groups.google.com/group/joomla-dev-framework?hl=en-GB.

[Gergo Erdosi]

Hi Christophe,

Thanks for your reply. As I said, I read the manual. I just don't see what's exactly the issue which restricts the use of this format in our case. However I agree, it's not easy to work with native INI files. So I support JSON. It can be used in a lot of cases, it is able to differentiate string and numeric types etc. So it's not a problem for me if we use JSON instead of INI, I just wanted to know what was the problem with it.

---
Gergo Erdosi

[Ian MacLennan]

@Gergo

The issue is, as Andrew noted above, it is awkward to include quotes in language strings in PHP 5.2.  5.3 allows you to use "" to represent a double quote in a value, but this does not work in PHP 5.2.  A possibility is to use a constant (so you would have "QUOTE") but this is uglier.

Ian

[Ian MacLennan]

Sure...  take a look at libraries/joomla/language/language.php in the jstring_16 branch.

1. Unhappily no one has proposed a script to port 1.5 lang files to 1.6 format, whichever it is, so TTs will anyway have to redo most of the work, string per string. As the tokens/keys will all be non-explicit as to the value given to them, it will be a tedious one.

Once we decide on a format, I will provide a script.  I have one that I used for profiling with the new INI format, but it does not properly convert things in light of these rules.  I left it at home unfortunately, but I can post the quick and dirty one tonight (proper use relies on the find command, which isn't available on Windows, but I can make a version that will work better on Windows as well).

And, this could actually be a part of the migration process, and would be a rather easy step.  This is certainly a huge advantage of going with JSON rather than INI, because it doesn't require keys to be changed.

Ian

 

--

You received this message because you are subscribed to the Google Groups "Joomla! Framework Development" group.
To post to this group, send an email to joomla-dev...@googlegroups.com.

To unsubscribe from this group, send email to joomla-dev-frame...@googlegroups.com.

[Gergo Erdosi]

I see, thanks! But can't it be solved by escaping the quote (\")? However it doesn't really matter, because as I can see all of us support to go with JSON.

---

Gergo Erdosi

[Christophe Demko]

I've made a test for estimating the gain if we generate a .php file
for each .ini file

So these are the results:

on administrator/index.php page

The gain is 11%.
The lost to generate the .php file is 35% (but it is lost only one
time)
The ini file are compatible with 1.5 .ini file


It can be testing using the chdemko branch (http://joomlacode.org/svn/
joomla/development/branches/chdemko)

On 14 déc, 15:27, Ian MacLennan <ian.maclen...@joomla.org> wrote:
> Sure...  take a look at libraries/joomla/language/language.php in the
> jstring_16 branch.
>
> 1. Unhappily no one has proposed a script to port 1.5 lang files to 1.6
>
> > format, whichever it is, so TTs will anyway have to redo most of the work,
> > string per string. As the tokens/keys will all be non-explicit as to the
> > value given to them, it will be a tedious one.
>
> Once we decide on a format, I will provide a script.  I have one that I used
> for profiling with the new INI format, but it does not properly convert
> things in light of these rules.  I left it at home unfortunately, but I can
> post the quick and dirty one tonight (proper use relies on the find command,
> which isn't available on Windows, but I can make a version that will work
> better on Windows as well).
>
> And, this could actually be a part of the migration process, and would be a
> rather easy step.  This is certainly a huge advantage of going with JSON
> rather than INI, because it doesn't require keys to be changed.
>
> Ian
>

> On Mon, Dec 14, 2009 at 2:49 AM, infograf768 <infograf...@gmail.com> wrote:
>
> > On 14 Dec, 04:58, Ian MacLennan <ian.maclen...@joomla.org> wrote:
> > > > Could also profile JM's other suggestion using preg matching
> > > >http://mach13.com/loose-and-multiline-parse_ini_file-function-in-php
>
> > >  We could...  if somebody wants to rebuild the _load function with that
> > code
>
> > Could you share the changes you introduced to use parse_ini_file in he
> > hope that a volunteers shows up?
>
> > > > You received this message because you are subscribed to the Google
> > Groups
> > > > "Joomla! Framework Development" group.
> > > > To post to this group, send an email to
> > > > joomla-dev...@googlegroups.com.
> > > > To unsubscribe from this group, send email to
> > > > joomla-dev-frame...@googlegroups.com<joomla-dev-framework%2Bunsu...@googlegroups.com>

> > <joomla-dev-framework%2Bunsu...@googlegroups.com<joomla-dev-framework%252Buns...@googlegroups.com>

[Niels Braczek]

Ian MacLennan schrieb:

> The issue is, as Andrew noted above, it is awkward to include quotes in
> language strings in PHP 5.2.

That's not true - if you use the INI format *correctly*, there's no
problem with quotes. I use it at some places.

Example:
CLICK_CONTINUE="Please click on the \"continue\" button."

works as expected.

Regards,
Niels

--
| http://www.kolleg.de · Das Portal der Kollegs in Deutschland |
| http://www.bsds.de · BSDS Braczek Software- und DatenSysteme |
| Webdesign · Webhosting · e-Commerce · Joomla! Content Management |
------------------------------------------------------------------

[Ian MacLennan]

Hmmm...  for some reason I couldn't get it to work.  It must have been late :)  I'll have to play with it some more.  I did try escaping it.

Ian

[Niels Braczek]

Ian MacLennan schrieb:

> Hmmm... for some reason I couldn't get it to work. It must have been late
> :) I'll have to play with it some more. I did try escaping it.

Did you enclose the string with quotes?

[Ian MacLennan]

On 14/12/2009 9:55 AM, Niels Braczek wrote:
> Ian MacLennan schrieb:
>
>> Hmmm... for some reason I couldn't get it to work. It must have been late
>> :) I'll have to play with it some more. I did try escaping it.
>
> Did you enclose the string with quotes?
>
> Regards,
> Niels
>

Yes, I did. What may have happened is that I may have been testing
quotes and parse_ini_file may have been returning false. I might have
assumed that it was returning false because of the unescaped quotes
where in fact it was actually due to keywords.

Ian

[Gergo Erdosi]

I tested now and it appears to be a bug: http://bugs.php.net/bug.php?id=38052

---
Gergo Erdosi

[Andrew Eddie]

2009/12/14 JM Simonet <infog...@gmail.com>:

> Andrew, here are my first thoughts:
> A. Concerning TTs, looks like it would not be a major issue -at this stage-
> using traditional tools (UTF8 Text Editors or com_localize) for 2 reasons:

> 1. Unhappily no one has proposed a script to port 1.5 lang files to 1.6
> format, whichever it is, so TTs will anyway have to redo most of the work,
> string per string. As the tokens/keys will all be non-explicit as to the
> value given to them, it will be a tedious one.

Happily we are looking into that option and someone is preparing a
proof of concept :)

It would also not be too much trouble for com_localise to load in .ini
format and output JSON format, if that's the way we feel is best to
go.

[JM Simonet]

>Ian MacLennan schrieb:
>
>> The issue is, as Andrew noted above, it is awkward to include quotes in
>> language strings in PHP 5.2.
>
>That's not true - if you use the INI format *correctly*, there's no
>problem with quotes. I use it at some places.
>
>Example:
>CLICK_CONTINUE="Please click on the \"continue\" button."
>
>works as expected.
>
>Regards,
>Niels

It works on php 5.3 but not when php < 5.3 as we discovered with
Andrew yesterday.

>However it doesn't really matter, because as I can see all of us
>support to go with JSON.

I have posted above some concerns about using JSON.

>--
>
>You received this message because you are subscribed to the Google
>Groups "Joomla! Framework Development" group.
>To post to this group, send an email to joomla-dev...@googlegroups.com.
>To unsubscribe from this group, send email to
>joomla-dev-frame...@googlegroups.com.
>For more options, visit this group at
>http://groups.google.com/group/joomla-dev-framework?hl=en-GB.


--

[JM Simonet]

Sure...  take a look at libraries/joomla/language/language.php in the jstring_16 branch.

1. Unhappily no one has proposed a script to port 1.5 lang files to 1.6 format, whichever it is, so TTs will anyway have to redo most of the work, string per string. As the tokens/keys will all be non-explicit as to the value given to them, it will be a tedious one.

Once we decide on a format, I will provide a script.  I have one that I used for profiling with the new INI format, but it does not properly convert things in light of these rules.  I left it at home unfortunately, but I can post the quick and dirty one tonight (proper use relies on the find command, which isn't available on Windows, but I can make a version that will work better on Windows as well).

And, this could actually be a part of the migration process, and would be a rather easy step.  This is certainly a huge advantage of going with JSON rather than INI, because it doesn't require keys to be changed.

Ian

Not taking into account a possible JSON format or even a new format, the keys/token have already started to be changed to the underscores and prefix format.

How is it possible to migrate 1.5 strings to 1.6?

 

--

You received this message because you are subscribed to the Google Groups "Joomla! Framework Development" group.
To post to this group, send an email to joomla-dev...@googlegroups.com.
To unsubscribe from this group, send email to joomla-dev-frame...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/joomla-dev-framework?hl=en-GB.

[Christophe Demko]

I've made another test, generating a ".txt" file (an .ini file
compatible with parse_ini_file) from the original .ini file using the
same algorithm

The reduction is 14%
Ch.D

[Christophe Demko]

You can test the ".txt" version using that
http://groups.google.fr/group/joomla-dev-framework/web/language.php
librairie language file (libraries/joomla/language/language.php)

This patch can handle 1.5 version and generate a parse_ini_file
compatible file.

[infograf768]

I tested your solution, Christophe, and it works real fine.
The only warnings I got were from badly written tokens containing what
I guess are forbidden characters
examples:
(TODAY)=
ADMINISTRATION TEMPLATE FOR JOOMLA! 1.6=

I guess ! and ( or ) an some others should be forbidden in the tokens.

On 14 Dec, 17:52, Christophe Demko <chde...@gmail.com> wrote:
> You can test the ".txt" version using thathttp://groups.google.fr/group/joomla-dev-framework/web/language.php

[infograf768]

Hmm, sorry, The files are indeed correctly parsed to .txt but I tested
the double quote and as forecasted it will not work here with
parse_ini_file on php < 5.3

[Karol Čejka]

hi everybody.
I agree with Christophe on many points.
I think the best way is:
- Use INI format for translators' work. This format is ideal with text-
format work (notepad) and exist many IDEs for "properties" file from
another languages (for example java IDEs)
- INI file must have keys without spaces. It make mistake with
existing properties editors, and also this is better in many cases.
- INI file can have free of usage quotes. Quotes is not important in
this file, because INI file don't be used in production. This is only
file for compilation. Quotes can be replaced during the compilation
process to the correct format.
- INI file will be compiled to native PHP class. Native class must be
compiled by user fired from backend. Automatic compilation is nice
thing, but can produce mistakes in live web site. Compilation process
can have also validation part of INI file.
- Compiled version of lang file must be in native PHP class format.
JSON is popular and nice thing, but JSON is text representation of
PHP objects, and must be converted from text to object. For
performance reason is better use native PHP class.
- Migration translations from 1.5 to 1.6 will be easier with INI files
than use another format or process, current translators know how to
work with ini and many of us have tools for that work.

I haven't any performance test, but the logic is that native PHP class
shall be faster than JSON format (JSON is ideal for AJAX, and
persistence to database, but not in this case).
As Christophe wrote, compiled translation file to native PHP class is
a good way for everybody.

Karol

[Christophe Demko]

Hi Karol,

It seems that the reading of a ini file by the parse_ini_file function
is faster than its .php counterpart.
But some serious tests have to be done confirming this point.

Christophe

[Ian MacLennan]

That is actually fairly performant, although still slower than the native calls.  That function takes about 4 times as long as the natives (likely JSON or INI wouldn't make much difference), but still saves us a lot of processing time.

Ian

On Sun, Dec 13, 2009 at 11:56 AM, infograf768 <infog...@gmail.com> wrote:

Found this that could help overpass the value limitations:
http://mach13.com/loose-and-multiline-parse_ini_file-function-in-php

The above strings are OK (after adding the usual ENT_COMPAT, 'UTF-8'
in their test file.

[JM Simonet]

That is actually fairly performant, although still slower than the native calls.  That function takes about 4 times as long as the natives (likely JSON or INI wouldn't make much difference), but still saves us a lot of processing time.

Ian

Ian, could we have an idea on real-time differences and not percents?

For example: loading a specific admin page with cache off with each solution in seconds.

On Sun, Dec 13, 2009 at 11:56 AM, infograf768 <infog...@gmail.com> wrote:

Found this that could help overpass the value limitations:
http://mach13.com/loose-and-multiline-parse_ini_file-function-in-php

The above strings are OK (after adding the usual ENT_COMPAT, 'UTF-8'

in their test file.

--

You received this message because you are subscribed to the Google Groups "Joomla! Framework Development" group.
To post to this group, send an email to joomla-dev...@googlegroups.com.
To unsubscribe from this group, send email to joomla-dev-frame...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/joomla-dev-framework?hl=en-GB.

[Ian MacLennan]

On 16/12/2009 1:20 AM, JM Simonet wrote:
>> That is actually fairly performant, although still slower than the
>> native calls. That function takes about 4 times as long as the
>> natives (likely JSON or INI wouldn't make much difference), but
>> still saves us a lot of processing time.
>>
>> Ian
>
> Ian, could we have an idea on real-time differences and not percents?
> For example: loading a specific admin page with cache off with each
> solution in seconds.
>

I can do that, but it's a rather useless number because it is machine
dependent and thus isn't something I'd be too inclined to waste my time
on. A recent page run rendered in .161 seconds, so that would mean
currently language processing is taking about .04 seconds. With native
INI handling it would take about .12 seconds.

Ian

[Christophe Demko]

Hi to all,

I want to make several remarks concerning this point.

1. language files (when seen by third party developers, joomla core
translators and third party translators) must be as simple as
possible. The actual Joomla ini format is actually well known and easy
to read by human
2. language files processing must be as fast as possible. The native
ini php format or the json format are probably the fastest way to read
them. My previous tests have shown that the native ini files have some
problems in the characters which are allowed in the key part of a key-
value pair. So the .json format is certainly the most accurate

We have to combine these two issues because we have to facilitate the
work done by translators and to speed up Joomla site.

Actually, administrator/language/**-**/ and language/**-**/ are
writable directories for allowing installation of languages files.

We could have
* a global option for a joomla site: "do you want to automatically
create compiled version of language files?" If this option is set to
TRUE by the administrator, then the administrator/language/**-**/ and
language/**-**/ directories have to be writable and json files could
be created on the fly when a language file (the ini joomla version) is
loaded for the first time or when it has been modified by hand.
* an ability for the administrator to create (or update) these .json
language files using the admin ui (a well chosen button in the
language manager). The administrator/language/**-**/ and
language/**-**/ directories have also to be writable but only when the
admin update or create language files.
* a way in the install manager to create these .json files when an
extension is installed (or updated)

I could provide a patch for these features (if they are asked by the
community) at the begining of january.

Ch.D

[Hannes Papenberg]

Hi Christophe,
those points are valid and right. I want to make one other remark that
is crucial to me: All extensions have to ship with the final files used
by the system to translate strings.

This means, that I don't want to have a system where we have to check if
a file is up-to-date and may get created on the fly by our system. This
has two reasons, the first and most important one is the usability, the
second one being the performance. In terms of usability, it is one more
folder that needs to be writable by PHP if we start creating files on
the fly and I just want to remind people of how difficult it still is to
just get people through the normal installation with writing the
configuration.php. This is also related to security. Even if there is no
security problem, people will feel uneasy about having a folder made
writable permanently for normal operations. (Yes, we are having the
folder writable now already, but from the users perspective this is
quite different. Explaining that the folder needs to be writable during
installation of an extension and explaining the complicated reasons for
this system, are two entirely different things.)

In terms of performance it is again a whole series of filesystem
lookups, where we also need checks if a file has been changed and that
needs to be done on each pageload. It might not eat up all of our
performance gain by using the native functions, but it will deminish the
achievable result by a great deal.

That said, I want us to find a solution that solves this issue without
such an intricate system. Maybe we could encode the problematic
characters in our INI-strings and then still use the INI-format.

Hannes

> On 16 d�c, 15:38, Ian MacLennan <ian.maclen...@joomla.org> wrote:
>
>> On 16/12/2009 1:20 AM, JM Simonet wrote:>> That is actually fairly performant, although still slower than the
>>
>>>> native calls. That function takes about 4 times as long as the
>>>> natives (likely JSON or INI wouldn't make much difference), but
>>>> still saves us a lot of processing time.
>>>>
>>
>>>> Ian
>>>>
>>
>>> Ian, could we have an idea on real-time differences and not percents?
>>> For example: loading a specific admin page with cache off with each
>>> solution in seconds.
>>>
>> I can do that, but it's a rather useless number because it is machine
>> dependent and thus isn't something I'd be too inclined to waste my time
>> on. A recent page run rendered in .161 seconds, so that would mean
>> currently language processing is taking about .04 seconds. With native
>> INI handling it would take about .12 seconds.
>>
>> Ian
>>

[Christophe Demko]

Hi Hannes, I understand your remarks, especially the security and
performances issues.

My proposal was designed for two problematic:
-using ini (joomla format) files for developers and translators
-using json files for dispaying site pages

I answer inline

On 30 déc, 01:02, Hannes Papenberg <hackwa...@googlemail.com> wrote:
> Hi Christophe,
> those points are valid and right. I want to make one other remark that
> is crucial to me: All extensions have to ship with the final files used
> by the system to translate strings.

The installation manager could create these "compiled" files when
installation is done (since "language" folders are writable when
installing an extension)

>
> This means, that I don't want to have a system where we have to check if
> a file is up-to-date and may get created on the fly by our system. This
> has two reasons, the first and most important one is the usability, the
> second one being the performance. In terms of usability, it is one more
> folder that needs to be writable by PHP if we start creating files on
> the fly and I just want to remind people of how difficult it still is to
> just get people through the normal installation with writing the
> configuration.php.

This is why I suggest to have a global parameter "do you want to
automatically
create/update compiled version of language files?" which can be set to
false by default. May be this solution is not adapted and may be we
have to replace this solution by loading the json file if it exists
and the ini file else. The json files will be always located in
administrator/language/**-** or language/**-** (since created by the
installer) and the ini files will be located in the same folders or in
the third party folder.

> This is also related to security. Even if there is no
> security problem, people will feel uneasy about having a folder made
> writable permanently for normal operations. (Yes, we are having the
> folder writable now already, but from the users perspective this is
> quite different. Explaining that the folder needs to be writable during
> installation of an extension and explaining the complicated reasons for
> this system, are two entirely different things.)
>
> In terms of performance it is again a whole series of filesystem
> lookups, where we also need checks if a file has been changed and that
> needs to be done on each pageload. It might not eat up all of our
> performance gain by using the native functions, but it will deminish the
> achievable result by a great deal.

see my remark about the global parameter

[ErikThe3rd]

Compiled files can be created at any time the system is toggled from
offline to online state and also by an explicit backend command . We
need no at-runtime checker/plugin.

Erik

Christophe Demko schrieb: