1.6 and languages/internationalisation

50 views
Skip to the first unread message

infograf768

unread,
5 Aug 2009, 23:41:3805/08/2009
to Joomla! CMS Development
This is a proposal I made to Andrew in June about 1.6 and languages/
internationalisation.
Stayed a bit on the back burner but better put it here as a reminder
if considered as possible to implement.
A similar proposal was accepted in the whitepapers. Now that we see
1.6 coming to life, it can be looked at more concretely.

This implies adding language db fields in many parts of 1.6.
The purpose is not to replace a component as Joomfish (although it
would make its coding easier) but a basic implementation of language
choice for contents as this feature, added to our internationalisation
of the interface, is eagerly awaited for.

1. Create a language field in the category table, this choice goes
down to sub-categories
2. Automatically, when we create content in a category it adds the
language tag in a table field for the article (this may be used for
other purposes than com_content/com_contacts, etc.). Change is
possible.
3. When creating uncategorized articles, there is a choice for
language too in the db field
4. Depending on Category chosen (or simple article) menu item is
tagged with a language, also in db
5. Each module is tagged with a language (default = all)
6. List of available languages is taken from db and/or from language
packs installed in "site/language/
7. Frontpage component menu item would let choose one or many
languages to display
8. We do need filters per language in the admin lists of various items

I think that all this would be possible in 1.6 if our basic target is
limited to change languages globally through a module on frontend.
i.e. if we are not trying to provide a joomfish like language choice
on all pages which switches to the equivalent of that page in the said
language.
Therefore it should not require a specific router.

Andrew Eddie

unread,
5 Aug 2009, 23:55:4705/08/2009
to joomla-...@googlegroups.com
I think Christophe (unless my memory is lapsing, which it probably the
case) was going to try to add the DB driven list to com_languages (#6)
allowing additional languages to the localisation packs installed. If
that's the case (#8) is totally feasible.

However, I don't recall the other points - some of that cascading
behaviour would not be trivial to implement.

Regards,
Andrew Eddie
http://www.theartofjoomla.com - the art of becoming a Joomla developer




2009/8/6 infograf768 <infog...@gmail.com>:

Christophe Demko

unread,
6 Aug 2009, 05:59:0306/08/2009
to Joomla! CMS Development
I'm ok for developing these ideas, but I'm currently on summer
holidays (near the "cote d'azur"). I will be back 25th of august. I
will have a phone contact with Jean-Marie Simonet.

Cheers,
Ch.D

On 6 août, 05:55, Andrew Eddie <mambob...@gmail.com> wrote:
> I think Christophe (unless my memory is lapsing, which it probably the
> case) was going to try to add the DB driven list to com_languages (#6)
> allowing additional languages to the localisation packs installed.  If
> that's the case (#8) is totally feasible.
>
> However, I don't recall the other points - some of that cascading
> behaviour would not be trivial to implement.
>
> Regards,
> Andrew Eddiehttp://www.theartofjoomla.com- the art of becoming a Joomla developer
>
> 2009/8/6 infograf768 <infograf...@gmail.com>:

Hannes Papenberg

unread,
7 Aug 2009, 13:59:2207/08/2009
to joomla-...@googlegroups.com
At the moment I would opt for adding the language fields and filling it
by default with the default language for the site. Implementing complete
multi-linguality in 1.6 however seems to be too much. We would need a
lot more work on this and I'm not willing to wait for people to finish
this to ship 1.6. Not only would we have to add those extra fields, we
would also have to split up the #__content table to keep the translated
article under the same URL and with identical parameters, etc. At least
in my vision, the URL (and appearance) for an article should be
identical to the one of its translated counterpart, just differing on a
language string. That said, I see the benefit of adding the appropriate
fields to the modules and menu table and for the time being also to the
tables of the different content components shipped with the core. That
way extensions like Joomfish can do their magic. But I don't think that
in 1.6 we actively can implement the multi-lingual content/site.

Hannes

Christophe Demko schrieb:

Beat

unread,
7 Aug 2009, 19:52:1807/08/2009
to Joomla! CMS Development
Not sure the URL criteria is an important one.

Specially not in sefed ones, you would want the right language to
appear in the link.

But having a fixed clear id-relation between the translations is
needed.

So adding a language field to articles, categories, menus and modules
would probably be a good first step, but then you need also to have a
common "id", different from the table key, and which is same for the
same item in each language. And that's just to start, quite sure of
forgetting other things.

Best Regards,
Beat
http://www.joomlapolis.com/
> >> Andrew Eddiehttp://www.theartofjoomla.com-the art of becoming a Joomla developer

JM Simonet

unread,
8 Aug 2009, 01:29:1008/08/2009
to joomla-...@googlegroups.com
I was not in any way describing a Joomfish-like
structure where there are Translated
counterparts, Hannes.
The paper only deals with tagging items to a
specific language and displaying a site in one or
the other language.
In facts, contents can be totally different depending on language chosen.
JM

Hannes Papenberg

unread,
8 Aug 2009, 04:22:4208/08/2009
to joomla-...@googlegroups.com
I'm not against adding those fields to the tables, I just don't think
that we are able to implement any special code that goes further than
storing the current sites default language in that field. Everything
beyond that would be the task of a third party extension like Joomfish
or something similar.

Hannes

JM Simonet schrieb:

Amy Stephen

unread,
8 Aug 2009, 15:10:1308/08/2009
to Joomla! CMS Development
I welcome an improved basis for translation systems for Joomla!. I
want to suggest another approach that might make it easier to create
multi-lingual Joomla! Web sites.

In com_content, there are two unused data elements: parent_id and
language. A Translation Extension could use the parent_id data element
to hold the primary key of the original document, thus keeping
together source document and all translations. If those two data
elements were added to other source document tables, like banners,
contact_details, newsfeeds, web links, and third party extension
tables, that method could be used by Translation Extensions, across
the board.

The other core change required would be to data retrieval queries.
Those would have to be changed to include a where clause that matches
the user's frontend language to the content language. (It would be up
to the translation extension to make certain a row exists for every
language offered or query results would not be returned.)
Using this approach should enable the Interface to continue to operate
in a language-agnostic fashion. That would simplify the process of
building a multi-lingual site since a Web Site Developer would
continue to create only one Menu Item and Module, etc., for each
function needed, without regard to language, and let Joomla! return
language-specific results.

There are always lots of ways to do things and tackling this challenge
with Categories certainly could get the job done. Doing so, however,
introduces 3NF data model issues which translates to less application
flexibility. For that reason, I wanted to throw another idea out there
that might be worth considering.

Gergő Erdősi

unread,
12 Aug 2009, 14:51:5512/08/2009
to joomla-...@googlegroups.com
Hi,

Just a note: The fact that we have an unused parent_id column doesn't
mean that we can use it for something that has nothing to do with
parents. :) If a column is required for multilingual content, then we
can add a new, more meaningful column. Migration is not problem, we
just need to alter the table.

--
Gergő Erdősi



2009/8/8 Amy Stephen <amyst...@gmail.com>:

Amy Stephen

unread,
12 Aug 2009, 15:24:3312/08/2009
to Joomla! CMS Development
I created a patch http://joomlacode.org/gf/project/joomla/tracker/?action=TrackerItemEdit&tracker_item_id=17601

Purpose: Prepare database for third party translation extensions.

Specifics: Add a language field that identifies ISO 639-2 values (or,
alternatively, can handle the ISO 639-1 value which is one character
shorter).
Also, include a data element that can be used to point to the
translated source
document contained within the same table.

Added two data elements:
`language` CHAR( 6 ) NOT NULL DEFAULT '',
`source_id` integer NOT NULL default '0',

To the following tables:
- banner
- bannerclient
- categories
- contact_details
- content
- menu
- modules
- newsfeeds
- weblinks

References:

- ISO Codes http://www.loc.gov/standards/iso639-2/php/code_list.php

- Alex Kempkens White Paper
http://forum.joomla.org/viewtopic.php?f=502&t=276920&start=0 -
deviated
from Alex's proposal by adding a source_id column to track the source
content
and did not add a language table. That could be done by the third
party
translation extension or might already be underway in the com_localise
for 1.6
project (uncertain) http://joomlacode.org/gf/project/com_localise/

- Spoke with JM about this patch and will email Alex for his
feedback.

Question:

- Should source queries be changed to restrict results based on
Language? I
recommend doing so in order to prepare the queries for multi-language
use.
(Such changes would be transparent for single-language sites.)

With the language key in place within the source data, and then used
as a WHERE
condition in the source queries, a Site Visitor's Language could be
used to
return appropriate results.

Translation Extensions would then focus on copying source documents
into a new
row and presenting work to the appropriate translator for other
languages. No
additional logic should be required to present the language needed for
the site
visitor.

I can work through those queries once I know this work is agreed upon.

Airton Torres

unread,
12 Aug 2009, 17:13:3712/08/2009
to joomla-...@googlegroups.com
Just to add to the discussion, although I can see that Amy's patch already
provided for this case, I think it should be standardized the combined use of
the ISO 639-1 or 639-2 language codes and the ISO 3166-1 alpha-2 country codes,
forming language codes like en-GB, pt-BR, eng-US, spa-ES, etc.

This is to force 3pds to use the same schema, since I've seen 3pd extensions
using only the ISO 639-1 / 639-2 codes, because it's the only one alluded to in
any Joomla doc on languages I could find so far.


2009/8/12 Amy Stephen <amyst...@gmail.com>

infograf768

unread,
13 Aug 2009, 03:08:2313/08/2009
to Joomla! CMS Development
While we are at ISO code, let me remind here we need to find a
solution to install Joomla with 3 letters iso code.
Please read
http://groups.google.com/group/joomla-dev-cms/browse_thread/thread/8115a39b78b1abbe?hl=en-GB#

On 12 Aug, 23:13, Airton Torres <airton.tor...@gmail.com> wrote:
> Just to add to the discussion, although I can see that Amy's patch already
> provided for this case, I think it should be standardized the combined use of
> the ISO 639-1 or 639-2 language codes and the ISO 3166-1 alpha-2 country codes,
> forming language codes like en-GB, pt-BR, eng-US, spa-ES, etc.
>
> This is to force 3pds to use the same schema, since I've seen 3pd extensions
> using only the ISO 639-1 / 639-2 codes, because it's the only one alluded to in
> any Joomla doc on languages I could find so far.
>
> 2009/8/12 Amy Stephen <amystep...@gmail.com>
>
>
>
> > I created a patch
> >http://joomlacode.org/gf/project/joomla/tracker/?action=TrackerItemEd...
>
> > Purpose: Prepare database for third party translation extensions.
>
> > Specifics: Add a language field that identifies ISO 639-2 values (or,
> > alternatively, can handle the ISO 639-1 value which is one character
> > shorter).
> > Also, include a data element that can be used to point to the
> > translated source
> > document contained within the same table.
>
> > Added two data elements:
> >  `language` CHAR( 6 ) NOT NULL DEFAULT '',
> >  `source_id` integer NOT NULL default '0',
>
> > To the following tables:
> >  - banner
> >  - bannerclient
> >  - categories
> >  - contact_details
> >  - content
> >  - menu
> >  - modules
> >  - newsfeeds
> >  - weblinks
>
> > References:
>
> >  - ISO Codeshttp://www.loc.gov/standards/iso639-2/php/code_list.php
>
> >  - Alex Kempkens White Paper
> >http://forum.joomla.org/viewtopic.php?f=502&t=276920&start=0-
> > deviated
> > from Alex's proposal by adding a source_id column to track the source
> > content
> > and did not add a language table. That could be done by the third
> > party
> > translation extension or might already be underway in the com_localise
> > for 1.6
> > project (uncertain)http://joomlacode.org/gf/project/com_localise/

Amy Stephen

unread,
13 Aug 2009, 09:01:2013/08/2009
to Joomla! CMS Development
There should be a discussion on standardizing the language codes
consistently. That's probably a good "other subject" since it has
implications to installation, etc. This patch can accomodate any
standard we select. I do think it's important that Joomla! settle on a
standard.

For this patch, I'd like feedback on my recommendation to modify core
queries to restrict result sets based on the frontend language.
Comments?

akede

unread,
13 Aug 2009, 10:13:1913/08/2009
to Joomla! CMS Development
Hi all,

First of all, thx Amy for the note on Twitter ;-)

The white paper I wrote is only a replacement for the very unlucky
implementation of a language tag in the params of com_content
currently. For any 3rd party application that deals with multi lingual
content this would be already perfect.

@Hannes
On 8 Aug, 10:22, Hannes Papenberg <hackwa...@googlemail.com> wrote:
> I'm not against adding those fields to the tables, I just don't think
> that we are able to implement any special code that goes further than
> storing the current sites default language in that field. Everything
> beyond that would be the task of a third party extension like Joomfish
> or something similar.

Please beware that tagging with the "current" code must mean to tag it
with the user code or better the value you get out of
JFactory::getLanguage()->getTag(). What ever a 3rd party system does
with the language object should be fine for you to add while writing
new content or other information.

What I mean - it's not the site default language, it should be the
language the user is using at a specific moment.

@Amy
The most important aspect of the white paper was that it also extends
the language reference to the other tables as you mentioned in your
patch.
As JM said we have discussed already the usage of the ISO tag as the
reference and Sam already added the languages as rows for the new
table extensions. We decided to keep the ISO tag as the reference as
this will be valid even so a language might be deleted.

Related your idea with the source field.
I'm not sure if this is a beneficial at this state. This field will
create a kind of revision reference to the original content which has
a lot of dependencies or additional requirements. Personally I would
leave the handling of these revisions to the 3rd party extensions as
you can either solve it the way Joom!Fish is doing it or you can also
use something like the currently developed versioning system for such
a solution. It depends on the needs. In all cases having a source_id
field will not be relevant for the extension as they need to maintain
the references themselves. I know its a change to my original white
paper.

@Gergo
Yes we basically said already that there should be a new column called
language_tag


My notes about the questions:
> - Should source queries be changed to restrict results based on Language?

I would say no.

The reason is that adding a where element for only the articles will
create a complex understanding why it is being implemented for
articles but not for other tables/information. The other topic is that
the queries can become really complex and in these cases the condition
will not be simple anymore. Side effects are possible.

>Translation Extensions would then focus on copying source documents into a new
>row and presenting work to the appropriate translator for other
>languages. No
>additional logic should be required to present the language needed for
>the site
>visitor.

I would not second this. There are many more issues you need to focus
on and just doing a copy of an row is also not causing the fact that
less additional logic is used. For one table such as content this
might be right, but still the result for the user and admin might be
quite complicate. The primary issue at this point is the reference to
other entry in our database (e.g. menus, categories, ...).

Planing this for 2.0 from scratch will help and I know Louis has
already some ideas.

So far my first comments

Alex

PS: @Christophe, I hope after my vacation (15th of Sept) I will have
some more time to help you with that. Sorry. my plan wasto have this
done yet.

Amy Stephen

unread,
13 Aug 2009, 11:22:3713/08/2009
to joomla-...@googlegroups.com
Thanks so much, Alex for joining in.

I want to continue discussing a couple of items with you, please.

On Thu, Aug 13, 2009 at 9:13 AM, akede <alex.k...@gmail.com> wrote:

Hi all,

First of all, thx Amy for the note on Twitter ;-)

The white paper I wrote is only a replacement for the very unlucky
implementation of a language tag in the params of com_content
currently. For any 3rd party application that deals with multi lingual
content this would be already perfect.

Related your idea with the source field. 

I'm not sure if this is a beneficial at this state. This field will
create a kind of revision reference to the original content which has
a lot of dependencies or additional requirements. Personally I would
leave the handling of these revisions to the 3rd party extensions as
you can either solve it the way Joom!Fish is doing it or you can also
use something like the currently developed versioning system for such
a solution. It depends on the needs. In all cases having a source_id
field will not be relevant for the extension as they need to maintain
the references themselves. I know its a change to my original white
paper.

When I think "revision" I think version control. source_id would not be used to track changes but rather to identify the primary source document and it's translations. This is true not only for com_content but for EACH source table identified in my patch.

For example, let's assume a multi-language implementation with English as the primary language and German and French as translations.

Author writes article entitled "Hello World" in English - and the id is 1 - and the source_id is 0.

The translation extension identifies the "Hello World" article with a source_id of 0 and recognizes there are no French or German translations. So, the translation extension  makes a copy available for the French and German translators. Each copy has a source_id value of 1 (the primary key of the original source) and the language attribute identifies French for one copy and German, for the other.

Now, we have three rows in the source table for the same article - each in a different language.

 
My notes about the questions:
>  - Should source queries be changed to restrict results based on Language?

I would say no.

The reason is that adding a where element for only the articles will
create a complex understanding why it is being implemented for
articles but not for other tables/information. The other topic is that
the queries can become really complex and in these cases the condition
will not be simple anymore. Side effects are possible.


Adding the query conditions is extremely important.  If we don't change the query to restrict results to the frontend language of the user, three items will return for the "Hello World" document. We only want one to return, and that is the item in the visitor's language.

It is not complex, in fact, it is very simple. Determine the visitors frontend language. Put that value into a variable. Use the variable in a WHERE clause. Piece of cake. Certainly side effects are possible - but those side effects would come from a bug to this simple logic.

What is the User's Frontend Language?

1. If they are logged on, it's defined in the user object.
2. If they are not logged in, it would be the default site language.(If you wanted to get REALLY fancy, you could use the visitor's default browser language - see if that language was offered by the site and use it, instead of the site default, if desired.)


If the queries are not changed, then some type of core hack would be required for multi-language environments.

If the queries are changed, this change will be transparent for single language implementations.

 


>Translation Extensions would then focus on copying source documents into a new
>row and presenting work to the appropriate translator for other
>languages. No
>additional logic should be required to present the language needed for
>the site
>visitor.

I would not second this. There are many more issues you need to focus
on and just doing a copy of an row is also not causing the fact that
less additional logic is used. For one table such as content this
might be right, but still the result for the user and admin might be
quite complicate. The primary issue at this point is the reference to
other entry in our database (e.g. menus, categories, ...).

Menus and categories are also included in the patch. The Site Developer *can* change the ACL for those objects, as required. In fact, that's the point. By recognizing language as an attributes of the source document and not treating it differently than other attributes, nothing special has to be done to use features, like ACL, for those objects.

In other words:
  • There is a menu item for each language.
  • The text and alias can be translated for each menu item - but the pointer to the same source and parameters ensures it is the same menu item programmatically.
  • The same menu item (translated differently) also returns different source data because - the language value filters those results, as needed.
  • But, ACL can be managed for each.
What that should do is make certain the translation extension doesn't need to deal with source data queries or menu items or ACL - those application functions are in place and can be used on rows in those content tables. Then, the translation extension could focus on the translation function.

Thanks very much for your feedback. I realize it might take a bit of back and forth to understand one another. We might even want to keep prototyping. Often, a simple solution, like adding these two fields to each source table, can reduce application logic significantly. I think such is the case, here.
 

diri

unread,
15 Aug 2009, 06:57:3415/08/2009
to Joomla! CMS Development
Hi and hello,

after watching 1.5 and 1.6 long time now I like to recommend two
additional fields at each item for internationalization:

1. language tag
2. an ID being under control of Joomla! (no auto increment!) for each
item

This additional ID is ment the way to give every item (content, menu,
link, ...) ONE and only one ID no matter which language is choosen. It
can be identified without any problem with this. Language tag should
be used to show item in language choosen by user.

To prevent problems in multi-user environment an additional table
should be created to have better chance to control last ID used.

Effort to code it is almost zero - depending on coding style basic
functionality needs one function for all items. Further improvements
are a performance boost in frontend while offering real multi-language
sites. Additional it can be implement with full portable SQL. And - it
is backward compatible very much.
>    - There is a menu item for each language.
>    - The text and alias can be translated for each menu item - but the
>    pointer to the same source and parameters ensures it is the same menu item
>    programmatically.
>    - The same menu item (translated differently) also returns different
>    source data because - the language value filters those results, as needed.
>    - But, ACL can be managed for each.

Sam Moffatt

unread,
15 Aug 2009, 07:46:0915/08/2009
to joomla-...@googlegroups.com
Presuming I read you correctly, you propose that two rows in
com_content could look like this:
ID: 1
Title: What is your name?
Lang: en-AU

ID: 1
Title: Siapa nama mu?
Lang: id-ID


Actually the effort to synchronise commits to ensure collisions don't
occur with a web based system is quite complicated when you stop
asking the database to hand it to you. The best way of doing this is
actually with a sequence which unfortunately isn't supported natively
by MySQL. There are ways and means of emulating it, as you suggest
with an extra table, however to handle this properly you need a new
table for every one that you're disabling the autoincrement field.
There are some techniques that utilise a single counter table (with
rows for each counter) though this can run into synchronisation issues
as well leading to potential problems with consistency. Of course the
worst thing with that method is that there is no easy way to ensure
consistency either - presume you have a simple uniqueness constraint
on ID-Language, it doesn't prevent two articles being created in two
different languages with the same ID at the same time but completely
different contents. Furthermore it adds a large layer of complexity
for single language sites which I'd suggest is the majority not the
minority - if JoomlaCode is to be used as a crude measure, people want
a forum more than they want a multilingual solution.

Sam Moffatt
http://pasamio.id.au

Christophe Demko

unread,
15 Aug 2009, 09:47:2615/08/2009
to Joomla! CMS Development
Hi amy,

I make some suggestions on your ideas
I think it's better to set the source_id to be the same as id (for the
original item). It allows to be be more general:
-selecting all the items (including the original one) is done by
"SELECT * FROM #__something WHERE source_id=somevalue".
-selecting all items which are a source one is done by "SELECT * FROM
#__something WHERE source_id=id"
-selecting the source item is done by "SELECT * FROM #__something
WHERE id=somevalue"
-selecting all items from a specific language is done by "SELECT *
FROM #__something WHERE lang=sometag"
Visitors could have the possibility to choose their language (by
clicking on some flag icon or selecting an available language)
> If the queries are not changed, then some type of core hack would be
> required for multi-language environments.
>
> If the queries are changed, this change will be transparent for single
> language implementations.
>
>
>
>
>
> > >Translation Extensions would then focus on copying source documents into a
> > new
> > >row and presenting work to the appropriate translator for other
> > >languages. No
> > >additional logic should be required to present the language needed for
> > >the site
> > >visitor.
>
> > I would not second this. There are many more issues you need to focus
> > on and just doing a copy of an row is also not causing the fact that
> > less additional logic is used. For one table such as content this
> > might be right, but still the result for the user and admin might be
> > quite complicate. The primary issue at this point is the reference to
> > other entry in our database (e.g. menus, categories, ...).
>
> Menus and categories are also included in the patch. The Site Developer
> *can* change the ACL for those objects, as required. In fact, that's the
> point. By recognizing language as an attributes of the source document and
> not treating it differently than other attributes, nothing special has to be
> done to use features, like ACL, for those objects.
>
> In other words:
>
>    - There is a menu item for each language.
>    - The text and alias can be translated for each menu item - but the
>    pointer to the same source and parameters ensures it is the same menu item
>    programmatically.
>    - The same menu item (translated differently) also returns different
>    source data because - the language value filters those results, as needed.
>    - But, ACL can be managed for each.

Amy Stephen

unread,
15 Aug 2009, 11:17:1615/08/2009
to joomla-...@googlegroups.com
Christophe -

Yes, that is *exactly* right. The value stored in the new source_id data element would be the primary key value for the original document.

That way, all items  which are the same Article, Web link, Newsfeed, or Menu Item, are tied together. 

Changing the queries is an important part, too. That way, the item needed for the Visitor's language is automatically returned by the query.

Thanks! Glad you agree on that source_id - it's a small change, but very powerful.

Amy

On Sat, Aug 15, 2009 at 8:47 AM, Christophe Demko <chd...@gmail.com> wrote:
Ithink it's better to set the source_id to be the same as id (for the

diri

unread,
16 Aug 2009, 04:53:1116/08/2009
to Joomla! CMS Development
> Presuming I read you correctly, you propose that two rows in
> com_content could look like this:
> ID: 1
> Title: What is your name?
> Lang: en-AU
>
> ID: 1
> Title: Siapa nama mu?
> Lang: id-ID

That's correct so far. Difference is: I would not disable
autoincrement field.

See discussion about source_id and parent_id which is nothing else
because you have to have one "anchor" for each item. Risks related to
consistency are the very same when an item is created first time.

In reality it does not matter which data are used to identify the
parent item. It could be the autoincremented id being created at very
first saving of an item as well as an independent id which is build by
the system.

Complexity for single language sites could be ignored because they
have those additional identifier and language tag in db only.

diri

unread,
16 Aug 2009, 04:53:2716/08/2009
to Joomla! CMS Development
> Presuming I read you correctly, you propose that two rows in
> com_content could look like this:
> ID: 1
> Title: What is your name?
> Lang: en-AU
>
> ID: 1
> Title: Siapa nama mu?
> Lang: id-ID

Amy Stephen

unread,
16 Aug 2009, 11:17:1116/08/2009
to joomla-...@googlegroups.com
Diri -

The autoincrement on the source data tables (ex. jos_content and jos_weblinks, etc.) would continue to be the primary key, the id field. The id for the original copy (be it an article, or a Web link, etc.), would serve as the source_id for all translations. The language value would differentiate the language.

Is that what you are also calling for? If so, great! That's what the patch calls for, too.

The final piece I am hoping we can implement is to change the source queries to restrict the results by the visitors language.

Amy

Amy Stephen

unread,
19 Aug 2009, 17:13:1919/08/2009
to Joomla! CMS Development

Amy Stephen

unread,
21 Aug 2009, 22:37:3521/08/2009
to Joomla! CMS Development
Patch http://joomlacode.org/gf/project/joomla/tracker/?action=TrackerItemEdit&tracker_item_id=17723

I found a problem in the Category listbox for the Article Layout with
the Chinese character set. This can be reproduced by adding a Category
with the value 孫子兵法 (The Art of War).

The listbox item returns incorrectly as:
<option
value="3">&aring;&shy;&laquo;&aring;&shy;�&aring;�&micro;&aelig;&sup3;�</
option>

I traced the problem to the encoding in the options function in
JHtmlSelect class. (libraries/joomla/html/html/select.php)

When line 569 is changed: ($options['option.text.toHtml'] ?
htmlentities(html_entity_decode($text)) : $text)

To this: ($options['option.text.toHtml'] ? htmlspecialchars($text) :
$text)

The listbox returns correctly: <option value="3">孫子兵法</option>

Amy Stephen

unread,
22 Aug 2009, 10:00:4422/08/2009
to Joomla! CMS Development
Regarding the two, versus three character language code - the W3's
Internationalization site < http://www.w3.org/International/questions/qa-lang-2or3#answer
> pointed out that in 2006 the IETF published RFC 4646 <
http://www.rfc-editor.org/rfc/rfc4646.txt > that no longer refers to
ISO code lists, but instead recommends using the appropriate subtags
in the new IANA Language Subtag Registry < http://www.iana.org/assignments/language-subtag-registry
>. That resource provides the relevant 2 or 3 character language code,
thus removing guesswork on which to use.

Ste

unread,
25 Aug 2009, 04:30:1025/08/2009
to Joomla! CMS Development
Hi all, I agree with Jean-Marie, a basic implementation of language
choice for contents is a feature eagerly awaited for.
I'm happy to see that you're looking for a solution about
multilanguage. Unfortunately I can't suggest anything about the code,
but I can say that this feature is very important expecially in non
spoking english nations, where multilanguage sites are very used.
If this feature will be implemented in 1.6 version, it will be a great
achievement for Joomla.

Amy Stephen

unread,
25 Aug 2009, 08:45:5925/08/2009
to Joomla! CMS Development
I agree that strengthening Joomla! for it's international community is
important, as well.

To make certain this discussion doesn't mislead anyone, we are talking
about setting the foundations for a translation system. That would
include "invisible" structure like additional columns to hold the
language (ex. en-GB) and a column that could be used to link a
translation to it's source document. Additionally, I am recommending
changes to the queries in order to facilitate the return of content in
the appropriate language.

Those changes are the scope of this discussion and will make it
*easier* for third party developers to provide elegant translation
solutions for 1.6.

Thanks for your support!
Amy :)

Valdis Ozols

unread,
26 Aug 2009, 17:37:0726/08/2009
to Joomla! CMS Development
Regarding implementation of queries it would be great if language
conditions would not be simply hardcoded. So, for instance if one
wants to retrieve/work with set of articles in any language/some
languages - then there is some way to impact filters without going for
full set of custom component/modules/etc.

Valdis

Amy Stephen

unread,
26 Aug 2009, 18:47:5426/08/2009
to joomla-...@googlegroups.com
We could come up with some formula for how that is determined. For example, the list length is is determined from a set of possible values, listed below in ascending order of priority.

1. Global configuration length.
2. Menu Item Parameter length.
3. User selected length from the front-end.

We could use:

1. Default frontend language.
2. Browser language, if available as an installed option.
3. User's selected frontend language, if logged on.
4. An override to that could be added by a third party tool for "language choice" offered in an Module.

Is that what you are thinking?

Amy Stephen

unread,
26 Aug 2009, 18:49:4026/08/2009
to joomla-...@googlegroups.com
To be clear - the #4 override would not be a part of core - and it could be implemented as a "Flag Module" that presented language options available on the site. But, again, it would be a third party extension. The core could simply make it possible to override the other settings.

Valdis Ozols

unread,
27 Aug 2009, 11:13:0927/08/2009
to Joomla! CMS Development
Yes, something like that. Regarding #4 - for instance if override will
be used by 3rd party module - will it make it possible to switch
between content filtered by specific language and content in any
language in standard components?


On 27 Aug, 01:49, Amy Stephen <amystep...@gmail.com> wrote:
> To be clear - the #4 override would not be a part of core - and it could be
> implemented as a "Flag Module" that presented language options available on
> the site. But, again, it would be a third party extension. The core could
> simply make it possible to override the other settings.
>
> On Wed, Aug 26, 2009 at 5:47 PM, Amy Stephen <amystep...@gmail.com> wrote:
> > We could come up with some formula for how that is determined. For example,
> > the list length is is determined from a set of possible values, listed below
> > in ascending order of priority.
>
> > 1. Global configuration length.
> > 2. Menu Item Parameter length.
> > 3. User selected length from the front-end.
>
> > We could use:
>
> > 1. Default frontend language.
> > 2. Browser language, if available as an installed option.
> > 3. User's selected frontend language, if logged on.
> > 4. An override to that could be added by a third party tool for "language
> > choice" offered in an Module.
>
> > Is that what you are thinking?
>

Christophe Demko

unread,
28 Aug 2009, 04:17:2728/08/2009
to Joomla! CMS Development
After studying proposed solutions, it seems that it cannot be done for
Joomla!1.6 (or it will be largely delayed...). Adding 'source_id' and
'language' fields is not sufficient; for example in #__content, there
are some fields which are language independent (publish_up and
publish_down). It could introduce incoherent data in the database.
IMO, I think that it MUST be a principal feature of Joomla!1.7 as ACL
have been for Joomla!1.6.

I agree with Hannes: adding language and source_id is a temporary
solution and translation have to be managed by third party extensions.

Amy Stephen

unread,
28 Aug 2009, 09:28:2928/08/2009
to joomla-...@googlegroups.com
Christophe -

The patch adds source_id and language fields to the listed tables. Sounds like you see that as acceptable.

The second change I recommend is modifying the SQL to conditionally return results based on language. That way, the frontend is ready for site visitors.

That's the extent of my proposal.

Doing those two changes should allow Translation Extensions to add interface changes to manage translation processes - but not have to hack core queries or add additional tables to manage relationship between translations and the source document. 

Regarding date issues, I assume a translator would be using their language as they create a translation - so - date issues shouldn't be a problem as Joomla! already manages this.

But, the point to understand is that I am *not* recommending a full-blown translation environment, but rather the basic core pieces, that being the addition of two data elements and selection criteria in queries, to set up the possibility for extensions to do the rest.

I'm not sure from your response that the scope of my proposal was clear.

Thanks!
Amy :)

Christophe Demko

unread,
28 Aug 2009, 15:16:1228/08/2009
to Joomla! CMS Development
I agree with Amy, unfortunately, I will not have the time to develop
the patch concerning the queries before the beta will be out (I will
try to develop this on monday, but I'm not sure to terminate it). If
someone else can develop this patch...

Amy Stephen

unread,
28 Aug 2009, 16:31:4328/08/2009
to joomla-...@googlegroups.com
I am happy to work on the patch. I would like to know, first, if the Release Team supports this approach.

Christophe Demko

unread,
29 Aug 2009, 00:07:5929/08/2009
to Joomla! CMS Development
Before developing, I would like to know too.

On 28 août, 22:31, Amy Stephen <amystep...@gmail.com> wrote:
> I am happy to work on the patch. I would like to know, first, if the Release
> Team supports this approach.
>

diri

unread,
29 Aug 2009, 02:43:1929/08/2009
to Joomla! CMS Development


On 28 Aug., 15:28, Amy Stephen <amystep...@gmail.com> wrote:
> Christophe -
>
> The patch adds source_id and language fields to the listed tables. Sounds
> like you see that as acceptable.
>
> The second change I recommend is modifying the SQL to conditionally return
> results based on language. That way, the frontend is ready for site
> visitors.
>
> That's the extent of my proposal.
>
> Doing those two changes should allow Translation Extensions to add interface
> changes to manage translation processes - but not have to hack core queries
> or add additional tables to manage relationship between translations and the
> source document.
>
> Regarding date issues, I assume a translator would be using their language
> as they create a translation - so - date issues shouldn't be a problem as
> Joomla! already manages this.
>
> But, the point to understand is that I am *not* recommending a full-blown
> translation environment, but rather the basic core pieces, that being the
> addition of two data elements and selection criteria in queries, to set up
> the possibility for extensions to do the rest.
>
> I'm not sure from your response that the scope of my proposal was clear.

Amy, I support this proposal. It offers basic international
capabilities without imposing to any real load.

Only problematic area might be com_contact where it would be nice to
have chance to show addresses according local requirenments. This
might need some more coding later on.

Problem with com_contacts is easy to understand when you look at
current implementation:
It is american only.

I.e. in Germany zipcode should be in front of city. Some additional
parameters could make live more easy when there is need for another
format.

Amy Stephen

unread,
29 Aug 2009, 09:03:0129/08/2009
to joomla-...@googlegroups.com
Addresses are a real challenge, aren't they?

I assume all of the necessary data elements for address collected? If not, it might be good to identify and add to this patch.

For now, at least with layout overrides, presentation can be changed, as needed.

For a better, long-term solution, I wonder if the translation/Internationalization group might be able to define layouts for their language. Those specs would make it possible to format the output accordingly.

Thanks Deri.

Amy Stephen

unread,
29 Aug 2009, 09:05:3429/08/2009
to joomla-...@googlegroups.com
On Fri, Aug 28, 2009 at 11:07 PM, Christophe Demko <chd...@gmail.com> wrote:

Before developing, I would like to know too.

Makes sense. Maybe they will respond with the direction, soon.
 

Gergő Erdősi

unread,
30 Aug 2009, 17:56:1030/08/2009
to joomla-...@googlegroups.com
Hi,

We had a release team meeting yesterday and discussed this topic as
well. We plan to completely rewrite com_content in a future version of
Joomla! (probably 1.7), and that's why we don't want to add extra
columns to the database in 1.6. There are currently very good
extensions for multilingual content, just like Nooku Content, which
handles multilingual content perfectly with the current database
scheme, so I think we don't need to rush and implement a partial
solution.

--
Gergő Erdősi



2009/8/29 Amy Stephen <amyst...@gmail.com>:

Amy Stephen

unread,
30 Aug 2009, 17:57:5330/08/2009
to joomla-...@googlegroups.com
Thanks for the response, Gergo. I agree this would make sense to do with a  com_content overhaul.

Alex Kempkens

unread,
23 Nov 2009, 02:16:3223/11/2009
to joomla-...@googlegroups.com
Hi all,

I personally feel that this thread drifted into a direction that was not intended by me with writing my white paper in the first place. Now the response from Gergö was from end of August and now I hear that com_content might get some over-whole already in 1.6.

In order to have a better understanding I would like to get a branch and implement a first draft of my white paper there. Everybody interested to join in is welcome.

I started a page on the wiki (http://docs.joomla.org/Improving_the_language_awareness) which is to summarize the white paper as well as the results of the discussions we have here in the mailing list.

So if one of the joomlacode admins is so kind to give us a branch we can get this started. I suggest we think about a version to integrate this when it is done. Any time before I think its just the wrong discussion.

Regards

Alex

Amy Stephen

unread,
23 Nov 2009, 11:08:1123/11/2009
to joomla-...@googlegroups.com
On Mon, Nov 23, 2009 at 1:16 AM, Alex Kempkens <alex.k...@gmail.com> wrote:
Hi all,

I personally feel that this thread drifted into a direction that was not intended by me with writing my white paper in the first place. Now the response from Gergö was from end of August and now I hear that com_content might get some over-whole already in 1.6.

Alex - I thought this was waiting for 1.7? Where are you hearing this? Is there a link?
 

Andrew Eddie

unread,
23 Nov 2009, 17:56:2923/11/2009
to joomla-...@googlegroups.com
Hi Alex.

The translation team was asked some time ago what are the absolute
must-haves they can't live without for 1.6. Those were:

1. Extra db-fields to allow content/menus/etc. to be presented
according to language.
2. Transliteration
3. Allowing 3-letter language codes in installer.

#3 works I believe, #1 I have in progress and #2 I've not touched yet
but I believe JM has done some work on this as have others. It's best
tackled anyone once the entire code stack is standarised.

Hope this helps.

Regards,
Andrew Eddie
http://www.theartofjoomla.com - the art of becoming a Joomla developer




2009/11/23 Alex Kempkens <alex.k...@gmail.com>:

Amy Stephen

unread,
23 Nov 2009, 18:09:4823/11/2009
to joomla-...@googlegroups.com
Andrew -

We had a patch for #1 with two fields for each content table. We spent a bit of time discussing it, but the Release Team asked that we wait until 1.7 when the content model was to be enhanced.

So, is this now a go, again? And if so, what is being done? I'd like to revisit it, myself, and see if it's different than what we were doing in August.

Thanks!
Amy

Andrew Eddie

unread,
23 Nov 2009, 18:23:1523/11/2009
to joomla-...@googlegroups.com
Maybe talk to Ole?

The fields aren't so much the problem, it the "awareness" logic on the
frontend that is going to be fun. What were the two fields?

Regards,
Andrew Eddie
http://www.theartofjoomla.com - the art of becoming a Joomla developer


2009/11/24 Amy Stephen <amyst...@gmail.com>:

Amy Stephen

unread,
23 Nov 2009, 18:31:4823/11/2009
to joomla-...@googlegroups.com
The two fields were 1) the language code and 2) the original content key. The link to the patch is in this thread. Yea, what you are talking about is certainly more complex. I have not heard of that yet.

So, is the release team still involved? Or, are we using different approaches now?  We all stopped on this topic when the release team set it aside. Might be a good time to get a handle on what's all remaining. Alex added the plugins refactoring to the Docs today. Maybe we should update the 1.6 status page? See where everything is at?

I know Gergo stepped away for awhile. Might be time to bring in a closer - get Wilco to get his bossy Project Mgr self back in here and organize things. He drove a hard whip getting 1.5 in, that's for sure! hehe!.

Andrew Eddie

unread,
23 Nov 2009, 19:43:0823/11/2009
to joomla-...@googlegroups.com
2009/11/24 Amy Stephen <amyst...@gmail.com>:

> The two fields were 1) the language code and 2) the original content key.
> The link to the patch is in this thread. Yea, what you are talking about is
> certainly more complex. I have not heard of that yet.

Ok, I remember now. Let me continue to get the lay of the land on
things and see where it heads. I’m really mid-stream at the moment
and I don’t really know where it’s going to end up so can’t give you
an opinion one way or the other. Sorry.

> So, is the release team still involved?

Yes, but are busy with their studies just at the moment.

> Or, are we using different
> approaches now?

You’d have to ask the Production Leadership Team and/or the Release Team.

> We all stopped on this topic when the release team set it
> aside. Might be a good time to get a handle on what's all remaining. Alex
> added the plugins refactoring to the Docs today. Maybe we should update the
> 1.6 status page? See where everything is at?

There are a number of things in the works to provide more status
information for the general public. I suggest those with development
time on their hands introduce themselves on this list (if you haven't
already) and ask where they can be most help or read the several posts
already active with task lists. As you can appreciate things can
change from day to day at the detail level that development status
pages can't really convey. If in doubt, ask.

> I know Gergo stepped away for awhile. Might be time to bring in a closer -
> get Wilco to get his bossy Project Mgr self back in here and organize
> things. He drove a hard whip getting 1.5 in, that's for sure! hehe!.

Personally, and this is just me, I’m quite happy with the way things
are closing. More bodies would be better and probably change what
those higher up in the leadership chain do (eg, I can mentor more).
Can’t really give you a definitive answer on that one because it’s
relative to who’s available at any point in time. Regarding Wilco,
his expertise and experience in unquestioned and when he’s ready to
get involved again I’m sure he’ll act on it in his own time.

Amy Stephen

unread,
23 Nov 2009, 21:04:1123/11/2009
to joomla-...@googlegroups.com
On Mon, Nov 23, 2009 at 6:43 PM, Andrew Eddie <mamb...@gmail.com> wrote:

2009/11/24 Amy Stephen <amyst...@gmail.com>:
> The two fields were 1) the language code and 2) the original content key.
> The link to the patch is in this thread. Yea, what you are talking about is
> certainly more complex. I have not heard of that yet.

Ok, I remember now.  Let me continue to get the lay of the land on
things and see where it heads.  I’m really mid-stream at the moment
and I don’t really know where it’s going to end up so can’t give you
an opinion one way or the other. Sorry.

That's okay. When you know what direction you need, just say what you need.

This is an example where volunteers were working, but we were asked to stop. If it is determined later that the work is needed, just bump the thread and we'll get started again.

We want to help. :)
 

> So, is the release team still involved?

Yes, but are busy with their studies just at the moment.

> Or, are we using different
> approaches now?

You’d have to ask the Production Leadership Team and/or the Release Team.

I did ask. This wasn't addressed to you, although I admire your dedication to responding and appreciate it very much.

I will keep encouraging developers to get involved. Thank you *very much* for punching out the plugin work for us. When we get our SVN access to the plugin_dirs branch, we'll get started on that.

JM Simonet

unread,
24 Nov 2009, 01:41:2324/11/2009
to joomla-...@googlegroups.com
Andrew,

1. This has started to be implemented indeed. Getting also parameters
in each UI to directly tag not only contents (the param exists there
since 1.5) but modules and menus would be an enormous step forward.

2. Custom ASCII Transliteration has already been implemented in 1.6
by Ercan, by letting use a specific xx-XX.transliterate.php file in
language packs.
I have proposed an en-GB.transliterate.php covering European latin
languages which we suggest to include as default in en-GB.
File is ready to get in whenever a committer gets it is.
See
http://groups.google.com/group/joomla-dev-cms/browse_thread/thread/c8abbff7f67f9a85#

3. I am not aware that this has been implemented as it requires a new
specific metadata in the language xml file in order to let choose a 2
letters fallback language to cope with browsers limitations. Details
of the issue have also been posted here.

4. let me add here a fourth aspect we did not think of at the time as
must-have: letting the choice for admins to use unicode slugs
(percent encodings) instead of ASCII transliteration for the aliases.
I have posted elsewhere about that.

>Hi Alex.
>
>The translation team was asked some time ago what are the absolute
>must-haves they can't live without for 1.6. Those were:
>
>1. Extra db-fields to allow content/menus/etc. to be presented
>according to language.
>2. Transliteration
>3. Allowing 3-letter language codes in installer.
>
>#3 works I believe, #1 I have in progress and #2 I've not touched yet
>but I believe JM has done some work on this as have others. It's best
>tackled anyone once the entire code stack is standarised.
>
>Hope this helps.
>
>Regards,
>Andrew Eddie
>http://www.theartofjoomla.com - the art of becoming a Joomla developer
>
>
>

--
>Please keep the Subject wording in your answers
This e-mail and any attachments may be confidential. You must not
disclose or use the information contained in this e-mail if you are
not the
intended recipient. If you have received this e-mail in error, please
notify us immediately and delete the e-mail and all copies.
-----------------------------------------------------------
Jean-Marie Simonet / infograf768
Joomla! Translation Coordination Team


Reply all
Reply to author
Forward
0 new messages