Multilingual website plans

[Berteh]

Hello,

I'm interested in getting an improved support for multilingual website
in SilverStripe. I noticed some related bugs have been milestoned to
2.2.3... and would really like these (& others) to be sooner than
later ;o)

Therefore I'd be willing to discuss topics relating to page
translation support; relevant admin & nav & search functionalities;
integration into SST core and more... both as a final user with no
programming skill (got of few of them available for testing &
requesting ;o) and a webmaster with some IT competencies.

Is there any related wiki/forum?


In case it's welcome I could join the dev effort in these regards... I
got some background in PHP, clean OO Java, most XML languages and a
few scripting languages, a.o. Let me know if someone's already in the
place and wants some help... By the way, as I'm working full time (and
bit more) I don't want to take everything on my own side, but rather
contribute to the common effort.

I experienced with quite a few CMS (on production sites) before, none
of them being fully satisfactory from the multilingual perspective
(phpnuke, cmsmadesimple, drupal, mambo, joomla)... even if the
"joomfish" extension for joomla was maybe the most relevant.


Have a nice day,
Berteh.

[Ingo Schommer]

Hey Berteh,

help is always welcome! :)

Fixing translation bugs in most cases means "diving in the deep end"
in terms of developer requirements, but not impossible.
We can give you some pointers to help you get started
if you tell us which tickets you're interested in
(you most likely have some immediate problems in
your projects that you'd like fixed, right?)

Have a look at pending bugs/enhancements for i18n:
http://open.silverstripe.com/query?status=assigned&status=new&status=reopened&group=type&component=i18n&order=priority&col=id&col=summary&col=status&col=type&col=priority&col=milestone

Cheers
Ingo

-------
Ingo Schommer | Senior Developer
SilverStripe
http://silverstripe.com

Phone: +64 4 978 7330 ext 42
Skype: chillu23

Level 3, 97-99 Courtenay Place
Wellington, New Zealand

[Berteh]

Hello Ingo,

Thanks for the answer.

Yes, I got some particular bugs bothering me (eg. #2129, #2406,
#1803)... but all of these seem more deep-rooted than just "light
syntactical bug fixing"... as you mentionned.

So, before diving further, I'd like to get a rough idea of the way you
want multilingual content to work, and therefore give a few ideas
below, let me know which ones you wanna keep or drop.

-----
1) multilingual content means you can have different articles in lang1
(eg EN) and lang2 (eg DE)

Even if the categories (parent pages) are expected to match and be
translated 1-1, the articles (leaf pages) should be allowed to exist
in only one language... just because content editors don't always know
all languages.

This implies to drop the idea of a common (an required) "main"
language (EN) and "secondary" languages (DE), but rather adopt and
support an "original" language (different for all pages) and
"translation" pages.

This obviously arises some troubles with respect to
- the site tree (my idea: display the tree in a selected language, and
display in grey all pages not translated in that language... filtering
would only filter out leaf pages to keep tree consistent among
languages)
- navigation: if a category (parent) page is not translated one should
display its "original" title and content... but translated sub-menus
and children links.


----
2) i18n should be site-wide rather than page specific
The whole template should be internationalized/able... footer, site
teaser, eventually site name

----
3) support side-wide widgets
Language switching widget is cool, but it is a pain in the *** to
include it in all original and translated page. Putting the same
dialog in the template is not accessible to non-tech webmasters. So I
guess having a function stating "add this widget to all pages" could
be an option.



Let me know... including for additional pointers about the current
i18n implementation
(besides
http://doc.silverstripe.com/doku.php?id=i18n and
http://doc.silverstripe.com/doku.php?id=multilingualcontent
http://api.silverstripe.com/sapphire/core/Translatable.html

you got some graphical view of the core model or explanations about
translatable rationales?)


have a nice day,
Berteh.

[Berteh]

in the same thread, other concerns I forgot to mention when going for
the perspective of "original" (non imposed) language page and its
translations is the issue of consistency:

- do you delete all translations when you delete the original
- do you delete the original and other translations when you delete a
translation
- do you always leave these choices to the user, who will generally
choose not to remove things ("... just in case..."), and come to a
cluttered website.
- do you apply the same behavior for parent and leaf pages

there are other open questions. If you think/know of any feel free to
drop me a few lines.
Berteh.

On 22 mai, 11:49, Berteh <ber...@gmail.com> wrote:
> -----
> 1) multilingual content means you can have different articles in lang1
> (eg EN) and lang2 (eg DE)
>

(...)

[Ingo Schommer]

Hey Berteh,

On 22/05/2008, at 9:49 PM, Berteh wrote:

Yes, I got some particular bugs bothering me (eg. #2129, #2406,

#1803)... but all of these seem more deep-rooted than just "light

syntactical bug fixing"... as you mentionned.

Yes, all those require some upfront research into the "Translatable" DataObjectDecorator,
the SiteTree javascript and the i18n internals - but nothing impossible.

-----

1) multilingual content means you can have different articles in lang1

(eg EN) and lang2 (eg DE)

Even if the categories (parent pages) are expected to match and be

translated 1-1, the articles (leaf pages) should be allowed to exist

in only one language... just because content editors don't always know

all languages.

This implies to drop the idea of a common (an required) "main"

language (EN) and "secondary" languages (DE), but rather adopt and

support an "original" language (different for all pages) and

"translation" pages.

This would make the system a lot more flexible, but as you noted will
break a lot of assumptions. The table structure currently has
a SiteTree table for the "main" records (with assumption of default language),
and different translations in SiteTree_lang (with a RecordID pointing to the SiteTree table).
This relation is required
We could add a "Lang" column to the SiteTree table which would store
the language the page is created in. The "main" language could be optionally stored
as a new row in SiteTree_lang. This means:
1. SELECT * FROM SiteTree will give you mixed languages (for non i18n-enabled code)
2. Fetching a record in a specific language gets more difficult
3. All the UI problems you mentioned before

Alternatively, we could ask authors to hide pages in the "main" language if they
just want the translations to show up. This already works AFAIK, is a bit counter-intuitive,
but far less work. What do you think?

----

2) i18n should be site-wide rather than page specific

The whole template should be internationalized/able... footer, site

teaser, eventually site name

Thats already the case: http://doc.silverstripe.com/doku.php?id=i18n (look for "templates")

----

3) support side-wide widgets

Language switching widget is cool, but it is a pain in the *** to

include it in all original and translated page. Putting the same

dialog in the template is not accessible to non-tech webmasters. So I

guess having a function stating "add this widget to all pages" could

be an option.

We have a widget system which supports this kind of setting,
just need somebody to make an actualy widget out of the current code :)
Plus, with Silverstripe template includes, you need to add
the switcher template code in one place (depending on the design of course).

you got some graphical view of the core model or explanations about

translatable rationales?)

I've sent you our notes from the original GSOC project by email -
they are probably a bit outdated, but show the motiviations and
assumptions with which the project was started last year.

in the same thread, other concerns I forgot to mention when going for

the perspective of "original" (non imposed) language page and its

translations is the issue of consistency:

- do you delete all translations when you delete the original

I'd say we do.

- do you delete the original and other translations when you delete a

translation

No

- do you always leave these choices to the user, who will generally

choose not to remove things ("... just in case..."), and come to a

cluttered website.

I'd keep the user informed of the consequences of his action (e.g. deleting
the original), but don't give him too much choice.

- do you apply the same behavior for parent and leaf pages

Deleting the parent should delete all children and all translations,
leaf pages shouldn't have this effect.

Thanks for your help!
Ingo

[Berteh]

Hello Ingo,

Thanks for the notes of the GSOC i18n and multilingual content
project.
To answer your questions:

> Alternatively, we could ask authors to hide pages in the "main"
> language if they
> just want the translations to show up. This already works AFAIK, is a
> bit counter-intuitive,
> but far less work. What do you think?

From a developper point of view I'd say: easier but more bug-prone
(filtering things out in a core-functionality just sounds dangerous)
From a user perspective I'd say: not ergonomic nor intuitive.

> I'd keep the user informed of the consequences of his action (e.g.
> deleting the original), but don't give him too much choice.

>(..)

> Deleting the parent should delete all children and all translations,
> leaf pages shouldn't have this effect.

Fine with me.



To stick with the great first feeling I had with SST (namely: non-geek
friendly and working interface and functionnalities), and if it
doesn't seem overwhelming to you, I'd prefer going for the "original"
and "translation" page option (only when i18n is enabled).
Unfortunately I'm close to nowhere in javascript (just familiar with
DOM, prototype and ajax rationales, but not -yet- with JS
datastructures or functions) and would probably need some help there.

What svn repository should I look at? http://svn.silverstripe.com/open/phpinstaller/trunk/
had even more translation-realted problems than stable 2.2.1... or
should I start from something else than the svn?


I think I'm fine with the Tanslatable, Bernat did a *great* job there
(clean and all). Would he happen to still be involved/available?
I think we'd have mostly nothing to change to the i18n internals about
entity translation. wrong?
and as for the TreeSite... got no clue about the amount of work
there... more below.


From what I got so far:

The changes you mentionned in model structure (i.e. Translatable)
should be enough model-wize, verbatim:

> We could add a "Lang" column to the SiteTree table which would store
> the language the page is created in. The "main" language could be

> optionally be stored as a new row in SiteTree_lang.

* most small impacts on DataObjectDecorator overriden methods, e.g. to
initialize the "main_language" row values to the default site language
when Translatable support is enabled; and correct "Translatable.
$default_lang" and "Translatable.is_default_lang()" uses;

* most big impacts on
- "Translatable.get_by_lang()" and its uses, that would return mixed
language objects;
- "Translatable.get_one_by_lang()" that should eventually return the
original when no translation exists... without changing
"$reading_lang" in this later case;
- Hierarchy & SiteTree... I guess the selection of children is
somehow impacted (to trace which ones are given in the right -reading-
language, and which ones are originals with lacking translation), but
couldn't figure where this is already done (for current Translatable
pages).
- UI-related changes in "Translatable.updateCMSFields()" and SiteTree
handling... didn't look at it at all, and got no clue where to look
at, pointer are welcome.


More pointers and remarks are welcome.

>Thanks for your help!
hum... I still did nothing so: you're welcome ;o) and thanks for being
interested in improving that part of SST.
(... I just don't like the SS abbreviation).

Berteh.

[Ingo Schommer]

Hey again,

On 23/05/2008, at 10:22 PM, Berteh wrote:
>> Alternatively, we could ask authors to hide pages in the "main"
>> language if they
>> just want the translations to show up. This already works AFAIK, is a
>> bit counter-intuitive,
>> but far less work. What do you think?
>
>> From a developper point of view I'd say: easier but more bug-prone
> (filtering things out in a core-functionality just sounds dangerous)
>> From a user perspective I'd say: not ergonomic nor intuitive.

Well, its standard functionality, which should already work on a
language-by-language basis.
But I get your point, its of course best to implement properly
(if somebody has the time to do it, that is...*g*)

>
>
>
> To stick with the great first feeling I had with SST (namely: non-geek
> friendly and working interface and functionnalities), and if it
> doesn't seem overwhelming to you, I'd prefer going for the "original"
> and "translation" page option (only when i18n is enabled).

Sounds good - are you familiar with unit testing?
i18n is a pretty complex piece of functionality, unfortunately not
covered by any tests.
This is not ideal of course, but when changing deep functionality,
we would require at least partial test coverage to ensure existing
assumptions don't break.

>
> Unfortunately I'm close to nowhere in javascript (just familiar with
> DOM, prototype and ajax rationales, but not -yet- with JS
> datastructures or functions) and would probably need some help there.

We might be able to help out here, I expect the bulk of work
to be on the serverside anyway. Any tree-related changes will
probably coincide with switching the tree javascript to a different
codebase anyway (most likely an extjs solution).

>
>
> What svn repository should I look at? http://svn.silverstripe.com/open/phpinstaller/trunk/
> had even more translation-realted problems than stable 2.2.1... or
> should I start from something else than the svn?

You should work on trunk, correct. After you get started,
we might be able to provide a development branch with write access,
depending on the amount of your changes.

>
>
>
> I think I'm fine with the Tanslatable, Bernat did a *great* job there
> (clean and all). Would he happen to still be involved/available?

I wouldn't count on it - he's snowed with uni work (he's just
graduating afaik).

Your development plan from the last post seems reasonable, let me know
how you get on - great to see some help from the community!

Cheers
Ingo

[Berteh]

On May 24, 1:29 am, Ingo Schommer <i...@silverstripe.com> wrote:
>
> i18n is a pretty complex piece of functionality, unfortunately not
> covered by any tests.
> This is not ideal of course, but when changing deep functionality,
> we would require at least partial test coverage to ensure existing
> assumptions don't break.

OK. Do you got some list of particular calls/assumptions/uses you want
to cover/protect? I could start by writing those tests, preferably for
Translatable than for i18n though.

Berteh.

[Ingo Schommer]

Hello Berteh,

I think all augument*() calls are most critical - comparing raw query
strings against assumptions, optimally in different scenarios
(partial translations, fallbacks if language doesn't exist etc).
Second priority would be all get* calls to see if the generated
DataObjects contain the correct information.

Thanks
Ingo