Better language support

[pierreth]

Hello,

It seems that web2py takes care of only the first language listed in
the browser request. The other languages are not taken into account.

For example, in my browser preferences, I have:

fr-ca
fr
fr-fr
en
en-us

But there is only one value for T.accepted_language and in my case I
have fr-ca.

I propose a modification for web2py to take in consideration other
languages:

- T.accepted_language would stay the same.
- A new T.accepted_languages, with an 's', would be created to list
all the languages accepted by the users.
- When searching for a translation, web2py will go through all the
files for T.accepted_languages to find the best match

What do you think of this idea?

[pierreth]

I would like to further express why this feature would be a great
benefit.

People doing translation are having a hard time the way web2ppy is
working now. I will take the example of translating in French but it
seems to be the same for about any languages. One of the problem we
have to deal with as translator is that we have few big files that are
mostly identical fr.py, fr-cap.y, fr-fr.py. Each time we want to add a
new translated term, we must add it to all different language files.
On top of that, it becomes hard to see what are the differences in all
these files.

So what I propose with the mechanism I described in the last message
is to have one big file fr.py and only have the differences in fr-
fr.py and fr-ca.py. That would already make life a lot easier for
translators.

The second thing that I propose is that web2py put the term is unable
to translate in other files like "missing-translations-fr.py". Right
now, web2py adds these translation automatically in the file related
to T.accepted_language. It does so whiteout even adding a comment so
it is difficult for translators to know that the translation is coming
from web2py. The "missing-translations-xx.py" would only exist for
main languages like 'fr' but not for 'fr-ca' or 'fr-fr'. Web2py would
automatically consider the main language if this one is missing from
the request by appending the main language if needed to follow more
specific languages (for example, imagine a request accepting fr-ca but
not fr, web2py would add fr after fr-ca).

Any other ideas?

[pierreth]

On 17 mai, 13:54, pierreth <pierre.thibau...@gmail.com> wrote:
> It seems that web2py takes care of only the first language listed in
> the browser request. The other languages are not taken into account.

I made some tests and web2py choose the T.accepted_language as the
first language listed in the request for which there is a translation
file or match the current language (the language in which the
application is written for which no translation is needed). This match
must be perfect: web2py won't accept fr as a language if the request
only specify fr-ca.

[pierreth]

On 17 mai, 13:54, pierreth <pierre.thibau...@gmail.com> wrote:

> - T.accepted_language would stay the same.
> - A new T.accepted_languages, with an 's', would be created to list
> all the languages accepted by the users.
> - When searching for a translation, web2py will go through all the
> files for T.accepted_languages to find the best match

After further reflexion, I realize that it would be unreasonable to go
through all the languages files accepted by the user. What would be
more reasonable is to try a more generic language if a translation is
not found in the language defined by T.accepted_language. So if
T.accepted_language == 'fr-ca' and the translation is not present in
'fr-ca.py' then we try to find it in 'fr.py'. Of course we stop the
translation as soon as the language we are searching is the current
one (the language of the app itself).

[Jonathan Lundell]

On May 17, 2011, at 5:35 PM, pierreth wrote:
>
> Any other ideas?

One. Now that we can have translations in gluon, it'd be nice to have global (that is, not app-specific) translation files. My thought was that the translator would look first in the current app's translations, and if it's not found, check the global translation files.

That way we could accumulate standard translation for gluon messages (and perhaps other common messages as well). Developers could override the global translations if necessary by simply supplying their own version. It's conceptually aligned with the scheme you're proposing, in the sense that it relies on primary and fallback translation files, so I think it might fit in nicely.

BTW, the parametric (newish) URL router has some language selection support, but it's not hooked up: all that happens is that the language specified in the the URL ends up in the request object. That's because I wasn't using translations enough to know what the best approach would be. It'd be nice to complete that loop.

[Massimo Di Pierro]

+1 for this. I would take a patch. ;-)

[pierreth]

On 17 mai, 23:46, Massimo Di Pierro <massimo.dipie...@gmail.com>
wrote:

> +1 for this. I would take a patch. ;-)
>

+1 for this one too? :

> The second thing that I propose is that web2py put the term is unable
> to translate in other files like "missing-translations-fr.py". Right
> now, web2py adds these translation automatically in the file related
> to T.accepted_language. It does so whiteout even adding a comment so
> it is difficult for translators to know that the translation is coming
> from web2py. The "missing-translations-xx.py" would only exist for
> main languages like 'fr' but not for 'fr-ca' or 'fr-fr'. Web2py would
> automatically consider the main language if this one is missing from
> the request by appending the main language if needed to follow more
> specific languages (for example, imagine a request accepting fr-ca but
> not fr, web2py would add fr after fr-ca).

And for Jonathan's idea? It seems to be a very good idea for me.

Any way, this time I'll try to be test driven. I was lazy on that with
my last patches.

[Pierre Thibault]

Hello,

I didn't have a lot of feedback on this proposal. I thought it will
help translators but now I am wondering if it would be time well
spent.

Please post a comment if this idea attracts interest.

[Massimo Di Pierro]

I am for this and for Jonathan idea. I am a little worried about the
level of complexity this may bring, so I would like to see an
implementation before committing to it. Too many file look-ups could
cause a slow down.

About Jonathan proposal. It is ok to have a set of global translations
but we need to brainstorm it more.
Should global translations just stay global or propagate to apps?
What if the translation changes in global but not in local app?

[Christopher Steel]

Hi Pierre,

I think you have touched on an important issues and although I am also
wondering if Jonathan proposal might provide for similar (although
less "blingy") effect with less effort, it is clear that the way we do
translations now can make things a bit tedious and contributing new
translations back to the project requires a high level of motivation.

If I understand it correctly, Jonathan's suggestion has a lower bling
factor but depending on how it is implemented could make translating
and perhaps as important, contributing new translations much easier
for people who are new to Web2py.

As it stands now when you do a translation on a "per application"
basis your application translation is mixed in with the Web2py
translation. So in order to contribute all your hard work you need to
cleanup the file or copy and paste your Web2py translations into a
"clean" file in order to contribute them. This makes for a lot of
extra work so contributors submit "dirty" files or need to be very
motivated to do Web2py translations.

If we had two files and a diff setup or perhaps even three files we
might be able to substantially increase the amount of high quality
Web2py translations that get submitted to the Web2py project in
addition to offering developers / translators an great deal of
flexibility as far as language is concerned.

Off the top of my head if...

nv.py <- global, contains only translations for Web2py itself and the
Welcome application.
nv.py <-per appliciation # a "copy" of the global file.
nv-app.py <- per-applicaiton , application specific translations
(custom).

This would keep the applications nv.py "clean" so that it could be
quickly and easily submitted back to the Web2py project itself with no
cleanup of the application specific translations required, we could
even have an "email" translation contribution button if we wanted.






On May 19, 8:39 am, Pierre Thibault <pierre.thibau...@gmail.com>
wrote:

[Pierre Thibault]

2011/5/19 Massimo Di Pierro <massimo....@gmail.com>

I am for this and for Jonathan idea. I am a little worried about the
level of complexity this may bring, so I would like to see an
implementation before committing to it. Too many file look-ups could
cause a slow down.

This stress seems to hide a deeper problem: the lack of automated tests. Personally, and others are too, I am not very satisfy with what web2py has to offer for testing purposes. It an issue I want to explore further before committing other developments.

I believe that a good layer of tests would remove a lot a stress when developers are re-factoring the code. It will ensure a smooth evolution with web2py, improve the design and the quality of its implementation.

Has for the problem of speed, we can cache things in memory. But I think the first problem to deal with is the stress of re-factoring.
 

About Jonathan proposal. It is ok to have a set of global translations
but we need to brainstorm it more.
Should global translations just stay global or propagate to apps?
What if the translation changes in global but not in local app?

I have some second thought on Jonathan proposal. I have some concerns:

- By nature, a change in the global dictionary may affect many applications;
- A single word in English may have to be translated in many different words in another language;
- Once we are commited to support a language, if you add a new word, we must add the word in all the languages supported;

- How do we deal with a word that would not be appropriate for a sub-dialect?

I think the risks are important. I might not be a very good idea after all.


A+

-------------
Pierre
My blog and profile (http://pierrethibault.posterous.com)
YouTube page (http://www.youtube.com/user/tubetib)
Twitter (http://twitter.com/pierreth2)

[Pierre Thibault]

2011/5/19 Christopher Steel <chris...@gmail.com>

Hi Pierre,

I think you have touched on an important issues and although I am also
wondering if Jonathan proposal might provide for similar (although

less "blingy")...

Hi Chris,

No, I think mixing translations is not a good thing for the reasons I mention above.

--

[mart]

not sure if this applies, but just in case...

years ago (i think 12, so this may be outdated), i worked in a
localization team, (although it was a purely windows shop where words
like dll was all that anyone talked about). Anyways, we adopted a
method for 1) re-using translations (from translation tool kits (aka.
TTKs) 2) standardizing on commonly accepted strings (i.e. there aren't
too many ways to translate a yes/no button ;)) & 3) the language
specific string tables would grow over time and served the task very
well considering... (we only had 4 or 5 languages ;) english, swedish,
japanese, simplified chinese, and french (fr_ca was deemed unworthy at
the time ;))

anyways, the moral is, that all strings going to UI were ripped out of
the code and replaced with IDs where the users language selection
would force load the language specific "satellite dll" @ run time -
was just a dll with the resource strings in it.

perhaps adopting something similar (but no dll's ;) ) would be
beneficial?

Mart :)


On May 19, 6:52 pm, Pierre Thibault <pierre.thibau...@gmail.com>
wrote:
> 2011/5/19 Christopher Steel <chris.st...@gmail.com>

>
> > Hi Pierre,
>
> > I think you have touched on an important issues and although I am also
> > wondering if Jonathan proposal might provide for similar (although
> > less "blingy")...
>
> Hi Chris,
>
> No, I think mixing translations is not a good thing for the reasons I
> mention above.
>
> --
>
> A+
>
> -------------
> Pierre
> My blog and profile

> (http://pierrethibault.posterous.com)<http://pierrethibault.posterous.com>
> YouTube page (http://www.youtube.com/user/tubetib)<http://www.youtube.com/user/tubetib>
> Twitter (http://twitter.com/pierreth2) <http://twitter.com/pierreth2>

[Pierre Thibault]

2011/5/20 mart <msenecal.sc@gmail.com>

not sure if this applies, but just in case...

years ago (i think 12, so this may be outdated), i worked in a
localization team, (although it was a purely windows shop where words
like dll was all that anyone talked about). Anyways, we adopted a
method for 1) re-using translations (from translation tool kits (aka.
TTKs) 2) standardizing on commonly accepted strings (i.e. there aren't
too many ways to translate a yes/no button ;)) & 3) the language
specific string tables would grow over time and served the task very
well considering... (we only had 4 or 5 languages ;) english, swedish,
japanese, simplified chinese, and french (fr_ca was deemed unworthy at
the time ;))

anyways, the moral is, that all strings going to UI were ripped out of
the code and replaced with IDs where the users language selection
would force load the language specific "satellite dll" @ run time -
was just a dll with the resource strings in it.

perhaps adopting something similar (but no dll's ;) ) would be
beneficial?

Mart :)

Hi Mart,

Thank you for sharing this experience. I guess you had the professional resources to do all the translations. If we use a common dictionary for web2py, we need all the translations for the languages we support for all the words before doing a new release. Do we have to resources to do that? I have some doubts.

--


A+

-------------
Pierre
My blog and profile (http://pierrethibault.posterous.com)

[mart]

Hi Pierre,

Agreed, and yes the money was there for the resources, but doesn't
mean we were smart about it (I.e. of couple of local Japanese
restaurant employees had much to do with a couple of of product
releasees;) - true story)

I think the issue here is not lack of knowledge on how-best-to-go-
about-it (there are many smart people here to pick from for process
ideas), but lack of knowledge about the translations (the strings). My
reasoning is that, if a language is supported by web2py, its probably
because one if its members requires that language and he/she (or
someone else) has already contributed to that languages string table.

That said, why not put a bit of the onus on the users? if we treat
each language as a plugin, and each plugin can be versioned, then each
web2py release can have access to the latest and greatest of each of
the language dictionaries/plugins (does not mean they all have to be
at the version). we publish language dictionaries based on
requirement, and not as a default (because keeping up may be need to
be some one's full time job)

here's an example. lets say we have N languages (including EN). EN
(being the prime language of release) would set the bar. Lets say, the
EN plugin is stamped with version 1.96.1 . if a user requires a German
dictionary for his app, then he may agree to update the German
dictionary and bring it to the level on the EN dictionary. Otherwise,
the _de_de dict would remain at its previous version (maybe 1.95.2).

This is open source, so obviously and as you say, the funds are not
there, but there are many users, native speakers for each of the
supported languages. If there are many users of any given language,
then that's great! Just means that there are many users to help keep
things in check. Otherwise, the assumption can be that if any language
is failing to keep up the EN, its probably that it is not being used,
therefore no harm is done by having a language at a lower version. If
at any point, someone requires the language, then he/she should do
their best to update those un-translated strings where, once vetted,
that language dict could be stamped at the same version as EN.

I think by introducing and supporting l18n in this type software
development environment, users who use the product can be expected to
contribute, lest it become unmanageable.

besides, i know for a fact, that even large corporation with way too
many funds at their disposal release localized product with the worst
possible practices (and questionable string equivalents). At least
here, there may be no funds, but there is discussion and many who may
be willing and able to help (i think).

Maybe we can have a "members only" app on web2py.com, where folks can
approved members can go when time permits and update strings. @
release time, Massimo could simply dump those string table to... well
to what ever format is required. Versioning for the lang plugins can
be set by comparing against the model (EN) --> 1.96.1.<number of
translated strings>


just a thought,
Mart :)

On May 20, 2:46 am, Pierre Thibault <pierre.thibau...@gmail.com>
wrote:
> 2011/5/20 mart <msenecal...@gmail.com>

[Christopher Steel]

The important issue being, making it easier for people to contribute
additional translations...

On May 19, 6:52 pm, Pierre Thibault <pierre.thibau...@gmail.com>
wrote:
> 2011/5/19 Christopher Steel <chris.st...@gmail.com>
>

> > Hi Pierre,
>
> > I think you have touched on an important issues and although I am also
> > wondering if Jonathan proposal might provide for similar (although
> > less "blingy")...
>
> Hi Chris,
>
> No, I think mixing translations is not a good thing for the reasons I
> mention above.
>
> --
>
> A+
>
> -------------
> Pierre
> My blog and profile

[Pierre Thibault]

You made good points. I think that it is important to have a process that can support the community we are.

Here is my last idea on the subject:

We can support a global translation repository just like Jonathan was proposing. But if an application has not for example a custom dictionary for German, then web2py will not continue its search to try to get another translation in the global repository.

Also, when a new application is created with the web interface, web2py will ask what are the languages to be available for the new app. This will avoid trailing dictionaries.

This way of doing will avoid partial translations where only a word up and there is translated. Instead, there will be no translation at all and I think this is a lot nicer.

Sounds good?

A+

-------------
Pierre
My blog and profile (http://pierrethibault.posterous.com)

[mart]

that makes a lot of sense! removes confusion and leaves little room
for doubt.

Mart :)




On May 20, 9:23 pm, Pierre Thibault <pierre.thibau...@gmail.com>
wrote: