Use of Gettext / PyBabel proposal for translations
Carlos Perelló Marín
I would like to start a thread to discuss the use of gettext/pybabel use
in Tryton to handle translations.
I added a brief explanation at
https://bugs.tryton.org/roundup/issue1163
I will appreciate if translators and developers read it and provide me
with comments, I would like to get this done for 1.4.
Cheers.
--
Carlos Perelló Marín
[P+] SERVICIOS PROFESIONALES
http://www.pemas.es
mailto:car...@pemas.es || mailto:car...@gnome.org
Este mensaje y los ficheros anexos son confidenciales. Los mismos
contienen información reservada de la empresa que no puede ser
difundida. Si usted ha recibido este correo por error, tenga la
amabilidad de eliminarlo de su sistema y avisar al remitente mediante
reenvío a su dirección electrónica; no deberá copiar el mensaje ni
divulgar su contenido a ninguna persona.
Su dirección de correo electrónico junto a sus datos personales forman
parte de un fichero titularidad de PEMAS Servicios Profesionales, S.L.
cuya finalidad es la de mantener el contacto con Ud. De acuerdo con la
Ley Orgánica 15/1999, usted puede ejercitar sus derechos de acceso,
rectificación, cancelación y, en su caso, oposición enviando una
solicitud por escrito, acompañada de una fotocopia de su DNI dirigida a
PEMAS Servicios Profesionales, S.L. C/ Santos Justo y Pastor, 72 - 5,
C.P. 46022 de Valencia (España).
Mathias Behrle
08 Sep 2009 14:12:03 +0200):
Hi Carlos,
thx a lot for your effort to improve the translation, sounds promising.
Some first comments, or rather questions (I am not very familiar with gettext
translation and its possibilities):
> I added a brief explanation at
>
> https://bugs.tryton.org/roundup/issue1163
>
> I will appreciate if translators and developers read it and provide me
> with comments, I would like to get this done for 1.4.
Assuming the kernel freeze for 14.09.09 this appears to be quite a challenge
(implementation, regression tests, performance tests, ... ) ...
> Hi,
>
> I'm working on a branch to move Tryton to use Gettext's .po files to handle
> its translations.
>
> Today, I finished a really basic prototype and you can get it from:
>
> http://hg.pemas.es/tryton/gettext
>
> The approach I'm taking and the proposal I'm doing is:
>
> - We are explicit about the strings we want to translate (I still wonder why
> do we get class docstrings in the list of strings to translate). So we tag
> them as translatable, but on execution time it doesn't looks for any
> translation (that's the wrapper call N_())
> - We delay those translations until we actually use them (that's the real
> call to gettext or _()), just like we do right now to retrieve the
> translation from the database.
> - All Python code stops using the database for translations.
> - All report strings will stop using the database for translations.
> - All translatable xml strings used in Tryton to 'construct' the UI (for
> instance, the menuitem tags), are handled like Python code.
> - All translatable xml strings used to import data into Tryton (<record>
> tag), remains unchanged, except that the translations are provided inside
> the .po files so we stop using .csv for translations.
> - Given that translations are not handled always inside the database, all
> translations inside __tryton__.py could be removed and handled inside the .po
> files too, so we only have a single place to store translations.
> - The translations are handled per thread, so each thread is able to handle
> different translations depending on the language sent by the client.
> - The main server uses translations in the server locale (mainly for log
> output).
>
> I think this approach will help a lot us (translators) to improve Tryton
> translations because:
>
> - We could use any standard tool out there to do the translation work.
> - Duplicated strings are only translated once per Tryton module.
This could be a drawback, if you need to translate the same string different at
different places.
Another very important issue for me:
The current interface provides an easy way to lookup strings application wide
(application means trytond + installed modules). This is very important to get
a consistent translation over all modules and it is easy to search for (and
change) certain strings in all installed modules in one go. Can this be done
with those standard tools, too?
> - We will be able to use plural forms, so some translations will be seen more
> natural.
> - We will be able to see developer comments so we understand a bit better
> what means the developer with that string.
> - We will be able to get context strings, so we know how to translate a
> string which may mean two different things in your language but in English
> uses the same word.
How will this get along with 'Duplicated strings are only translated once per
Tryton module'?
> - We will have an exact reference to the source code from where the string
> comes from.
> - We are able to get any new string easily without having to do the update /
> export dance using the server, just a python command from a fresh checkout
> would be enough.
> - We will be able to use standard tools to see translation status (like
> http://l10n.gnome.org/).
>
> and the most important....
>
> - We will be able to do translations reviews / improvements even once the
> module is released as stable, because we will not be changing the database at
> all.
>
> I think this approach will help a lot us (developers) to improve Tryton
> because:
>
> - We can add some comments in the source code that will be seen by
> translators, so they understand better the meaning of a phrase.
> - Easier to translate means more translation contributors.
> - Less database queries, even if we use a cache, means a more scalable server.
Did you already some performance tests?
> - The logic behind gettext is much more simple than with current system so
> the code size will be reduced a bit.
>
>
> I would like to get some input about this initial implementation and
> comments / suggestions.
Just my first immediate questions.
> I know I'm a bit late, but if this proposal is approved me/my company will
> take care to do all the migration work so all current translations are reused
> and we get all this in time for 1.4.
--
Mathias Behrle
MBSolutions
Gilgenmatten 10 A
D-79114 Freiburg
Tel: +49(761)471023
Fax: +49(761)4770816
http://mbsolutions.selfip.biz
UStIdNr: DE 142009020
Daniel Baumann
>> - We could use any standard tool out there to do the translation work.
>> - Duplicated strings are only translated once per Tryton module.
>
> This could be a drawback, if you need to translate the same string different at
> different places.
not if you use po4a to manage the po files.
> Another very important issue for me:
> The current interface provides an easy way to lookup strings application wide
> (application means trytond + installed modules). This is very important to get
> a consistent translation over all modules and it is easy to search for (and
> change) certain strings in all installed modules in one go. Can this be done
> with those standard tools, too?
po4a is pretty much standard i think.
--
Address: Daniel Baumann, Burgunderstrasse 3, CH-4562 Biberist
Email: daniel....@panthera-systems.net
Internet: http://people.panthera-systems.net/~daniel-baumann/
Carlos Perelló Marín
> * Betr.: " [tryton] Use of Gettext / PyBabel proposal for translations" (Tue,
> 08 Sep 2009 14:12:03 +0200):
>
> Hi Carlos,
>
> thx a lot for your effort to improve the translation, sounds promising.
>
> Some first comments, or rather questions (I am not very familiar with gettext
> translation and its possibilities):
>
Sure, let's see...
>
>> I added a brief explanation at
>>
>> https://bugs.tryton.org/roundup/issue1163
>>
>> I will appreciate if translators and developers read it and provide me
>> with comments, I would like to get this done for 1.4.
>>
>
> Assuming the kernel freeze for 14.09.09 this appears to be quite a challenge
> (implementation, regression tests, performance tests, ... ) ...
>
Hmm, I missed that date.... anyway, I will try to do my best on it...
but yeah, it's going to be a bit hard... anyway, the hardest part was to
check that the system would work in a multi language environment, and
that's exactly what I did, the remaining part is to tag all strings and
substitute the sql lookups with the gettext call.
That's already solved, it's called 'context' so you provide the string
and a context string, so it will be handled as a different string in
gettext and the translator knows a bit more how to translate it.
> Another very important issue for me:
> The current interface provides an easy way to lookup strings application wide
> (application means trytond + installed modules). This is very important to get
> a consistent translation over all modules and it is easy to search for (and
> change) certain strings in all installed modules in one go. Can this be done
> with those standard tools, too?
>
There is something called translation memory which is a single .po file
with everything included from where some tools could extract suggestions.
Also Launchpad's translation option (I was one of the developers until
last year) also shows you how something is translated on different .po
files for the same product so you can detect incosistencies. As an added
bonus, you can see how it's translated in other modules, for instance,
openERP.
>
>> - We will be able to use plural forms, so some translations will be seen more
>> natural.
>> - We will be able to see developer comments so we understand a bit better
>> what means the developer with that string.
>> - We will be able to get context strings, so we know how to translate a
>> string which may mean two different things in your language but in English
>> uses the same word.
>>
>
> How will this get along with 'Duplicated strings are only translated once per
> Tryton module'?
>
As I already pointed before, the context is used when you don't want to
translate it in the same way.
>
>> - We will have an exact reference to the source code from where the string
>> comes from.
>> - We are able to get any new string easily without having to do the update /
>> export dance using the server, just a python command from a fresh checkout
>> would be enough.
>> - We will be able to use standard tools to see translation status (like
>> http://l10n.gnome.org/).
>>
>> and the most important....
>>
>> - We will be able to do translations reviews / improvements even once the
>> module is released as stable, because we will not be changing the database at
>> all.
>>
>> I think this approach will help a lot us (developers) to improve Tryton
>> because:
>>
>> - We can add some comments in the source code that will be seen by
>> translators, so they understand better the meaning of a phrase.
>> - Easier to translate means more translation contributors.
>> - Less database queries, even if we use a cache, means a more scalable server.
>>
>
> Did you already some performance tests?
>
No, I didn't, but given that database is current bottleneck, as many sql
queries we could save the better it will be.
But yes, some performance tests would be interesting I will add it to
the TODO list if the initial proposal is accepted.
>
>> - The logic behind gettext is much more simple than with current system so
>> the code size will be reduced a bit.
>>
>>
>> I would like to get some input about this initial implementation and
>> comments / suggestions.
>>
>
> Just my first immediate questions.
>
Thanks!
>
>
>> I know I'm a bit late, but if this proposal is approved me/my company will
>> take care to do all the migration work so all current translations are reused
>> and we get all this in time for 1.4.
>>
>
>
>
--
Cédric Krier
I know I will disappoint a lot of people but here is some remarks that I
already made in the past about this subject.
On 08/09/09 14:12 +0200, Carlos Perelló Marín wrote:
>The approach I'm taking and the proposal I'm doing is:
>
>- We are explicit about the strings we want to translate (I still wonder why
> do we get class docstrings in the list of strings to translate). So we tag
>them as translatable, but on execution time it doesn't looks for any
>translation (that's the wrapper call N_())
docstrings are stored in ir.model in a descrition field which is translatable.
>- We delay those translations until we actually use them (that's the real call
> to gettext or _()), just like we do right now to retrieve the translation
>from the database.
>- All Python code stops using the database for translations.
mo files are a _custom database_ which will not be shared between different
server.
>- All report strings will stop using the database for translations.
idem.
>- All translatable xml strings used in Tryton to 'construct' the UI (for
> instance, the menuitem tags), are handled like Python code.
>- All translatable xml strings used to import data into Tryton (<record> tag),
> remains unchanged, except that the translations are provided inside the .po
>files so we stop using .csv for translations.
How do you know that a field of a record is translatable ?
Per example, we create an act_window (ir.action.act_window)
<record model="ir.action.act_window">
<field name="name">Run wizard</field>
</record>
It is possible to detect that the field name is translatable if you have a
Tryton server running to query it.
But later when we read this record how do you know that you must call gettext
on the value of name to have the translation? Because there will be two kind
of translation (gettext and ir.translation).
>- Given that translations are not handled always inside the database, all
> translations inside __tryton__.py could be removed and handled inside the
>.po files too, so we only have a single place to store translations.
In fact, we will always have the current method because we must handle
translation of translatable fields of records (like name of product).
>- The translations are handled per thread, so each thread is able to handle
> different translations depending on the language sent by the client.
In Tryton, translation are handle by context. Sometimes inside the same
thread, we have different languages. (ex: french user that makes a invoice for
english customer).
>- The main server uses translations in the server locale (mainly for log
> output).
Why not. But it must not be changed for the logs from threads.
>
>I think this approach will help a lot us (translators) to improve Tryton
>translations because:
>
>- We could use any standard tool out there to do the translation work.
Don't use standard but common. :-)
>- Duplicated strings are only translated once per Tryton module.
Except that a same string in different place can have a different translation.
>- We will be able to use plural forms, so some translations will be seen more
> natural.
I don't see where it is useful?
>- We will be able to see developer comments so we understand a bit better what
> means the developer with that string.
For me, you must translate with the client user interface to have a proper
translation (fit well in the screen).
>- We will be able to get context strings, so we know how to translate a string
> which may mean two different things in your language but in English uses the
>same word.
I think it is more important to see the GUI than the code.
>- We will have an exact reference to the source code from where the string
> comes from.
We already have it (not in the form file/line)
>- We are able to get any new string easily without having to do the update /
> export dance using the server, just a python command from a fresh checkout
>would be enough.
So you skip the check in GUI.
>- We will be able to use standard tools to see translation status (like
> http://l10n.gnome.org/).
>
>and the most important....
>
>- We will be able to do translations reviews / improvements even once the
> module is released as stable, because we will not be changing the database
>at all.
And what about the custom translation ?
>
>I think this approach will help a lot us (developers) to improve Tryton
>because:
>
>- We can add some comments in the source code that will be seen by
> translators, so they understand better the meaning of a phrase.
>- Easier to translate means more translation contributors.
>- Less database queries, even if we use a cache, means a more scalable server.
I think that PostgreSQL will handle better the query than the embeded database
of gettext and more over there is cache.
>- The logic behind gettext is much more simple than with current system so the
> code size will be reduced a bit.
No, because we need to keep it for translatable fields.
And how do you handle same string in different modules because gettext doesn't
work with module?
>
>I know I'm a bit late, but if this proposal is approved me/my company will
>take care to do all the migration work so all current translations are reused
>and we get all this in time for 1.4.
I think it is really too late.
--
Cédric Krier
B2CK SPRL
Rue de Rotterdam, 4
4000 Liège
Belgium
Tel: +32 472 54 46 59
Email: cedric...@b2ck.com
Jabber: cedric...@b2ck.com
Website: http://www.b2ck.com/
Carlos Perelló Marín
[...]
>
>> Another very important issue for me:
>> The current interface provides an easy way to lookup strings application wide
>> (application means trytond + installed modules). This is very important to get
>> a consistent translation over all modules and it is easy to search for (and
>> change) certain strings in all installed modules in one go. Can this be done
>> with those standard tools, too?
>>
>>
>
> There is something called translation memory which is a single .po file
> with everything included from where some tools could extract suggestions.
>
> Also Launchpad's translation option (I was one of the developers until
> last year) also shows you how something is translated on different .po
> files for the same product so you can detect incosistencies. As an added
> bonus, you can see how it's translated in other modules, for instance,
> openERP.
>
>
Also, there is a gettext tool that allows you to 'merge' all .po files
in a single one so you could translate a single .po file for all modules
you are interested/available and then split them out again for commit.
The command is intelligent so it will handle conflicts and merge/tag
them for review .
Cheers.
Daniel Baumann
> Also, there is a gettext tool that allows you to 'merge' all .po files
> in a single one so you could translate a single .po file for all modules
> you are interested/available and then split them out again for commit.
> The command is intelligent so it will handle conflicts and merge/tag
> them for review .
aka po4a (which can does exactely that, amongst other things); or did
you have another one in mind?
Cédric Krier
> >> - We could use any standard tool out there to do the translation work.
> >> - Duplicated strings are only translated once per Tryton module.
> >>
> >
> > This could be a drawback, if you need to translate the same string different at
> > different places.
> >
>
> That's already solved, it's called 'context' so you provide the string
> and a context string, so it will be handled as a different string in
> gettext and the translator knows a bit more how to translate it.
But it is difficult in practice. The developper will not know that the concept
is different in some other languages. And so he will not add a context or he
will add a context to every string.
Carlos Perelló Marín
> I know I will disappoint a lot of people but here is some remarks that I
> already made in the past about this subject.
>
this is the start to see whether it's possible or not ;-)
> On 08/09/09 14:12 +0200, Carlos Perelló Marín wrote:
>
>> The approach I'm taking and the proposal I'm doing is:
>>
>> - We are explicit about the strings we want to translate (I still wonder why
>> do we get class docstrings in the list of strings to translate). So we tag
>> them as translatable, but on execution time it doesn't looks for any
>> translation (that's the wrapper call N_())
>>
>
> docstrings are stored in ir.model in a descrition field which is translatable.
>
Is it useful/used?
If it's useful / used, then we will not save that, we will still need to
translate it...
>
>> - We delay those translations until we actually use them (that's the real call
>> to gettext or _()), just like we do right now to retrieve the translation
>>
> >from the database.
>
>> - All Python code stops using the database for translations.
>>
>
> mo files are a _custom database_ which will not be shared between different
> server.
>
I'm not sure I'm understanding you... The .mo files are shared in the
same way that the .py code is shared.
>
>> - All report strings will stop using the database for translations.
>>
>
> idem.
>
Same here.
>
>> - All translatable xml strings used in Tryton to 'construct' the UI (for
>> instance, the menuitem tags), are handled like Python code.
>> - All translatable xml strings used to import data into Tryton (<record> tag),
>> remains unchanged, except that the translations are provided inside the .po
>> files so we stop using .csv for translations.
>>
>
> How do you know that a field of a record is translatable ?
> Per example, we create an act_window (ir.action.act_window)
>
> <record model="ir.action.act_window">
> <field name="name">Run wizard</field>
> </record>
>
> It is possible to detect that the field name is translatable if you have a
> Tryton server running to query it.
> But later when we read this record how do you know that you must call gettext
> on the value of name to have the translation? Because there will be two kind
> of translation (gettext and ir.translation).
>
For that kind of translations, we keep current system. .po file will be
used as the translation source, but the translation will be stored
inside the database and work as the .csv translations work right now. In
that case, you don't win anything except the metadata that .po files
provide you and the .po translation tools.
>
>> - Given that translations are not handled always inside the database, all
>> translations inside __tryton__.py could be removed and handled inside the
>> .po files too, so we only have a single place to store translations.
>>
>
> In fact, we will always have the current method because we must handle
> translation of translatable fields of records (like name of product).
>
You mean for translations added directly inside Tryton using the client?
In that case, we only need to handle __tryton__.py files like the xml's
record case. Where the .po file provide the translations to be stored in
the database.
>
>> - The translations are handled per thread, so each thread is able to handle
>> different translations depending on the language sent by the client.
>>
>
> In Tryton, translation are handle by context. Sometimes inside the same
> thread, we have different languages. (ex: french user that makes a invoice for
> english customer).
>
Then, we could add a way to override the translation. Current
implementation has a global translation catalog per thread, but the
gettext call could accept an optional argument to get the translation
for any other catalog.
>
>> - The main server uses translations in the server locale (mainly for log
>> output).
>>
>
> Why not. But it must not be changed for the logs from threads.
>
What do you mean here? that thread logs should keep the global server
language or the client language? I guess you mean the global server
language, don't you?
>
>> I think this approach will help a lot us (translators) to improve Tryton
>> translations because:
>>
>> - We could use any standard tool out there to do the translation work.
>>
>
> Don't use standard but common. :-)
>
Ok... the most common tool in Free Software world ;-)
>
>> - Duplicated strings are only translated once per Tryton module.
>>
>
> Except that a same string in different place can have a different translation.
>
Which is why the context string from gettext exists.
>
>> - We will be able to use plural forms, so some translations will be seen more
>> natural.
>>
>
> I don't see where it is useful?
>
In some languages, the language changes whether there is 0 or 1 or 12
items so it's not enought to have
no items
%d items
because in some languages for different %d values, the 'items' word may
be different.
You can read more about this at:
http://www.gnu.org/software/gettext/manual/html_chapter/gettext_10.html#Plural-forms
>
>> - We will be able to see developer comments so we understand a bit better what
>> means the developer with that string.
>>
>
> For me, you must translate with the client user interface to have a proper
> translation (fit well in the screen).
>
Indeed, that's part of the QA work, but that doesn't mean is the best
way to do the whole translation work.
>
>> - We will be able to get context strings, so we know how to translate a string
>> which may mean two different things in your language but in English uses the
>> same word.
>>
>
> I think it is more important to see the GUI than the code.
>
Right, but is impossible to cover all code paths to do a full
translation at least in a reasonable amount of time.
>
>> - We will have an exact reference to the source code from where the string
>> comes from.
>>
>
> We already have it (not in the form file/line)
>
Right, and you need to know a bit about internal code layout to know
where you must look for. However, there are tools (I think Emacs and
other editors do it) allow you to go directly to the file reference like
a cross code editor.
>
>> - We are able to get any new string easily without having to do the update /
>> export dance using the server, just a python command from a fresh checkout
>> would be enough.
>>
>
> So you skip the check in GUI.
>
As I said, the GUI check is part of the QA work, but it doesn't mean it
must be the unique way to do it. Being able to script the process allows
you also to get statistics on the website without having to run a server
and much more funny things, I'm sure.
>
>> - We will be able to use standard tools to see translation status (like
>> http://l10n.gnome.org/).
>>
>> and the most important....
>>
>> - We will be able to do translations reviews / improvements even once the
>> module is released as stable, because we will not be changing the database
>> at all.
>>
>
> And what about the custom translation ?
>
I'm talking about python and XML UI and report translations, that are
not in the database, the ones in the database (<record> translations)
will not be updated and may be handled in a different .po file to make
it clear and allow Mercurial to commit to those files.
>
>> I think this approach will help a lot us (developers) to improve Tryton
>> because:
>>
>> - We can add some comments in the source code that will be seen by
>> translators, so they understand better the meaning of a phrase.
>> - Easier to translate means more translation contributors.
>> - Less database queries, even if we use a cache, means a more scalable server.
>>
>
> I think that PostgreSQL will handle better the query than the embeded database
> of gettext and more over there is cache.
>
I doubt it, because with PostgreSQL you have locks, and take care of
writes, while the other is a read only 'database' optimised exactly to
do that, retrieve translations.
>
>> - The logic behind gettext is much more simple than with current system so the
>> code size will be reduced a bit.
>>
>
> No, because we need to keep it for translatable fields.
> And how do you handle same string in different modules because gettext doesn't
> work with module?
>
Gettext doesn't work with module, PyBabel does. There were several
options, but I took an approach similar to the current one used by
Tryton, the last module in the dependency tree is the one that has
preference. Tryton just gets the first row returned by the database query.
>
>> I know I'm a bit late, but if this proposal is approved me/my company will
>> take care to do all the migration work so all current translations are reused
>> and we get all this in time for 1.4.
>>
>
> I think it is really too late.
>
>
I know... but even in that case, I would prefer to talk about this and
get it done, if it's not possible for 1.4 for 1.6
Carlos Perelló Marín
> Carlos Perelló Marín wrote:
>
>> Also, there is a gettext tool that allows you to 'merge' all .po files
>> in a single one so you could translate a single .po file for all modules
>> you are interested/available and then split them out again for commit.
>> The command is intelligent so it will handle conflicts and merge/tag
>> them for review .
>>
>
> aka po4a (which can does exactely that, amongst other things); or did
> you have another one in mind?
>
>
Carlos Perelló Marín
> On 08/09/09 15:22 +0200, Carlos Perelló Marín wrote:
>
>>>> - We could use any standard tool out there to do the translation work.
>>>> - Duplicated strings are only translated once per Tryton module.
>>>>
>>>>
>>> This could be a drawback, if you need to translate the same string different at
>>> different places.
>>>
>>>
>> That's already solved, it's called 'context' so you provide the string
>> and a context string, so it will be handled as a different string in
>> gettext and the translator knows a bit more how to translate it.
>>
>
> But it is difficult in practice. The developper will not know that the concept
> is different in some other languages. And so he will not add a context or he
> will add a context to every string.
>
>
>
And that's ok (the fact that he doesn't know it). Then, when a
translator finds the problem it's reported as a bug, the context is
added and the system improved. We don't expect that a developer borns
knowing all kind of conflicts in all languages.... but we need a way to
provide a way to fix it without having to translate the same string a
thousand of times just in case it may be a problem...
Cheers.
Cédric Krier
>
> Cédric Krier escribió:
> > On 08/09/09 15:22 +0200, Carlos Perelló Marín wrote:
> >
> >>>> - We could use any standard tool out there to do the translation work.
> >>>> - Duplicated strings are only translated once per Tryton module.
> >>>>
> >>>>
> >>> This could be a drawback, if you need to translate the same string different at
> >>> different places.
> >>>
> >>>
> >> That's already solved, it's called 'context' so you provide the string
> >> and a context string, so it will be handled as a different string in
> >> gettext and the translator knows a bit more how to translate it.
> >>
> >
> > But it is difficult in practice. The developper will not know that the concept
> > is different in some other languages. And so he will not add a context or he
> > will add a context to every string.
> >
> >
> >
>
> And that's ok (the fact that he doesn't know it). Then, when a
> translator finds the problem it's reported as a bug, the context is
> added and the system improved. We don't expect that a developer borns
> knowing all kind of conflicts in all languages.... but we need a way to
> provide a way to fix it without having to translate the same string a
> thousand of times just in case it may be a problem...
The translation of the same string can be (has partially been) fixed. A
wizard can do the job.
Cédric Krier
> >> - We are explicit about the strings we want to translate (I still wonder why
> >> do we get class docstrings in the list of strings to translate). So we tag
> >> them as translatable, but on execution time it doesn't looks for any
> >> translation (that's the wrapper call N_())
> >>
> >
> > docstrings are stored in ir.model in a descrition field which is translatable.
> >
>
> Is it useful/used?
>
> If it's useful / used, then we will not save that, we will still need to
> translate it...
Yes, it is useful. We have a picture of the models in database. So we can fill
the reference field, we can display a "UML" kind of graph, we can handle the
access rules etc.
In Tryton, all is a record.
And later, it will be used to create custom model (or custom field) from the
interface.
> >> - We delay those translations until we actually use them (that's the real call
> >> to gettext or _()), just like we do right now to retrieve the translation
> >>
> > >from the database.
> >
> >> - All Python code stops using the database for translations.
> >>
> >
> > mo files are a _custom database_ which will not be shared between different
> > server.
> >
>
> I'm not sure I'm understanding you... The .mo files are shared in the
> same way that the .py code is shared.
I mean that we can have many server running on the same database. So if the
traduction are in database, you share the same value.
> >> - All translatable xml strings used in Tryton to 'construct' the UI (for
> >> instance, the menuitem tags), are handled like Python code.
> >> - All translatable xml strings used to import data into Tryton (<record> tag),
> >> remains unchanged, except that the translations are provided inside the .po
> >> files so we stop using .csv for translations.
> >>
> >
> > How do you know that a field of a record is translatable ?
> > Per example, we create an act_window (ir.action.act_window)
> >
> > <record model="ir.action.act_window">
> > <field name="name">Run wizard</field>
> > </record>
> >
> > It is possible to detect that the field name is translatable if you have a
> > Tryton server running to query it.
> > But later when we read this record how do you know that you must call gettext
> > on the value of name to have the translation? Because there will be two kind
> > of translation (gettext and ir.translation).
> >
> For that kind of translations, we keep current system. .po file will be
> used as the translation source, but the translation will be stored
> inside the database and work as the .csv translations work right now. In
> that case, you don't win anything except the metadata that .po files
> provide you and the .po translation tools.
This translations are almost 30% of translations.
So you still need to have current translation process (what is the benefit?)
> >> - Given that translations are not handled always inside the database, all
> >> translations inside __tryton__.py could be removed and handled inside the
> >> .po files too, so we only have a single place to store translations.
> >>
> >
> > In fact, we will always have the current method because we must handle
> > translation of translatable fields of records (like name of product).
> >
>
> You mean for translations added directly inside Tryton using the client?
Like the product name, it must be translatable.
> In that case, we only need to handle __tryton__.py files like the xml's
> record case. Where the .po file provide the translations to be stored in
> the database.
So you need to use the .mo of modules that are not installed which is not
clean.
> >
> >> - The translations are handled per thread, so each thread is able to handle
> >> different translations depending on the language sent by the client.
> >>
> >
> > In Tryton, translation are handle by context. Sometimes inside the same
> > thread, we have different languages. (ex: french user that makes a invoice for
> > english customer).
> >
>
> Then, we could add a way to override the translation. Current
> implementation has a global translation catalog per thread, but the
> gettext call could accept an optional argument to get the translation
> for any other catalog.
Ok, but how do you know the global language?
> >
> >> - The main server uses translations in the server locale (mainly for log
> >> output).
> >>
> >
> > Why not. But it must not be changed for the logs from threads.
> >
>
> What do you mean here? that thread logs should keep the global server
> language or the client language? I guess you mean the global server
> language, don't you?
Yes. (global server).
>
> In some languages, the language changes whether there is 0 or 1 or 12
> items so it's not enought to have
>
> no items
> %d items
>
> because in some languages for different %d values, the 'items' word may
> be different.
>
> You can read more about this at:
>
> http://www.gnu.org/software/gettext/manual/html_chapter/gettext_10.html#Plural-forms
I don't know any case like this in the current code. As Tryton is very
modular, we use generic message.
> >
> >> - We will be able to see developer comments so we understand a bit better what
> >> means the developer with that string.
> >>
> >
> > For me, you must translate with the client user interface to have a proper
> > translation (fit well in the screen).
> >
>
> Indeed, that's part of the QA work, but that doesn't mean is the best
> way to do the whole translation work.
It is best to have the good one earlier.
>
> >
> >> - We will be able to get context strings, so we know how to translate a string
> >> which may mean two different things in your language but in English uses the
> >> same word.
> >>
> >
> > I think it is more important to see the GUI than the code.
> >
>
> Right, but is impossible to cover all code paths to do a full
> translation at least in a reasonable amount of time.
Of course you must switch between the list of translation and forms.
But it is long only the first time.
>
> >
> >> - We will have an exact reference to the source code from where the string
> >> comes from.
> >>
> >
> > We already have it (not in the form file/line)
> >
>
> Right, and you need to know a bit about internal code layout to know
> where you must look for. However, there are tools (I think Emacs and
> other editors do it) allow you to go directly to the file reference like
> a cross code editor.
> >
> >> - We are able to get any new string easily without having to do the update /
> >> export dance using the server, just a python command from a fresh checkout
> >> would be enough.
> >>
> >
> > So you skip the check in GUI.
> >
>
> As I said, the GUI check is part of the QA work, but it doesn't mean it
> must be the unique way to do it. Being able to script the process allows
> you also to get statistics on the website without having to run a server
> and much more funny things, I'm sure.
What is the benefit of stats ?
It is possible to have those stats. We can add an instance on demo host to
generate stats (if someone write a script for that).
>
> >
> >> - We will be able to use standard tools to see translation status (like
> >> http://l10n.gnome.org/).
> >>
> >> and the most important....
> >>
> >> - We will be able to do translations reviews / improvements even once the
> >> module is released as stable, because we will not be changing the database
> >> at all.
> >>
> >
> > And what about the custom translation ?
> >
>
> I'm talking about python and XML UI and report translations, that are
> not in the database, the ones in the database (<record> translations)
> will not be updated and may be handled in a different .po file to make
> it clear and allow Mercurial to commit to those files.
Why splitting translations? It is the same, it is part of the code (menu name,
action name, etc.).
>
> >
> >> I think this approach will help a lot us (developers) to improve Tryton
> >> because:
> >>
> >> - We can add some comments in the source code that will be seen by
> >> translators, so they understand better the meaning of a phrase.
> >> - Easier to translate means more translation contributors.
> >> - Less database queries, even if we use a cache, means a more scalable server.
> >>
> >
> > I think that PostgreSQL will handle better the query than the embeded database
> > of gettext and more over there is cache.
> >
> I doubt it, because with PostgreSQL you have locks, and take care of
> writes, while the other is a read only 'database' optimised exactly to
> do that, retrieve translations.
There is no locks (except for ir.sequence but doesn't have any effect for
translation).
We had the right indexes on ir.translation, I don't think it can be faster.
>
> >
> >> - The logic behind gettext is much more simple than with current system so the
> >> code size will be reduced a bit.
> >>
> >
> > No, because we need to keep it for translatable fields.
> > And how do you handle same string in different modules because gettext doesn't
> > work with module?
> >
>
> Gettext doesn't work with module, PyBabel does. There were several
> options, but I took an approach similar to the current one used by
> Tryton, the last module in the dependency tree is the one that has
> preference. Tryton just gets the first row returned by the database query.
You talk about a rare case in current implemenation because almost every
string is unique.
But in your case, it will happen more often.
Cédric Krier
current infrastructure it must at least handle the same goals:
- translate all base terms.
- don't mix up record translations and base translations (if separate tools).
But I don't see how it is possible because most of base translations are
also record translations.
- provide at least the same granularity.
Carlos Perelló Marín
> On 08/09/09 15:55 +0200, Carlos Perelló Marín wrote:
>
>> Cédric Krier escribió:
>>
>>> On 08/09/09 15:22 +0200, Carlos Perelló Marín wrote:
>>>
>>>
>>>>>> - We could use any standard tool out there to do the translation work.
>>>>>> - Duplicated strings are only translated once per Tryton module.
>>>>>>
>>>>>>
>>>>>>
>>>>> This could be a drawback, if you need to translate the same string different at
>>>>> different places.
>>>>>
>>>>>
>>>>>
>>>> That's already solved, it's called 'context' so you provide the string
>>>> and a context string, so it will be handled as a different string in
>>>> gettext and the translator knows a bit more how to translate it.
>>>>
>>>>
>>> But it is difficult in practice. The developper will not know that the concept
>>> is different in some other languages. And so he will not add a context or he
>>> will add a context to every string.
>>>
>>>
>>>
>>>
>> And that's ok (the fact that he doesn't know it). Then, when a
>> translator finds the problem it's reported as a bug, the context is
>> added and the system improved. We don't expect that a developer borns
>> knowing all kind of conflicts in all languages.... but we need a way to
>> provide a way to fix it without having to translate the same string a
>> thousand of times just in case it may be a problem...
>>
>
> The translation of the same string can be (has partially been) fixed. A
> wizard can do the job.
>
>
I guess you talk about the fuzzy addition you did, right?
Yes, it's quite useful, however, I think you are thinking on a
'Translate once use for ever' use case, which is not valid for software
translation. The translations evolve over time, because is really hard,
if not impossible, to get a near perfect translation the first time you
translate software. Next time you do a review, you need to go string by
string, I did it for the es_ES when I started from es_CO, and is quite
boring and resource wasting when you see the same string over, and over
again...
I know you are doing the best to get Tryton l10n/i18n process as
painless as possible, but really, I think that we should find a solution
that is already tested, used and even maintained outside Tryton, so you
and others expend that time in more useful things directly related with
Tryton. That's one of my main goals, the other main goal is that
translators life is as easy as possible.
Carlos Perelló Marín
> On 08/09/09 15:51 +0200, Carlos Perelló Marín wrote:
>
>>>> - We are explicit about the strings we want to translate (I still wonder why
>>>> do we get class docstrings in the list of strings to translate). So we tag
>>>> them as translatable, but on execution time it doesn't looks for any
>>>> translation (that's the wrapper call N_())
>>>>
>>>>
>>> docstrings are stored in ir.model in a descrition field which is translatable.
>>>
>>>
>> Is it useful/used?
>>
>> If it's useful / used, then we will not save that, we will still need to
>> translate it...
>>
>
> Yes, it is useful. We have a picture of the models in database. So we can fill
> the reference field, we can display a "UML" kind of graph, we can handle the
> access rules etc.
> In Tryton, all is a record.
>
> And later, it will be used to create custom model (or custom field) from the
> interface.
>
I see, it's a low level priority translation, but yes, something that
should be translated.
>
>>>> - We delay those translations until we actually use them (that's the real call
>>>> to gettext or _()), just like we do right now to retrieve the translation
>>>>
>>>>
>>> >from the database.
>>>
>>>
>>>> - All Python code stops using the database for translations.
>>>>
>>>>
>>> mo files are a _custom database_ which will not be shared between different
>>> server.
>>>
>>>
>> I'm not sure I'm understanding you... The .mo files are shared in the
>> same way that the .py code is shared.
>>
>
> I mean that we can have many server running on the same database. So if the
> traduction are in database, you share the same value.
>
Well, I never used several servers using the same database, however, I
expect that all those servers use the same version or it would be a
madness... If all use the same code, they will use the same version of
translations.
I know the database schema would be the same for all 1.2 server
versions, however, if someone runs different 1.2 versions against the
database, the last thing that he may worry about is getting different
translations, the fact that one server has a bug fixed and other server
doesn't would produce a bunch of weird situations...
>
>>>> - All translatable xml strings used in Tryton to 'construct' the UI (for
>>>> instance, the menuitem tags), are handled like Python code.
>>>> - All translatable xml strings used to import data into Tryton (<record> tag),
>>>> remains unchanged, except that the translations are provided inside the .po
>>>> files so we stop using .csv for translations.
>>>>
>>>>
>>> How do you know that a field of a record is translatable ?
>>> Per example, we create an act_window (ir.action.act_window)
>>>
>>> <record model="ir.action.act_window">
>>> <field name="name">Run wizard</field>
>>> </record>
>>>
>>> It is possible to detect that the field name is translatable if you have a
>>> Tryton server running to query it.
>>> But later when we read this record how do you know that you must call gettext
>>> on the value of name to have the translation? Because there will be two kind
>>> of translation (gettext and ir.translation).
>>>
>>>
>> For that kind of translations, we keep current system. .po file will be
>> used as the translation source, but the translation will be stored
>> inside the database and work as the .csv translations work right now. In
>> that case, you don't win anything except the metadata that .po files
>> provide you and the .po translation tools.
>>
>
> This translations are almost 30% of translations.
> So you still need to have current translation process (what is the benefit?)
>
The main benefit I see is (the first one is the one that motivated me to
work on this):
1. The file format is much more sane than current .csv file and even if
you really love Tryton client for doing translations, every translator
already have his prefered tool to translate and you(we) don't really
need to expend resources on doing Tryton also the perfect translation
editor.
2. The number of SQL queries will be reduced even if the performance of
the gettext method is the same as the same queries against the SQL
server, you will free resources that are tagged as the bottle neck, so
that's an improvement.
>
>>>> - Given that translations are not handled always inside the database, all
>>>> translations inside __tryton__.py could be removed and handled inside the
>>>> .po files too, so we only have a single place to store translations.
>>>>
>>>>
>>> In fact, we will always have the current method because we must handle
>>> translation of translatable fields of records (like name of product).
>>>
>>>
>> You mean for translations added directly inside Tryton using the client?
>>
>
> Like the product name, it must be translatable.
>
From what I read in other parts of this discussion, I think that your
target is that we are able to develop a whole Tryton module from the
Tryton client, right?
Although I don't really see it's as a need here (for the use cases I'm
handling), I understand it may be useful for others and I should say
that I didn't pay too much attention to that use case, given that I
didn't even think on it, so I would need to mature a bit more the
proposal to cope with that use case.
>
>> In that case, we only need to handle __tryton__.py files like the xml's
>> record case. Where the .po file provide the translations to be stored in
>> the database.
>>
>
> So you need to use the .mo of modules that are not installed which is not
> clean.
>
Good point. We could explore the option to use different translation
domains in this case but I would like to do some performance checks to
see whether it would affect performance in any way.
>
>>>
>>>
>>>> - The translations are handled per thread, so each thread is able to handle
>>>> different translations depending on the language sent by the client.
>>>>
>>>>
>>> In Tryton, translation are handle by context. Sometimes inside the same
>>> thread, we have different languages. (ex: french user that makes a invoice for
>>> english customer).
>>>
>>>
>> Then, we could add a way to override the translation. Current
>> implementation has a global translation catalog per thread, but the
>> gettext call could accept an optional argument to get the translation
>> for any other catalog.
>>
>
> Ok, but how do you know the global language?
>
You get it from the global process, just like any other application in
your system. If the sysadmin configures the system to use the C, en_US,
en_UK, es_ES, etc... locales that's the global language.
>
>>>
>>>
>>>> - The main server uses translations in the server locale (mainly for log
>>>> output).
>>>>
>>>>
>>> Why not. But it must not be changed for the logs from threads.
>>>
>>>
>> What do you mean here? that thread logs should keep the global server
>> language or the client language? I guess you mean the global server
>> language, don't you?
>>
>
> Yes. (global server).
>
Not a big problem, for syslog calls we would use a function that uses
the global server language.
>
>> In some languages, the language changes whether there is 0 or 1 or 12
>> items so it's not enought to have
>>
>> no items
>> %d items
>>
>> because in some languages for different %d values, the 'items' word may
>> be different.
>>
>> You can read more about this at:
>>
>> http://www.gnu.org/software/gettext/manual/html_chapter/gettext_10.html#Plural-forms
>>
>
> I don't know any case like this in the current code. As Tryton is very
> modular, we use generic message.
>
I guess that may be ok for the core, but for vertical modules... it may
not be enough.
>
>>>
>>>
>>>> - We will be able to see developer comments so we understand a bit better what
>>>> means the developer with that string.
>>>>
>>>>
>>> For me, you must translate with the client user interface to have a proper
>>> translation (fit well in the screen).
>>>
>>>
>> Indeed, that's part of the QA work, but that doesn't mean is the best
>> way to do the whole translation work.
>>
>
> It is best to have the good one earlier.
>
Well, that's in your mind process, and is ok, but it doesn't mean all
translators follow the same process you have in your mind for
translations (Myself is a good example), so is good to give choices that
fit better in different translation procedures.
>
>>>
>>>
>>>> - We will be able to get context strings, so we know how to translate a string
>>>> which may mean two different things in your language but in English uses the
>>>> same word.
>>>>
>>>>
>>> I think it is more important to see the GUI than the code.
>>>
>>>
>> Right, but is impossible to cover all code paths to do a full
>> translation at least in a reasonable amount of time.
>>
>
> Of course you must switch between the list of translation and forms.
> But it is long only the first time.
>
Well, Tryton is quite modular, so one module doesn't affect a single
form but it may change a bunch of them, having to hunt one by one all
those forms is a bit hard IMHO. Remember, you know quite well, the
system but there are people that don't even know about programming, so
many of them will not know which forms should be loaded based just on
the information inside the .csv file.
>
>>>
>>>
>>>> - We will have an exact reference to the source code from where the string
>>>> comes from.
>>>>
>>>>
>>> We already have it (not in the form file/line)
>>>
>>>
>> Right, and you need to know a bit about internal code layout to know
>> where you must look for. However, there are tools (I think Emacs and
>> other editors do it) allow you to go directly to the file reference like
>> a cross code editor.
>>
>>>
>>>
>>>> - We are able to get any new string easily without having to do the update /
>>>> export dance using the server, just a python command from a fresh checkout
>>>> would be enough.
>>>>
>>>>
>>> So you skip the check in GUI.
>>>
>>>
>> As I said, the GUI check is part of the QA work, but it doesn't mean it
>> must be the unique way to do it. Being able to script the process allows
>> you also to get statistics on the website without having to run a server
>> and much more funny things, I'm sure.
>>
>
> What is the benefit of stats ?
> It is possible to have those stats. We can add an instance on demo host to
> generate stats (if someone write a script for that).
>
Well, stats is useful for two things:
1. You know the status of your translation without having to download
and check for new strings, just take a look to the statistics and you
will know whether there is something you need to update.
2. Translation 'ego', you will think this is stupid, but from my
experience in GNOME and Ubuntu, translation teams are quite competitive
to be one of the best supported languages ;-).
This is not the most important part of the proposal, but yet again, your
solution is to implement something specific for Tryton, while using a
'common' used format would provide us with the tools, and let us focus
on the important part, Tryton functionality development.
>
>>>
>>>
>>>> - We will be able to use standard tools to see translation status (like
>>>> http://l10n.gnome.org/).
>>>>
>>>> and the most important....
>>>>
>>>> - We will be able to do translations reviews / improvements even once the
>>>> module is released as stable, because we will not be changing the database
>>>> at all.
>>>>
>>>>
>>> And what about the custom translation ?
>>>
>>>
>> I'm talking about python and XML UI and report translations, that are
>> not in the database, the ones in the database (<record> translations)
>> will not be updated and may be handled in a different .po file to make
>> it clear and allow Mercurial to commit to those files.
>>
>
> Why splitting translations? It is the same, it is part of the code (menu name,
> action name, etc.).
>
Well, one of the main things I was thinking on was that for a proper QA,
translations should match a given version, so if someone calls you for
support and you ask them for the value of the field 'Nombre' (Name in
Spanish), they should see the field 'Nombre' not something different
that someone decided to change for some obscure reason.
In the other hand, we have the data handled by Tryton, the feature of
supporting the translation of that data is quite interesting and of
course, depends on the customer to be able to translate it (names of
products, names of categories, etc...).
That's why I talk about the split. Said that, we have the 'problem' of
the use case you gave me about being able to create Tryton modules from
the Tryton client, which I still need to think about it.
Btw, this doesn't mean that with my proposal you are not able to
override/customise core translations, you can still do it installing a
Tryton module with your customised translations, however is not a
functionality that I planned.
>
>>>
>>>
>>>> I think this approach will help a lot us (developers) to improve Tryton
>>>> because:
>>>>
>>>> - We can add some comments in the source code that will be seen by
>>>> translators, so they understand better the meaning of a phrase.
>>>> - Easier to translate means more translation contributors.
>>>> - Less database queries, even if we use a cache, means a more scalable server.
>>>>
>>>>
>>> I think that PostgreSQL will handle better the query than the embeded database
>>> of gettext and more over there is cache.
>>>
>>>
>> I doubt it, because with PostgreSQL you have locks, and take care of
>> writes, while the other is a read only 'database' optimised exactly to
>> do that, retrieve translations.
>>
>
> There is no locks (except for ir.sequence but doesn't have any effect for
> translation).
> We had the right indexes on ir.translation, I don't think it can be faster.
>
Well, in the same way you believe that, I believe the opposite (about
which one would be faster), so instead of losing our time arguing about
it, I will wait until I have some performance checks done so we talk
about facts instead of feelings (I'm not saying you are wrong and I'm
right, I'm just saying that I feel the oposite you feel so I don't have
solid arguments right now).
>
>>>
>>>
>>>> - The logic behind gettext is much more simple than with current system so the
>>>> code size will be reduced a bit.
>>>>
>>>>
>>> No, because we need to keep it for translatable fields.
>>> And how do you handle same string in different modules because gettext doesn't
>>> work with module?
>>>
>>>
>> Gettext doesn't work with module, PyBabel does. There were several
>> options, but I took an approach similar to the current one used by
>> Tryton, the last module in the dependency tree is the one that has
>> preference. Tryton just gets the first row returned by the database query.
>>
>
> You talk about a rare case in current implemenation because almost every
> string is unique.
> But in your case, it will happen more often.
>
>
Sure, in my case it will happen more often, but we have a way to avoid
bad results and reduce the number of strings at the same time, just
handling every single string as different from the rest is not the the
right approach from the translator point of view. Remember, we should
find a balance between the development effort and the translation effort.
Carlos Perelló Marín
> To summarize, I have nothing against gettext but if you want to replace the
> current infrastructure it must at least handle the same goals:
>
the technical side of things, but that's not a bad thing, given that
it's your role as the project lead. This discussion showed me some
Tryton details I didn't know or didn't realise and helped me also to
think about ways to improve the proposal, which is exactly what I wanted.
I wished to get this done for 1.4, but I don't expected to get it done ;-).
> - translate all base terms.
> - don't mix up record translations and base translations (if separate tools).
> But I don't see how it is possible because most of base translations are
> also record translations.
> - provide at least the same granularity.
>
There is an idea I discarded which was to use gettext by default and use
current database ir_translation table to override translations. I don't
think the performance win will be high, given that you still need to
check the database for strings overriding the default one, but I will
introduce this also in my performance check. The fact that the table is
smaller should also speed it a bit but we need to know the gettext
performance to be sure it isn't worse than just using database.
Anyway, I still need to mature this a bit more.
Thanks for the input!