i18n state of affairs

[Hans-Peter Jansen]

Hi,

today, I dug into the various packages, that use i18n:

deform
deformdemo
colander

and I noticed, that they're missing a unified approach.

Some use lingua, some babel, and this results in inconsistent i18n state and
handling. I propose to use the current i18n.sh script of lingua for all Pylons
packages, similar to https://github.com/Pylons/deform/pull/332 .

Why: due to Pyramids range of supported templating systems, lingua is far
better suited to fulfill our needs. It's also much easier to remember (and to
document) to run a single script, then three of them in order to update
translations, not to speak of the unfortunate setup integration..

The nice side of this move:

$ ./i18n.sh
Extract messages
No changes found - not replacing deform/locale/deform.pot
Update translations
......... done.
......... done.
......... done.
......... done.
......... done.
......... done.
......... done.
......... done.
......... done.
......... done.
... done.
......... done.
......... done.
......... done.
......... done.
.......... done.
......... done.
......... done.
Compile message catalogs
cs: 19 translated messages, 2 fuzzy translations, 5 untranslated messages.
da: 11 translated messages, 1 fuzzy translation, 14 untranslated messages.
de: 26 translated messages.
el: 24 translated messages, 2 untranslated messages.
es: 11 translated messages, 1 fuzzy translation, 14 untranslated messages.
fi: 25 translated messages, 1 untranslated message.
fr: 19 translated messages, 2 fuzzy translations, 5 untranslated messages.
it: 19 translated messages, 2 fuzzy translations, 5 untranslated messages.
ja: 25 translated messages, 1 untranslated message.
nl: 19 translated messages, 2 fuzzy translations, 5 untranslated messages.
pl: 19 translated messages, 2 fuzzy translations, 5 untranslated messages.
pt: 19 translated messages, 2 fuzzy translations, 5 untranslated messages.
pt_BR: 11 translated messages, 1 fuzzy translation, 14 untranslated messages.
ru: 19 translated messages, 2 fuzzy translations, 5 untranslated messages.
sv: 18 translated messages, 2 fuzzy translations, 6 untranslated messages.
zh: 19 translated messages, 2 fuzzy translations, 5 untranslated messages.
zh_CN: 21 translated messages, 1 fuzzy translation, 4 untranslated messages.
zh_Hans: 21 translated messages, 1 fuzzy translation, 4 untranslated messages.

As a consequence, all these packages will depend on lingua then..

One downside of this move: jinja2 support depends on babel, and needs some
configuration file support for pot-create, that has to be patched into i18n.sh
before:

$ cat lingua.cfg
[extractor:xml]
default-engine = tales

[extractor:chameleon]
default-engine = tales

[extensions]
.jinja2 = babel-jinja2

What do you think?

Cheers,
Pete

[Michael Merickel]

On Wed, Feb 8, 2017 at 11:36 AM, Hans-Peter Jansen <h...@urpla.net> wrote:

As a consequence, all these packages will depend on lingua then..

Do the packages actually need to depend on lingua? I see that your PR to deform doesn't add it, however deform expects it to in the dev environment (tests, running i18n.sh, etc). I'm just trying to clarify if you're saying it would be a new runtime dependency.

 

One downside of this move: jinja2 support depends on babel, and needs some
configuration file support for pot-create, that has to be patched into i18n.sh
before:

The quick history here, as I know it:

- Pyramid, etc shipped using babel for everything.

- Babel didn't support py3.

- For that reason and I'm sure many other reasons, Wichert created lingua and submitted a few PRs to pyramid to start using it in the documentation.

I personally don't know what the state of the art is now that babel supports py3 I haven't needed to write a fully i18n app myself and so my knowledge is only limited to diving in and fixing certain bugs or reviewing PRs.

I very much appreciate your work on this and would love to see things get standardized and modernized. I've had several people ask in #pyramid on IRC why things are not working as our docs kinda sort of tell you to use both babel and lingua without definitively explaining anything as I think they were only partially updated when switching to lingua.

[Hans-Peter Jansen]

Hi Michael,

On Mittwoch, 8. Februar 2017 12:22:27 Michael Merickel wrote:
> On Wed, Feb 8, 2017 at 11:36 AM, Hans-Peter Jansen <h...@urpla.net> wrote:
> > As a consequence, all these packages will depend on lingua then..
>
> Do the packages actually need to depend on lingua? I see that your PR to
> deform doesn't add it, however deform expects it to in the dev environment
> (tests, running i18n.sh, etc). I'm just trying to clarify if you're saying
> it would be a new runtime dependency.

AFAICS, it's not a runtime dependency, it's needed for i18n work only (as long
as the mo files are tarred up)..

> > One downside of this move: jinja2 support depends on babel, and needs some
> > configuration file support for pot-create, that has to be patched into
> > i18n.sh
>
> > before:
> The quick history here, as I know it:
>
> - Pyramid, etc shipped using babel for everything.
> - Babel didn't support py3.
> - For that reason and I'm sure many other reasons, Wichert created lingua
> and submitted a few PRs to pyramid to start using it in the documentation.

Thanks for the intro. Yeah, I know, Armin Ronacher add his Py3 crusade..

> I personally don't know what the state of the art is now that babel
> supports py3 I haven't needed to write a fully i18n app myself and so my
> knowledge is only limited to diving in and fixing certain bugs or reviewing
> PRs.

I'm just starting with Pyramid (ahem, websauna) in a project, that essentially
needs full i18n support.. Therefor, I try to wrap my head around all these
loose ends ATM. Not that easy, because the web is full of misleading
information in this regard as well.

Since I cannot depend on hearsay, I started with the simplest possible task,
having a pyramid-cookiecutter-starter project localized. Since that uses
jinja2 templating only, babel is sufficient. Adding deform makes lingua
necessary, since babel doesn't support chameleon templates (to my knowledge).
lingua depends on babel, when it comes to jinja2 extraction. Oh, well..

Sorry for not having expressed that from the beginning.

> I very much appreciate your work on this and would love to see things get
> standardized and modernized. I've had several people ask in #pyramid on IRC
> why things are not working as our docs kinda sort of tell you to use both
> babel and lingua without definitively explaining anything as I think they
> were only partially updated when switching to lingua.

Maybe, with your insight into Pyramid and my intention to have a properly
working i18n in _all_ these packages, we can tidy up that mess a bit.

As of now, I don't see the need for babel and lingua as runtime dependencies
at all. babel brings in a lot of useful things, that a fully localized app can
make use of, but none of that is used in the mentioned packages ATM. lingua is
created for message extraction only (with a script called pot-create). I can
add a test to i18n to check, if pot-create exists, and bail out with an
installation hint otherwise.

In order to make the translation infrastructure universally useful, we can:
* add the i18n.sh script
* ask for installation of babel and lingua, if one is missing
* add a lingua.cfg:

[extractor:xml]
default-engine = tales

[extractor:chameleon]
default-engine = tales

[extensions]
.jinja2 = babel-jinja2

.mako = babel-mako


Working with translations would boil down to:
* create language: ./i18n.sh lang
* everything else: ./i18n.sh

By default, neither babel nor lingua is a runtime dependency.

Cheers,
Pete

[Michael Merickel]

One more thing I should add since you mentioned jinja2 and you may not have seen this yet:

https://github.com/domenkozar/pyramid-cookiecutter-jinja2

The pyramid_jinja2 cookiecutter actually has some integrated i18n support that works. We have thrown around some ideas on merging that into the default starter scaffold [1], but I'm afraid that we don't yet have the "best practice" defined and would want it to be synchronized with whatever pyramid discusses in its docs [2].

[1] https://github.com/Pylons/pyramid_jinja2/issues/139

[2] http://docs.pylonsproject.org/projects/pyramid/en/latest/narr/i18n.html

Cheers,
Pete

--
You received this message because you are subscribed to the Google Groups "pylons-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-devel+unsubscribe@googlegroups.com.
To post to this group, send email to pylons...@googlegroups.com.
Visit this group at https://groups.google.com/group/pylons-devel.
For more options, visit https://groups.google.com/d/optout.

[Hans-Peter Jansen]

On Mittwoch, 8. Februar 2017 15:41:29 Michael Merickel wrote:
> One more thing I should add since you mentioned jinja2 and you may not have
> seen this yet:
>
> https://github.com/domenkozar/pyramid-cookiecutter-jinja2
>
> The pyramid_jinja2 cookiecutter actually has some integrated i18n support
> that works. We have thrown around some ideas on merging that into the
> default starter scaffold [1], but I'm afraid that we don't yet have the
> "best practice" defined and would want it to be synchronized with whatever
> pyramid discusses in its docs [2].
>
> [1] https://github.com/Pylons/pyramid_jinja2/issues/139
> [2] http://docs.pylonsproject.org/projects/pyramid/en/latest/narr/i18n.html

Thanks for the pointers. I've left a comment on the issue and invited Steve to
this thread.

There's one decision to make as the next step:

Does the involved parties agree to leave the translation procedures somewhat
outside the normal workflow and supply a i18n.sh script, accompanied with a
default lingua.cfg file, that checks and proposes to install "lingua" and
"babel", if missing?

"lingua" will be used for extraction, "babel" provides the extraction plugins
for "mako" and "jinja2" templates, that are used by lingua's pot-create
script.

Does anybody foresee any problems with this approach?

Cheers,
Pete

[Michael Merickel]

On Thu, Feb 9, 2017 at 6:02 AM, Hans-Peter Jansen <h...@urpla.net> wrote:

There's one decision to make as the next step:

Does the involved parties agree to leave the translation procedures somewhat
outside the normal workflow and supply a i18n.sh script, accompanied with a
default lingua.cfg file, that checks and proposes to install "lingua" and
"babel", if missing?

"lingua" will be used for extraction, "babel" provides the extraction plugins
for "mako" and "jinja2" templates, that are used by lingua's pot-create
script.

Does anybody foresee any problems with this approach?

I don't see any problem with this. It sounds great. A few requests below (not requirements, just my picture of what the i18n story would ideally turn into as we move forward).

1) Update the pyramid documentation to explain what's going on and how things fit together, including an example i18n.sh or i18n.py.

2) Try to support windows as much as possible and so i18n.py might be worth considering instead, along with an attempt to explain what is necessary to get things to work on windows.

3) I'd like things to work for arbitrary projects and not just colander and deform. It'd be great if they served as best practice examples.

- Michael