Some Questions about L20n

[Glen De Cauwsemaecker]

I am the (currently only :() developer of "L20n for games", trying to
bring l20n to games, as you guys bring l20n to the web. Games are
getting more social as well (especially Mobile F2P games), and thus
will require even more complex translations (if that wasn't already
the case).

Between February and May of this year I developed the C# L20n Library
and Unity plugin based on those specs. You can find the Unity Plugin
Repository here: https://github.com/GlenDC/L20n.unity (The L20n.cs
repository is included as a submodule). All of the current published
work is open source, free and gratis.

Looking back at it, and after several conversations with some of my
users, I learned that Mozilla is now pushing L20n, with the FTL format
(btw what does FTL stand for?). Therefore I am now working again on
L20n.cs to support L20n (FTL) and have moved all old code to the L10n
namespace. That work in progress can be followed at
https://github.com/GlenDC/L20n.cs

I mostly understand L20n (FTL) based on the grammar.ebnf file (found
in l20n/spec), l20n.org/learn and some other places (such as
l20n/l20n.js).

I do have a couple of unresolved questions, for which I couldn't find an answer:

1) L10n had the concept of private entities, where the identifier
started with an underscore `_`. Those could only be referenced from
within the same or another l10n file. I see that with L20n an
identifier can still start with an underscore `_`, but I don't see any
reference of a private entity anywhere, so am I correct to assume that
this has been removed as a feature in this new version?
2) I have seen on l20n.org/learn that sections are a way to group
entities (messages) together, but I have never seen an example how
this influences the translation calls. Do they work as namespaces,
such that you have to call `section.identifier` instead of simply
`identifier`?
3) What happened to globals? In L10n the library could provide globals
that were accessible similar to variables, but starting with `@`
instead of `$`. Examples of those were screen information, OS
information and so on. I used this to expose info given by Unity that
could be used by the localizer. Users of the L20n.Unity library could
also add their own globals custom to their games. Would this now have
to be done by using builtins?
4) My last question, I promise. Is there anywhere information
available about what types are allowed as external variables?

With my new push for support I'm trying to stay 100% compatible with
what you guys do. I also have a repository where I collect my
understanding as documentation, you can see it at
https://github.com/GlenDC/L20n.design/blob/master/design/l20n.md

I know this is a lot to ask for a first question, but I would be
really grateful if I could get some answers on these questions.

cheers
Glen

[Axel Hecht]

Am 24/11/16 um 01:57 schrieb Glen De Cauwsemaecker:

This still works, it's just not an explicit concept for now. We might
add that back if it turns out to be important in practice. Much of what
we've done is trying to make things simpler, and this was one of those
things.

You can have entities that are only within a particular localization,
but you'd need a way to tool that. Tooling is one of the things that's
much easier if you just need to deal with 1-1 corresponding entries in
en-US and each locale.

> 2) I have seen on l20n.org/learn that sections are a way to group
> entities (messages) together, but I have never seen an example how
> this influences the translation calls. Do they work as namespaces,
> such that you have to call `section.identifier` instead of simply
> `identifier`?

Sections only affect which comments are in scope for which entity. If
they exist or not doesn't play a role for the developer or for how you
reference the entities.

> 3) What happened to globals? In L10n the library could provide globals
> that were accessible similar to variables, but starting with `@`
> instead of `$`. Examples of those were screen information, OS
> information and so on. I used this to expose info given by Unity that
> could be used by the localizer. Users of the L20n.Unity library could
> also add their own globals custom to their games. Would this now have
> to be done by using builtins?

Yes, builtins are the new globals. Convention is that they're ALLCAPS.

In our implementation, we pass them as additional information from the
platform binding to the MessageContext objects. See
https://github.com/l20n/l20n.js/blob/master/src/runtime/gecko/l20n.js#L37
for an example.

> 4) My last question, I promise. Is there anywhere information
> available about what types are allowed as external variables?

We currently start with JSON-ish, but we're thinking about more.

https://github.com/l20n/l20n.js/blob/master/src/intl/types.js is the
start, with String, Number, DateTime, List (of the other).

We think that it'll make sense to have Number-with-formatting-annotation
later, too. So you'd pass in that this is a currency, Dollars, two
significant digits. Only some of that would be allowed to be overwritten
by the localizer. In particular, the currency wouldn't :-).

We're building on top of ICU/CLDR for many of the formatting operations
now, instead of having localizers write macros to do that.

>
> With my new push for support I'm trying to stay 100% compatible with
> what you guys do. I also have a repository where I collect my
> understanding as documentation, you can see it at
> https://github.com/GlenDC/L20n.design/blob/master/design/l20n.md

We did remove PLURAL, as it's the same as NUMBER. That's required so
that you can match explicit numbers in the keys.

> I know this is a lot to ask for a first question, but I would be
> really grateful if I could get some answers on these questions.

Happy to see your interest, thanks for the support.

Axel
>
> cheers
> Glen
>

[Glen De Cauwsemaecker]

> Sections only affect which comments are in scope for which entity. If they exist or not doesn't play a role for the developer or for how you reference the entities.

Why are comments actually stored in the final Object Tree? When I was
working on L10n (C#/Unity), I simply omitted them after parsing. What
use case do they have for the user of the L20n Libraries?

> We're building on top of ICU/CLDR for many of the formatting operations now, instead of having localizers write macros to do that.

Could you collaborate on this? So that I can make sure I provide all
the formatters that you guys will support by default?
Also Builtins are both formatters and utility functions, correct? Or
do you call stuff like `DROP` and `TAKE` also formatters?

> This still works, it's just not an explicit concept for now. We might add that back if it turns out to be important in practice. Much of what we've done is trying to make things simpler, and this was one of those things.

I like that, for now I'll also ignore it, keeping it simple sounds
good. L10n was really heavy on some parts.

> We did remove PLURAL, as it's the same as NUMBER. That's required so that you can match explicit numbers in the keys

Does that mean that when a NUMBER is used as a SELECTOR it is now not
anymore explicitly converted to `other`, `one`, `two`, etc? Or does it
just mean that PLURAL is merged into NUMBER?

Thanks for answering the questions so fast! :)

[Axel Hecht]

Am 24/11/16 um 18:02 schrieb Glen De Cauwsemaecker:

>> Sections only affect which comments are in scope for which entity. If they exist or not doesn't play a role for the developer or for how you reference the entities.
>
> Why are comments actually stored in the final Object Tree? When I was
> working on L10n (C#/Unity), I simply omitted them after parsing. What
> use case do they have for the user of the L20n Libraries?

There are two use-cases for l20n libraries. Applications localized with
l20n, and tooling to help localization infrastructure. Editors, linters,
dashboards and such.

For the applications, you only need the entries, whereas tools want the
full AST.

For JS, we have both an entries parser and an ast parser. For python, we
only have an AST parser so far.

>> We're building on top of ICU/CLDR for many of the formatting operations now, instead of having localizers write macros to do that.
>
> Could you collaborate on this? So that I can make sure I provide all
> the formatters that you guys will support by default?
> Also Builtins are both formatters and utility functions, correct? Or
> do you call stuff like `DROP` and `TAKE` also formatters?

https://github.com/l20n/l20n.js/blob/master/docs/hacking.rst is probably
a useful pointer at how we're doing parameters. This isn't really solid
yet, as we're not using any of those features in practice.

Most uses of our builtins are very similar to filters in other
templating languages. So there's no clear constraint of input and
output, but in general, the output of one should make sense as the input
of another.

From an implementation-point-of-view, I'd suggest to annotate the
original data in each builtin, and only at the point where you convert a
placable to a string to be used as an individual text fragment, you take
all those annotations into account.

Related, if your platform supports BIDI separation markers, this is also
the point where adding those makes sense. So that you can do your best
for the BIDI algorithm if you mix up Arabic and English multi-word data
in a list passed in to a placable in l20n.

>> This still works, it's just not an explicit concept for now. We might add that back if it turns out to be important in practice. Much of what we've done is trying to make things simpler, and this was one of those things.
>
> I like that, for now I'll also ignore it, keeping it simple sounds
> good. L10n was really heavy on some parts.
>
>> We did remove PLURAL, as it's the same as NUMBER. That's required so that you can match explicit numbers in the keys
>
> Does that mean that when a NUMBER is used as a SELECTOR it is now not
> anymore explicitly converted to `other`, `one`, `two`, etc? Or does it
> just mean that PLURAL is merged into NUMBER?

Right. The selector code matches a number to a number, and if that
fails, it also matches the pluralform of the number to the literal. See
the selectors section in
https://github.com/l20n/l20n.js/blob/master/docs/syntax.rst.

> Thanks for answering the questions so fast! :)
>

YW.

Axel

[Axel Hecht]

Am 24/11/16 um 18:02 schrieb Glen De Cauwsemaecker:

>> Sections only affect which comments are in scope for which entity. If they exist or not doesn't play a role for the developer or for how you reference the entities.
>
> Why are comments actually stored in the final Object Tree? When I was
> working on L10n (C#/Unity), I simply omitted them after parsing. What
> use case do they have for the user of the L20n Libraries?

There are two use-cases for l20n libraries. Applications localized with
l20n, and tooling to help localization infrastructure. Editors, linters,
dashboards and such.

For the applications, you only need the entries, whereas tools want the
full AST.

For JS, we have both an entries parser and an ast parser. For python, we
only have an AST parser so far.

>> We're building on top of ICU/CLDR for many of the formatting operations now, instead of having localizers write macros to do that.
>
> Could you collaborate on this? So that I can make sure I provide all
> the formatters that you guys will support by default?
> Also Builtins are both formatters and utility functions, correct? Or
> do you call stuff like `DROP` and `TAKE` also formatters?

https://github.com/l20n/l20n.js/blob/master/docs/hacking.rst is probably
a useful pointer at how we're doing parameters. This isn't really solid
yet, as we're not using any of those features in practice.

Most uses of our builtins are very similar to filters in other
templating languages. So there's no clear constraint of input and
output, but in general, the output of one should make sense as the input
of another.

From an implementation-point-of-view, I'd suggest to annotate the
original data in each builtin, and only at the point where you convert a
placable to a string to be used as an individual text fragment, you take
all those annotations into account.

Related, if your platform supports BIDI separation markers, this is also
the point where adding those makes sense. So that you can do your best
for the BIDI algorithm if you mix up Arabic and English multi-word data
in a list passed in to a placable in l20n.

>> This still works, it's just not an explicit concept for now. We might add that back if it turns out to be important in practice. Much of what we've done is trying to make things simpler, and this was one of those things.
>
> I like that, for now I'll also ignore it, keeping it simple sounds
> good. L10n was really heavy on some parts.
>
>> We did remove PLURAL, as it's the same as NUMBER. That's required so that you can match explicit numbers in the keys
>
> Does that mean that when a NUMBER is used as a SELECTOR it is now not
> anymore explicitly converted to `other`, `one`, `two`, etc? Or does it
> just mean that PLURAL is merged into NUMBER?

Right. The selector code matches a number to a number, and if that
fails, it also matches the pluralform of the number to the literal. See
the selectors section in
https://github.com/l20n/l20n.js/blob/master/docs/syntax.rst.

> Thanks for answering the questions so fast! :)
>

YW.

Axel