Bi-annual review of the FTL syntax

[zbran...@mozilla.com]

When we closed the design phase of the FTL syntax for L20n, we decided to force ourselves to work with the new syntax for half a year, collect feedback from adopters and other stakeholders like tool authors, and only then come back to review the syntax.

It's been over 6 months since and we (mostly :stas, :matjaz and :gandalf) accumulated a number of items to review that are worth considering.
Most of the feedback comes from user testing, implementation woes, feedback from potential users and conversations with experts.

My personal opinion is that for the 6 months of working with the syntax, the number and severity of items is low, but I believe it's a good time to consider making any changes resulting form them rather now than later.

====================================

1) Suboptimal selector and trait reference resolution when no default index is available

Currently, our grammar allows but doesn't enforce default value for traits and variants.

In the process of developing runtime resolution, we identified that lack of that annotation significantly reduces how resilient our resolver can be.

Without it, the entity is not "resolvable" within itself - if external arguments or other entities are not resolved, there is no way to indicate which variant should be selected and the fallback stringification of the selector is limited.

For traits, lack of default trait means that if the entity has no value, and there is no default trait, and caller selects trait that is not available, then the reference expression cannot be resolved.

Beyond error recovery, this impacts environments that would benefit from ability to display whole or parts of the entity without surrounding context available.
Example of such environment are localization tools. Having a default variant would allow for an easy display of a "folded" selector, and availability of the default trait would allow for an easy display of a string representation of the entity value when value itself is not present.

2) Forward-compatible functions

During concaptualization of how to make builtin functions forward compatible, we identified that we cannot ensure that the contract between the provider of the function and the localizer using it will be preserved.
One can imagine a builtin function like `PLATFORM()`, `OS()` or `SCREEN_WIDTH()`, that will change its signature over time.
Same can be said for the environment in which the entity is being used. At the time of creation of the entity, maybe just 2 platforms are considered, but later the project target platforms may change.

In such cases, additional bit of information allowing for better fallbacking in case of such incompatibilities would be of high value to the quality of the output.

3) Confusing similarity of traits and variants

While variants are not a completely new feature in l20n - they exist in a similar form in MessageFormat, traits are, as far as we know, unprecedented localization entity features.
That means that most users are unfamiliar with the concept and how to use it when they encounter it, and the feedback we collected indicates that the syntactic similarity of the representation adds to that confusion and hides semantic differences.

An example of such an entity would be:

brandName =
[nominative] Firefox
[possesive] Firefoksa

welcome = { $num ->
[one] Witaj
[other] Witajcie
}

There can be more extreme (and less realistic) examples created where both trait and variants are next to each other visually overlapping and differentiated only by indentation and syntax in surrounding lines of code.

We believe it would be worth revisiting the syntax around them and consider changing one or the other to indicate the difference.

4) Traits are a syntactic cluster of semantically different concepts

Explaining traits and reasoning about them has been found to be particularly challenging. The feature feels very open-ended, ambiguous and seems to be of a "catch-all" nature.

Over last 6 months (and previous work with FxOS and Vaani), we really encountered three classes of linguistic concepts that are served by traits:

a) Multi-value translations

For example a widget with a label, ok and cancel button may be localized with a single entity with a value and two traits:

install-widget = Do you want to install { brandName }?
[ui/ok] OK
[ui/cancel] Cancel

b) Multi-variant values

When the entity value has multiple potential variants:

brandName =
[nominative] Firefox
[possesive] Firefoksa

or:

welcome =
[written] Welcome!
[spoken] Welcome to { brandName }


c) Meta-information

When we want to encode additional information on the entity that will be useful for further translation:

brandName = Firefox
[gender] masculine

froze = { brandName[gender] ->
[masculine] { brandName } zawiesil sie
[feminine] { brandName } zawiesila sie
}

Those three use cases are of vastly different nature and clustering them into a single syntactic structure seems to be a workaround rather than a solution, making each three work "well-enough", but none of them working really intuitively for the reader.

On top of that, it creates a possibility to end up with a really confusing entity:

key =
[gender] masculine
*[nominative] Firefox
[dom/tooltip] Nazwa programu


This becomes especially visible while working on GUI localization tool that is supposed to handle such cases and aid localizer in making right decisions.

The multi-variant value case would benefit from the default index, but of course gender trait nor dom/tooltip traits should have it. The visual representation of this is at best confusing.

5) We don't seem to have a way to represent a combination of L20n and Intl API using `formatToParts` feature.

Although this did not come up in our prototyping yet, I believe that it is the last moment to try to make space for such use case in our syntax.

The common case would be an attempt to create a localization phrase with a unit or date formatted and styled.

Below are three examples with ftl/js solutions *without* the formatToParts part.

a) "You have to pick up your package by December <b>2nd</b> 2016"

ftl:
key = "You have to pick up your package by { $date }"

js:
let date = new Date();
ctx.formatValue('key', {
date: FTL.DateTime(date, {month: 'long', day: 'short', year: 'numeric'})
});

b) "The fastest route will get you to your destination in <b>30</b> minutes"

ftl:
key = "The fastest route will get you to your destination in { $time }"

js:
ctx.formatValue('key', {
time: FTL.Unit(30, {unit: 'minute', style: 'long'})
});

c) "This email has been sent to you 2 <b>hours</b> ago"

ftl:
key = "This email has been sent to you { $relative_time }"

js:
ctx.formatValue('key', {
time: FTL.RelativeTime(-2, {unit: 'hour', style: 'long'})
});

The question is how can we incorporate formatToParts approach to solve them *with* formatting.

===================================

My experience with UX is that the first study is the most significant one, since it changes the sample tested from 0 to non-zero and that's a 100% change.

>From there, each further sample is giving diminishing returns, so I expect that the number of items we'll collect over the next 6 months will be lower.

This thread is for discussing the validity of the problem statements and their severity, rather than proposing solutions. I'd like to ask everyone to kick of new threads for each solution proposal, so that we can discuss each separately.

zb.

[zbran...@mozilla.com]

6) Nested selectors are hard to read/edit/write

ICU recommends [0] to write full sentences instead of chunking messages to minimize redundancy.
In other words, instead of:

key = { $num ->
[one] She likes
[other] They like
} when { $gender ->
[masculine] he
[feminine] she
} dances

people should write:

key = { $num ->
[one] { $gender ->
[masculine] She likes when he dances
[feminine] She likes when she dances
}
[other] { $gender ->
[masculine] They like when he dances
[feminine] They like when she dances
}
}

This kind of approach to complex selectors is recommended so we should look for ways to optimize for it.

zb.

[0] http://userguide.icu-project.org/formatparse/messages

[Staś Małolepszy]

Hey Zibi, thanks for taking the time to write all of this down! I
replied in the topic-specific threads with more details. Here I'll
comment about the two other topics.

On Fri, Dec 16, 2016 at 9:47 AM, <zbran...@mozilla.com> wrote:

> 2) Forward-compatible functions
>
> During concaptualization of how to make builtin functions forward compatible, we identified that we cannot ensure that the contract between the provider of the function and the localizer using it will be preserved.
> One can imagine a builtin function like `PLATFORM()`, `OS()` or `SCREEN_WIDTH()`, that will change its signature over time.
> Same can be said for the environment in which the entity is being used. At the time of creation of the entity, maybe just 2 platforms are considered, but later the project target platforms may change.
>
> In such cases, additional bit of information allowing for better fallbacking in case of such incompatibilities would be of high value to the quality of the output.

Can you specify what you mean by "better fallback" here? To my
understanding, if a BUILTIN starts with 'foo' and 'bar' as possible
values and we enforce a default value on variants, then the variants
of the translation are exhaustive. Later, if a new value, 'baz',
becomes available, the current translation still is exhaustive (with
'baz' resulting in the default variant). The localizer can improve
the localization, but no extra fallback is needed.

> 5) We don't seem to have a way to represent a combination of L20n and Intl API using `formatToParts` feature.
>
> Although this did not come up in our prototyping yet, I believe that it is the last moment to try to make space for such use case in our syntax.

Do you think this should be solved in the syntax? The point of
solving it in the syntax is to allow the localizer to control the
formatting of each part. Instead, would it be enough to allow the
developer to do it? It does seem like something related to the
implementation of the UI.

> a) "You have to pick up your package by December <b>2nd</b> 2016"
>
> ftl:
> key = "You have to pick up your package by { $date }"
>
> js:
> let date = new Date();
> ctx.formatValue('key', {
> date: FTL.DateTime(date, {month: 'long', day: 'short', year: 'numeric'})
> });

formatToParts was designed with map() and reduce() in mind:

dtf.formatToParts(date).map(({type, value}) => {
switch (type) {
case 'day': return `<b>${value}</b>`;
default : return value;
}
}).reduce((string, part) => string + part);

FTL arguments like FTL.DateTime could be instrumented to use
formatToParts() instead of format() and would then require two
functions as additional arguments. Here's an idea of what this could
look like in JS:

let date = new Date();
ctx.formatValue('key', {
date: FTL.DateTime(date, {month: 'long', day: 'short', year:

'numeric'}).toParts(mapper, reducer)
});

or maybe:

FTL.DateTimeParts(date, {month: 'long', day: 'short', year:
'numeric'}, mapper, reducer)

I don't have an opinion on either of these options yet, but the
important part is -- is this enough? Or do we need to allow
localizers to format individual parts somehow? I don't think we do,
but I might be missing some use-cases.

-stas

[Axel Hecht]

Am 16/12/16 um 09:47 schrieb zbran...@mozilla.com:

> When we closed the design phase of the FTL syntax for L20n, we decided to force ourselves to work with the new syntax for half a year, collect feedback from adopters and other stakeholders like tool authors, and only then come back to review the syntax.
>
> It's been over 6 months since and we (mostly :stas, :matjaz and :gandalf) accumulated a number of items to review that are worth considering.

I'd like to respond to the nomenclature here.

I'm looking at the suite of threads and the "bi-annual review of the ftl
syntax" name for it, from the POV of a tool author or programm owner.

I'd turn around and leave, and look for an l10n tech that values my
investments for more than 6 months.

If we intend other people to put mindshare, time, and money into l20n,
we cannot have something like a bi-annual syntax review.

Axel

[Staś Małolepszy]

I'll respond for Zibi, because I feel like I know the answer to this
too. I think Zibi's use of "biannual" wasn't intentional, as in,
there isn't a strict process of giving our syntax a do-over twice a
year! When I talked to him about it, I meant the fact that FTL (the
syntax) isn't stable yet. We want to converge and stabilize, and make
adjustments where needed, and giving ourselves a timeline for
gathering feedback helps inform our decisions.

I recall that we did settle some of our syntax discussions about FTL
with "let's go with this for now and revisit in half a year after
we've got used to it and gathered some feedback." It's been a bit over
half a year and the time is ripe for taking a step back and
readjusting the design to the feedback we've seen. I talked to
localizers at two hackathons about it, wrote the menubar patch and
wrote the migration script using the FTL AST. We have an effort
underway to design a UI for authoring translations using FTL in two
GUI tools and got feedback there as well from Matjaz and Ryan. Flod
has been thinking about adding FTL to Transvision. Zibi has been
pushing the ECMA402 agenda and talked to Unicode about the future of
MessageFormat.

We have a lot of feedback to digest and having these discussions right
now is great. That's what we planned and that's what we should do if
we see shortcomings in the current syntax. Which I and Zibi do.

I'll admit that we're bad at versioning. FTL should be versioned
separately from L20n and still be a 0.x. It would make it clearer
from consumers' perspective that we're still working on it and that
they have an impact of what it ends up looking like! Perhaps that's
something we should talk about in another thread?

HTH,
-stas

On Fri, Dec 16, 2016 at 4:33 PM, Axel Hecht <l1...@mozilla.com> wrote:
> Am 16/12/16 um 09:47 schrieb zbran...@mozilla.com:
>>

>> When we closed the design phase of the FTL syntax for L20n, we decided to
>> force ourselves to work with the new syntax for half a year, collect
>> feedback from adopters and other stakeholders like tool authors, and only
>> then come back to review the syntax.
>>
>> It's been over 6 months since and we (mostly :stas, :matjaz and :gandalf)
>> accumulated a number of items to review that are worth considering.
>
>
>

> I'd like to respond to the nomenclature here.
>
> I'm looking at the suite of threads and the "bi-annual review of the ftl
> syntax" name for it, from the POV of a tool author or programm owner.
>
> I'd turn around and leave, and look for an l10n tech that values my
> investments for more than 6 months.
>
> If we intend other people to put mindshare, time, and money into l20n, we
> cannot have something like a bi-annual syntax review.
>
> Axel
>

> _______________________________________________
> tools-l10n mailing list
> tools...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/tools-l10n

[zbran...@mozilla.com]

Yeah, sorry for confusion Axel. I don't think I chose the title properly.

Back when we came up with FTL we were worried that we'll end up constantly changing it almost every week with new tweaks, so we negotiated the half-a-year embargo to give ourselves time to get settled in the new syntax and gather feedback.

My goal is by no mean to review and change the syntax every 6 months.

If I understand your response in this thread correctly, you're not discussing the nature of the problems we raised, but rather you're asking if we should be making any changes to FTL at all at this stage.

I believe it to be a question about our release strategy and as such, worth a separate thread to plan the release.

In this thread I'd like to focus exclusively on the question whether what I listed above are problems that we should be tackling.

>From what I collected so far, :stas, :matjaz and me believe they are which imho warrants the conversation to be had.

zb.

[zbran...@mozilla.com]

7) Every use of a string can be a complex one

At the moment, our syntax does have a concept of a "simple string", which means that every string can have placeables in it.

For example:

key2 = inal

key = { PLURAL($num, type: "card{ key2 }") ->
[one] One
[other] Other
}

will work properly.

I believe it's worth simplifying it and at the very least introducing simple string as a literal that can be used for argument values.

zb.

[Staś Małolepszy]

On Sat, Dec 17, 2016 at 11:01 AM, <zbran...@mozilla.com> wrote:

> I believe it's worth simplifying it and at the very least introducing simple string as a literal that can be used for argument values.

Is the fact that "every string can be complex" the problem that you'd
like to solve? From your description I can't really tell what makes
it a problem. Would you mind elaborating?

[zbran...@mozilla.com]

Thanks for asking,

So, I believe that the fact that we allow for function argument values to be complex strings is an unnecessary vector for complicated parser/resolver code and is not something FTL users would expect.

Does it answer your question?
zb.

[Staś Małolepszy]

On Sat, Dec 17, 2016 at 9:06 PM, <zbran...@mozilla.com> wrote:

> So, I believe that the fact that we allow for function argument values to be complex strings is an unnecessary vector for complicated parser/resolver code and is not something FTL users would expect.
>
> Does it answer your question?

To some extent :) It tells me that we haven't seen enough usage of
this feature yet to really make good decisions. From my point of view,
I'd say it would be surprising if strings marked with "" could or
could not have placeables in them depending on where they are used.
But I'm aware that this might just be my inclination towards
consistency and not something users of FTL would have an issue with.

Regarding your other point about complicated parser/resolver code, I
would think that the current design which simply uses the existing
TextElemenet syntax with all its semantics is already very lean
implementation-wise.

I agree that it's rather useless to allow for complex strings in
function arguments. I don't know right now if it's better to simplify
by removing this feature (and possibly introducing new syntax) or to
keep it consistent with the current semantics of "". Perhaps there is
a middle-ground to be found in that we don't change the syntax but
instead impair our implementation (via a ParseError), so that we can
decide later and not have to deal with existing translations already
using the feature.

Also, we already have a symbol which has a value and doesn't allow
interpolations: keywords. We could consider the following as an
alternative: NUMBER($num, style: [percent]).

[zbran...@mozilla.com]

On Monday, December 19, 2016 at 8:51:59 AM UTC-8, Staś Małolepszy wrote:
> Also, we already have a symbol which has a value and doesn't allow
> interpolations: keywords. We could consider the following as an
> alternative: NUMBER($num, style: [percent]).

Good point. My worry here is that it does look awkward from the first persona that I use to validate syntax choices against (someone who sees FTL for the first time and has to read it).

`style: "short"` seems the natural choice unless we're ok with limiting to single words in which case `style: short` could also work.

I think I like "short" here tbh, but we can also just run a small study on localizers and see how they react.

Additional rationale for introducing some sort of SimpleText node that doesn't have placeholders is that it could allow for fastpaths in both parser and resolver and would avoid a PITA for GUI tool authors.

Cheers,
zb.