Handling plural forms in moz2po and po2moz

[Khaled Hosny]

Hi all,

I've been looking into adding support for handling Mozilla's plural
forms in translate toolkit conversion scripts so that they are properly
handled by PO editors (less error prone etc.).

The problem is that there is no systematic way to tell when a semi colon
in a string is plural delimiter or not; comments are not formalized, and
several consecutive strings will get only one comment (I can imagine
this being an issue when working with PO files right now, for example if
the translator is starting after the first string, for any reason, he
might not see the comment).

How should I proceed with this, should we agree on a magic word to be
required in the comments of plural forms and start submitting patches,
or will the new style plural forms adopted soon enough to make such an
effort not worth it?

Regards,
Khaled

[flod]

Il 01/05/13 18:13, Khaled Hosny ha scritto:

> The problem is that there is no systematic way to tell when a semi colon
> in a string is plural delimiter or not; comments are not formalized, and
> several consecutive strings will get only one comment (I can imagine
> this being an issue when working with PO files right now, for example if
> the translator is starting after the first string, for any reason, he
> might not see the comment).

I think every string with a plural form has a comment with a link like this
# See: http://developer.mozilla.org/en/docs/Localization_and_Plurals
<http://developer.mozilla.org/en/docs/Localization_and_Plurals>

Or at least I remember Axel commenting on this matter in a bug lately.
https://bugzilla.mozilla.org/show_bug.cgi?id=834721#c9

Francesco

[Khaled Hosny]

That was not the case for many of the files I checked, for example:

toolkit/chrome/mozapps/downloads/downloads.properties

has four strings with just one comment:

# LOCALIZATION NOTE (seconds, minutes, hours, days): Semi-colon list of plural
# forms. See: http://developer.mozilla.org/en/docs/Localization_and_Plurals
seconds=second;seconds
minutes=minute;minutes
hours=hour;hours
days=day;days

Regards,
Khaled

[Francesco Lodolo [:flod]]

Il 01/05/13 19:31, Khaled Hosny ha scritto:

> That was not the case for many of the files I checked, for example

You're absolutely right, I never noticed that.

To be honest I don't understand Python enough to understand how
compare-locales works here, but I think this is the part of the code
involved
http://hg.mozilla.org/l10n/compare-locales/file/55c0d0a185ae/lib/Mozilla/Checks.py#l55

Is it just searching for " Localization_and_Plurals" in order to avoid
warnings about missing/extra variables?

I definitely need to learn Python at some point... :-(

Francesco

[Khaled Hosny]

On Wed, May 01, 2013 at 07:46:24PM +0200, Francesco Lodolo [:flod] wrote:
> Il 01/05/13 19:31, Khaled Hosny ha scritto:
> >That was not the case for many of the files I checked, for example
> You're absolutely right, I never noticed that.
>
> To be honest I don't understand Python enough to understand how
> compare-locales works here, but I think this is the part of the code
> involved
> http://hg.mozilla.org/l10n/compare-locales/file/55c0d0a185ae/lib/Mozilla/Checks.py#l55
>
> Is it just searching for " Localization_and_Plurals" in order to
> avoid warnings about missing/extra variables?

Seems so. But it doesn't seem to be handling the case of multiple keys
with a single comment (unless the parser duplicates the comments
somehow).

Regards,
Khaled

[Axel Hecht]

Yes, compare-locales doesn't switch on the plurals check on any entity
but the first in the block.

I wholeheartly agree that the situation regarding plurals is less than
optimal.

It's, like many other l10n infrastructure pieces, put on top of
something that exists, in a "works today" fashion.

Another level of complexity is that the plural support within the
gettext world may differ from gecko, in terms of how many plural forms
are expected and how they're ordered. It's not something we'd do
lightly, but it's also not something where we'd, say, revert to gettext
behavior added after gecko support was added with a different form.

Axel

[Khaled Hosny]

On Wed, May 01, 2013 at 09:51:15PM +0200, Axel Hecht wrote:
> On 5/1/13 8:02 PM, Khaled Hosny wrote:
> >On Wed, May 01, 2013 at 07:46:24PM +0200, Francesco Lodolo [:flod] wrote:
> >>Il 01/05/13 19:31, Khaled Hosny ha scritto:
> >>>That was not the case for many of the files I checked, for example
> >>You're absolutely right, I never noticed that.
> >>
> >>To be honest I don't understand Python enough to understand how
> >>compare-locales works here, but I think this is the part of the code
> >>involved
> >>http://hg.mozilla.org/l10n/compare-locales/file/55c0d0a185ae/lib/Mozilla/Checks.py#l55
> >>
> >>Is it just searching for " Localization_and_Plurals" in order to
> >>avoid warnings about missing/extra variables?
> >
> >Seems so. But it doesn't seem to be handling the case of multiple keys
> >with a single comment (unless the parser duplicates the comments
> >somehow).
> >
> >Regards,
> >Khaled
> >
>
> Yes, compare-locales doesn't switch on the plurals check on any
> entity but the first in the block.
>
> I wholeheartly agree that the situation regarding plurals is less
> than optimal.
>
> It's, like many other l10n infrastructure pieces, put on top of
> something that exists, in a "works today" fashion.

So, do you suggest we try to address this by, say, added a comment with
the MDN link to each plural string?

> Another level of complexity is that the plural support within the
> gettext world may differ from gecko, in terms of how many plural
> forms are expected and how they're ordered. It's not something we'd
> do lightly, but it's also not something where we'd, say, revert to
> gettext behavior added after gecko support was added with a
> different form.

I'm aware of that because Arabic is one of the languages affected by
this. I was going to first try to convince you to revert to to gettext
formula, otherwise implement some sort of conversion table.

Regards,
Khaled

[Axel Hecht]

I don't think that makes the files more readable. Being better at
parsing localization notes and generally migrating the checks logic to
run across the file and not across single entities would help them a
great deal, on the compare-locales side, that is.

>> Another level of complexity is that the plural support within the
>> gettext world may differ from gecko, in terms of how many plural
>> forms are expected and how they're ordered. It's not something we'd
>> do lightly, but it's also not something where we'd, say, revert to
>> gettext behavior added after gecko support was added with a
>> different form.
>
> I'm aware of that because Arabic is one of the languages affected by
> this. I was going to first try to convince you to revert to to gettext
> formula, otherwise implement some sort of conversion table.

How do they differ?

Axel

[Khaled Hosny]

On translate toolkit I'm now doing the following:
* consider the unit immediately after a "Localization_and_Plurals" comment
* parse that comment for "LOCALIZATION NOTE (key_1, ... , key_n)" and
consider the units with those keys
* if any of those units have semicolon, then it is a plural unit.

This seems to catch the greater number of plural units with no false
positive, but manual checking should a few false negatives all of them
are missing "Localization_and_Plurals" in the comment:

browser/chrome/browser/browser.properties editBookmark.removeBookmarks.label
browser/chrome/browser/browser.properties popupWarning.message
browser/chrome/browser/preferences/aboutPermissions.properties cookiesCount
browser/chrome/browser/preferences/aboutPermissions.properties passwordsCount
browser/chrome/browser/preferences/aboutPermissions.properties visitCount
browser/chrome/browser/syncSetup.properties addonsCount.label
toolkit/chrome/mozapps/extensions/extensions.properties numReviews
toolkit/chrome/mozapps/extensions/extensions.properties showAllSearchResults

> >>Another level of complexity is that the plural support within the
> >>gettext world may differ from gecko, in terms of how many plural
> >>forms are expected and how they're ordered. It's not something we'd
> >>do lightly, but it's also not something where we'd, say, revert to
> >>gettext behavior added after gecko support was added with a
> >>different form.
> >
> >I'm aware of that because Arabic is one of the languages affected by
> >this. I was going to first try to convince you to revert to to gettext
> >formula, otherwise implement some sort of conversion table.
>
> How do they differ?

In Gettext zero case is the first, in Mozilla it is the last.

Regards,
Khaled

[Axel Hecht]

Mind filing bugs on these, and CC me? Gettig those fixed would be great.

>
>>>> Another level of complexity is that the plural support within the
>>>> gettext world may differ from gecko, in terms of how many plural
>>>> forms are expected and how they're ordered. It's not something we'd
>>>> do lightly, but it's also not something where we'd, say, revert to
>>>> gettext behavior added after gecko support was added with a
>>>> different form.
>>>
>>> I'm aware of that because Arabic is one of the languages affected by
>>> this. I was going to first try to convince you to revert to to gettext
>>> formula, otherwise implement some sort of conversion table.
>>
>> How do they differ?
>
> In Gettext zero case is the first, in Mozilla it is the last.

Yeah, that's unfortunate. The arabic localization is apparently the only
one that's using this plural form, so we could change that on our side,
if you want to.

It's a rather massive undertaking, though, as we need to coordinate the
landing in gecko with reordering all existing entries in the arabic
localization.

Axel

[Khaled Hosny]

I filed one bug for them all, is this OK?
https://bugzilla.mozilla.org/show_bug.cgi?id=869907

> >>>>Another level of complexity is that the plural support within the
> >>>>gettext world may differ from gecko, in terms of how many plural
> >>>>forms are expected and how they're ordered. It's not something we'd
> >>>>do lightly, but it's also not something where we'd, say, revert to
> >>>>gettext behavior added after gecko support was added with a
> >>>>different form.
> >>>
> >>>I'm aware of that because Arabic is one of the languages affected by
> >>>this. I was going to first try to convince you to revert to to gettext
> >>>formula, otherwise implement some sort of conversion table.
> >>
> >>How do they differ?
> >
> >In Gettext zero case is the first, in Mozilla it is the last.
>
> Yeah, that's unfortunate. The arabic localization is apparently the
> only one that's using this plural form, so we could change that on
> our side, if you want to.
>
> It's a rather massive undertaking, though, as we need to coordinate
> the landing in gecko with reordering all existing entries in the
> arabic localization.

It won't be the first time, we had a different formula earlier with 4
instead of 6 cases. Actually it seems that the deviation from gettext
was intentional, as shown here:
https://bugzilla.mozilla.org/show_bug.cgi?id=467347

Though I don't agree with the stated reasons, the benefit of being
compatible with gettext outweighs any backward compatibility concerns
(which I never understood, once the strings are fixing there is no
longer any backward compatibility to worry about). The less special
cases we have the less potential errors to worry about.

Regards,
Khaled

[Axel Hecht]

Yep, thanks.

Well, back in the day we had a 100% incremental change, which is an
order of magnitude simpler than what you're proposing.

I'd love to get more detail on how you want this to practically happen,
with landings on central, aurora, on l10n etc.

Axel

[Khaled Hosny]

May be I'm missing something, but it does not seem that complex to me.
The change would be done in central, probably short before aurora, is
branched. Once the new aurora is there I’ll update the translation to
use the new order, and it should propagate to beta as usual. If having
the correct forms is needed in central, it shouldn’t be had for me to
backport translation from aurora (we never updated Arabic translation on
central since the introduction of rapid release)

Regards,
Khaled

[Axel Hecht]

We should back-port to central, probably.

Anywho, I'd expect the build to be pretty frankenbuild as long as plural
forms don't match, do you have any idea how long the adjustment of the
'ar' localization would take? Also, we'll need to do that for all the
things in l10n, so not just Firefox, but also the fennec files and
thunderbird. calendar seems to be never started? Maybe we should drop
building that?

Axel