Updating l10n-merge in production, switching on l10n-merge in general, all of Firefox, TB, SeaMonkey, whatnot
Axel Hecht
the version of compare-locales/l10n-merge we use in release production
is rather dated, and we should update. We should also switch on
l10n-merge across the board on all builds. This is a bit of a can of
worms. I'll try to untangle that, but you know me.
Features of the new code:
- catches more errors (yet more in the works right now, thanks android)
- *fixes* more errors by removing bad strings from existing files and
replacing them with en-US
Boundary conditions:
- all release automations (moco, momo, seamonkey) refer to the same
version (tagged,
http://hg.mozilla.org/build/compare-locales/file/RELEASE_AUTOMATION)
- the new error code finds stuff in stable releases that we've been
shipping for a while with those errors
- fixing them all is a lot of communication effort (that we should focus
on fx4?)
The bad:
If we just bump the tag, previously "green" localizations go "red", and
the builds break.
If we switch on l10n-merge, it won't break, but it will change the code
we ship. Even for a chem-spill that doesn't take l10n-related changes.
A similar change would be changing the optimization flags in our
mozconfigs, even if no code changes would be taken, the shipped code
would differ.
The good:
The errors that the new version finds have been smoketested on the
dashboard for a while now, and it's finding errors for stuff where the
code really breaks, mostly [1].
It's the code that enables me to write a check that fixes the
multi-locale android builds (in the works).
We'd be using the same code generation path for nightlies and release
builds. Both easier releng-wise and sounder process-wise.
The risk:
The actual merge code is not generating builds anywhere right now. I'm
only smoke testing the error detection code. That is, once we bump, we
should make some serious QA efforts.
The non-goal:
This thread is about changing the build process, and simplifying it.
It's not about changing the release criteria, i.e., if we'd ship Firefox
4 locales with missing strings. That's a separate discussion we can (and
probably should) have after this one.
l10n-drivers would like to do this, any objections? Questions? I'll also
take questions at the product planning meeting today.
Axel
[1] There's one exception to the rule, the xml error message string
changed from signed to unsinged line numbers and columns. This is an
error in theory and code, though usually the builds are just fine.
compare-locales does report it as an error, just because it's consistent
with what we do elsewhere.
Christian Legnitto
* We don't use l10n-merge on branches, correct? Even if the code changes it isn't used by RelEng, right?
* How would failure show? Unable to build/repack? Incorrect strings in localized builds? Something more subtle?
* How easy to rollback if needed?
* How long will verification that the changes worked take?
Thanks,
Christian
> _______________________________________________
> dev-planning mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-planning
Axel Hecht
> I have a couple of questions:
>
> * We don't use l10n-merge on branches, correct? Even if the code changes it isn't used by RelEng, right?
That's the subject to change here, really. We should switch l10n-merge
on, for all products on all branches, aka, fx, fennec, tb, seamonkey,
down to 1.9.1.
The other part of this change is to change the actual merge code, too,
which is part of compare-locales.
> * How would failure show? Unable to build/repack? Incorrect strings in localized builds? Something more subtle?
I can only phrase that in terms of build steps, which isn't really what
all release factories do right now.
compare-locales is a step, which fails if not switching on l10n-merge.
The localizations we ship have errors that are undetected by the current
RELEASE_PRODUCTION version of compare-locales, and thus is shipping.
Updating compare-locales will thus turn the compare-locales steps for
these locales on those stable branches red, and the repack builds will
abort at that point, i.e., not generate binary bits.
Once we switch on l10n-merge, two things happen:
- the build actually continues to create binary bits
- the strings with errors get removed by the compare-locales code and
replaced with working en-US strings.
> * How easy to rollback if needed?
The rollback is as easy as move the tag on the compare-locales hg repo
back to where it was.
The switch to have l10n-merge on everywhere across the table doesn't
make any difference as long as the version of compare-locales used
doesn't find any errors or missing strings. (Caveat, under the
assumption that the mergedir is properly clobbered.)
> * How long will verification that the changes worked take?
>
That verification comes is several depths: One would be quick, and can
be done on nightlies. The output of compare-locales that we have on the
dashboard gives pretty focused information on which functionality would
break.
Example: Verify on a current 3.6 welsh (cy) nightly that all the errors
showing up on
https://l10n-stage-sj.mozilla.org/dashboard/compare?run=112484 actually
bust the functionality. Either bump the tag for a nightly, or do some
one-off builds with a new version of compare-locales and l10n-merge, and
verify that the functionality works again, plus, that nothing in the
area regressed, which could happen from removing bad stuff.
Now, as we're dealing with a few hundred bugs on stable releases here,
verifying each and every one independently is much more work. That's
toughened by the fact that a bunch of those are actually down in pipnss
error messages and such, so one would actually need to know how to even
trigger those.
We're also doing some good 1.5k or something builds, so in the end,
we're fuzz testing the infra code.
On the good side, this code isn't really rocket science either. You find
a bad string, you remove it, you paste a good string to the end. 'nuff said.
The error detection itself has been tested for a good while now.
Mike Shaver
> That's the subject to change here, really. We should switch l10n-merge on,
> for all products on all branches, aka, fx, fennec, tb, seamonkey, down to
> 1.9.1.
>
> The other part of this change is to change the actual merge code, too, which
> is part of compare-locales.
Naively, it seems like these would be changes that we should decouple:
make the change to the code, and then roll it out incrementally across
branches, perhaps?
Mike
Axel Hecht
So, yes, we can somewhat decouple this:
First step would be to make sure that all our builds on all branches and
orgs have l10n-merge set on. For some that's a configuration change, not
sure about others.
The next step would be to update the version of compare-locales used.
The first step shouldn't affect release bits, as they're not finding
anything to merge with the old version of compare-locales anyway. Beta 8
on the other hand showed that there's risk in any change.
We can't roll out the update to compare-locale branch by branch. That's
one single monolithic step, as all the factories all refer to the same
tag in the compare-locales repo.
Axel
Robert Kaiser
> That's the subject to change here, really. We should switch l10n-merge
> on, for all products on all branches, aka, fx, fennec, tb, seamonkey,
> down to 1.9.1.
I disagree here. We never should ship potentially incomplete L10n in
stable releases unless we make that an explicit policy (but then I
wonder why we need our complicate L10n signoff stuff anyhow).
Up to now our strict policy was to not ship any incomplete L10n as
official stable releases, you would just turn that over. We need to
think about that carefully before doing such a thing.
Robert Kaiser
--
Note that any statements of mine - no matter how passionate - are never
meant to be offensive but very often as food for thought or possible
arguments that we as a community needs answers to. And most of the time,
I even appreciate irony and fun! :)
Axel Hecht
> Axel Hecht schrieb:
>> That's the subject to change here, really. We should switch l10n-merge
>> on, for all products on all branches, aka, fx, fennec, tb, seamonkey,
>> down to 1.9.1.
>
> I disagree here. We never should ship potentially incomplete L10n in
> stable releases unless we make that an explicit policy (but then I
> wonder why we need our complicate L10n signoff stuff anyhow).
> Up to now our strict policy was to not ship any incomplete L10n as
> official stable releases, you would just turn that over. We need to
> think about that carefully before doing such a thing.
>
Switching on l10n-merge doesn't imply taking signoffs on missing
strings, or not cancelling those from beta that remain to have some. To
quote my initial post:
> The non-goal:
> This thread is about changing the build process, and simplifying it.
> It's not about changing the release criteria, i.e., if we'd ship Firefox
> 4 locales with missing strings. That's a separate discussion we can (and
> probably should) have after this one.
de-facto, we're shipping "red" builds, and that's not dramatically
improving on our stable branches.
Ftr, my policy for signoffs on stable branches is to not take
regressions, but I don't reject signoffs on fx 3.5 or 3.6 because they
still have the errors that we're shipping with for months.
Axel
Robert Kaiser
> Switching on l10n-merge doesn't imply taking signoffs on missing
> strings
It implies that we can ship builds on stable branches that miss some
strings, and that's what I don't think is a good idea without a thorough
discussion. I'm not a friend of doing that at this time.
Axel Hecht
> Axel Hecht schrieb:
>> Switching on l10n-merge doesn't imply taking signoffs on missing
>> strings
>
> It implies that we can ship builds on stable branches that miss some
> strings, and that's what I don't think is a good idea without a thorough
> discussion. I'm not a friend of doing that at this time.
Not true. It implies that we're using l10n-merge to fix up broken
strings which we've shipped for a while. Broken in the sense of anything
between "kinda nothing" like the xml error signed vs unsigned to xml
parsing errors to broken functionality to crashers (using a %s instead
of %u).
Axel
Christian Legnitto
For a stable branch I'd be worried about taking these changes wholesale. It's an all-or-nothing approach, correct? My general attitude for branches is if there is not a bug on file from users or developers it isn't an issue we should fix (even if it is technically broken).
Christian
Axel Hecht
ship with different versions of our toolchain on different branches.
Axel
Robert Kaiser
Erm, we still ship an untranslated string instead of the erroneous
string, right? (That's what I meant with "miss some strings", I probably
should have added "in the correct locale" or something like that.)
Axel Hecht
> Axel Hecht schrieb:
>> On 07.01.11 20:50, Robert Kaiser wrote:
>>> Axel Hecht schrieb:
>>>> Switching on l10n-merge doesn't imply taking signoffs on missing
>>>> strings
>>>
>>> It implies that we can ship builds on stable branches that miss some
>>> strings, and that's what I don't think is a good idea without a thorough
>>> discussion. I'm not a friend of doing that at this time.
>>
>> Not true. It implies that we're using l10n-merge to fix up broken
>> strings which we've shipped for a while. Broken in the sense of anything
>> between "kinda nothing" like the xml error signed vs unsigned to xml
>> parsing errors to broken functionality to crashers (using a %s instead
>> of %u).
>
> Erm, we still ship an untranslated string instead of the erroneous
> string, right? (That's what I meant with "miss some strings", I probably
> should have added "in the correct locale" or something like that.)
>
Well, what are the alternatives? At this point, we're just pretending
that we wouldn't know that things are broken on the branches.
Axel
Axel Hecht
> Axel Hecht schrieb:
>> On 07.01.11 20:50, Robert Kaiser wrote:
>>> Axel Hecht schrieb:
>>>> Switching on l10n-merge doesn't imply taking signoffs on missing
>>>> strings
>>>
>>> It implies that we can ship builds on stable branches that miss some
>>> strings, and that's what I don't think is a good idea without a thorough
>>> discussion. I'm not a friend of doing that at this time.
>>
>> Not true. It implies that we're using l10n-merge to fix up broken
>> strings which we've shipped for a while. Broken in the sense of anything
>> between "kinda nothing" like the xml error signed vs unsigned to xml
>> parsing errors to broken functionality to crashers (using a %s instead
>> of %u).
>
> Erm, we still ship an untranslated string instead of the erroneous
> string, right? (That's what I meant with "miss some strings", I probably
> should have added "in the correct locale" or something like that.)
Robert and I chatted about this irl.
Robert, mind updating us here if you still have concerns?
Axel
Robert Kaiser
I think you were right, let's go for it.