Coupled Train Model v2

[Alex Keybl]

I've tried to collect all of the feedback that's been provided by developers in the community (thanks!!), and put it on a new schedule. Here's my attempt: https://docs.google.com/spreadsheet/ccc?key=0Av9ejaFxk72tdHgwWS14d1NVbkFSd0xjZzRzNXl2VGc

It still accomplishes the same goals as the original Coupled Train Proposal (changing the time spent solely with Aurora, fewer Gecko versions in flight, overlapping beta/release for dot releases) but also has the added benefit of Aurora population testing throughout the development cycle.

Some notes:
* it assumes we perform nightly->aurora merges every ~2 weeks (this has impact to QA, so is not a final proposal)
* it assumes we'll change to a 9 week cycle (may not be the case when we decide on how to align with B2G)
* it assumes we'll have the ability to push RCs to the Beta channel by this point (incoming, thanks to RelEng)
* I'm looking at the schedule on the granularity of 1 week, for sake of visual sanity
* I threw this together fairly quickly, so there may still be mistakes
* This model assumes we'd kick off as of Firefox 32 on m-c to allow time for dust to settle around early B2G releases

-Alex

[Chris Peterson]

hi Alex, your proposal looks very promising. I like that Aurora gets new
code sooner while also increasing Beta overlap and bake time.

On 10/24/13, 12:31 PM, Alex Keybl wrote:
> * it assumes we perform nightly->aurora merges every ~2 weeks (this has impact to QA, so is not a final proposal)

Why every 2 weeks instead of 1? To reduce the QA impact?

> * it assumes we'll change to a 9 week cycle (may not be the case when we decide on how to align with B2G)

Why a 9 week cycle instead of a multiple of 2 like 8? The ninth week
adds an unnecessary discontinuity to the nightly->aurora merge cycle. :)


cpeterson

[Alex Keybl]

> Why every 2 weeks instead of 1? To reduce the QA impact?

Correct, I'm guessing the Aurora mini-merges less than once a week due to QA bandwidth. Might not be the case.

>> * it assumes we'll change to a 9 week cycle (may not be the case when we decide on how to align with B2G)
>
> Why a 9 week cycle instead of a multiple of 2 like 8? The ninth week adds an unnecessary discontinuity to the nightly->aurora merge cycle. :)

A couple of reasons for 9 weeks:

* It maintains an 18 week end to end - code won't take longer to ship than today
* It juggles security (which would rather have a shorter cycle), l10n (who want more time to localize), and amount of time we need with our beta population to collect feedback (somewhere in between)

-Alex

> _______________________________________________
> dev-planning mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-planning

[Nicolas B. Pierron]

I can understand the intent to have an overlap between Beta and Release, but
I am not sure to understand the usefulness of it between Aurora and Beta.
Couldn't we just let Aurora settle for the 2 last weeks and then merge once
into beta?

Just a side note, if we do only 3 merges from nightly to aurora, then we can
have a 2 weeks overlap without the inconsistency of the week 8 of nightly.

--
Nicolas B. Pierron

[Alex Keybl]

> I can understand the intent to have an overlap between Beta and Release, but I am not sure to understand the usefulness of it between Aurora and Beta. Couldn't we just let Aurora settle for the 2 last weeks and then merge once into beta?

The intent here would be to give QA the opportunity to confidently put out a beta without worrying about the next Aurora merge quite yet.

> Just a side note, if we do only 3 merges from nightly to aurora, then we can have a 2 weeks overlap without the inconsistency of the week 8 of nightly.

We could consider this, but for sake of quality (as opposed to QA bandwidth), more merges is better.

-Alex

[Lawrence Mandel]

----- Original Message -----
> > I can understand the intent to have an overlap between Beta and Release,
> > but I am not sure to understand the usefulness of it between Aurora and
> > Beta. Couldn't we just let Aurora settle for the 2 last weeks and then
> > merge once into beta?
>
>

> The intent here would be to give QA the opportunity to confidently put out a
> beta without worrying about the next Aurora merge quite yet.
>

> > Just a side note, if we do only 3 merges from nightly to aurora, then we
> > can have a 2 weeks overlap without the inconsistency of the week 8 of
> > nightly.
>

> We could consider this, but for sake of quality (as opposed to QA bandwidth),
> more merges is better.

For this reason and to avoid Aurora diverging too much from Nightly and therefore, in this model, reducing the usefulness of testing, I would like to consider the weekly merge. As has been pointed out earlier, Aurora users are used to daily updates. Moving to weekly is already a significant reduction. My initial thought is we want to update Aurora more than bi-weekly.

Lawrence

>
> -Alex
>
> On Oct 24, 2013, at 4:28 PM, "Nicolas B. Pierron"
> <nicolas....@mozilla.com> wrote:
>

[Chris Hofmann]

On 10/24/13 12:31 PM, Alex Keybl wrote:
> I've tried to collect all of the feedback that's been provided by developers in the community (thanks!!), and put it on a new schedule. Here's my attempt: https://docs.google.com/spreadsheet/ccc?key=0Av9ejaFxk72tdHgwWS14d1NVbkFSd0xjZzRzNXl2VGc
>
> It still accomplishes the same goals as the original Coupled Train Proposal (changing the time spent solely with Aurora, fewer Gecko versions in flight, overlapping beta/release for dot releases) but also has the added benefit of Aurora population testing throughout the development cycle.
>
> Some notes:
> * it assumes we perform nightly->aurora merges every ~2 weeks (this has impact to QA, so is not a final proposal)
> * it assumes we'll change to a 9 week cycle (may not be the case when we decide on how to align with B2G)
> * it assumes we'll have the ability to push RCs to the Beta channel by this point (incoming, thanks to RelEng)
> * I'm looking at the schedule on the granularity of 1 week, for sake of visual sanity
> * I threw this together fairly quickly, so there may still be mistakes
> * This model assumes we'd kick off as of Firefox 32 on m-c to allow time for dust to settle around early B2G releases
>

> -Alex
>

Its hard for me to read what this means for a few important goals that I
think we often lose sight of.

1) Create system that impose more discipline and order on the
development process, taking high volume and frequent changes early, and
then reducing the number of changes as soon as possible, then
dramatically as we get closer to release. This kind of disapline allows
us to create the highest quality software possible and introduce it to
increasingly wider audiences and check at each stage. That was the
basic principal behind the 100k, million, 10 million people on each of
the rapid release channels.

2) Create a solid baseline for the next release with all the changes we
intend to release (central and very early aurora)

3) then allow time localize and prepare for world wide release. (aurora,
and hopefully covered under a new firefox for web developer release)

4) Also check out that software with addon and web developers to ensure
compatibility, or give them time to adjust to new features, or report
back bugs. (aurora, and hopefully covered under a new firefox for web
developer release)

5) When we are sure we have the highest quality in terms of
compatibility, stability, performance release to the biggest beta
audience we can muster, for as long as we possibly can test. (beta)

6) When the beta audience tells us the software is great release it to
the rest of the world.

All the merging up, bug fix builds, and constant changes possibly
implied by the diagram makes me nervous about not paying attention to
these basic principals and needs. Especially 2, 3, and 4. It opens
opportunities to expose a lot of people to regresssions, and there by
possibly loosing those people as active testers at each stage of the
process. It also makes it confusing about what we are trying to do at
each stage of the process other than just shoving software out to a lot
of people in anticipation of trying to get their feedback, and fixing
things as they seem busted.

-chofmann

[Karl Tomlinson]

Lawrence Mandel writes:

>> > I can understand the intent to have an overlap between Beta and Release,
>> > but I am not sure to understand the usefulness of it between Aurora and
>> > Beta. Couldn't we just let Aurora settle for the 2 last weeks and then
>> > merge once into beta?
>>
>> The intent here would be to give QA the opportunity to confidently put out a
>> beta without worrying about the next Aurora merge quite yet.
>>
>> > Just a side note, if we do only 3 merges from nightly to aurora, then we
>> > can have a 2 weeks overlap without the inconsistency of the week 8 of
>> > nightly.
>>
>> We could consider this, but for sake of quality (as opposed to QA bandwidth),
>> more merges is better.
>
> For this reason and to avoid Aurora diverging too much from
> Nightly and therefore, in this model, reducing the usefulness of
> testing, I would like to consider the weekly merge. As has been
> pointed out earlier, Aurora users are used to daily
> updates. Moving to weekly is already a significant reduction. My
> initial thought is we want to update Aurora more than bi-weekly.

I assume fixes can still land on aurora with approval.

Skipping the merge from nightly to aurora at the end of week 9
would mean no last minute features, but it would still receive
fixes before going to beta, increasing the settling/fixing time
from 1 to 2 weeks, and so *increasing* quality on beta.

Also skipping the merge at the end of week 8 would increase
quality further by increasing fixing time to 4 weeks.

[Axel Hecht]

On behalf of l10n, I veto this proposal.

This is not a modification of our existing release schedule, it's merely
reusing a few words with completely different meaning.

Aurora in the current scheme is the most powerful tool we have to
deliver localized software, and there's nothing left of that that matters.

It just won't create localized software, and honestly, I don't have to
prove that it doesn't. You have to prove that it would.

Axel

[Alexander Keybl]

I'll try to address 2, 3, and 4 since you feel these are the largest holes in the proposal.

Can you clarify #2? What do you believe constitutes a solid baseline, and in what ways do you feel this proposal changes that?

#3 obviously still needs consensus, but I'm optimistic we can work something out.

Jorge has already addressed #4 for add-on developers, who suggests that they will welcome these changes. The lengthened cycle with Beta users will mitigate any website bustage, since we don't typically hear about them until Beta anyway.

The idea is to get our code out to as many pre-release users as early as possible without breaking our quality contract with them. My team has looked at tracking bugs to evaluate how many would block beta after a week on Aurora, and we found none. I also do not see how more frequent central to Aurora merges would be of significantly lower quality when it's basically the same process that we accomplish once a cycle, just with less code change.

-Alex

----- Original Message -----
From: Chris Hofmann <chof...@mozilla.com>
To: Alex Keybl <ake...@mozilla.com>, dev-pl...@lists.mozilla.org group <dev-pl...@lists.mozilla.org>
Sent: Thu, 24 Oct 2013 15:10:45 -0700 (PDT)
Subject: Re: Coupled Train Model v2

On 10/24/13 12:31 PM, Alex Keybl wrote:
> I've tried to collect all of the feedback that's been provided by developers in the community (thanks!!), and put it on a new schedule. Here's my attempt: https://docs.google.com/spreadsheet/ccc?key=0Av9ejaFxk72tdHgwWS14d1NVbkFSd0xjZzRzNXl2VGc
>
> It still accomplishes the same goals as the original Coupled Train Proposal (changing the time spent solely with Aurora, fewer Gecko versions in flight, overlapping beta/release for dot releases) but also has the added benefit of Aurora population testing throughout the development cycle.
>
> Some notes:
> * it assumes we perform nightly->aurora merges every ~2 weeks (this has impact to QA, so is not a final proposal)
> * it assumes we'll change to a 9 week cycle (may not be the case when we decide on how to align with B2G)
> * it assumes we'll have the ability to push RCs to the Beta channel by this point (incoming, thanks to RelEng)
> * I'm looking at the schedule on the granularity of 1 week, for sake of visual sanity
> * I threw this together fairly quickly, so there may still be mistakes
> * This model assumes we'd kick off as of Firefox 32 on m-c to allow time for dust to settle around early B2G releases
>
> -Alex
>

[Alexander Keybl]

Correct, Aurora approvals between merges will improve quality over each N week mini cycle.

-Alex

----- Original Message -----
From: Karl Tomlinson <moz...@karlt.net>
To: dev-pl...@lists.mozilla.org
Sent: Thu, 24 Oct 2013 15:14:44 -0700 (PDT)
Subject: Re: Coupled Train Model v2

[chris hofmann]

On 10/25/13 7:50 AM, Alexander Keybl wrote:
> I'll try to address 2, 3, and 4 since you feel these are the largest holes in the proposal.
>
> Can you clarify #2? What do you believe constitutes a solid baseline, and in what ways do you feel this proposal changes that?

One in which we have stopped adding features to. One in which we have
seen a decline in the number of changes, number of regressions found and
fixed, and for which we have enough time to gather good crash and other
feedback data (about a week or so).

If I read the proposal and the diagram right its mostly a constant
changing piece of software that is harder to localize, gain an
understanding of the state of the software, and harder to measure
feedback and quality since its under change.

-chofmann

>
> #3 obviously still needs consensus, but I'm optimistic we can work something out.
>
> Jorge has already addressed #4 for add-on developers, who suggests that they will welcome these changes. The lengthened cycle with Beta users will mitigate any website bustage, since we don't typically hear about them until Beta anyway.
>
> The idea is to get our code out to as many pre-release users as early as possible without breaking our quality contract with them. My team has looked at tracking bugs to evaluate how many would block beta after a week on Aurora, and we found none. I also do not see how more frequent central to Aurora merges would be of significantly lower quality when it's basically the same process that we accomplish once a cycle, just with less code change.
>

> -Alex
>
> ----- Original Message -----

> From: Chris Hofmann <chof...@mozilla.com>
> To: Alex Keybl <ake...@mozilla.com>, dev-pl...@lists.mozilla.org group <dev-pl...@lists.mozilla.org>
> Sent: Thu, 24 Oct 2013 15:10:45 -0700 (PDT)
> Subject: Re: Coupled Train Model v2
>

[Alexander Keybl]

We're going to be stricter about feature backouts around merges, for sure, and the early and frequent Aurora population feedback will go a long way to guide those decisions. Using Aurora as time padding to make those decisions shouldn't be necessary.

[Chris Peterson]

On 10/25/13, 4:19 AM, Axel Hecht wrote:
> Aurora in the current scheme is the most powerful tool we have to
> deliver localized software, and there's nothing left of that that matters.

hi Alex, for those of us that are not familiar with Firefox's
localization workflow, can you briefly describe the timeline for a new
feature's localization?

Neither of these recent train proposals have a good answer for string
freezes or localization time.

What are the average and longtail of localization times? Are most
features localization-complete before Beta? Our localizers are tireless
volunteers, so some locales are probably completed much more quickly
than others.


thanks,
chris

[Alex Keybl]

I'd actually prefer to hear typical timelines, average l10n complete % at Nightly->Aurora and Aurora->Beta, etc. from the l10n team since I don't have that data. Agreed this would be great data to tweak the model.

-Alex

[Alex Keybl]

To give a couple of examples of anecdotal evidence as to why we should really be considering a longer beta cycle, we just ran into https://bugzilla.mozilla.org/show_bug.cgi?id=927901 in the last week of FF25's beta after we'd already spun our RC for FF25. My conversation with dougt:

> 1:16 PM <akeybl> is this a breakdown of pre-release user testing?
> 1:17 PM <akeybl> if we think we need to respin for this, we obviously need to lengthen the beta cycle
> 1:17 PM <akeybl> because there's no other way to find/fix in time
> 1:17 PM <dougt> i think you want to remove m-a, and make beta longer
> 1:18 PM <akeybl> we're doing something similar in that Couple Train Model thread on dev.planning

On a separate note, in terms of Aurora vs Beta user feedback:

> 1:18 PM <dougt> johns tells me all of the time he only hears about plugin bugs in beta
> 1:18 PM <dougt> so, basically he fixes stuff, forgets about the fixes for 6 weeks, then gets a bunch of new bug reports about the stuff he forgot about :)

Seems like all the more reason to pursue this model, allow devs to focus on fewer Gecko versions, and provide them with more feedback earlier in the release.

-Alex

On Oct 24, 2013, at 3:31 PM, Alex Keybl <ake...@mozilla.com> wrote:

> I've tried to collect all of the feedback that's been provided by developers in the community (thanks!!), and put it on a new schedule. Here's my attempt: https://docs.google.com/spreadsheet/ccc?key=0Av9ejaFxk72tdHgwWS14d1NVbkFSd0xjZzRzNXl2VGc
>
> It still accomplishes the same goals as the original Coupled Train Proposal (changing the time spent solely with Aurora, fewer Gecko versions in flight, overlapping beta/release for dot releases) but also has the added benefit of Aurora population testing throughout the development cycle.
>
> Some notes:
> * it assumes we perform nightly->aurora merges every ~2 weeks (this has impact to QA, so is not a final proposal)
> * it assumes we'll change to a 9 week cycle (may not be the case when we decide on how to align with B2G)
> * it assumes we'll have the ability to push RCs to the Beta channel by this point (incoming, thanks to RelEng)
> * I'm looking at the schedule on the granularity of 1 week, for sake of visual sanity
> * I threw this together fairly quickly, so there may still be mistakes
> * This model assumes we'd kick off as of Firefox 32 on m-c to allow time for dust to settle around early B2G releases
>
> -Alex

[Axel Hecht]

On 10/25/13 7:06 PM, Chris Peterson wrote:

On 10/25/13, 4:19 AM, Axel Hecht wrote:

Aurora in the current scheme is the most powerful tool we have to
deliver localized software, and there's nothing left of that that matters.

hi Alex, for those of us that are not familiar with Firefox's localization workflow, can you briefly describe the timeline for a new feature's localization?

I'm taking this as an "Axel" question.

I'll need to turn this question around a bit. Feature-based l10n is something that a few communities do. The large-scale effort we're doing is working on code dumps. Someone (not me) goes in, imports our string into the db of a webtool, localizers work on that, someone (same not me) exports that to a hg repo, commits, nags me and Jeff.

In our current release process, we're putting out a statement of work to our l10n community every six weeks. That's the thing we're exposing on the aurora channel.

The localization teams have some 5 1/2 weeks to get to work on that.

The actual work to get from English to localized strings is probably less than a day.

What really makes localizing aurora attractive today is that a localizer can occasionally scratch their itch and attend mozilla l10n. And they'll spend a few hours, and they can be confident that their work creates a good product for the officially released product. They'll watch community feedback, and they'll know if they need to get back to it or not.


The challenge to put out is:

In the proposal, what's the value of contributing to aurora l10n for 7 out of the 9 weeks?

My concern is that this can be anything between zero and hero, depending on what lands in the last merge. The l10n contributor seems to have little impact on that. If the first 7 weeks were https elliptic curve stuff, and the last merge impacts the file menu, your l10n work sucks. The l10n-drivers team has a lot of feedback that localizers are optimizing for what happens at the end of the cycle.

I'd love a change to our release schedule that gave me more leverage to get l10n teams to work early in the cycle.

Neither of these recent train proposals have a good answer for string freezes or localization time.
What are the average and longtail of localization times? Are most features localization-complete before Beta? Our localizers are tireless volunteers, so some locales are probably completed much more quickly than others.

Thanks, in particular on behalf of the many teams that are active early.

The aspect that I'd like to put front and center here is contributor experience. What we have today is: A string localized in aurora is a string localized in release. At the point you contribute, you can evaluate the impact of your contributions to the released product.

In the Model v2, neither of those hold. The strings that an l10n contributor localized in week 1-7 don't represent the strings that users gonna see in release.

Axel

[Chris Peterson]

On 10/25/13, 4:44 PM, Axel Hecht wrote:
> In our current release process, we're putting out a statement of work to
> our l10n community every six weeks. That's the thing we're exposing on
> the aurora channel.
>
> The localization teams have some 5 1/2 weeks to get to work on that.
>
> The actual work to get from English to localized strings is probably
> less than a day.

hi Axel, thanks for the explanation. I see how sustaining a community of
l10n contributors is different than hiring contractors to do a job.

Today, contributors don't use all 6 weeks on Aurora for localizing, but
the time gives them flexibility to schedule *when* they do the
localization. And Aurora is always open, so new l10n projects can start
any time.

While the coupled Aurora/Beta train model (with a 2 week Aurora) might
*technically* be enough time to localize, we would be asking
contributors to rush for 2 weeks and then not provide them much
opportunity for contributing during the other 7 weeks of the cycle.

Chrome can impose code freezes on developers and hire an army of l10n
contractors to rush as the end of the cycle. Mozilla can't do that if
developers need mozilla-central to always be open for checkins and the
l10n community needs complete string dumps and flexible scheduling.
Aurora is Mozilla's code and string freeze.


chris

[Alexander Keybl]

In my response to Chris, I noted that we will be much stricter about feature backout based upon QA testing both during development and upon the final merge to Aurora before Beta. That saves about 7 weeks to localize (forget the week before release). That's obviously less time than today. But the real question is, can we agree on an acceptable beta string quality (perhaps lower than today) and can we tweak the process to ensure the same release quality at the end of the day.

I think the main piece of data we're missing is the % completion of string translation upon the current Aurora to Beta merge and the Beta to Release merge. That data will be very telling, and help us to understand what our targets need to be.

As noted previously, we've never blocked an Aurora to Beta merge based upon localization completeness, so I'm hoping we can focus on Release localization completeness instead.

-Alex

----- Original Message -----
From: Axel Hecht <l1...@mozilla.com>
To: dev-pl...@lists.mozilla.org
Sent: Fri, 25 Oct 2013 16:44:48 -0700 (PDT)
Subject: Re: Coupled Train Model v2

*The challenge to put out is*:

[Axel Hecht]

Sorry, but here's going to be a metric shitload of data:

We had ~70 people pushing to Firefox 25, 23 on central, 58 on aurora, 38
on beta. There were 19 active locales on central, 75 on aurora, 35 on
beta. 35 is surprisingly also the number of localizations that have
recently got non-merge-or-release pushes.

To slice the data differently:

The count of locales that are currently tracking are roughly ~80.

We have less-than 20 locales on central.

We have some 40-50 locales that are translating through pootle,
https://mozilla.locamotion.org/. That's where we're making numbers these
days.
Those don't interact with version control themselves, but only come into
our repositories through mentors, mostly Dwayne. That means, there's on
person in the 58 people mentioned above that stands for half our our
localizations in the data I have.
They're only exposed to aurora, the beta repositories don't even show
up. There's benefit to that. People showing up to contribute don't need
to figure out what they want to contribute to, they just pick "Firefox",
and know their work is good. Some activity stats, based on which of his
irregular landings picked up changes from his system: 1 landings for 9
localizations, 2 for 10, 3 for 4 and 4 landing for 4 localizations, 6
and 7 landings for one locale only. One third/one half of Dwayne's
ecosystem are actually one-hit wonders per cycle. I bet some of those
are actually follow-ups for technical problems, but I don't know how to
pick them right now.


Bottom line: The bread and butter of Firefox localization is working on
a single contribution channel, many of which contribute once or twice
per cycle.

I've focused this post on contribution metrics, mostly because I think
they matter most. Engaged people with rewarding contribution patterns
are most likely to create good products. I realize that I'm not talking
about completion metrics at all. They've sadly lost their meaning over
the course of the past releases, one devtool by the other. One third of
the Firefox user interface is actually developer interface, and that
follows different rules. Getting back comparable completion metrics is
something we've started talking about, but we're not close.

Now I'm taking a deep breath and send out just the data. If you're
interested in different slices or sets or patterns, feel free to ping
me, I'll try best to find something in the data that I have that's close.

I'll add my personal opinions on how the v2 proposal affects the
contribution patterns in a follow-up.

Axel

[Axel Hecht]

Here's my take on how this proposal affects localizers. It's based on
those that are outspoken to me on what they do, and when, and what
they're asking for.

Bi-weekly dev channel:

First off, it could be that any of what we have today doesn't fit the
contribution opportunities that localizers want. And the summit
reiterated that that is true to a good extent. The reported mismatch and
ask for change is not about cadence itself, though. I don't have a
localizer on my mind that had been asking for a biweekly-ish
contribution rhythm.

For our frequent contributors, I think those are on central anyway now.
I don't know of one that's easy about merges and pushing to multiple
upstream repos. The six-week central-aurora merge is an uneasy step for
them, and already a dump of files or just massive transplants instead of
merges for many. Doing this biweekly will add technical hurdles more
often, I don't see this group engaging with l10n of a bi-weekly dev channel.

For the big chunk of occasional contributors, the rather frequent
updates are watering down the value of their contribution at that point
in time. Given that at the same point, we'd have beta open, and expect
people to work on beta, I'd expect that the large amount of folks is
gonna neglect the bi-weekly dev channel l10n, and go straight for beta.

Most localizers are also expecting that a developer-focused channel is
going to be used in English builds, and not localized builds. That would
add to the emotional distance.


Beta channel:

This is based on my expectations on the dev-channel l10n.

When the beta phase starts, we'll have a minority of localizations
coming from Nightly. Not really sure when, though, as nightly doesn't
have a stable state on either l10n or en-US, of course.
The big chunk of l10n is going to start over the course of the next few
weeks then.
Beta builds update every few days, and only ship off of given revisions
at this point. At least today we don't have builds there that allow for
easy and timely testing of a localization.

I personally don't think we should brand such a product as Firefox Beta.

Independently, this opens the question of how much we want to know about
the l10n state of release, and how to get from something we build at the
beginning of beta to something we build at the end of beta.


From my understanding of the contributors that are doing the l10n work
for us, I don't expect to see an engaged l10n community around a
bi-weekly developer channel. I think there are many open questions on
how to get to a good localized release with just Beta.

Axel

[Alex Keybl]

A few notes I want to point out:

* I don't think we need to assume that localizers would need to be engaged on Aurora specifically. With the Beta cycle elongated, I think it's reasonable for localizers to the longer Beta if they're concerned with Nightly churn.
* I'm wondering if 3 more weeks of change, with less frequent localization deadlines, may actually be a positive for localizers. Have you discussed with them more string changes with less frequent deadlines? If not, I can start a separate thread.
* We also have only discussed a subset of process improvements that could alleviate string churn and pull more localizers to earlier channels. I know I mentioned utilizing QA to perform feature backouts quickly before the merge to beta. We can also explore all string/UX changes requiring review prior to landing on m-c to prevent unnecessary churn. Up until now, we've been reactive as opposed to proactive when it comes to string change, and that probably pushed localizers into later channels.
* For large forward-facing features (for instance Australis) we should explore localizing on the project branch. Most new user facing features are significantly smaller (for instance, dev tools) and wouldn't have significant user impact with an unlocalized string on the pre-release Beta channel.
* I think we still need better data for the localization requirements that we should meet if we ever choose to change our release model. Currently that's not clear to me (% of strings localized that should be localized at each merge, amount of time needed to localize per string, etc.), so I don't know how to move forward with compromise.

That last point is most important. What's the bar that we need to hit in order to make the l10n community happy with any new release process changes? "No different than today" is tough to accommodate when we're trying to improve quality and developer/QA focus.

-Alex

[chris hofmann]

On 10/29/13 9:46 AM, Alex Keybl wrote:
> A few notes I want to point out:
>
> * I don't think we need to assume that localizers would need to be engaged on Aurora specifically. With the Beta cycle elongated, I think it's reasonable for localizers to the longer Beta if they're concerned with Nightly churn.

You've highlighted the impact on localizers, but we need to also assess
the impact on localized build users for aurora and beta.

If we kick out poorly localized builds into beta on 9 weeks cycles the
impact will likely be lost users on the beta channel, even if the builds
slowly get better over the course of the beta. We need to be finding
ways to make betas more stable right from the start, not just shifting
the sorting out that happens in aurora to early stages of the beta. I
think the proposal works against that.

In fact we have lost about 70,000 users on aurora over the last year.
Do we have any understanding of why that happened?

Its it due to poor quality and unmet expectations about what aurora is
supposed to be? other reasons?

> * I'm wondering if 3 more weeks of change, with less frequent localization deadlines, may actually be a positive for localizers. Have you discussed with them more string changes with less frequent deadlines? If not, I can start a separate thread.
> * We also have only discussed a subset of process improvements that could alleviate string churn and pull more localizers to earlier channels. I know I mentioned utilizing QA to perform feature backouts quickly before the merge to beta. We can also explore all string/UX changes requiring review prior to landing on m-c to prevent unnecessary churn. Up until now, we've been reactive as opposed to proactive when it comes to string change, and that probably pushed localizers into later channels.
> * For large forward-facing features (for instance Australis) we should explore localizing on the project branch. Most new user facing features are significantly smaller (for instance, dev tools) and wouldn't have significant user impact with an unlocalized string on the pre-release Beta channel.

this is pretty well covered. we have advanced teams localizing on
central, and they provide feedback to feature teams and uncover problems
before they get exposed to the bulk of teams.
> * I think we still need better data for the localization requirements that we should meet if we ever choose to change our release model. Currently that's not clear to me (% of strings localized that should be localized at each merge, amount of time needed to localize per string, etc.), so I don't know how to move forward with compromise.

As Axel has mentioned, its hard to over generalize for a group of 1000
contributors in localization. Probably just as hard or harder than it
would be to generalize for any other development group. We have a wide
set of diversity in workstyles with everything from active translation
within hours of landing on nighly, to some teams that some times need to
skip a release because the team is small and life
(vacations/school/work/other) calls.

These things work to the advantage of all teams, and we have
communicated them several times.

1) Provide advance warning on big new features and get
internationalization/localzation feedback in advance of landing.

2) Have clear schedules and cut off dates for when strings stop changing
and we promise they will not change.

3) Give teams an opportunity to initially respond to string changes,
get out a cut at translation, and be able to get a round of feedback and
response from the communities that follow translation work.

Advance teams on central and aurora the way it is set up right now helps
to meet those goals, Cutting the Aurora cycle from 6 to 2 weeks, and
introducing the potential for aurora to be in a pretty confused state
for its entire lifetime work against those goals.

I've asked a few times for us to assess and document what our
expectations are for user users of nightly, aurora, and beta and I hope
that we can spend some time doing that as part of this planing.

-chofmann

>
> That last point is most important. What's the bar that we need to hit in order to make the l10n community happy with any new release process changes? "No different than today" is tough to accommodate when we're trying to improve quality and developer/QA focus.
>
> -Alex
>
> On Oct 29, 2013, at 10:38 AM, Axel Hecht <l1...@mozilla.com> wrote:
>

[Randell Jesup]

I'll note that the main feedback at the Summit in Toronto was that the
proposal shown at the time was a real problem for localizers - and I
think it may have been an easier one for them to deal with than the
current proposal. Even a 9 and 9 schedule was seen as causing serious
problems.

Bhavana should have more detailed notes.

--
Randell Jesup, Mozilla Corp
remove "news" for personal email

[Gervase Markham]

On 29/10/13 16:46, Alex Keybl wrote:
> That last point is most important. What's the bar that we need to hit
> in order to make the l10n community happy with any new release
> process changes? "No different than today" is tough to accommodate
> when we're trying to improve quality and developer/QA focus.

That sounds a bit like "it's OK to make life harder for localizers if it
improves developer/QA focus". I'm not sure that's a given.

Perhaps we need to step back a bit and try and work out what your
average localizer (i.e. not the keen ones who localize any string within
38 minutes of it hitting the repo, and not the ones who skip releases)
needs. Is the answer something like "an N-week string freeze where they
can localize before builds of their locale are then shipped to a wide
audience"? If so, what's the value of N, and where in the current plan
does this happen? AIUI, at the moment it happens on Aurora, and N is 6,
or perhaps a bit less if they wait a week for Aurora to settle down.

Gerv

[Chris Hofmann]

these are actually pretty ineffective ways, and maybe even destructive
ways, to think about what we are asking localizers to do.

It sets the exception that localization work is just a calculated
conversion step to any given set of core design and development activity
thats done in English largely for a US audiance. That's definitely not
what we are asking of localization teams for.

We are asking them to look at what has been done in English for the US
audience, and first gather context of what the feature is about, then
figure out if and how that feature might work for speakers of their
language. Then if it does, figure out how it might work, how it might
be explained to users, and how those users can know and understand the
feature. The the lead localizer takes a crack at the translation, then
gets feedback.

We often use metaphores in the US centric design that really make no
sense in a large number of locales. When this happens we some times
request to go back into the design to have it modified so it works or
translates differently, and in some cases this also improves the
understanding of the feature and text in English!

The workflow that we try to promote and design allows for this
interaction, and iteration. The workflow you imply below is here is a
load of stuff to localize, here is a load of translated material back.
That's not how we envision this working and leads to lower quality
software for the 80% of our users who don't live in the US, and the 70%
that prefer a language other than English.

-chofmann

On 10/25/13 10:21 AM, Alex Keybl wrote:
> I'd actually prefer to hear typical timelines, average l10n complete % at Nightly->Aurora and Aurora->Beta, etc. from the l10n team since I don't have that data. Agreed this would be great data to tweak the model.
>
> -Alex
>
> On Oct 25, 2013, at 1:06 PM, Chris Peterson <cpet...@mozilla.com> wrote:
>

[Mike Connor]

On 2013-10-30 9:56 AM, Gervase Markham wrote:
> On 29/10/13 16:46, Alex Keybl wrote:
>> That last point is most important. What's the bar that we need to hit
>> in order to make the l10n community happy with any new release
>> process changes? "No different than today" is tough to accommodate
>> when we're trying to improve quality and developer/QA focus.
> That sounds a bit like "it's OK to make life harder for localizers if it
> improves developer/QA focus". I'm not sure that's a given.

Not without quantification. A trivial win for developers and QA
wouldn't justify a massive negative hit to l10n, but on the other hand
if compressing a string freeze window means we get a better product for
most/all of our users, it would be hard to convince me that we shouldn't
do it. The end goal has to be a process that yields the best product
experience for our users, while still being possible for our localizers
to work with in an effective manner.

Given the data cited elsewhere in this thread, it seems like we have a
solid chunk of locales tracking central (19 at last count, per Axel).
chofmann is right to cite feedback from localizers as a positive factor
in improving l10n support in core, and my experience is that it's the
localizers who are actively translating as we build who provide all (or
close enough) of that timely feedback. Unless there's a reasonable
belief that this change will cause those locales to move off central, I
don't think we're negatively impacting that set of locales, or putting
that feedback cycle at risk. So that's one concern I think is
reasonable, but should be a non-issue with this model.

The next question is about the other 3/4 of localizers who tend to
localize in a single burst. I'm going to assert that the primary
benefit to a longer string freeze for those localizers is a greater
degree of flexibility as to when they power through a full release worth
of string changes. Five weeks is better than five days in terms of
picking a time to do this, but no one has ever told me that it's a five
week process, rather it's just a matter of finding a chunk of
consecutive hours to make that happen. In the past, having a two week,
three weekend period of string freeze prior to builds/releases has been
more than sufficient for localizers, and having this happen on a
predictable schedule would likely make that even better for localizers.
I'd be curious to hear from localizers who can't make that sort of
timeframe work.

Finally, there's a key assumption we should assess and understand,
namely whether all locales must be treated equally. If we assume that
weeks three and four of the cycle (first two weeks of beta) are when the
bulk of locales are expected to land strings, what's the success rate we
consider acceptable? If we're covering 95% of beta users with fully
localized builds by the start of week five, and fill in the last locales
by release, is that sufficient? (Keep in mind that gives us five weeks
of bake time, basically the same as we have now, the earlier locales
will just get extra beta time.)

tl;dr I don't think this proposal is a major risk to the quality of l10n
in Firefox, in that we can likely make it work if we challenge some
assumptions.

-- Mike

[Axel Hecht]

Right.

> The next question is about the other 3/4 of localizers who tend to
> localize in a single burst. I'm going to assert that the primary
> benefit to a longer string freeze for those localizers is a greater
> degree of flexibility as to when they power through a full release worth
> of string changes. Five weeks is better than five days in terms of
> picking a time to do this, but no one has ever told me that it's a five
> week process, rather it's just a matter of finding a chunk of
> consecutive hours to make that happen. In the past, having a two week,
> three weekend period of string freeze prior to builds/releases has been
> more than sufficient for localizers, and having this happen on a
> predictable schedule would likely make that even better for localizers.
> I'd be curious to hear from localizers who can't make that sort of
> timeframe work.

The beta period doesn't seem to be a time problem right now. My heads
going back and forth about technical details here. Do we have enough
builds, off of the right versions, how do we get the right stuff to
testers of the localization, and how do we prepare to gather user
feedback on what we put on the release channel.

One possible way to work around this is to create an aurora-like channel
with build and update characteristics of what we have on aurora today,
but off of beta repos. I'm also not fond of this, though.

At that point we'd have 5 channels, and only localize 2 and 1/3rd of
that. We'd localize release, l10n-on-beta, and some of nightly. If we
should even do l10n for dev and beta in such a scheme, I don't know.

> Finally, there's a key assumption we should assess and understand,
> namely whether all locales must be treated equally. If we assume that
> weeks three and four of the cycle (first two weeks of beta) are when the
> bulk of locales are expected to land strings, what's the success rate we
> consider acceptable? If we're covering 95% of beta users with fully
> localized builds by the start of week five, and fill in the last locales
> by release, is that sufficient? (Keep in mind that gives us five weeks
> of bake time, basically the same as we have now, the earlier locales
> will just get extra beta time.)

I don't think we need to treat all locales equally. That doesn't mean
that we can get 95% of our beta population early in the cycle. I have a
list of locales that aren't set up for this historically, and that we've
tried to help in the past, with limited success. I don't think that
.planning is the right forum to go into the details of each, though.

> tl;dr I don't think this proposal is a major risk to the quality of l10n
> in Firefox, in that we can likely make it work if we challenge some
> assumptions.

I think we need to get more detailed on which assumptions we're actually
challenging.

Axel

[Alex Keybl]

Hi Chris,

> You've highlighted the impact on localizers, but we need to also assess the impact on localized build users for aurora and beta.

What feature from the last year do you expect would have caused localized Beta build users to leave the product over? As mentioned before, I could see Australis causing this, but very few other features. I'd rather we treated them on a case by case basis, if this is in fact your biggest objection.

> In fact we have lost about 70,000 users on aurora over the last year. Do we have any understanding of why that happened?

We can wait to try to make that determination (do you know how to survey users that have left? I don't) or we can react to the trend.

> Advance teams on central and aurora the way it is set up right now helps to meet those goals, Cutting the Aurora cycle from 6 to 2 weeks, and introducing the potential for aurora to be in a pretty confused state for its entire lifetime work against those goals.

We're cutting the hard string freeze before Beta from 6 down to 2, but the Aurora population will be parallel to Nightly for the majority of the cycle. I think we can make efforts to pull localizers back further in the cycle, to mitigate this change. I also believe that the bar we're setting for Beta (unlocalized features lose users) is much too high for a pre-release product and needs to be re-evaluated. The number of major forward facing features that would impact a person's daily browsing is very small, and could be communicated to localizers prior.

-Alex

On Oct 29, 2013, at 2:40 PM, chris hofmann <chof...@meer.net> wrote:

> On 10/29/13 9:46 AM, Alex Keybl wrote:
>> A few notes I want to point out:
>>
>> * I don't think we need to assume that localizers would need to be engaged on Aurora specifically. With the Beta cycle elongated, I think it's reasonable for localizers to the longer Beta if they're concerned with Nightly churn.
> You've highlighted the impact on localizers, but we need to also assess the impact on localized build users for aurora and beta.
>
> If we kick out poorly localized builds into beta on 9 weeks cycles the impact will likely be lost users on the beta channel, even if the builds slowly get better over the course of the beta. We need to be finding ways to make betas more stable right from the start, not just shifting the sorting out that happens in aurora to early stages of the beta. I think the proposal works against that.
>
> In fact we have lost about 70,000 users on aurora over the last year. Do we have any understanding of why that happened?
>
> Its it due to poor quality and unmet expectations about what aurora is supposed to be? other reasons?
>> * I'm wondering if 3 more weeks of change, with less frequent localization deadlines, may actually be a positive for localizers. Have you discussed with them more string changes with less frequent deadlines? If not, I can start a separate thread.
>> * We also have only discussed a subset of process improvements that could alleviate string churn and pull more localizers to earlier channels. I know I mentioned utilizing QA to perform feature backouts quickly before the merge to beta. We can also explore all string/UX changes requiring review prior to landing on m-c to prevent unnecessary churn. Up until now, we've been reactive as opposed to proactive when it comes to string change, and that probably pushed localizers into later channels.
>> * For large forward-facing features (for instance Australis) we should explore localizing on the project branch. Most new user facing features are significantly smaller (for instance, dev tools) and wouldn't have significant user impact with an unlocalized string on the pre-release Beta channel.
> this is pretty well covered. we have advanced teams localizing on central, and they provide feedback to feature teams and uncover problems before they get exposed to the bulk of teams.
>> * I think we still need better data for the localization requirements that we should meet if we ever choose to change our release model. Currently that's not clear to me (% of strings localized that should be localized at each merge, amount of time needed to localize per string, etc.), so I don't know how to move forward with compromise.
>
> As Axel has mentioned, its hard to over generalize for a group of 1000 contributors in localization. Probably just as hard or harder than it would be to generalize for any other development group. We have a wide set of diversity in workstyles with everything from active translation within hours of landing on nighly, to some teams that some times need to skip a release because the team is small and life (vacations/school/work/other) calls.
>
> These things work to the advantage of all teams, and we have communicated them several times.
>
> 1) Provide advance warning on big new features and get internationalization/localzation feedback in advance of landing.
>
> 2) Have clear schedules and cut off dates for when strings stop changing and we promise they will not change.
>
> 3) Give teams an opportunity to initially respond to string changes, get out a cut at translation, and be able to get a round of feedback and response from the communities that follow translation work.
>
> Advance teams on central and aurora the way it is set up right now helps to meet those goals, Cutting the Aurora cycle from 6 to 2 weeks, and introducing the potential for aurora to be in a pretty confused state for its entire lifetime work against those goals.
>
> I've asked a few times for us to assess and document what our expectations are for user users of nightly, aurora, and beta and I hope that we can spend some time doing that as part of this planing.
>
> -chofmann
>>

>> That last point is most important. What's the bar that we need to hit in order to make the l10n community happy with any new release process changes? "No different than today" is tough to accommodate when we're trying to improve quality and developer/QA focus.
>>

>> -Alex
>>
>> On Oct 29, 2013, at 10:38 AM, Axel Hecht <l1...@mozilla.com> wrote:
>>

[Alex Keybl]

>> tl;dr I don't think this proposal is a major risk to the quality of l10n
>> in Firefox, in that we can likely make it work if we challenge some
>> assumptions.
>
> I think we need to get more detailed on which assumptions we're actually challenging.

I tried to be as clear as possible with my bulleted list, but I'll shorten it further. Here are the assumptions we're challenging.

#1: All features must be localized before Beta or we'll lose localized users (we don't meet this bar today, so I'm still challenging this)
#2: Localizers would choose more frequent deadlines over more string change
#3: We can't do more to harden strings prior to landing on Nightly, so most localizers will have to continue working off of later branches

-Alex

[chris hofmann]

On 10/31/13 7:05 AM, Alex Keybl wrote:
>>> tl;dr I don't think this proposal is a major risk to the quality of l10n
>>> in Firefox, in that we can likely make it work if we challenge some
>>> assumptions.
>> I think we need to get more detailed on which assumptions we're actually challenging.

> I tried to be as clear as possible with my bulleted list, but I'll shorten it further. Here are the assumptions we're challenging.
>
> #1: All features must be localized before Beta or we'll lose localized users (we don't meet this bar today, so I'm still challenging this)

This is an extreme way to state the problem. We are talking about
direction, not absolutes. Of course we don't meet that bar. We don't
meet the bar of shipping completely bug free betas either. Does that
mean we should set up a systematic way of shipping betas with more
bugs? I think you'll have a hard time convincing people that's a good
idea.

We could try an experiment. Lets do our featrure development in Arabic
in the en-US builds, and have new features show up in Arabic. I'd
venture to say two things would happen. Some people would be upset and
annoyed at the fact we would ship a beta that's not useful to them, and
in addition we would lose the opportunity to get feedback on the new
feature from all the users of the en-US builds.

When we continue to annoy and upset users they eventually go find
different software to run. International users are especially sensitive
to this as they see English invading their language and culture more and
more. We need to be sensitive to those ideas if we want to reach them
effectively. Also, when we give them software that's hard or
impossible to understand we loose the opportunity to get good feedback.

Lets avoid going in all those directions because its counter productive
to our goals. Both Axel and I, and many others are advocating that we
develop a plan that gets us more high quality localization when we enter
beta and reach out to a wider audience. We want you to help us in
reaching that plan. What will it take to convince you this is a good
plan and get you to participate in it?

> #2: Localizers would choose more frequent deadlines over more string change

As stated. Localizers have a variety of different work patterns they
find effective in getting localization work done. The more we can
accommodate, the more localizations we end up being ready to ship and
get feedback on.

There are high start up costs for localizers that don't follow the
project and work on it every day or every week. We have heard that
feedback countless times from some teams. We continually try to reduce
those start up costs and make it faster to get context on whats changed,
get the source to make changes, allow first cuts at translation, and set
up ways to get quick feedback. Bug we still get feedback from some
teams that they prefer to work in big chucks of work as opposed to small
a frequent chunks.

> #3: We can't do more to harden strings prior to landing on Nightly, so most localizers will have to continue working off of later branches

Sure, we can always do more. We have set up programs like
https://wiki.mozilla.org/L10n/WorldReady to help with goals like your
talking about here. But each feature requires some level of
stability before efficient localization work can happen. That
stability usually happens not on development branches, but on nightly
and aurora.

If we solve for 1 and 2, then solving for 3 is not that critical and not
that disruptive of current work patterns for core feature developers and
designers to make them artificially freeze concepts and strings on
development branches, creating a condition of premature optimization and
freezing just for localization.

-chofmann

> -Alex
>
> On Oct 30, 2013, at 8:24 PM, Axel Hecht <l1...@mozilla.com> wrote:
>

[Chris Peterson]

On 10/31/13, 9:51 AM, chris hofmann wrote:
> Lets avoid going in all those directions because its counter productive
> to our goals. Both Axel and I, and many others are advocating that we
> develop a plan that gets us more high quality localization when we enter
> beta and reach out to a wider audience. We want you to help us in
> reaching that plan. What will it take to convince you this is a good
> plan and get you to participate in it?

I believe you and Axel are correct, that a high-quality, localized Beta
is important for Mozilla.

To flip this conversation around, what "blue sky" changes would
localizers like to see in our current release process? I'm sure
localizers have feature requests and papercuts they would like
addressed. What do people grumble about at l10n meetups? ;)

The localization teams are diverse and distributed, so I think people on
this list may have a hard time seeing the process and problems through
their eyes. With a new perspective, we all may be able to find a middle
ground or a totally new path together.


cpeterson

[Alex Keybl]

This would deserve its own thread.

-Alex

[Alex Keybl]

> This is an extreme way to state the problem. We are talking about direction, not absolutes. Of course we don't meet that bar.

It's not actually an absolute or extreme. I didn't say how many users we'd lose, I just noted your assertion that we would lose some amount of users without full localization at the Beta step.

Even still, I disagree that we'd lose any significant number of localized build users. I'm looking forward to seeing your opinion on a feature in the last year, that if a beta user saw it unlocalized, would cause them to change browsers. I don't think our users are so fickle (as proof by our experience with unlocalized Beta builds).

> We don't meet the bar of shipping completely bug free betas either. Does that mean we should set up a systematic way of shipping betas with more bugs? I think you'll have a hard time convincing people that's a good idea.

Are you suggesting that any plan moving forward must be better for all parties involved at all steps in the process, regardless of the overall benefit to a release user? If so, no progress towards improving our release process could ever be accomplished.

My primary goal is improving release channel quality while maintaining our existing prerelease populations. It's my opinion that you're overemphasizing the impact changing our release process would have on the beta population using l10n builds.

> As stated. Localizers have a variety of different work patterns they find effective in getting localization work done. The more we can accommodate, the more localizations we end up being ready to ship and get feedback on.

Wouldn't a longer cycle accommodate their schedules even more?

> If we solve for 1 and 2, then solving for 3 is not that critical and not that disruptive of current work patterns for core feature developers and designers to make them artificially freeze concepts and strings on development branches, creating a condition of premature optimization and freezing just for localization.

We disagree here. If we could harden strings before landing, localizers wouldn't care whether or not we were shipping the feature in that release. They would just localize on Nightly in preparation for the feature's release, whenever that is.

-Alex

On Oct 31, 2013, at 12:51 PM, chris hofmann <chof...@meer.net> wrote:

> On 10/31/13 7:05 AM, Alex Keybl wrote:

>>>> tl;dr I don't think this proposal is a major risk to the quality of l10n
>>>> in Firefox, in that we can likely make it work if we challenge some
>>>> assumptions.
>>> I think we need to get more detailed on which assumptions we're actually challenging.

>> I tried to be as clear as possible with my bulleted list, but I'll shorten it further. Here are the assumptions we're challenging.
>>
>> #1: All features must be localized before Beta or we'll lose localized users (we don't meet this bar today, so I'm still challenging this)
>
> This is an extreme way to state the problem. We are talking about direction, not absolutes. Of course we don't meet that bar. We don't meet the bar of shipping completely bug free betas either. Does that mean we should set up a systematic way of shipping betas with more bugs? I think you'll have a hard time convincing people that's a good idea.
>
> We could try an experiment. Lets do our featrure development in Arabic in the en-US builds, and have new features show up in Arabic. I'd venture to say two things would happen. Some people would be upset and annoyed at the fact we would ship a beta that's not useful to them, and in addition we would lose the opportunity to get feedback on the new feature from all the users of the en-US builds.
>
> When we continue to annoy and upset users they eventually go find different software to run. International users are especially sensitive to this as they see English invading their language and culture more and more. We need to be sensitive to those ideas if we want to reach them effectively. Also, when we give them software that's hard or impossible to understand we loose the opportunity to get good feedback.
>

> Lets avoid going in all those directions because its counter productive to our goals. Both Axel and I, and many others are advocating that we develop a plan that gets us more high quality localization when we enter beta and reach out to a wider audience. We want you to help us in reaching that plan. What will it take to convince you this is a good plan and get you to participate in it?
>

>> #2: Localizers would choose more frequent deadlines over more string change
>
> As stated. Localizers have a variety of different work patterns they find effective in getting localization work done. The more we can accommodate, the more localizations we end up being ready to ship and get feedback on.
>
> There are high start up costs for localizers that don't follow the project and work on it every day or every week. We have heard that feedback countless times from some teams. We continually try to reduce those start up costs and make it faster to get context on whats changed, get the source to make changes, allow first cuts at translation, and set up ways to get quick feedback. Bug we still get feedback from some teams that they prefer to work in big chucks of work as opposed to small a frequent chunks.
>
>> #3: We can't do more to harden strings prior to landing on Nightly, so most localizers will have to continue working off of later branches
> Sure, we can always do more. We have set up programs like https://wiki.mozilla.org/L10n/WorldReady to help with goals like your talking about here. But each feature requires some level of stability before efficient localization work can happen. That stability usually happens not on development branches, but on nightly and aurora.
>
> If we solve for 1 and 2, then solving for 3 is not that critical and not that disruptive of current work patterns for core feature developers and designers to make them artificially freeze concepts and strings on development branches, creating a condition of premature optimization and freezing just for localization.
>
> -chofmann
>> -Alex
>>
>> On Oct 30, 2013, at 8:24 PM, Axel Hecht <l1...@mozilla.com> wrote:
>>

[jsmith....@gmail.com]

Small note I want to slip in this discussion since I haven't heard many QA opinions on this proposal yet:

I'm hearing a lot of references to benefits, tradeoffs, etc to QA processes in the modified train model. I'm going to suggest that we take that piece offline with some of the QA leads & Alex to get a clear understanding of the QA perspective of this modified model. There were opinions on this when this was brought up in the roundtable, so I think that perspective needs to be thought about here a bit more.

I'm just a bit concerned that we're making conclusions here on the QA benefits without actually talking to QA team.

[Karen Rudnitski]

I'd like to suggest a similar discussion with Product, User Advocacy and
UX teams, too. I think it worth having this conversation off the email
thread as it is a difficult forum to follow for such a large topic that
is, indeed, a huge change that we're suggesting (both to how we are
developing internally but also what can impact our users as a result -
whether beta or release).

I'm thinking along the lines of consumer-facing feature development,
impact, updates, PR cycles (and any PR fatigue?) - areas that I know I
haven't been able to explore thoroughly due to competing priorities, but
these are also areas I think are worth a good discussion having on before
any decisions are made. I also don't think everyone who would be impacted
- again whether internal impact or the face to any external impact - are
on this thread to provide any useful insight and dialogue.

This is, unfortunately, a bit of a difficult forum for having this
discussion on, and I just want to ensure other voices are heard before
moving forward in whatever direction.

Thanks,
Karen

-----Original Message-----
From: dev-planning
[mailto:dev-planning-bounces+krudnitski=mozil...@lists.mozilla.org] On
Behalf Of jsmith....@gmail.com
Sent: Friday, November 1, 2013 4:29 PM
To: dev-pl...@lists.mozilla.org
Subject: Re: Coupled Train Model v2

[Robert Kaiser]

Karen Rudnitski schrieb:

> I'd like to suggest a similar discussion with Product, User Advocacy and
> UX teams, too.

I think it's good to involve multiple levels and teams here, but please
makes sure that all the arguments in either direction are open to the
community - which they are when discussion here, actually.
So, if you have discussions elsewhere, could you post the results here
to make them transparent for the community?

Thanks,
KaiRo

[Karen Rudnitski]

Of course. One of my reasonings here is that many key groups aren't represented on this mailing list so they are missing out on providing input to the discussion. I also think it key that their points of view are passed back to this thread, too, to add further dimension to the topic.

[Alexander Keybl]

Jason,

We already met with Clint/Marc and the desktop QA team. I'll let them write their own opinions, but we left the meeting feeling positive about the impact to release quality and QA analyst focus.

-Alex

----- Original Message -----

From: jsmith mozilla <jsmith....@gmail.com>
To: dev-pl...@lists.mozilla.org
Sent: Fri, 01 Nov 2013 13:28:58 -0700 (PDT)
Subject: Re: Coupled Train Model v2

[Alexander Keybl]

I have a meeting with Chad coming up, and we've discussed shortly with PR already. User Advocacy is aware of the thread and its contents, with no objection raised thus far.

-Alex

----- Original Message -----

From: Karen Rudnitski <krudn...@mozilla.com>
To: jsmith mozilla <jsmith....@gmail.com>, dev-pl...@lists.mozilla.org
Sent: Fri, 01 Nov 2013 13:38:57 -0700 (PDT)
Subject: RE: Coupled Train Model v2

I'd like to suggest a similar discussion with Product, User Advocacy and

UX teams, too. I think it worth having this conversation off the email
thread as it is a difficult forum to follow for such a large topic that
is, indeed, a huge change that we're suggesting (both to how we are
developing internally but also what can impact our users as a result -
whether beta or release).

I'm thinking along the lines of consumer-facing feature development,
impact, updates, PR cycles (and any PR fatigue?) - areas that I know I
haven't been able to explore thoroughly due to competing priorities, but
these are also areas I think are worth a good discussion having on before
any decisions are made. I also don't think everyone who would be impacted
- again whether internal impact or the face to any external impact - are
on this thread to provide any useful insight and dialogue.

This is, unfortunately, a bit of a difficult forum for having this
discussion on, and I just want to ensure other voices are heard before
moving forward in whatever direction.

Thanks,
Karen

-----Original Message-----
From: dev-planning
[mailto:dev-planning-bounces+krudnitski=mozil...@lists.mozilla.org] On
Behalf Of jsmith....@gmail.com
Sent: Friday, November 1, 2013 4:29 PM
To: dev-pl...@lists.mozilla.org

Subject: Re: Coupled Train Model v2

[Alexander Keybl]

Definitely, our goal is to make this as transparent a process as possible, with feedback taken at all the major stages of planning.

-Alex

----- Original Message -----

From: Robert Kaiser <ka...@kairo.at>
To: dev-pl...@lists.mozilla.org
Sent: Sun, 03 Nov 2013 18:52:32 -0800 (PST)
Subject: Re: Coupled Train Model v2

[Axel Hecht]

On 10/31/13 3:05 PM, Alex Keybl wrote:
>>> tl;dr I don't think this proposal is a major risk to the quality of l10n
>>> in Firefox, in that we can likely make it work if we challenge some
>>> assumptions.
>> I think we need to get more detailed on which assumptions we're actually challenging.
> I tried to be as clear as possible with my bulleted list, but I'll shorten it further. Here are the assumptions we're challenging.
>
> #1: All features must be localized before Beta or we'll lose localized users (we don't meet this bar today, so I'm still challenging this)

I think that our quality on beta today can be higher on various locales,
and I wouldn't be surprised if someone brought up data that shows that
some of our beta l10n builds are such that we're actually loosing users
today. I'm not saying that each and every l10n team has the same
problems, or that they're having the same problems each cycle.

I'd love to work on fixing that, and the chained train model gives me
less tools to do that.

> #2: Localizers would choose more frequent deadlines over more string change
> #3: We can't do more to harden strings prior to landing on Nightly, so most localizers will have to continue working off of later branches

These two don't affect the people working on l10n. They want to work on
the translation of Firefox 3x, and be done.

Asking them to jump in every other day is not matching their lives and
the role that contributing to mozilla plays in those lives.

Human life also comes with various disruptions, like vacation, exams,
sickness, etc. If each of those need a mitigation strategy or degrade
the shipping product, we'll not win.

Axel

[Marc Schifer]

Well I wouldn't put it as "feeling positive" as much as we were feeling less negative than we were. We still have concerns over the volume of change coming into Aurora on a constant basis and our ability to keep up with the churn in development and execution of test plans on Aurora. We are also still concerned over the possibility of having a lesser quality Beta 1 which could drive off some of our Beta population. We don't have any data one way or another on how likely this would be, but it's something that worries us. Making sure there is time for L10N to do its thing is also a worry or us, (Worrying about things is just part of our nature as QA). V2 of this plan is not as scary for us as V1 was. The current model gives us the time to examine changes and have a good start at least on a test plans for everything we know about for the start of Aurora and Nightly. We have been working on moving our focus down to Aurora so we have more time with new features and are testing closer to the code changes than we have been doing on Beta. We are currently working on putting together a plan for what a QA cycle on this proposal would look like and what impacts, both positive and negative that it may have. Once we have that, we will have a bit more to say on the subject.

Marc S.

----- Original Message -----
From: "Alexander Keybl" <ake...@mozilla.com>
To: "jsmith mozilla" <jsmith....@gmail.com>
Cc: dev-pl...@lists.mozilla.org
Sent: Monday, November 4, 2013 6:06:20 AM
Subject: Re: Coupled Train Model v2

Jason,

We already met with Clint/Marc and the desktop QA team. I'll let them write their own opinions, but we left the meeting feeling positive about the impact to release quality and QA analyst focus.

-Alex

----- Original Message -----
From: jsmith mozilla <jsmith....@gmail.com>
To: dev-pl...@lists.mozilla.org
Sent: Fri, 01 Nov 2013 13:28:58 -0700 (PDT)
Subject: Re: Coupled Train Model v2