Fwd: TB Summary of release discussion

[Kent James]

At the recent Thunderbird summit in Warsaw, there were discussions about the future release cycle of Thunderbird. Wayne prepared the attached spreadsheet trying to summarize this complex subject, and there are also some meeting notes available at https://etherpad.mozilla.org/qIkqbtHTSV I'm going to try to clarify the discussion some, and reflect some understandings as well from an IRC discussion that we had today.

As a general summary, there are a number of parallel issues.

1) Rate of release of feature editions (mapped to Wayne's options)

Option 0: Currently, we do one release per six-week Gecko cycle/7 releases per ESR cycle. This is not going to happen beyond TB 17.

Option A: We only do ESR releases (1 release per ESR cycle)

Option B: We do 2 releases per ESR cycle. (This is the level of release that Mozilla commits to providing staff support for).

(not discussed by Wayne): We do more than two releases per ESR cycle. Standard8 in earlier memos discussed this as an option that would be possible with "community support". From the development perspective, the main issue is whether we allow intermediate releases at all, so if we do option B) it is only a resource question whether to add more intermediate releases in the future. So I will redefine B) to mean "two or more releases per ESR cycle".

Wayne listed an option C) and D) that mentioned some additional issues here, but in IRC we clarified those issues. This obsoletes the concept of a separate C) and D) option, and the issues discussed in C) and D) are discussed in the various issues below.

If we choose option A), then most other issues go away. That is, new work is landed on central (following current Gecko), central->aurora->beta->release transitions occur as now every 6 weeks (but the release channel does not get updated from the release repo every six weeks). The rate of new beta on the beta channel is reduced to about once per Gecko cycle except as we approach TB 24, when that rate may increase. The current release channel is identical to the current esr channel, but is maintained only to allow us to reconsider this decision later.

Assuming though that we choose B) and allow intermediate releases, a number of additional issues come up:

2) What Gecko repository do we use for intermediate feature releases?

Earlier there was some discussion about not using the Gecko ESR repository. I think there is a consensus that using the ESR Gecko repository for all future releases is the only viable approach.

3) Do we create a separate parallel intermediate feature branch only on demand and then backport desired features with an a+ patch approval, or do we create this branch immediately and land most new features simultaneously on both trunk and on the release branch (probably without needing an a+ approval initially)? Let me use an "fr" suffix for the feature release branch.

If we do the "create immediately" option, it seems to me the repo and channel flows would go like this:

central-fr ==> aurora-fr ==> beta ==> release
                               |        |
                               |        -> release update channel
                               -> beta update channel

central ==> aurora
   |            |
   |            -> aurora update channel (daily)
   -> daily update channel (daily)

After the last intermediate release, we abandon central-fr && aurora-fr, and update aurora==>beta

I'm less clear of the "on demand" variation. Perhaps like this:

central ==> aurora ==> beta (with l10n complete) => release
   |            |
   |            -> aurora update channel
   -> daily update channel

comm-fr is created on demand from comm-esr. Patches with a-tb-next+ are landed. It might be good to have an additional channel with completed localization, maybe earlybird-l10n, which would come from the beta

comm-fr
 |   |
 |   -> release update channel
 |-> beta update channel

DISCUSSION:

It seems to me that in either case, there is a serious deterioration in the available testing channels at a time when our paid staff support for support and QA will be dramatically reduced. In both options, there is no prior testing of the beta channel on the immediately upstream repo, so that channel will be less stable than in the past, which will drive testers away.

I think that we are going to be hard pressed to keep up with quality in the new model in any case. One path to failure would be if future releases start to have serious bugs and regressions. That would kill us faster than slow release of features. So I think that the quality risk, as well as the extra effort, of the intermediate feature release is not worth the benefit. I would rather encourage early adopters to use a beta channel which has new features instead, and maintain a stable product at release.

In either case, SeaMonkey would I assume use the release repo for their releases. The new model gives them the possibility to approve patches for that channel independently of TB, except near ESR releases.

:rkent

-------- Original Message --------

Subject:	TB Summary of release discussion
Date:	Fri, 07 Sep 2012 21:14:38 -0400
From:	Wayne Mery <wayne...@gmail.com>

Attached please find spreadsheet summarizing, I hope, all the options 
and very compactly the issues, pros and cons.  I have not attempted to 
include all comments for reasons of space, but they are very nicely 
noted in the etherpad https://etherpad.mozilla.org/qIkqbtHTSV by our 
wonderful note takers. Thanks to Kent and Mark who went over an early draft.

If I have missed any options or any important issues in the summary 
please let me know and I will amend the document.

W.

[Vincent]

Thanks for this post Kent.

I want to add that the feeling on the French forum is also that just 1 release a year is a good idea. No need for extra releases, especially if there is a beta channel which should be as stable as the current release channel (in my opinion)...
- Normal user will go to the ESR channel
- Advanced users will go to the beta channel (Thunderbird Feature Edition)

We should have less work on support's forum, because no new feature means less new questions. This is also a good point.

--
Vincent (caméléon)

[Kent James]

I have an alternate proposal for the release planning.

One issue that we are facing now, approaching ESR for TB 17, is that we
are discovering issues too late for any l10n work to be done. We are
going to face that issue much worse in the ESR TB-24 time frame.

So my proposal is that we do "intermediate releases" to the main release
channel starting at either TB 22 or TB 23. These would be releases from
the main central/aurora/beta/release repositories so would not need
additional repos with all of the complications of that.

Two big advantages:

1) We get one or two cycles of testing with the release channel before
we lock down a ESR release.

2) We get one or two cycles of l10n that we could use to fix any issues
that are discovered.

:rkent
_______________________________________________
tb-planning mailing list
tb-pl...@mozilla.org
https://mail.mozilla.org/listinfo/tb-planning

[Kent James]

On 9/14/2012 9:30 PM, Unicorn.Consulting wrote:

On 15/09/2012 1:42 PM, Kent James wrote:

I have an alternate proposal for the release planning.

So my proposal is that we do "intermediate releases" to the main release channel starting at either TB 22 or TB 23. These would be releases from the main central/aurora/beta/release repositories so would not need additional repos with all of the complications of that.

I think that an ESR release should not become an ESR release at the same time the general release occurs. ESR is really aimed at Business and they are in no hurry for new releases, so we do what business has been doing for almost as long as they have had computers, wait for the .1 release.  That is there will be a 17.1 ESR and a 24.1 ESR or whatever no time frame as such, but released reasonably soon after the main release with fixes as required, including a roleup of the now almost mandatory 0.1 and 0.2. Releases.  The general user base can come along for the ride to point one, but point one is the ESR release.

Matt

OK I like that - and it could be done for the 17 series as well.

So to interpret what I think you are saying in terms of repos and channels, we proceed with 17.0 as planned - but it is not the ESR release. We freeze mozilla-central in comm-aurora, release, and beta so that eventually release, beta, and aurora are all on gecko17. New work is landed in comm-central (with current gecko), and selected patches are landed in aurora (with a+) as needed in 17.1 (which will also be ESR). After 17.1 releases, we restart the central->aurora->beta migrations as now (but no new releases until 24.0 then 24.1 ESR)

So we do the allowed 1 intermediate release, but we do it immediately so that hopefully central and aurora are close enough together that the backport is easy.

If that is your proposal, I like it!

:rkent

[Unicorn.Consulting]

Your interpretation is it  I never really got the channels but it sounds exactly what I was trying to say.  The only one that roles along undaunted is nightly and the point 1 release never contains new features, only bug fixes (although they are one and the same sometimes)  I think also that it lives up to the intent better.   ESR is really aimed at those wanting tried and tested products, to deliver on that expectation we should not be offering new features to them at the time of release.

:rkent

_______________________________________________
tb-planning mailing list
tb-pl...@mozilla.org
https://mail.mozilla.org/listinfo/tb-planning

-- 
 “Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety.” Benjamin Franklin

[Mark Banner]

On 11/09/2012 23:13, Kent James wrote:
> Option B: We do 2 releases per ESR cycle. (This is the level of
> release that Mozilla commits to providing staff support for).

...

> Assuming though that we choose B) and allow intermediate releases, a
> number of additional issues come up:
>
> 2) What Gecko repository do we use for intermediate feature releases?

The thought here was that we would use something like a *-esr17.1 set of
repositories. The procedure would be to land on central and esr17.1 at
least 12 weeks before the intermediate release, which would give time
for l10n to do their work when the feature got to aurora.

The beta channel would be hijacked from the *-beta repositories onto the
*-esr17.1 ones straight after 17 is released, which would give the beta
coverage necessary for the features landing in the 17.1 repo. Once 17.1
is released, the beta channel would revert to the existing *-beta
repositories.

> DISCUSSION:
>
> It seems to me that in either case, there is a serious deterioration
> in the available testing channels at a time when our paid staff
> support for support and QA will be dramatically reduced. In both
> options, there is no prior testing of the beta channel on the
> immediately upstream repo, so that channel will be less stable than in
> the past, which will drive testers away.

The above model could be tweaked, so that the feature/function would
only land on the esr17.1 repos after it has migrated to aurora/beta in
the main repositories, which I think would alleviate some of these issues.

> I think that we are going to be hard pressed to keep up with quality
> in the new model in any case. One path to failure would be if future
> releases start to have serious bugs and regressions. That would kill

> us faster than slow release of features.So I think that the quality

> risk, as well as the extra effort, of the intermediate feature release
> is not worth the benefit. I would rather encourage early adopters to
> use a beta channel which has new features instead, and maintain a
> stable product at release.

Personally, I think the beta should only be recommended for testing new
features, as has been previously mentioned - we need to be careful to
set the expectations of quality correctly.

Whilst regressions during an intermediate release would concern me, I'd
also expect there to be a significantly smaller amount of changes
compared to a new gecko-based release, especially as we can do some
picking and choosing of what we believe will be stable.

Mark.

[Mark Banner]

On 15/09/2012 06:14, Kent James wrote:

On 9/14/2012 9:30 PM, Unicorn.Consulting wrote:

On 15/09/2012 1:42 PM, Kent James wrote:

I have an alternate proposal for the release planning.

So my proposal is that we do "intermediate releases" to the main release channel starting at either TB 22 or TB 23. These would be releases from the main central/aurora/beta/release repositories so would not need additional repos with all of the complications of that.

Assuming these are based on the equivalent Gecko versions, then we would also need to consider back-porting of the core security fixes to those releases for each of the .0.1 releases.

I think that an ESR release should not become an ESR release at the same time the general release occurs. ESR is really aimed at Business and they are in no hurry for new releases, so we do what business has been doing for almost as long as they have had computers, wait for the .1 release.  That is there will be a 17.1 ESR and a 24.1 ESR or whatever no time frame as such, but released reasonably soon after the main release with fixes as required, including a roleup of the now almost mandatory 0.1 and 0.2. Releases.  The general user base can come along for the ride to point one, but point one is the ESR release.

ESR has been designed with new features at the time of the ESR being considered. There is an overlap of 2 releases on the ESR channel to allow organisations time to switch over and for any critical issues to be resolved. The ESR FAQ has a diagram showing this overlap. Hence, this allows for the .0.1 and .0.2 releases.

So to interpret what I think you are saying in terms of repos and channels, we proceed with 17.0 as planned - but it is not the ESR release. We freeze mozilla-central in comm-aurora, release, and beta so that eventually release, beta, and aurora are all on gecko17. New work is landed in comm-central (with current gecko), and selected patches are landed in aurora (with a+) as needed in 17.1 (which will also be ESR). After 17.1 releases, we restart the central->aurora->beta migrations as now (but no new releases until 24.0 then 24.1 ESR)

I think my other message to this thread covers this slightly differently. What I think I could do with though, is a clear statement of the problem you're trying to solve here, as this doesn't just sound like just allowing an intermediate release of features. Earlier in the thread you mentioned there's issues fixing bugs due to L10n. Are these flagged as tracking-thunderbird17? Can you also provide some examples?

Thanks
Mark.

[Gervase Markham]

On 18/09/12 09:00, Mark Banner wrote:
> On 11/09/2012 23:13, Kent James wrote:
>> Option B: We do 2 releases per ESR cycle. (This is the level of
>> release that Mozilla commits to providing staff support for).
> ...
>> Assuming though that we choose B) and allow intermediate releases, a
>> number of additional issues come up:
>>
>> 2) What Gecko repository do we use for intermediate feature releases?

At yesterday's MoCo meeting, it was suggested that B2G are going to need
to do releases faster than ESR but slower than rapid release. It _could_
be that this will result in longer-lived maintained Gecko branches that
we could use.

Of course, it could be that those branches carry too many B2G-specific
hacks for us to use. Just an idea...

Gerv

[Kent James]

On 9/18/2012 1:11 AM, Mark Banner wrote:

On 15/09/2012 06:14, Kent James wrote:

On 9/14/2012 9:30 PM, Unicorn.Consulting wrote:

On 15/09/2012 1:42 PM, Kent James wrote:

I have an alternate proposal for the release planning.

So my proposal is that we do "intermediate releases" to the main release channel starting at either TB 22 or TB 23. These would be releases from the main central/aurora/beta/release repositories so would not need additional repos with all of the complications of that.

Assuming these are based on the equivalent Gecko versions, then we would also need to consider back-porting of the core security fixes to those releases for each of the .0.1 releases.

At the moment I prefer Matt's proposal and not this one. But in defense of my old position, all that I was suggesting is that we do the "intermediate release" one or two cycles prior to the ESR release, and that we use the same procedures that we have in place today to do those. I would do a TB22 and TB 23 or just a TB 23.  In neither case do we need backporting, just like we don't need backporting today - we just start the old rapid release train one or two releases before ESR.

I think that an ESR release should not become an ESR release at the same time the general release occurs. ESR is really aimed at Business and they are in no hurry for new releases, so we do what business has been doing for almost as long as they have had computers, wait for the .1 release.  That is there will be a 17.1 ESR and a 24.1 ESR or whatever no time frame as such, but released reasonably soon after the main release with fixes as required, including a roleup of the now almost mandatory 0.1 and 0.2. Releases.  The general user base can come along for the ride to point one, but point one is the ESR release.

ESR has been designed with new features at the time of the ESR being considered. There is an overlap of 2 releases on the ESR channel to allow organisations time to switch over and for any critical issues to be resolved. The ESR FAQ has a diagram showing this overlap. Hence, this allows for the .0.1 and .0.2 releases.

I think that the concern that Matt and I are showing is that each new *. release seems to come with a few new critical issues that are not resolved until the *.0.1 or *.0.2 release.  Currently ESR 17 is the same as TB17 and has the same initial instability.

Why is that needed? Why not delay the ESR release relative to the main release until some period of time has elapsed for those instabilities to get worked out? ESR has no rush to it.

So to interpret what I think you are saying in terms of repos and channels, we proceed with 17.0 as planned - but it is not the ESR release. We freeze mozilla-central in comm-aurora, release, and beta so that eventually release, beta, and aurora are all on gecko17. New work is landed in comm-central (with current gecko), and selected patches are landed in aurora (with a+) as needed in 17.1 (which will also be ESR). After 17.1 releases, we restart the central->aurora->beta migrations as now (but no new releases until 24.0 then 24.1 ESR)

I think my other message to this thread covers this slightly differently. What I think I could do with though, is a clear statement of the problem you're trying to solve here, as this doesn't just sound like just allowing an intermediate release of features. Earlier in the thread you mentioned there's issues fixing bugs due to L10n. Are these flagged as tracking-thunderbird17? Can you also provide some examples?

The problem we are trying to solve is preventing the *.0.0 instabilities from hitting the ESR channel, so that ESR users do not have to go through a period of pain before the release stabilizes. Matt's proposal does this better than my initial proposal.

The current example of a bug where potentially better fixes were not considered due to L10n constraints is "[Bug 791311] Don't disable the menubar for existing users", thought that is just a current example. The more general point is that we should not let L10n constraints prevent us from presenting ESR users with the optimal fixes for bugs. That may or may not be an issue in the current ESR release, if not then ESR could happen at 17.0.1 or 17.0.2

Maybe you are doing that anyway.

:rkent

[Mark Banner]

On 18/09/2012 17:03, Kent James wrote:

On 9/18/2012 1:11 AM, Mark Banner wrote:

On 15/09/2012 06:14, Kent James wrote:

On 9/14/2012 9:30 PM, Unicorn.Consulting wrote:

On 15/09/2012 1:42 PM, Kent James wrote:

I think that an ESR release should not become an ESR release at the same time the general release occurs. ESR is really aimed at Business and they are in no hurry for new releases, so we do what business has been doing for almost as long as they have had computers, wait for the .1 release.  That is there will be a 17.1 ESR and a 24.1 ESR or whatever no time frame as such, but released reasonably soon after the main release with fixes as required, including a roleup of the now almost mandatory 0.1 and 0.2. Releases.  The general user base can come along for the ride to point one, but point one is the ESR release.

ESR has been designed with new features at the time of the ESR being considered. There is an overlap of 2 releases on the ESR channel to allow organisations time to switch over and for any critical issues to be resolved. The ESR FAQ has a diagram showing this overlap. Hence, this allows for the .0.1 and .0.2 releases.

I think that the concern that Matt and I are showing is that each new *. release seems to come with a few new critical issues that are not resolved until the *.0.1 or *.0.2 release.  Currently ESR 17 is the same as TB17 and has the same initial instability.

Why is that needed? Why not delay the ESR release relative to the main release until some period of time has elapsed for those instabilities to get worked out? ESR has no rush to it.

I can understand the concern, but I don't believe it is really critical to ESR. Despite our mentions about testing the changes, I'm pretty sure some organizations won't start widely testing until the next ESR is actually released. So if there are issues that affect them specifically, they may not get reported until we release the ESR. If we delayed releasing that ESR for a couple of point releases, then they would be in the case where the old ESR version would be unsupported, but they might not be able to upgrade to the new version due to the issues.

The overlap is there for exactly this reason - upgrade straight away if you can, but if not, you've got 12 weeks to resolve any issues (or get us to resolve them).

The problem we are trying to solve is preventing the *.0.0 instabilities from hitting the ESR channel, so that ESR users do not have to go through a period of pain before the release stabilizes. Matt's proposal does this better than my initial proposal.

Something else to point out here I think, is that when we release ESR 17.0, only syadmins that choose to upgrade their uses to the new ESR version will do so. As there will be two more ESR 10.0 releases (one in sync with ESR 17, the second in sync with 17.0.1), I don't see us prompting 10.0 users to do a major upgrade until 17.0.2 is released and 10.0.x is obsolete.

The current example of a bug where potentially better fixes were not considered due to L10n constraints is "[Bug 791311] Don't disable the menubar for existing users", thought that is just a current example. The more general point is that we should not let L10n constraints prevent us from presenting ESR users with the optimal fixes for bugs. That may or may not be an issue in the current ESR release, if not then ESR could happen at 17.0.1 or 17.0.2

Just as an example, for specifically for an ESR, we might want to have an option whereby sysadmins can choose that Thunderbird does not automatically remove the menubars (most sysamdins have set-ups that allow them to set default prefs etc). We've done this type of thing before for them and it may be appropriate here.

Mark.

[Kent James]

On 9/19/2012 1:31 AM, Mark Banner wrote:

On 18/09/2012 17:03, Kent James wrote:

I think that the concern that Matt and I are showing is that each new *. release seems to come with a few new critical issues that are not resolved until the *.0.1 or *.0.2 release.  Currently ESR 17 is the same as TB17 and has the same initial instability.

Why is that needed? Why not delay the ESR release relative to the main release until some period of time has elapsed for those instabilities to get worked out? ESR has no rush to it.

I can understand the concern, but I don't believe it is really critical to ESR. Despite our mentions about testing the changes, I'm pretty sure some organizations won't start widely testing until the next ESR is actually released. So if there are issues that affect them specifically, they may not get reported until we release the ESR. If we delayed releasing that ESR for a couple of point releases, then they would be in the case where the old ESR version would be unsupported, but they might not be able to upgrade to the new version due to the issues.

The overlap is there for exactly this reason - upgrade straight away if you can, but if not, you've got 12 weeks to resolve any issues (or get us to resolve them).

That's a good point. I had not thought of the issues of the overlap.

The problem we are trying to solve is preventing the *.0.0 instabilities from hitting the ESR channel, so that ESR users do not have to go through a period of pain before the release stabilizes. Matt's proposal does this better than my initial proposal.

Something else to point out here I think, is that when we release ESR 17.0, only syadmins that choose to upgrade their uses to the new ESR version will do so. As there will be two more ESR 10.0 releases (one in sync with ESR 17, the second in sync with 17.0.1), I don't see us prompting 10.0 users to do a major upgrade until 17.0.2 is released and 10.0.x is obsolete.

OK that's news to me, thanks for pointing that out.

With this discussion,  I think that I am going back to my earlier position, that the best way to use an intermediate release would be to release a Thunderbird 23 or 22, so that we would get some additional user feedback on issues before we get locked in for another year.

What do you think, Matt?

:rkent

[Unicorn.Consulting]

I have no real objection.

:rkent

_______________________________________________
tb-planning mailing list
tb-pl...@mozilla.org
https://mail.mozilla.org/listinfo/tb-planning

[Ludovic Hirlimann]

On 9/20/12 1:10 AM, Unicorn.Consulting wrote:

On 20/09/2012 12:29 AM, Kent James wrote:

On 9/19/2012 1:31 AM, Mark Banner wrote:

On 18/09/2012 17:03, Kent James wrote:

I think that the concern that Matt and I are showing is that each new *. release seems to come with a few new critical issues that are not resolved until the *.0.1 or *.0.2 release.  Currently ESR 17 is the same as TB17 and has the same initial instability.

Why is that needed? Why not delay the ESR release relative to the main release until some period of time has elapsed for those instabilities to get worked out? ESR has no rush to it.

I can understand the concern, but I don't believe it is really critical to ESR. Despite our mentions about testing the changes, I'm pretty sure some organizations won't start widely testing until the next ESR is actually released. So if there are issues that affect them specifically, they may not get reported until we release the ESR. If we delayed releasing that ESR for a couple of point releases, then they would be in the case where the old ESR version would be unsupported, but they might not be able to upgrade to the new version due to the issues.

The overlap is there for exactly this reason - upgrade straight away if you can, but if not, you've got 12 weeks to resolve any issues (or get us to resolve them).

That's a good point. I had not thought of the issues of the overlap.

The problem we are trying to solve is preventing the *.0.0 instabilities from hitting the ESR channel, so that ESR users do not have to go through a period of pain before the release stabilizes. Matt's proposal does this better than my initial proposal.

Something else to point out here I think, is that when we release ESR 17.0, only syadmins that choose to upgrade their uses to the new ESR version will do so. As there will be two more ESR 10.0 releases (one in sync with ESR 17, the second in sync with 17.0.1), I don't see us prompting 10.0 users to do a major upgrade until 17.0.2 is released and 10.0.x is obsolete.

OK that's news to me, thanks for pointing that out.

With this discussion,  I think that I am going back to my earlier position, that the best way to use an intermediate release would be to release a Thunderbird 23 or 22, so that we would get some additional user feedback on issues before we get locked in for another year.

What do you think, Matt?

I have no real objection.

So you guys aren't worried that 6 or 12 weeks would be very short to gather feedback bubble it up and have things fixed ? I'd rather have 20 or 21 as a target train.

-- 
@lhirlimann on twitter
https://wiki.mozilla.org/Thunderbird:Testing

my photos http://www.flickr.com/photos/lhirlimann/collections/

[Kent James]

Oh yes, I'm worried. I think I've said before that the future of Thunderbird will be determined by Thunderbird 24, and there are reasons to worry.

But it is also true that we are constrained by resources, and releases take a lot of resources. Also we need to see the effect of Gecko changes to really get stable, and those arrive late. So given those constraints, I think that we will have to see in the 20 or 21 time frame where we stand before we can decide - and also leave open the possibility of a 24.1 if needed.

But the main point in all of this is that if we can only do one or a few intermediate releases, it is probably better to use those releases to improve that stability of the annual release, rather than to push features to users a little earlier.

:rkent