Some of you may have noticed that we're doing something a little bit different with our release candidate milestones for Firefox 3.5. I'd like to take a brief moment to explain what we're doing, why, and what the implications are for release dates.
// Who are our beta testers?
Our beta testing audience (defined as users who have downloaded a beta release of Firefox and thus put themselves on the "beta" update channel) is big; over 800,000 active daily users. These individuals have accepted the risks of running beta software, agreeing to help us by testing the breadth of the web with a stable version of the browser that, potentially, contains a few bugs.
// How do we make a development milestone, anyway?
Historically, we prepare development milestones (betas or release candidates) more or less as follows:
1. "freeze" code development 2. identify the last code change and use it as the basis for milestone builds 3. build the milestone on all operating systems 4. sign the builds on windows 5. generate update snippets 6. run a set of quality assurance tests on the builds 7. ship partial updates to beta-channel users and change the beta download page to point to the new development milestone
// So, what's new and exciting?
We've realized that we can run this process in a more parallel fashion by splitting step 7 into:
7.1 ship partial updates to beta-channel users 7.2 change the beta download page to point to the new development milestone
We'll be doing this for the Firefox 3.5 release candidate builds. As soon as QA performs basic smoketests and update tests on the milestone, we'll ship the partial updates to our existing beta-channel users. We feel confident in doing so since our nightly testing audience and unit test infrastructure have shown that by this point, the vast majority of serious stability bugs have already been identified and mitigated, so we know we're not putting our beta audience at any increased risk.
Then, while our quality assurance team finishes more in depth functional testing, our large beta audience will be helping us by using the builds across the breadth of the web. We'll get wider feedback earlier, giving us even more confidence before we declare a milestone as final. Once QA - and in effect our beta-audience - sign off on this testing phase, we can update the beta download page.
The release drivers feel that this parallelization helps us get the most out of our beta community, increasing the quality of our development milestones faster. Further, at this release candidate stage, it allows users to decide if they want to be beta testers or not (release candidate builds use the standard product update channel, not the beta channel)
// What does this mean for the upcoming Firefox 3.5 release candidate?
We've just released partial updates for Firefox 3.5 beta users which will update them to an early version of the release candidate (the exact version reference is rc1build2: this means it is the first revision of the release candidate for which we'll ship updates, and it required two build attempts). This build is *not* being released on the beta download page; if someone out there wants to get at it, they should first become a beta tester by downloading a beta and then they will receive the updates.
As we find issues with the release candidate, we will fix and rebuild. We're already generating a new version (rc2build2) with several small, localized fixes. We can do this because the small-scope nature of the fixes means that we can re-run a segment of the quality assurance tests instead of starting over from scratch.
Our target is still to release the most up to date version of the release candidate for download on our website this Friday. It really depends on what we find between now and then, but with 800,000 people looking, it means that if nothing is found, we can be extremely confident of the quality of the release candidate code!
// I'm confused: what's the schedule?
June 16th: Firefox 3.5 beta users get updated to an early release candidate version June 19th: target public ship date for Firefox 3.5 Release Candidate 1 end of June: target ship date for Firefox 3.5
> June 16th: Firefox 3.5 beta users get updated to an early release > candidate version > June 19th: target public ship date for Firefox 3.5 Release Candidate 1 > end of June: target ship date for Firefox 3.5
So if RC1 is released in two days and 3.5 final at the end of the month, when will RC2 be released?? Are you saying that the current rc2buildX will never end as RC2 but as final directly?
> Le 17. 06. 09 11:16, Mike Beltzner a écrit : >> June 16th: Firefox 3.5 beta users get updated to an early release >> candidate version >> June 19th: target public ship date for Firefox 3.5 Release >> Candidate 1 >> end of June: target ship date for Firefox 3.5
> So if RC1 is released in two days and 3.5 final at the end of the > month, when will RC2 be released?? Are you saying that the current > rc2buildX will never end as RC2 but as final directly?
I wouldn't worry too much about the build reference versions as I've published them above in this note. Version numbering here gets confusing, really.
On June 19th, we hope to release (on our website, as a full download) a release candidate. Right now that would be the rc2build2 version, as it's the latest thing we have. So, in other words:
June 16th: Firefox 3.5 beta users get updated to an early release candidate (rc1build2) June 19th: Firefox 3.5 beta users get updated to the official release candidate, which is also put up for direct download (right now this would be rc2build2, but that may yet change!) end of June: target ship date for Firefox 3.5
> On 17-Jun-09, at 2:27 AM, Fr d ric Buclin wrote:
>> Le 17. 06. 09 11:16, Mike Beltzner a crit : >>> June 16th: Firefox 3.5 beta users get updated to an early release >>> candidate version >>> June 19th: target public ship date for Firefox 3.5 Release Candidate 1 >>> end of June: target ship date for Firefox 3.5
>> So if RC1 is released in two days and 3.5 final at the end of the >> month, when will RC2 be released?? Are you saying that the current >> rc2buildX will never end as RC2 but as final directly?
> I wouldn't worry too much about the build reference versions as I've > published them above in this note. Version numbering here gets > confusing, really.
Indeed; I thought I was reading that we already know there will be an RC2, but I see now that RC1 >= rc2build2. Thanks for clearing that up!
... ... This build is *not* being released on the beta download
> page; if someone out there wants to get at it, they should first become > a beta tester by downloading a beta and then they will receive the updates.
...
Perhaps some can explain this sentence to me. To me it says: "The build cannot be downloaded; if you want to download it, first download it."
I don't get the whole idea anyway. If I have a beta, I get the updates asap. If I don't have a beta, I can't get the updates asap (duh). So the only thing new here is that if I don't have a beta I also can't get one? How is that new and exciting?
> Mike Beltzner wrote: > ... > ... This build is *not* being released on the beta download >> page; if someone out there wants to get at it, they should first >> become a beta tester by downloading a beta and then they will >> receive the updates. > ...
> Perhaps some can explain this sentence to me. To me it says: "The > build cannot be downloaded; if you want to download it, first > download it."
It means that the build which we are shipping as an update to existing beta users (via the built in Firefox software update mechanism) is not yet available at http://www.mozilla.com/firefox/all-beta.html
The primary reason we're doing things this way, as explained in the original post, is to get our 800,000+ beta users helping us on daily testing as part of our QA phase on a milestone
> I don't get the whole idea anyway. If I have a beta, I get the > updates asap. If I don't have a beta, I can't get the updates asap > (duh). So the only thing new here is that if I don't have a beta I > also can't get one? How is that new and exciting?
The new thing here is that while (in the scenario you posit) you would get an update asap, you would not be able to download it from the Firefox Beta Download web page, yes.
I find it new and exciting as a process change. Others find it slightly odd. I'm glad that you find it warm, comforting and not disturbing. It means we can move on.
Mike Beltzner wrote: > On 17-Jun-09, at 8:31 AM, John J. Barton wrote:
>> Mike Beltzner wrote: >> ... >> ... This build is *not* being released on the beta download >>> page; if someone out there wants to get at it, they should first >>> become a beta tester by downloading a beta and then they will receive >>> the updates. >> ...
>> Perhaps some can explain this sentence to me. To me it says: "The >> build cannot be downloaded; if you want to download it, first download >> it."
> It means that the build which we are shipping as an update to existing > beta users (via the built in Firefox software update mechanism) is not > yet available at http://www.mozilla.com/firefox/all-beta.html
> The primary reason we're doing things this way, as explained in the > original post, is to get our 800,000+ beta users helping us on daily > testing as part of our QA phase on a milestone
FF3.5b4 and FF3.5b99 did not work with Firebug. So we found a nightly build from June 5 that worked and directed our users to that build. I would like to direct them to a newer build. I tested the RC1 build2 last night and it looks great. So I posted the link in our blog. Then I was told that I should not do that. So I am disappointed that there is not a way for Firebug users to get a newer beta as soon as possible. If 800,000 is good, wouldn't a few more just be better?
> FF3.5b4 and FF3.5b99 did not work with Firebug. So we found a > nightly build from June 5 that worked and directed our users to that > build. I would like to direct them to a newer build. I tested the > RC1 build2 last night and it looks great. So I posted the link in > our blog. Then I was told that I should not do that. So I am > disappointed that there is not a way for Firebug users to get a > newer beta as soon as possible. If 800,000 is good, wouldn't a few > more just be better?
I don't know who told you not to point your own testing audience to a specific build. We ask that people don't reference a specific build on the FTP site and say "this is a beta, go get it!" or the like, but that's mostly to protect our servers from overload. I don't think that will be a problem in the case you mention, though.
Be aware that if you point people to a nightly build, they'll start getting nightly updates. If you point people to that RC build, they won't be on the beta channel, they'll be on the release channel.
For your general user audience, I'd suggest you wait a couple of days until all-beta is updated with a pointer to the release candidate. Or, if you want to advise that they become Firefox beta testers, have them download Beta 4 and check for updates.
Neither of these things seem particularly hard to do, if you are willing.
Mike Beltzner wrote: > On 17-Jun-09, at 8:56 AM, John J. Barton wrote: > For your general user audience, I'd suggest you wait a couple of days > until all-beta is updated with a pointer to the release candidate. Or, > if you want to advise that they become Firefox beta testers, have them > download Beta 4 and check for updates.
I updated my advice with this information.
> Neither of these things seem particularly hard to do, if you are willing.
I know all of this stuff seems easy for you. But I could not figure out from your original post how to get a copy of the release candidate. Now I see that you can update 800,000 users in few minutes but to update the web page takes several days. To avoid waiting on the web page, users can load the old build and update it.
It's funny that browsers and software updates are getting faster but web page updates are getting slower.
> Mike Beltzner wrote: > > On 17-Jun-09, at 8:56 AM, John J. Barton wrote: > > For your general user audience, I'd suggest you wait a couple of days > > until all-beta is updated with a pointer to the release candidate. Or, > > if you want to advise that they become Firefox beta testers, have them > > download Beta 4 and check for updates.
> I updated my advice with this information.
> > Neither of these things seem particularly hard to do, if you are willing.
> I know all of this stuff seems easy for you. But I could not figure out > from your original post how to get a copy of the release candidate. Now > I see that you can update 800,000 users in few minutes but to update the > web page takes several days. To avoid waiting on the web page, users can > load the old build and update it.
> It's funny that browsers and software updates are getting faster but web > page updates are getting slower.
> jjb
Got it. Now this all makes sense. I will mainly be testing -moz-border- radius and-moz-box-shadow, so I will report with any bugs.
I guess it's good that our existing beta users are getting new builds more frequently. But why not also put those builds on the web page? Is putting new builds on the web page way more work than it should be?
----- Original Message ----- From: dev-planning-bounces+beltzner=mozilla....@lists.mozilla.org
<dev-planning-bounces+beltzner=mozilla....@lists.mozilla.org> To: dev-plann...@lists.mozilla.org <dev-plann...@lists.mozilla.org> Sent: Wed Jun 17 16:03:28 2009 Subject: Re: Firefox 3.5 release candidate builds: new and exciting!
I guess it's good that our existing beta users are getting new builds more frequently. But why not also put those builds on the web page? Is putting new builds on the web page way more work than it should be? _______________________________________________ dev-planning mailing list dev-plann...@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-planning
We're comfortable shipping it to 800,000 users who downloaded the last beta, but not to the next 200,000 users who come by to download the latest beta? This doesn't make sense to me. Is it just a PR thing?
This isn't a beta, it's a release candidate. There's a different expectation of quality, and more importantly, when we publish such a release we are effectively saying: "this is it; this is the version that we believe is Firefox 3.5"
So, yes, we're unwilling to take that step until we've finished all QA on the product.
Further, we're comfortable sending an incremental update to our beta testing audience, who have effectively agreed to receive in- development software; we do believe that the update we delivered this morning is at the quality level of our previous beta releases. There is additional quality assurance testing, though, that we perform on a release candidate as compared to a beta. We would like our beta audience to help with that additional testing.
Also, we already know that the final version of the release candidate (rc2build2 is presently the latest revision) will be slightly different than the update we just shipped (rc1build2). So I don't think we can state "this is our release candidate of Firefox 3.5." We know that it isn't yet. What we can, and have stated, is that this is an early version of our release candidate.
All of this - all of it - was in the original email I sent. I encourage you to re-read it.
cheers, mike
ps: I find your comment about this new mechanism being "just a PR thing" incredibly offensive. Not just to me, but to our PR team at Mozilla, who are the most open, honest and forthright group you'll ever care to meet.
> We're comfortable shipping it to 800,000 users who downloaded the last > beta, but not to the next 200,000 users who come by to download the > latest beta? This doesn't make sense to me. Is it just a PR thing? > _______________________________________________ > dev-planning mailing list > dev-plann...@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-planning
> It means that the build which we are shipping as an update to existing > beta users (via the built in Firefox software update mechanism) is not > yet available at http://www.mozilla.com/firefox/all-beta.html
> The primary reason we're doing things this way, as explained in the > original post, is to get our 800,000+ beta users helping us on daily > testing as part of our QA phase on a milestone
At least the terminology, if not the idea, seems a bit odd here (which I thought when I saw the first-run page text as well). "Our basic QA isn't done yet, but we've automatically updated you to this pre-release release candidate build anyway."
If you've shipped it to 800,000 users, is it really not a "milestone" or a "release"? (Although I guess maybe the idea is to avoid having "release candidate 1" on a web page which will get slashdotted and have 3,000,000 people run it, is that it?)
You said in the top post that the thing that is shipped in a couple of days will be "release candidate 1". How does that make sense when it will come after (or be identical to) "release candidate 2 build 2"?
If rc1build2 was a released release candidate, then wouldn't the next release have to be "release candidate 2". And if rc1build2 wasn't a release, then shouldn't rc2build1 have been rc1build3?
I guess the answer may be that the process was worked out while it was happening and the numbering happened before the sequence was decided, in which case I think I understand everything now you've posted about it, so that's fine.
Ok, I think I understand why ship this almost-release-candidate build to existing beta users but don't put it on the beta page. Giving it to existing beta users is a clear win; I think we all agree about that. But listing it on the beta page would cause confusion:
* Calling it "beta 7" would make some people expect new features, make others wonder why we're having so many betas, and confuse everyone when we recast the build as a release candidate.
* Calling it "release candidate 1 build 1" or something along those lines is confusing, and prevents users from joining the beta channel.
* Calling it "the latest beta" is vague.
* Additionally, it has been through slightly less QA (so far) than a normal beta, so it might be ok to give it to existing beta users (e.g. because they opted for an early beta) but not new beta users.
So, we don't list it on the beta page. This causes some confusion of its own, and some people end up testing an older build than we'd like, but neither of these problems is huge. It's also a great chance to shake out updater bugs: if updating fails for a small percentage of users, we can be pretty sure they'll complain.
Jesse Ruderman wrote: > Ok, I think I understand why ship this almost-release-candidate build > to existing beta users but don't put it on the beta page. Giving it > to existing beta users is a clear win; I think we all agree about > that. But listing it on the beta page would cause confusion:
> * Calling it "beta 7" would make some people expect new features, make > others wonder why we're having so many betas, and confuse everyone > when we recast the build as a release candidate.
Next time consider calling it beta 7. Then you can educate users that 'beta' means "fixing bugs, not adding features", and "more betas is better than less betas, because it means the final release will have high quality". Plus no one gets confused because there is no release candidate.
> * Calling it "release candidate 1 build 1" or something along those > lines is confusing, and prevents users from joining the beta channel.
So don't.
> * Calling it "the latest beta" is vague.
So don't.
> So, we don't list it on the beta page.
But if was called b7 you could list it. And not have all of this.
Barton<johnjbar...@johnjbarton.com> wrote: > Next time consider calling it beta 7. Then you can educate users that 'beta' > means "fixing bugs, not adding features", and "more betas is better than > less betas, because it means the final release will have high quality". Plus > no one gets confused because there is no release candidate.
We *did* consider calling it a beta, and decided not to. One reason is that we don't want people evaluating it as "beta software" -- they should be holding it to the same standards as released software in terms of what they would report as problems, and we have seen an unfortunate number of "I would have reported it, but I figured it was beta so I'd see if it just got fixed next time" on bugs we would have rather heard about earlier.
Linus talks about this phenomenon periodically, as it affects the Linux kernel -- you don't get that last push of testing until it's declared as "done", so we're trying to walk the right line between "testers, please be very picky" and "world, we're not yet ready to call it Firefox 3.5". I think we've made a *huge* step forward in letting our beta testers be closer to our key endgame QA cycle here, naming nitpicks aside.
Mike Beltzner wrote: > On 17-Jun-09, at 8:56 AM, John J. Barton wrote: ... > For your general user audience, I'd suggest you wait a couple of days > until all-beta is updated with a pointer to the release candidate. Or, > if you want to advise that they become Firefox beta testers, have them > download Beta 4 and check for updates.
I've read all of this thread again. I still can't make out what the deal is.
If we go to the all-beta site now we become beta testers but we get the RC build by check-for-updates. That much I tried and it works. A bit of silly extra pointless work but whose counting.
If we wait a few days and go to the all-beta site we get the RC build but don't become beta-testers? So the all-beta becomes the site for RC download at that point and the beta program closes? This does not mesh with the whole story of how great the beta program is, so I don't think that is what you meant to say. I guess.
John J. Barton wrote: > If we wait a few days and go to the all-beta site we get the RC build > but don't become beta-testers? So the all-beta becomes the site for RC > download at that point and the beta program closes? This does not mesh > with the whole story of how great the beta program is,
The beta program is great, but it's not for everyone. Since the point of a release "candidate" is to become the actual release (if successful) then the candidate bits can't have "beta" baked into the channel-prefs.js file.
The beta program continues for those already signed up for betas -- we'll use them for testing early releases of security updates. But signing up for the betas at that point will be more cumbersome, you'll have to really mean it. This isn't any different from how we're handling Firefox 3.0 -- you can still get onto the Firefox 3.0.x beta program, but you have to either go download a released beta, manually edit your channel-prefs.js file, or install the update channel selector addon.
Daniel Veditz wrote: > John J. Barton wrote: >> If we wait a few days and go to the all-beta site we get the RC build >> but don't become beta-testers? So the all-beta becomes the site for RC >> download at that point and the beta program closes? This does not mesh >> with the whole story of how great the beta program is,
> The beta program is great, but it's not for everyone. Since the point of > a release "candidate" is to become the actual release (if successful) > then the candidate bits can't have "beta" baked into the > channel-prefs.js file.
> The beta program continues for those already signed up for betas -- > we'll use them for testing early releases of security updates. But > signing up for the betas at that point will be more cumbersome, you'll > have to really mean it. This isn't any different from how we're handling > Firefox 3.0 -- you can still get onto the Firefox 3.0.x beta program, > but you have to either go download a released beta, manually edit your > channel-prefs.js file, or install the update channel selector addon.
Thanks, but we've already determined this stuff is too complicated for me to understand.
All I really want to know is what link I should give to Firebug users when I ask them to try 3.5. I want to do that as soon as possible so we can get some testing of Firebug 1.4 on Firefox 3.5. I thought that was possible now, but I'll just wait.
