64-bit Windows Nightlies

[Ed Morley]

64-bit MSVC Windows builds are only tier 2 [1], but yet over half of our Windows Nightly users are using them [2], meaning that they are:

* Not testing the actual bits shipped to aurora, beta or release channel Windows users [3], thereby reducing their use to us. In some cases they are unable to test entire features, for example the Mozilla Maintenance Service [4].

* Subject to significant performance regressions compared to 32-bit builds - which would have resulted in backouts had they occurred on any other platform. In the last 3 weeks alone, V8 regressed by 80% [5] and SunSpider by 20% [6].


Additional problems:

* Mozilla-central is the only tree that runs tests on Win64 - which with the increased usage of mozilla-inbound, means that regression ranges can be as large as a 100 changeset merge [7] and so are time-consuming to investigate (i.e.: often aren't).

* Win64 tests are hidden by default on TBPL (since it is not reasonable to back out an entire merge just because it originated from a tree that doesn't run those tests), so both intermittent and permanent oranges are being ignored.

* Amazingly, there is still the perception by some Windows Nightly users that the 64-bit MSVC builds are faster than their 32-bit counterparts [8], whereas even before the recent regressions that was not the case [5] [6].

* Users of 64-bit versions of Windows may be under the impression that 64-bit builds of Nightly are somehow "more correct" for their platform, especially since http://nightly.mozilla.org displays 32-bit and 64-bit Windows Nightly builds with equal prominence and no indication that the 64-bit builds are experimental/effectively untested.

* Whilst there was a thorough discussion of the Win64 pros/cons [9], a follow-up summary of facts [10], and internally most of us know that Win64 builds aren't something that we'll be focusing on near-term - there doesn't appear to have been any public newsgroup/blog postings communicating the final decision. So it's not surprising that the average Nightly user is not aware that Win64 builds are pretty much unsuited for public consumption.

--

As a result, I am concerned that we're not only shooting ourselves in the foot with respects to 32-bit Nightly testing coverage, but also doing over half of our Windows Nightly users a disservice by shipping effectively untested and likely slower builds - without many of them even being aware of it.

Ok, so what now?

Option A:

* Alter http://nightly.mozilla.org such that 64-bit Windows builds are clearly marked as experimental & link to newsgroup/blog posts with an explanation.
* Make the Win64 decision more widely visible using newsgroups and a Planet blog post (though I imagine this thread will cover the former).
* Consider reaching out to the current Win64 Nightly users using a custom 'What's New' page advising that they are using a tier 2 build and the implications of that.
* Either dedicate the resources so that we can run Win64 tests on all trees (and eventually unhide them), or else switch off Win64 tests on mozilla-central for now, since they are currently hidden and taking up slaves for little benefit.

Option B: (my personal preference)

* Switch off Nightly Win64 builds (at least for the short to medium term).
* Still do per-push tinderbox Win64 builds to make sure we don't break anything major, but switch off the all tests (if we're not going to run them on all trees).
* Remove the Win64 links on http://nightly.mozilla.org.
* Transition Win64 Nightly users to 32-bit Nightly builds, with an unprompted update and an appropriate 'What's New' page.

Any other ideas/thoughts?

Thank you for reading! :-)


Ed


[1] https://developer.mozilla.org/en/Supported_build_configurations

[2] 33,507 WINNT_x86-msvc vs. 37,460 WINNT_x86_64-msvc mean daily active installations of Firefox 13.0a1 over the last two weeks.

[3] Aurora, beta and release channel Windows builds (both per push and nightlies) are 32-bit only.

[4] The Mozilla Maintenance Service is currently only enabled for x86 builds, pending bug 715876.

[5] V8 MSVC 32bit vs 64bit:
http://graphs-new.mozilla.org/graph.html#tests=[[76,1,19],[76,1,12]]&sel=1328150633518.812,1330185701601.2605&displayrange=90&datatype=running

[6] SunSpider MSVC 32bit vs 64bit:
http://graphs-new.mozilla.org/graph.html#tests=[[75,1,19],[75,1,12]]&sel=1328278981105.6865,1330209574209.1348&displayrange=90&datatype=running

[7] Feb 17th Dromaeo regressions (V8 by 81.1%, CSS by 31.8%, String/Array/Eval/Regex by 20.6%, SunSpider by 10.8%), with an initial 102 changeset regression range:
https://groups.google.com/d/msg/mozilla.dev.tree-management/5VxnjZSzZOw/b7u69Rv0qsEJ

[8] http://www.reddit.com/r/firefox/comments/puaw3/firefox_nightly_is_blazing_fast/ + many examples within: http://forums.mozillazine.org/viewtopic.php?f=23&t=2239223

[9] https://groups.google.com/d/topic/mozilla.dev.planning/Mrba6hvl5-w/discussion

[10] https://groups.google.com/d/topic/mozilla.dev.planning/aeTXSZ_WFAs/discussion

[Asa Dotzler]

On 2/27/2012 5:58 PM, Ed Morley wrote:

> Option B: (my personal preference)
>
> * Switch off Nightly Win64 builds (at least for the short to medium
> term). * Still do per-push tinderbox Win64 builds to make sure we
> don't break anything major, but switch off the all tests (if we're
> not going to run them on all trees). * Remove the Win64 links on
> http://nightly.mozilla.org. * Transition Win64 Nightly users to
> 32-bit Nightly builds, with an unprompted update and an appropriate
> 'What's New' page.

From the Product standpoing, I recommend with this option. We will not
be shipping official Win64 builds any time soon (if ever) and so
dividing our testing is only cost with no reward.

- A

[Asa Dotzler]

That didn't come out well :) From the Product *standpoint* I recommend
*going* with this option.

-A

[KWierso]

> * Amazingly, there is still the perception by some Windows Nightly users that the 64-bit MSVC builds are faster than their 32-bit counterparts [8], whereas even before the recent regressions that was not the case [5] [6].

Could it be the case that the perceived performance increase is from not having as many worthless/leaking/performance-destroying addons/plugins available for 64-bit builds to be damaged by? (If all the junk out there only targets 32-bit Firefox, x64 builds would seem better.)

[Anthony Hughes]

I don't believe we (QA) have done any real amount of testing using the Win64 Nightlies (unless a situation arose that specifically called for it). So in terms of QA, proceeding down either path is no net loss, no net gain.

_______________________________________________
dev-planning mailing list
dev-pl...@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-planning

[Brian R. Bondy]

On Feb 27, 9:11 pm, Asa Dotzler <a...@mozilla.org> wrote:
> > ...Option B...
>  From the Product standpoint, I recommend with this option. We will not

> be shipping official Win64 builds any time soon (if ever) and so
> dividing our testing is only cost with no reward.

I support option B up until we decide to put resources towards making
64-bit MSVC Windows builds tier 1.
In addition to giving us extra early testing, there are also several
bugs (such Bug 715876, bug 727873, bug 711210) which can suck precious
development time that could be better spent elsewhere.

[Christopher Blizzard]

On 2/27/2012 6:11 PM, Asa Dotzler wrote:

Is this a change? I thought we were looking to move because major
plugins were largely available as 64 bit?

--Chris

[Asa Dotzler]

I did conduct a fact gathering exercise to try to better understand the
pros and cons. Right now, I don't believe it's worth making 64-bit
builds official. I've been meaning to come back to this group with my
recommendation in more detail. I'll try to get to that ASAP.

- A

[Dao]

I don't think these builds should be available on
http://nightly.mozilla.org/ at all, regardless of whether they're built.

[sag net]

On Feb 28, 2:58 am, Ed Morley <bmo.takethis...@edmorley.co.uk> wrote:
> Option B: (my personal preference)
>

> * Transition Win64 Nightly users to 32-bit Nightly builds, with an unprompted update and an appropriate 'What's New' page.

Back to 32-bit Nightly?, never! In this case i have only one option,
switching to Opera 12 64bit dev build.
The 64-bit build is for me the last reason using Firefox, why i should
switch to 32-bit Firefox, if i wanted fast 32-bit browser, i would use
Chrome.

[Benjamin Smedberg]

What exactly makes you think that the 64-bit build is better than the
32-bit build? We have initial data which shows that it in general the
64-bit build uses more memory and doesn't run much faster. This isn't
always true on certain testcases where SSE instructions can improve
things, but for the vast majority of browsing tasks there is no difference.

Do you have data which indicates otherwise?

--BDS

[sag net]

On Feb 28, 3:33 pm, Benjamin Smedberg <benja...@smedbergs.us> wrote:
> What exactly makes you think that the 64-bit build is better than the
> 32-bit build? We have initial data which shows that it in general the
> 64-bit build uses more memory and doesn't run much faster. This isn't
> always true on certain testcases where SSE instructions can improve
> things, but for the vast majority of browsing tasks there is no difference.
>
> Do you have data which indicates otherwise?
>
> --BDS

First, because is a Native program and secondly i think it has better
security.
I don't care if the 64-bit build use more memory, currently i have 8GB
memory on my PC and in near future i will upgrade to 16GB.

[Wayne Mery]

For me 64bit on win7 has been more stable with lots of tabs.
(using nightly since ~October 1)
Yes, it does use more memory - doesn't matter to me.

--
contribute ... http://wiki.mozilla.org/Thunderbird:Testing

[Christopher Blizzard]

On 2/28/2012 10:07 AM, Wayne Mery wrote:
> For me 64bit on win7 has been more stable with lots of tabs.
> (using nightly since ~October 1)
> Yes, it does use more memory - doesn't matter to me.
>

Do we have crash data that shows that it crashes less/more? (I'd guess
less because we're less likely to run out of memory!)

--Chris

[lordkit...@gmail.com]

I would rather you guys actually put more work into it, users have clearly been asking for a 64bit for a long time and when the nightly came out the users who wanted 64bit used it.

I am aware that 64bit is slightly slower (peacekeeper puts the 64bit build juuuuust behind the 32) but it in my experience has been much more stable when working with lots of tabs or sites with a large amount of content.

At the very least please dont remove the 64bit build or ill be forced to use something like waterfox or one of the many other 64bit variants floating around, and i would rather not since they seem to have issues.

The argument against 64 used to be plugin support but so far it seems a vast array of plugins support the 64bit build so that argument no longer flies.
It is possible to squeeze performance out of 64bit for a few select tasks (but it boils down to is the %ofincrease = or > time to make happen) but other advantages are there mainly memory being able to go as high as it damn well pleases.

Almost every computer being sold now (exept the lower end budget oc/laptops) are being shipped with 4or more GB of ram and a 64bit OS so i think its more then worth putting more time into the 64bit build and wish you would reconsider, as i would hate for you too loose users over this and i can promise you that you would.

[Dave Mandelin]

On Feb 28, 2:34 am, Dao <d...@design-noir.de> wrote:
> I don't think these builds should be available onhttp://nightly.mozilla.org/at all, regardless of whether they're built.

I agree with that.

My personal inclination is to keep supplying 64-bit builds somehow,
because it seems like a lot of enthusiastic users (including a power-
user friend of mine) really like having them. But we consider it an
experimental thing, so it shouldn't be out there in a place that
suggests it is a fully supported product.

I definitely do not like the idea of switching over users from 64-bit
to 32-bit Nightly builds without at least asking nicely and saying why
we would like people to do that (and why we think they would like to
do it).

Dave

[Ben Hearsum]

On 02/27/12 08:58 PM, Ed Morley wrote:
> Option B: (my personal preference)
>
> * Switch off Nightly Win64 builds (at least for the short to medium term).
> * Still do per-push tinderbox Win64 builds to make sure we don't break anything major, but switch off the all tests (if we're not going to run them on all trees).

From a RelEng standpoint there's almost no difference between this and
the current state of things. If we have _any_ builds at all happening
regularly, we have to maintain a pool of machines to support them.
Obviously that's not the only factor here, but for us, it really sucks
to have to maintain a platform that almost nobody else cares about.

[Asa Dotzler]

A lot of this was covered in earlier threads on this topic. See
"Firefox 64-bit for Windows: data gathering" and "Firefox 64-bit for
Windows: what we know"

I don't think we have crash data that can demonstrate this and I've been
told it's not necessarily a win.

> • With a larger virtual address space, people using a 64-bit Firefox
> on 64-bit Windows could see fewer out of memory crashes (actually,
> out-of-virtual-address-space crashes.)
>
> •• This is not necessarily a clear win because 64-bit builds will use
> more memory overall and we do not have a clear picture of Firefox's
> address space usage in the wild.

- A

[Boris Zbarsky]

On 2/28/12 10:41 AM, sag net wrote:
> First, because is a Native program

What does that mean, exactly?

> and secondly i think it has better security.

It doesn't.

The strongest argument for 64-bit builds continues to be their better
handling of situations where we end up using >2GB of address space...

-Boris

[James May]

On 29 February 2012 08:42, Boris Zbarsky <bzba...@mit.edu> wrote:

> On 2/28/12 10:41 AM, sag net wrote:
>
>> First, because is a Native program
>>
>
> What does that mean, exactly?
>
>

But... 64 is twice as many bits as 32!

Bigger numbers are better!

>
> and secondly i think it has better security.
>>
>
> It doesn't.
>

Isn't there some benefit with more room for ASLR and something about the
calling convention being better?

With the con of course that it gets less testing.

> The strongest argument for 64-bit builds continues to be their better
> handling of situations where we end up using >2GB of address space...

Do we still not have stats on how often this happens?



> -Boris


Two other points I can think of:

* x64 builds are used on many linux distros, increased testing for the
platform independent code couldn't hurt.
* testing out the SSE2 generating code in MSVC might be useful if/when we
become able to use more cpu specific binaries (ie. stub instaler?) , but
switching to a build config ala waterfox would probably be more effective
on this front.

If this were a vote, I'd go for putting up a warning firstrun and marking
the download with stronger language. I'm not familiar with how many releng
resources x64 nightlies take up though.


-- James

[Boris Zbarsky]

On 2/28/12 5:07 PM, James May wrote:
> Isn't there some benefit with more room for ASLR

Hmm... maybe.

> and something about the
> calling convention being better?

I don't think that affects security.

> * x64 builds are used on many linux distros, increased testing for the
> platform independent code couldn't hurt.

Since all our Mac OS 10.6 and 10.7 users (including all recent releases)
are on 64-bit builds, the platform-independent part is getting a
reasonable amount of testing, I suspect...

-Boris

[Ben Hearsum]

On 02/28/12 05:07 PM, James May wrote:
> If this were a vote, I'd go for putting up a warning firstrun and marking
> the download with stronger language. I'm not familiar with how many releng
> resources x64 nightlies take up though.

Doing nightlies only takes up marginally more resources if we're already
doing on-change builds. However, on-change builds take up a bunch of
resources already, which is pretty undesirable for a platform we're not
shipping IMO.

[Ben Hearsum]

On 02/28/12 05:07 PM, James May wrote:

> If this were a vote, I'd go for putting up a warning firstrun and marking
> the download with stronger language. I'm not familiar with how many releng
> resources x64 nightlies take up though.

[Justin Dolske]

On 2/27/12 5:58 PM, Ed Morley wrote:

> Option B: (my personal preference)
>
> * Switch off Nightly Win64 builds (at least for the short to medium term).

I have 4 concerns off the top of my head:

1) If we turn off Win64 builds now, how much extra effort will be
required later to get them working and shippable? And is that extra
delay (on top of all the existing ambiguity/delay/work) worth it? And
how about versus an Option C of getting Win64 builds/tests going in
inbound/try, such that it becomes more like a tier-1 platform?

2) Conversely, if there's a proposal to simply never ship Win64 in the
foreseeable future, that should be raised and decided before we spend a
lot of time in this thread. :) My understanding of the status quo is
that we're eventually going to ship it, but it's not a high priority or
on a specific schedule.

3) Competitive landscape. I would be be worried about further eroding
market/mindshare if competitive browsers are shipping 64-bit builds or
have a plan for doing so. [IE has a 64-bit version but defaults to 32,
what's the state of Chrome's and Opera's plans?] If people are under the
mistaken impression that the 64-bit builds are blazing fast, I'd really
love to use that little thread to help start unraveling the whole
"chrome is king" sweater (by fixing our 64 bit issues), and not just say
"wrong! those builds are terrible and slow!".

4) 64-bit everywhere? How long until the majority of our userbase is
64-bit capable, and supporting 32-bit code become a legacy headache?
Probably still a ways out, but the future seems clear enough to me --
eventually most platforms will be 64-bit. Linux seems to already be
there (distros are commonly shipping the 64-bit version, no?), and OS X
is almost there (istr an earlier thread about going 64-bit only once
10.5 support is dropped). I would assume the fraction of Windows users
with 64-bit capable systems is only going to grow. Not sure what the
outlook is for mobile.

I guess there's one other meta-issue that comes to mind... We generally
know that our Nightly population is quite often just not representative
of our general population (Farmville, anyone?). I don't want to ship
crap to our users, but it's also not clear to me that we would get a lot
of incremental value from moving some or all of them to 32bit builds.

Justin

[Justin Lebar]

> Not sure what the outlook is for mobile.

Mobile will be 32-bit for the next two or three years, at least.
ARMv8 is the first 64-bit architecture from ARM, and it exists only on
paper at the moment. It's not even clear to me whether ARMv8 is
targeting phones, or just servers. We'd need to wait until we no
longer support ARMv7 until we could drop 32-bit ARM.

(FWIW, I'm in favor of more 64-bit on desktop. The extra virtual
address space eliminates an entire class of crashes.)

-Justin

[sag net]

On Feb 29, 12:23 am, Justin Dolske <dol...@mozilla.com> wrote:

> 3) Competitive landscape. I would be be worried about further eroding
> market/mindshare if competitive browsers are shipping 64-bit builds or
> have a plan for doing so. [IE has a 64-bit version but defaults to 32,
> what's the state of Chrome's and Opera's plans?] If people are under the
> mistaken impression that the 64-bit builds are blazing fast, I'd really
> love to use that little thread to help start unraveling the whole
> "chrome is king" sweater (by fixing our 64 bit issues), and not just say
> "wrong! those builds are terrible and slow!".

Opera is Working on 64-bit build.
http://dev.opera.com/articles/view/64-bit-opera-and-out-of-process-plug-ins/

[Brian Smith]

Boris Zbarsky wrote:
> The strongest argument for 64-bit builds continues to be their better
> handling of situations where we end up using >2GB of address space...

We should investigate startup performance on Win8. It could be bad if WoW64 didn't get loaded at startup, forcing us to pay the cost of loading it during Firefox startup.

- Brian

[Henri Sivonen]

On Wed, Feb 29, 2012 at 1:23 AM, Justin Dolske <dol...@mozilla.com> wrote:
> 3) Competitive landscape. I would be be worried about further eroding
> market/mindshare if competitive browsers are shipping 64-bit builds or have
> a plan for doing so.

Indeed. Metro IE10 on 64-bit Windows 8 seems to be a 64-bit app. (The
desktop IE10 launches as 32-bit on 64-bit Windows 8, but it's
positioned as a backward compat mode anyway.) Opera is preparing a
64-bit version for Windows.

When those ship, Firefox and Chrome will look like they are lagging.
If Chrome ships 64-bit on Windows, too, Firefox will look like its
lagging even more badly. At that point, saying that we measured this
or that won't matter for mindshare.

(As a technical argument, running out of address space less often
would be nice.)

--
Henri Sivonen
hsiv...@iki.fi
http://hsivonen.iki.fi/

[Chris Peterson]

On 3/1/12 1:33 AM, Henri Sivonen wrote:
> If Chrome ships 64-bit on Windows, too, Firefox will look like its
> lagging even more badly. At that point, saying that we measured this
> or that won't matter for mindshare.

Many early adopters are leaving Firefox for Chrome. Shipping a 64-bit
Firefox before Chrome (even as a non-default install) would give these
trendsetting users a reason to choose (and evangelize) Firefox.

More than half of Windows Nightly users install 64-bit builds. Do they
really want Nightly builds or do they just want 64-bit builds? If 64-bit
Nightly builds went away, many of those users might too.


chris p.

[Asa Dotzler]

On 3/1/2012 10:02 AM, Chris Peterson wrote:
> On 3/1/12 1:33 AM, Henri Sivonen wrote:
>> If Chrome ships 64-bit on Windows, too, Firefox will look like its
>> lagging even more badly. At that point, saying that we measured this
>> or that won't matter for mindshare.
>
> Many early adopters are leaving Firefox for Chrome. Shipping a 64-bit
> Firefox before Chrome (even as a non-default install) would give these
> trendsetting users a reason to choose (and evangelize) Firefox.

I'd like to understand why you believe that. Do we have any data saying
that users want or need 64-bit Firefox?

> More than half of Windows Nightly users install 64-bit builds. Do they
> really want Nightly builds or do they just want 64-bit builds? If 64-bit
> Nightly builds went away, many of those users might too

It's not because they're seeking out 64-bit builds. It's because this
page http://nightly.mozilla.org/ puts 64-bit at the top and those with
64-bit systems think that we are recommending it for them. I'd wager
that if we took the 64-bit build off of http://nightly.mozilla.org/ that
we'd see those numbers revert to an overwhelming majority of 32-bit
downloads and users over time.

- A

[Gian-Carlo Pascutto]

On 1/03/2012 19:34, Asa Dotzler wrote:

>> Many early adopters are leaving Firefox for Chrome. Shipping a 64-bit
>> Firefox before Chrome (even as a non-default install) would give these
>> trendsetting users a reason to choose (and evangelize) Firefox.
>
> I'd like to understand why you believe that. Do we have any data saying
> that users want or need 64-bit Firefox?

Probably anecdotal data, but just like Dave Mandelin, some power-user
friends of mine like them. I know one of them uses it because he has
image discarding turned off, and can have Firefox use all his RAM to
switch quicker between tabs (which would just crash quickly on 32-bits).
So you could say he's using it as a workaround for our ImageSuck situation.

I wouldn't be surprised if other power-users came up with similar
things, nor would I be surprised if some are running it because 64 is
obviously more than 32.

>> More than half of Windows Nightly users install 64-bit builds. Do they
>> really want Nightly builds or do they just want 64-bit builds? If 64-bit
>> Nightly builds went away, many of those users might too
>
> It's not because they're seeking out 64-bit builds. It's because this
> page http://nightly.mozilla.org/ puts 64-bit at the top

The normal build is left. The 64-bit build is not on top of it. I agree
this page doesn't do a good job of telling users they don't really want
to download the 64-bit one unless they understand the disadvantages.

--
GCP

[spec...@gmx.de]

On Thursday, March 1, 2012 7:34:17 PM UTC+1, Asa Dotzler wrote:
> It's not because they're seeking out 64-bit builds. It's because this
> page http://nightly.mozilla.org/ puts 64-bit at the top and those with
> 64-bit systems think that we are recommending it for them. I'd wager
> that if we took the 64-bit build off of http://nightly.mozilla.org/ that
> we'd see those numbers revert to an overwhelming majority of 32-bit
> downloads and users over time.
>
> - A

In other words, you think the users are too stupid choose the right build?