The future of Linux32 builds and tests

[km...@mozilla.com]

We recently disabled Linux32 talos testing on all trees.

Going forward, the question remains, do we need to run Linux32 tests at all? This would reduce the complexity of our configs, as well as significantly reduce our AWS bill. Do the linux32 test results provide significant differences from the ones we run on on linux64? I see three options going forward

1) Run linux32 builds and tests periodically
2) Turn off tests but run the builds and make it a Tier 2 platform
3) Turn off Linux32 builds + tests entirely

Your data-driven input is sought,

cheers,
Kim

Bugs of note
Disable all 32-bit linux testing
https://bugzilla.mozilla.org/show_bug.cgi?id=1209932
Do we need linux32 talos?
https://bugzilla.mozilla.org/show_bug.cgi?id=1204920
Disable linux32 talos testing and reimage machines for windows testing
https://bugzilla.mozilla.org/show_bug.cgi?id=1208449

[Millwood]

"Upgrading" a 32 bit linux system to 64 is not an easy prospect so
please keep building at least product versions for 32 bits.

[Daniel Holbert]

On 10/08/2015 10:01 AM, km...@mozilla.com wrote:
> Going forward, the question remains, do we need to run Linux32 tests at all? This would reduce the complexity of our configs, as well as significantly reduce our AWS bill. Do the linux32 test results provide significant differences from the ones we run on on linux64? I see three options going forward
>
> 1) Run linux32 builds and tests periodically
> 2) Turn off tests but run the builds and make it a Tier 2 platform
> 3) Turn off Linux32 builds + tests entirely
>
> Your data-driven input is sought,

When the possibility of making Linux32 "Tier 2" was brought up in 2012
[1], cpearce brought up a good point [2] about data on Linux users:

"...if we *were* to consider dropping a platform to tier 2, we should
make that decision with data to back it up, which for Linux should also
include data regarding the Firefox x86/x64 split in the major distros,
since most roll their own Firefox packages which we don't track."

We should try to get data from some 3rd party (not our download tallies)
about the Firefox-on-Linux 32-bit vs. 64-bit user stats, before we make
a call about demoting Linux32 to Tier 2. (Maybe we already have such data?)

~Daniel

[1]
https://groups.google.com/d/msg/mozilla.dev.planning/ahyq19APxb4/tvkekYhQbhcJ
[2]
https://groups.google.com/d/msg/mozilla.dev.planning/ahyq19APxb4/KO-P4xnwvIYJ

[Gijs Kruitbosch]

Pending more data about how many people use 32-bit fx on linux distros,
we could conceivably do both (1) and turn off 32-bit *debug* testing
completely, as nobody really uses those as daily builds anyway, and most
any issue should get caught by the opt tests or the x64 debug ones?

~ Gijs

[Daniel Holbert]

On 10/08/2015 11:34 AM, Daniel Holbert wrote:
> On 10/08/2015 10:01 AM, km...@mozilla.com wrote:
> We should try to get data from some 3rd party (not our download tallies)
> about the Firefox-on-Linux 32-bit vs. 64-bit user stats, before we make
> a call about demoting Linux32 to Tier 2. (Maybe we already have such data?)

Ah -- looking at the bugs that kmoir linked to, it looks like we do have
some data on this, from telemetry (which is opt-in in release builds, so
it's not perfect but it's something):

https://bugzilla.mozilla.org/show_bug.cgi?id=1204920#c1

The numbers there show that ~40% of our Linux users are using the 32-bit
version. (on release & beta channels) That's more than I was expecting.

Given that, I'd be uncomfortable with option (3) ("turn off Linux32
builds + tests entirely"). I'd also be uncomfortable with option (2)
(doing builds but no tests), because then 40% of our linux users would
be getting un-tested bits. (Likely-working since they work on other
platforms, but who knows.)

Option (1) (running tests occasionally) seems doable though, and
depending on how infrequent "occasionally" is, it seems like it could
get us cost-savings that are close to those from option (2).

~Daniel

[Mike Hommey]

Now the interesting question, is how many of those users are running on
64-bits systems, and are only using 32-bits builds because that's what
they downloaded back when mozilla.org didn't even offer the 64-bits
version, which brings to the additional question: would it be worth
forcing 32->64-bit upgrades of Firefox.

Mike

[William Lachance]

Aren't the vast majority of users running a version of Firefox
distributed by their vendor (of which Ubuntu probably has the largest
share, see e.g.
http://stats.wikimedia.org/wikimedia/squids/SquidReportOperatingSystems.htm)?
I don't think any metrics gathered from the Firefox builds which we
distribute are going to be very useful for measuring the impact of this
change.

I'd suggest reaching out to at least Ubuntu, Debian, and Fedora on this
and see what they think, at least as input. They're the domain experts,
and we really depend on their goodwill for ensuring that Firefox remains
the default browser for their operating system.

Will

[Mike Hommey]

On Thu, Oct 08, 2015 at 05:43:48PM -0400, William Lachance wrote:
> On 2015-10-08 5:26 PM, Mike Hommey wrote:
> >On Thu, Oct 08, 2015 at 12:42:11PM -0700, Daniel Holbert wrote:
> >>On 10/08/2015 11:34 AM, Daniel Holbert wrote:
> >> https://bugzilla.mozilla.org/show_bug.cgi?id=1204920#c1
> >>
> >>The numbers there show that ~40% of our Linux users are using the
> >>32-bit version. (on release & beta channels) That's more than I was
> >>expecting.
> >>
> >>Given that, I'd be uncomfortable with option (3) ("turn off Linux32
> >>builds + tests entirely"). I'd also be uncomfortable with option (2)
> >>(doing builds but no tests), because then 40% of our linux users would
> >>be getting un-tested bits. (Likely-working since they work on other
> >>platforms, but who knows.)
> >>
> >>Option (1) (running tests occasionally) seems doable though, and
> >>depending on how infrequent "occasionally" is, it seems like it could
> >>get us cost-savings that are close to those from option (2).
> >
> >Now the interesting question, is how many of those users are running on
> >64-bits systems, and are only using 32-bits builds because that's what
> >they downloaded back when mozilla.org didn't even offer the 64-bits
> >version, which brings to the additional question: would it be worth
> >forcing 32->64-bit upgrades of Firefox.
>
> Aren't the vast majority of users running a version of Firefox distributed
> by their vendor (of which Ubuntu probably has the largest share, see e.g.
> http://stats.wikimedia.org/wikimedia/squids/SquidReportOperatingSystems.htm)?

That's the question I'd like to see answered the most. ISTR having been
said that a surprising large number of people are using our builds, but
I'm not sure I'm not making things up, and I don't know how large.

Mike

[Daniel Holbert]

On 10/08/2015 02:43 PM, William Lachance wrote:
> Aren't the vast majority of users running a version of Firefox
> distributed by their vendor (of which Ubuntu probably has the largest
> share, see e.g.
> http://stats.wikimedia.org/wikimedia/squids/SquidReportOperatingSystems.htm)?
> I don't think any metrics gathered from the Firefox builds which we
> distribute are going to be very useful for measuring the impact of this
> change.

Oh, right -- it looks like telemetry isn't enabled in my Ubuntu-provided
Firefox build. (The telemetry checkbox shown in the "how to enable
telemetry" article[1] just isn't present.) I assumed it would be
enabled (and opt-in for users), but I assumed wrong.

So, it seems we aren't getting telemetry from Ubuntu users at least. So,
unless we're getting platform info some other way (AUS pings?), sounds
like we can't be sure about 32-vs-64 market share without talking to
distros.

~Dnaiel

[1]
https://support.mozilla.org/en-US/kb/share-telemetry-data-mozilla-help-improve-firefox

[dand...@mozilla.com]

If AWS cost is a big concern, I wouldn't look at Linux 32 usage versus Linux 64 usage, but Linux versus other platforms. According to Telemetry, Linux has about 0.2% market share on Desktop, which means Windows has around 500X more users. Given that, it seems like we should not give Linux a large slice of infrastructure time, outside of the internal analysis builds like ASAN.

[Lawrence Mandel]

On Thu, Oct 8, 2015 at 5:43 PM, William Lachance <wlac...@mozilla.com>
wrote:

Sylvestre - Can you get input from the Debian community about this?

Lawrence

[Mike Hommey]

On Thu, Oct 08, 2015 at 06:01:50PM -0700, Daniel Holbert wrote:
> On 10/08/2015 02:43 PM, William Lachance wrote:

> > Aren't the vast majority of users running a version of Firefox
> > distributed by their vendor (of which Ubuntu probably has the largest
> > share, see e.g.
> > http://stats.wikimedia.org/wikimedia/squids/SquidReportOperatingSystems.htm)?
> > I don't think any metrics gathered from the Firefox builds which we
> > distribute are going to be very useful for measuring the impact of this
> > change.
>

> Oh, right -- it looks like telemetry isn't enabled in my Ubuntu-provided
> Firefox build. (The telemetry checkbox shown in the "how to enable
> telemetry" article[1] just isn't present.) I assumed it would be
> enabled (and opt-in for users), but I assumed wrong.

Telemetry is not enabled on our release builds either anyways.

Mike

[Mike Hommey]

On Thu, Oct 08, 2015 at 09:08:04PM -0400, Lawrence Mandel wrote:
> On Thu, Oct 8, 2015 at 5:43 PM, William Lachance <wlac...@mozilla.com>
> wrote:
>

> Sylvestre - Can you get input from the Debian community about this?

Debian is likely not very indicative of the overall Linux world. Taking
the primarily available figures from Debian and Ubuntu give completely
opposite pictures:
debian: i386: 51859, amd64: 132362
ubuntu: i386: 2065695, amd64: 673565

(from popcon.ubuntu.com and popcon.debian.org)

Mike

[Mike Hommey]

I'd argue that if we are concerned by AWS cost, we should instead
(additionally?) consider consolidating B2G builds. There is no reason
we can't:
- stop building the whole android base (gonk) for every push.
- build one opt gecko and one debug one instead of one per gonk version.

Mike

[Mike Hoye]

On 2015-10-08 9:07 PM, dand...@mozilla.com wrote:
> If AWS cost is a big concern, I wouldn't look at Linux 32 usage versus
> Linux 64 usage, but Linux versus other platforms. According to
> Telemetry, Linux has about 0.2% market share on Desktop, which means
> Windows has around 500X more users

A disproportionate number of our contributors (and current employees)
come from the Linux user community. About 25% more people use Nightly on
Linux than OSX.

- mhoye

[Chris Coulson]

On 09/10/15 02:01, Daniel Holbert wrote:
> On 10/08/2015 02:43 PM, William Lachance wrote:

>> Aren't the vast majority of users running a version of Firefox
>> distributed by their vendor (of which Ubuntu probably has the largest
>> share, see e.g.
>> http://stats.wikimedia.org/wikimedia/squids/SquidReportOperatingSystems.htm)?
>> I don't think any metrics gathered from the Firefox builds which we
>> distribute are going to be very useful for measuring the impact of this
>> change.

> Oh, right -- it looks like telemetry isn't enabled in my Ubuntu-provided
> Firefox build. (The telemetry checkbox shown in the "how to enable
> telemetry" article[1] just isn't present.) I assumed it would be
> enabled (and opt-in for users), but I assumed wrong.
>

> So, it seems we aren't getting telemetry from Ubuntu users at least. So,
> unless we're getting platform info some other way (AUS pings?), sounds
> like we can't be sure about 32-vs-64 market share without talking to
> distros.
>
> ~Dnaiel
>
> [1]
> https://support.mozilla.org/en-US/kb/share-telemetry-data-mozilla-help-improve-firefox

> _______________________________________________
> dev-planning mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-planning
Hi,

I'm not sure I can provide any better metrics. You get blocklist pings
from us though, don't you? Do these provide a useful metric?

Regards
- Chris

[Robert Kaiser]

dand...@mozilla.com schrieb:

> According to Telemetry, Linux has about 0.2% market share on Desktop, which means Windows has around 500X more users.

That number does not look right. Our ADI numbers say that 3.5% of our
overall ADI are on Linux. Given that a number of Linux distros may
switch off Telemetry and FHR by default, but leave the add-on blocklist
enabled (where ADI comes from), I'd trust that number some more - even
with the flaws that the ADI measurement might have.

That said, the fact that Linux distros might switch off Telemetry and
FHR by default causes issues right in that way: they are undercounted
and we may disregard them completely in data-driven decisions. If you
are a Linux distro package manager, think about that.

KaiRo

[Robert Kaiser]

Mike Hommey schrieb:

> Debian is likely not very indicative of the overall Linux world. Taking
> the primarily available figures from Debian and Ubuntu give completely
> opposite pictures:
> debian: i386: 51859, amd64: 132362
> ubuntu: i386: 2065695, amd64: 673565

And even other distros may give yet different numbers. For example, for
the openSUSE Leap distro, 32bit will not even be an officially supported
architecture any more, only x86_64 will be. Unfortunately, I have no
clue if Fedora, RedHat, openSUSE, Arch and others even do have numbers
like you point out above.

KaiRo

[Mike Connor]

I think ADI is likely the only reliable metric here, if FHR and Telemetry
are disabled by Linux distributions. That said, if folks have examples of
Linux distros explicitly disabling these features, please let me know.
Unless they've got written permission to disable those features, they
shouldn't be doing that. And ultimately it's self-defeating, as it means
Linux users (and their use cases) won't be measured or prioritized
effectively.

-- Mike

[William Lachance]

On 15-10-08 09:40 PM, Mike Hommey wrote:
> Debian is likely not very indicative of the overall Linux world. Taking
> the primarily available figures from Debian and Ubuntu give completely
> opposite pictures:
> debian: i386: 51859, amd64: 132362
> ubuntu: i386: 2065695, amd64: 673565
>
> (from popcon.ubuntu.com and popcon.debian.org)

Are those Ubuntu numbers for installations in general, or Firefox
specifically? I couldn't find out how to get architecture specific
information from popcon, and I'm worried that the overall results
include server installations (which would really skew them):

http://popcon.ubuntu.com/main/index.html

If that was referring to Firefox, it's probably representative. It
appears popularity contest is installed on every Ubuntu instance:

https://help.ubuntu.com/community/UbuntuPopularityContest

Even without specific numbers, the fact that Ubuntu continues to see a
32-bit desktop version of their desktop operating system as worth
shipping and supportive is probably indicative in itself.

Will

[Axel Hecht]

Note, even if telemetry was enabled, it'd be switched off by default, right?

In particular with linux, would we expect a "usual as on windows" opt-in
ratio for telemetry?

I still think we should have them, as they're part of our overall user
base, but I'm not sure if numbers behind opt-in are going to be as
reliable for linux as they'd be on, say Windows.

Axel

[Robert Kaiser]

Axel Hecht schrieb:

> Note, even if telemetry was enabled, it'd be switched off by default,
> right?

In release, yes. In Nightly, Dev Edition, and Beta, it should be on by
default. But I guess the distros usually ship release anyhow. :)

> In particular with linux, would we expect a "usual as on windows" opt-in
> ratio for telemetry?

I don't know.

KaiRo

[Mike Hommey]

On Fri, Oct 09, 2015 at 12:21:26PM -0400, William Lachance wrote:
> On 15-10-08 09:40 PM, Mike Hommey wrote:
> >Debian is likely not very indicative of the overall Linux world. Taking
> >the primarily available figures from Debian and Ubuntu give completely
> >opposite pictures:
> >debian: i386: 51859, amd64: 132362
> >ubuntu: i386: 2065695, amd64: 673565
> >
> >(from popcon.ubuntu.com and popcon.debian.org)
>
> Are those Ubuntu numbers for installations in general, or Firefox
> specifically? I couldn't find out how to get architecture specific
> information from popcon, and I'm worried that the overall results include
> server installations (which would really skew them):
>
> http://popcon.ubuntu.com/main/index.html

For both Debian and Ubuntu, you can't get per-package architecture
split, but you can start from this, from the Ubuntu data:
- the total number of submissions is 2747796
- the sum of i386 + amd64 is 2739260, which leaves us with 8536
submissions from other architectures.
- the number of submissions with Firefox is 2503536.

Even if 100% of the amd64 and other non-i386 architectures
submissions were with Firefox, that still would leave 1821435 i386
submissions with Firefox, which is still a whole lot more than the
amd64 ones.

Mike

[Ryan VanderMeulen]

Note that SM(arm) builds run on linux32 build slaves (and AFAIK, they
have to). Not sure where SM builds fit in with all of this, but I
figured it was at least worth explicitly calling out.

[Steve Fink]

I don't *think* they have to, since they run fine on my local (linux64)
box. But cross-compilation always confuses me.

When I run it locally, the arm-sim build (aka SM(arm)) uses my 64-bit
gcc with -m32 to generate a 32-bit JS shell whose JIT generates 32-bit
ARM code, and then runs it via a 32-bit ARM simulator. (I think that
rest of the shell would be fine if it were compiled as 64-bit, but the
simulator part only works when compiled as 32-bit x86 code?)

Like I said, cross-compilation is confusing, especially when you mix in
JIT and a simulator.

Also note that as of very recently there is an SM(arm64) build, which is
64-bit all the way through (including generating ARM64 code). But that
does not replace the need for the 32-bit ARM simulator. Generally
speaking, we can't just ditch 32-bit architectures entirely because we
need to be able to run on 32-bit ARM devices for the foreseeable future.

[Jed Davis]

Chris Coulson <chrisc...@ubuntu.com> writes:

> On 09/10/15 02:01, Daniel Holbert wrote:
>> So, it seems we aren't getting telemetry from Ubuntu users at least. So,
>> unless we're getting platform info some other way (AUS pings?), sounds
>> like we can't be sure about 32-vs-64 market share without talking to
>> distros.
>

> I'm not sure I can provide any better metrics. You get blocklist pings
> from us though, don't you? Do these provide a useful metric?

For reference, we also get FHR data from Ubuntu's Firefoxes. (I know
Debian disables FHR in Iceweasel, however; other distributions I haven't
checked.) The main question in this thread seems to have been
effectively answered by the popcon data, but in general it's worth
keeping FHR in mind for things like this, where generalizing from the
prerelease population doesn't work.

--Jed

[Terrence Cole]

On Mon, Oct 12, 2015 at 9:56 AM, Steve Fink <sf...@mozilla.com> wrote:

> On 10/12/2015 08:21 AM, Ryan VanderMeulen wrote:
>

> I don't *think* they have to, since they run fine on my local (linux64)
> box. But cross-compilation always confuses me.
>
> When I run it locally, the arm-sim build (aka SM(arm)) uses my 64-bit gcc
> with -m32 to generate a 32-bit JS shell whose JIT generates 32-bit ARM
> code, and then runs it via a 32-bit ARM simulator. (I think that rest of
> the shell would be fine if it were compiled as 64-bit, but the simulator
> part only works when compiled as 32-bit x86 code?)
>

32bit arches use a different Value packing format than 64bit. The JITs
assume at a very low level that Values (and values in general) have the
same layout and alignment requirements everywhere. As Steve noted, however,
cross compilation with -m32 works just fine.

> Like I said, cross-compilation is confusing, especially when you mix in
> JIT and a simulator.
>
> Also note that as of very recently there is an SM(arm64) build, which is
> 64-bit all the way through (including generating ARM64 code). But that does
> not replace the need for the 32-bit ARM simulator. Generally speaking, we
> can't just ditch 32-bit architectures entirely because we need to be able
> to run on 32-bit ARM devices for the foreseeable future.
>
>