revisiting disabling Linux32 testing

[km...@mozilla.com]

Last summer I posted a note requesting feedback regarding disabling tests on Linux32. To summarize, the feedback was

* We use Linux32 tests for sanity checking tests that aren't running on b2g
* Linux64 tests don't have parity with the test coverage on Linux32
* Linux32 is a very small user base compared to Windows yet we allocate expensive resources to it.
* Was mentioned that 25% of our contributor community run Linux, however it is unclear if this is Linux32 or Linux64
* Most people don't use our Linux32 builds, but rather the ones from their distro
* We do not have reliable telemetry numbers for Linux32 from distros for various reasons

https://groups.google.com/forum/#!searchin/mozilla.dev.planning/linux32/mozilla.dev.planning/wBgLRXCTlaw/-YzXkHxyBgAJ

I didn't proceed with the change at that time because of the test parity and b2g issues. However, since that time we have disabled b2g builds and tests and achieved test parity on linux64. Thus at this time, I'm looking to proceed to disable linux32 tests.

https://bugzilla.mozilla.org/show_bug.cgi?id=1209932


Kim
Mozilla Release Engineering

[Ralph Giles]

On Thu, Mar 10, 2016 at 7:29 AM, <km...@mozilla.com> wrote:

> Last summer I posted a note requesting feedback regarding disabling tests on Linux32. [...]

>
> I didn't proceed with the change at that time because of the test parity and b2g issues. However, since that time we have disabled b2g builds and tests and achieved test parity on linux64. Thus at this time, I'm looking to proceed to disable linux32 tests.

Seems reasonable. If we also drop support for MacOS X 10.6-10.8, does
this mean we'll be testing 32 bit builds only on Android and Windows
XP? That should be enough for shared code, but I assume we're
periodically break platform-specific stuff on linux32 and rely on
contributors to detect and patch this like any Tier-2 platform?

-r

[Ryan VanderMeulen]

Windows 7 testing is still on 32-bit as well.

[Nicolas B. Pierron]

Does this include the JS Shell test suite, as well as the arm simulator?
From what I understand we have no substitute yet.

--
Nicolas B. Pierron

[Gijs Kruitbosch]

Could we transition to having taskcluster-based builds and only the test
suites that have no equivalent (so dump e.g. all of mochitest(-*), but
not js shell as Nicolas brought up) ? Or would that be an inordinate
amount of work? It seems like that might lead to faster + cheaper
builds, plus a good amount of gain from being able to switch off the
majority of the testing immediately.

I have no particular insight in expenses here, but I do know that the
linux build results tend to be the fastest to show up as a result of a
(try) push, and esp. for 32-bit build failures having to wait for the
windows builds would delay when we notice that there are problems.

~ Gijs

[William Lachance]

On 2016-03-10 10:29 AM, km...@mozilla.com wrote:
> * We use Linux32 tests for sanity checking tests that aren't running on b2g
> * Linux64 tests don't have parity with the test coverage on Linux32
> * Linux32 is a very small user base compared to Windows yet we allocate expensive resources to it.
> * Was mentioned that 25% of our contributor community run Linux, however it is unclear if this is Linux32 or Linux64
> * Most people don't use our Linux32 builds, but rather the ones from their distro
> * We do not have reliable telemetry numbers for Linux32 from distros for various reasons

FWIW, there was pretty conclusive evidence produced at the time that the
vast majority of Linux users were running linux32, and I doubt that's
changed:

https://groups.google.com/d/msg/mozilla.dev.planning/wBgLRXCTlaw/NvV-05jQBgAJ

Just because someone doesn't get their builds from us, it doesn't mean
that they aren't important. I think it would look pretty bad on us if we
released a version of Firefox which was broken on linux32 as a result of
this change, though I'm not sure how likely that is. Perhaps the linux64
tests + whatever QA we get from the community is enough to catch those
kinds of issues.

Will

[Steve Wendt]

On 3/10/2016 2:09 PM, William Lachance wrote:

> vast majority of Linux users were running linux32

Just wild guesses here, but I suspect that was only true for Ubuntu, and
only because they made the 32-bit download the default for much longer
than they should have.

[km...@mozilla.com]

I have opened https://bugzilla.mozilla.org/show_bug.cgi?id=1255890 (Linux32 as tier 2 platform) to get more comprehensive numbers from relman on our Linux 32 numbers from multiple distros.

Nicolas, could you open a bug for the JS Shell test suite issue you mention and mark it as a blocker of bug 1209932.

Gijs, yes it would be possible to move these tests to taskcluster but we don't want to invest in that effort if it's not warranted by the number of users that actually use that platform.

Kim