Permanently dropping testing of 32-bit Linux desktop Firefox debug builds

[Karl Tomlinson]

Starting a new thread for this so that discussion doesn't get
lost/mixed with the **temporarily** discussion:

Karl Tomlinson wrote:

>> BTW, disabling linux32 debug tests long term is probably quite
>> reasonable as the extra coverage over linux32 opt and linux64
>> debug is quite minimal.

John O'Duinn writes:

> Karl: this is quite interesting - can you elaborate? If "linux32 debug"
> is not buying us much value, then I would assert that is a *lot* of CPU
> load savings for us all. Karl, is that something you can make
> determination on, or do you know who can?

This was a suggestion to deal with the problem that we are
continually adding new tests but resources to run the tests are
not increasing at the same rate.

There are many different permutations of configurations that we
could test but don't. Wherever we run the same set of tests in
different configurations there is considerable overlap in what is
being tested and the value returned diminishes as overlap
increases.

We could run tests against several distributions and versions of
distributions, for example. We run tests for OS X builds on 10.6,
10.7, and 10.8, and we run tests for NT builds on 5.1 and 6.1, but
we don't do this kind of thing for Linux desktop builds.

We run tests for Linux desktop and Android builds on two different
architectures, but IIUC we don't do this for Mac or NT. NT has
been discussed elsewhere. I assume the reason for not running
32-bit tests for Mac on each checkin is that we expect the number
of 32-bit users to be diminishing.

On Linux desktop we may still keep 32-bit builds for as long as
there are still many systems with limited RAM that would suffer
from the additional memory requirements of 64-bit builds, and
Linux distributions continue to support such systems.

If we consider what advantages testing debug builds gives us over
just testing release-configuration builds, I can think of two
things:
1) debug builds have assertions and other in-code sanity checks
2) debug builds can check for memory leaks.

Very little of our C/C++ code is written differently for 32-bit or
LP64 environments. Hopefully memory management is high enough
level that there are unlikely to be differences between 32 or LP64
builds.

The same C/C++ code does behave differently when compiled to
32-bit or LP64 environments, so in-code sanity checks can
sometimes detect architecture-specific problems. This might be
particularly relevant to our JavaScript JIT compiler.

However, there is also considerable overlap in what is tested
across different operating systems. 32-bit linux builds would
benefit from much of the debug build testing run on WINNT.

So what we'd be missing from dropping linux 32 debug build testing
is 32-bit-specific behaviour of in-code sanity checks in
Linux-desktop-specific portions of the code.

It's just a value-for-resources call whether we keep those tests
running. What I'm saying is, if we are going to reduce the
permutations of linux desktop test environments, then this is
where I think we are getting the smallest value for resources.

[Panos Astithas]

On Thu, Jan 10, 2013 at 2:42 AM, Karl Tomlinson <moz...@karlt.net> wrote:

> It's just a value-for-resources call whether we keep those tests
> running. What I'm saying is, if we are going to reduce the
> permutations of linux desktop test environments, then this is
> where I think we are getting the smallest value for resources.
>

I'd also add that mochitests and xpcshell tests seem to mostly be
platform-independent.

[Justin Lebar]

I'm here at the B2G work week with a table of platform engineers.
There was a near riot at the suggestion of permanently disabling these
tests. Everyone at the table was vehemently opposed, for a simple
reason:

Linux 32 is the closest x86 platform we have to B2G.

Even if we ran every single test that runs on Linux x86-32 on the B2G
emulator (this does not appear to be the case), I'd want to have much
more experience with the B2G tests before we established that we don't
need the Linux x86-32 tests for coverage.

I understand that we face trade-offs here, but at the very least, I
think we'd need to look very carefully at the list of tests run on
x86-32 but not on B2G before we disabled Linux x86-32 tests.

-Justin

> _______________________________________________
> dev-planning mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-planning

[Karl Tomlinson]

Justin Lebar writes:

> I'm here at the B2G work week with a table of platform engineers.
> There was a near riot at the suggestion of permanently disabling these
> tests. Everyone at the table was vehemently opposed, for a simple
> reason:
>
> Linux 32 is the closest x86 platform we have to B2G.
>
> Even if we ran every single test that runs on Linux x86-32 on the B2G
> emulator (this does not appear to be the case), I'd want to have much
> more experience with the B2G tests before we established that we don't
> need the Linux x86-32 tests for coverage.

So, given the importance of these tests to B2G and the
significance of this week to B2G:

1) Should we match the priority of b2g and linux 32 debug desktop jobs?

2) Is there a different set of tests we can stop running this
week, or at least run less frequently, to free up some machines
to run at least some Linux 32 debug tests at least every now and
then?

[Andrew Halberstadt]

Just pointing out that the B2G emulator tests consume Linux x86-32
slaves. What if turning off Linux x86-32 debug meant more capacity to
add emulator tests?

Hopefully pandaboards won't be too far off and this will be a non-issue,
but we may eventually need to make a tradeoff.

Andrew