Intent to not fix: Building with gcc-4.6 for Fx38+

[bo...@mozilla.com]

Hi,

In summary: Officially make gcc-4.7 our minimum supported version. Fx38 and 39 don't compile with 4.6 and none of the GNU/Linux package maintainers I have contacted have any major concerns over dropping it.


There are three outstanding bugs open over Fx38+ not compiling with gcc-4.6.
One of these is to do with an update to the Chromium sandbox code we use, which I landed. Chromium have already dropped support for 4.6.
The other two would be fairly easy to fix.

I could back-out parts of my patch to get it compiling with 4.6, but it is going to make it difficult to take new versions of the sandbox code.

I've contacted the package maintainers for Debian, Red Hat, openSUSE, SLES and Ubuntu and they either don't have a problem with dropping 4.6 or, at least, no more of a problem than the fact we've already dropped 4.4.
These distros were picked because they have longer support cycles and tend to have some older versions of gcc.

I've also posted to the mozilla-linux-taskforce mailing list.

If nobody has any issues with this, I'll look into sorting the patches to make the build failures explicit.

Cheers,
Bob

[Ehsan Akhgari]

Have you tested bumping the gcc min version here
<http://mxr.mozilla.org/mozilla-central/source/build/autoconf/toolchain.m4#104>
to see if there are any builders that still use gcc 4.6?

> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>

[bo...@mozilla.com]

On Tuesday, March 10, 2015 at 2:38:43 PM UTC, Ehsan Akhgari wrote:
> Have you tested bumping the gcc min version here
> <http://mxr.mozilla.org/mozilla-central/source/build/autoconf/toolchain.m4#104>
> to see if there are any builders that still use gcc 4.6?

I haven't, no.
I assume you mean by pushing to try.
Here's a push in case the bugs don't exist in certain builds:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=044e896fc6fa

Thanks.

[L. David Baron]

I was looking at the log for a B2G Device Image build (I think it
was the one for Nexus 5-L) a few days (less than a week) ago, and it
was using gcc 4.8 for the target and gcc 4.6 for the host (assuming
I was reading the logs correctly).

I point this out because:

(1) the device image build isn't in that try run, and

(2) it's not clear to me that the compiler version checks check
both the target and host compiler versions when cross-compiling

-David

--
𝄞 L. David Baron http://dbaron.org/ 𝄂
𝄢 Mozilla https://www.mozilla.org/ 𝄂
Before I built a wall I'd ask to know
What I was walling in or walling out,
And to whom I was like to give offense.
- Robert Frost, Mending Wall (1914)

[Ehsan Akhgari]

On 2015-03-10 12:50 PM, L. David Baron wrote:
> On Tuesday 2015-03-10 08:49 -0700, bo...@mozilla.com wrote:

> I was looking at the log for a B2G Device Image build (I think it
> was the one for Nexus 5-L) a few days (less than a week) ago, and it
> was using gcc 4.8 for the target and gcc 4.6 for the host (assuming
> I was reading the logs correctly).
>
> I point this out because:
>
> (1) the device image build isn't in that try run, and
>
> (2) it's not clear to me that the compiler version checks check
> both the target and host compiler versions when cross-compiling

That's a good point. I don't think we have any checks for the host
compiler.

[Brian Smith]

<bo...@mozilla.com> wrote:
> In summary: Officially make gcc-4.7 our minimum supported version. Fx38 and 39 don't compile with 4.6 and none of the GNU/Linux package maintainers I have contacted have any major concerns over dropping it.

I propose that either:

(1) GCC 4.8 be made the minimum supported version immediately, or

(2) The trychooser tool should be extended to make it possible to
build with GCC 4.7 on any platforms where it is supported, and
bootstrap.py be updated to install GCC 4.7 alongside the
currently-installed compiler.

I greatly prefer option (1).

It is very inconvenient to have a minimum supported compiler version
that we cannot even do test builds with using tryserver.

Note that recent Ubuntu releases make it very easy to install GCC 4.8
but not so easy (AFAICT) to install other versions like GCC 4.7.

> I've contacted the package maintainers for Debian, Red Hat, openSUSE, SLES and Ubuntu and they either don't have a problem with dropping 4.6 or, at least, no more of a problem than the fact we've already dropped 4.4.

Did any of them state a preference for not going to GCC 4.8? If so,
what was the reasoning?

Thanks,
Brian

[Ryan VanderMeulen]

On 3/10/2015 10:23 PM, Brian Smith wrote:
> (2) The trychooser tool should be extended to make it possible to
> build with GCC 4.7 on any platforms where it is supported, and
> bootstrap.py be updated to install GCC 4.7 alongside the
> currently-installed compiler.

All Android and B2G JB/KK emulator builds are on GCC 4.7. Linux Desktop
and B2G ICS/L emulator builds are GCC 4.8. All of the aforementioned are
available on Trychooser.

-Ryan

[Mike Hommey]

On Tue, Mar 10, 2015 at 07:23:36PM -0700, Brian Smith wrote:
> <bo...@mozilla.com> wrote:
> > In summary: Officially make gcc-4.7 our minimum supported version. Fx38 and 39 don't compile with 4.6 and none of the GNU/Linux package maintainers I have contacted have any major concerns over dropping it.
>

> I propose that either:
>
> (1) GCC 4.8 be made the minimum supported version immediately, or
>

> (2) The trychooser tool should be extended to make it possible to
> build with GCC 4.7 on any platforms where it is supported, and
> bootstrap.py be updated to install GCC 4.7 alongside the
> currently-installed compiler.
>

> I greatly prefer option (1).
>
> It is very inconvenient to have a minimum supported compiler version
> that we cannot even do test builds with using tryserver.

Why this sudden requirement when our *current* minimum "supported"
version is 4.6 and 4.6 is nowhere close to that on try. That is also
true for older requirements we had for gcc. That is also true for clang
on OSX, and that was also true for the short period we had MSVC 2012 as
a minimum on Windows. I'm not saying this is an ideal situation, but I'd
like to understand why gcc needs to suddenly be treated differently.

> Note that recent Ubuntu releases make it very easy to install GCC 4.8
> but not so easy (AFAICT) to install other versions like GCC 4.7.
>

> > I've contacted the package maintainers for Debian, Red Hat, openSUSE, SLES and Ubuntu and they either don't have a problem with dropping 4.6 or, at least, no more of a problem than the fact we've already dropped 4.4.
>

> Did any of them state a preference for not going to GCC 4.8? If so,
> what was the reasoning?

At least for Debian, current stable can't build security updates with
more than 4.7.

Mike

[bo...@mozilla.com]

On Tuesday, March 10, 2015 at 5:06:31 PM UTC, Ehsan Akhgari wrote:

> >> Here's a push in case the bugs don't exist in certain builds:
> >> https://treeherder.mozilla.org/#/jobs?repo=try&revision=044e896fc6fa
> >
> > I was looking at the log for a B2G Device Image build (I think it
> > was the one for Nexus 5-L) a few days (less than a week) ago, and it
> > was using gcc 4.8 for the target and gcc 4.6 for the host (assuming
> > I was reading the logs correctly).
> >
> > I point this out because:
> >
> > (1) the device image build isn't in that try run, and
> >
> > (2) it's not clear to me that the compiler version checks check
> > both the target and host compiler versions when cross-compiling
>
> That's a good point. I don't think we have any checks for the host
> compiler.

I'll admit to knowing nothing about B2G Device Image builds, but I see that setting the default host compiler is in the instructions on the MDN B2G build instructions [1].
What are the reasons for this?
The Firefox OS instructions [2] seem to suggest this is only needed for older versions of B2G.
Are there automated Device Image builders that we will need to get updated?
Do we need to inform partners, who might be using gcc-4.6?

Am I right in thinking that B2G only currently uses up to Gecko 37?
This may explain why we haven't seen a problem, but we might start seeing one since a merge a week ago.
If we do a patch (for bug 1134487) is on mozilla-inbound and has been requested for uplift to beta.

[1] https://developer.mozilla.org/en-US/Firefox_OS/Preparing_for_your_first_B2G_build
[2] https://developer.mozilla.org/en-US/Firefox_OS/Firefox_OS_build_prerequisites

[Brian Smith]

Mike Hommey <m...@glandium.org> wrote:

> Brian Smith wrote:
>> It is very inconvenient to have a minimum supported compiler version
>> that we cannot even do test builds with using tryserver.
>
> Why this sudden requirement when our *current* minimum "supported"
> version is 4.6 and 4.6 is nowhere close to that on try. That is also
> true for older requirements we had for gcc. That is also true for clang
> on OSX, and that was also true for the short period we had MSVC 2012 as
> a minimum on Windows. I'm not saying this is an ideal situation, but I'd
> like to understand why gcc needs to suddenly be treated differently.

The current situation is very inconvenient. To improve it, all
compilers should be treated the same: Code that builds on
mozilla-inbound/central/tryserver is good enough to land, as far as
supported compiler versions are concerned. So, for example, if clang
3.7 is what is used on the builders, then clang 3.6 would be
unsupported. And the same with GCC and MSVC.

Further, it is best to upgrade compiler versions as fast as possible,
so that we can make more use of newer C++ features. I contributed many
patches in bug 1119072 so that MSVC 2015 can become the minimum MSVC
version ASAP. The same should happen with GCC and clang so that we can
write better code using newer C++ features ASAP. (This also requires
replacing STLPort with a reasonable C++ standard library
implementation on Android/B2G.)

>> Did any of them state a preference for not going to GCC 4.8? If so,
>> what was the reasoning?
>
> At least for Debian, current stable can't build security updates with
> more than 4.7.

Isn't this a chicken and egg problem? If Firefox required GCC 4.9 then
Debian would figure out a way to build security updates using GCC 4.9.
It is easier for Debian to insist on GCC 4.7 so that's what Debian
asks for. But, it is better to optimize for Mozilla developer
efficiency than any Linux distros' efficiency. In particular, things
like minimum compiler versions affect every Mozilla developer's
efficiency, which affects the rate at which we can ship improvements
to 100% of Mozilla's users. But, Linux-distro-packaged Firefox makes
up less than 1% of the userbase.

Note that I'm not saying Debian is unimportant. I'm saying that
Mozilla should focus on what's best for developer productivity, and
then assist Debian and others cope with whatever inconvenience that
that strategy causes them.

Cheers,
Brian

[bo...@mozilla.com]

On Wednesday, March 11, 2015 at 8:40:46 AM UTC, Mike Hommey wrote:
> On Tue, Mar 10, 2015 at 07:23:36PM -0700, Brian Smith wrote:
> > <bo...@mozilla.com> wrote:
> > > In summary: Officially make gcc-4.7 our minimum supported version. Fx38 and 39 don't compile with 4.6 and none of the GNU/Linux package maintainers I have contacted have any major concerns over dropping it.
> >
> > I propose that either:
> >
> > (1) GCC 4.8 be made the minimum supported version immediately, or

*snip*

> > > I've contacted the package maintainers for Debian, Red Hat, openSUSE, SLES and Ubuntu and they either don't have a problem with dropping 4.6 or, at least, no more of a problem than the fact we've already dropped 4.4.
> >
> > Did any of them state a preference for not going to GCC 4.8? If so,
> > what was the reasoning?
>
> At least for Debian, current stable can't build security updates with
> more than 4.7.

In addition, as I understand it:
* openSUSE still has supported versions with gcc-4.7
* SLES (on older versions without 4.6 or 4.7) use a special stack of packages to build Firefox which includes gcc-4.7
* Ubuntu 12.04 LTS has 4.6, but are happy to build with a later version as long as there are no runtime issues
* Older versions of RHEL are still on gcc-4.4, so they are already going to have to do something for Fx37 anyway and moving to 4.7 instead of 4.6 would probably not cause too much extra pain.

I don't know how much more work moving to gcc-4.8 would cause, but it would certainly be some.

Also, from what I can tell of the C++ features that gcc-4.8 enables (from [1]), none of them are available until MSVC 2015.
It seems likely that we'll be supporting MSVC 2013 until the next ESR, so I don't see that moving to 4.8 gives us any immediate benefits.

[1] https://developer.mozilla.org/en-US/docs/Using_CXX_in_Mozilla_code

[Brian Smith]

Thanks for sharing that. For the patches that I contribute to Gecko,
that actually works out OK, because it seems like all my code is built
on all those platforms. I The overall issue of wasting time supporting
relatively unimportant configurations, which aren't checked during
tryserver/inbound/central builds still applies, though.

Cheers,
Brian

[Brian Smith]

<bo...@mozilla.com> wrote:
> Also, from what I can tell of the C++ features that gcc-4.8 enables (from [1]), none of them are available until MSVC 2015.
> It seems likely that we'll be supporting MSVC 2013 until the next ESR, so I don't see that moving to 4.8 gives us any immediate benefits.
>
> [1] https://developer.mozilla.org/en-US/docs/Using_CXX_in_Mozilla_code

ESR is also an incredibly wasteful drag on Mozilla development. When
ESR was first proposed we agreed, as far as I understand, that we
would NOT hold back improvements to the real Firefox for the sake of
ESR. In fact, it is counterproductive to hold back changes like this
for ESR's benefit because doing so makes it *harder* to backport
changes to ESR. For example, imagine if you had updated the Chromium
code after the next ESR branched, and then 12 months later a serious
and hard-to-fix security bug in the old Chromium code was found*. It
would be much less work to backport the patch is the minimum compiler
version for ESR was as similar to the minimum compiler version for
real Firefox.

In the particular case of MSVC on Windows, it would be particularly
good to make MSVC 2015 the minimum compiler version ASAP for this
reason, assuming it will be possible at all. That's why I contributed
all the patches in bug 1119072 elsewhere to facilitate that.

Cheers,
Brian

* Of course, security researchers won't be looking at that old
Chromium code because so few people use Firefox ESR that it isn't
worth doing it. So, users of Firefox ESR will generally be less secure
than users of the real Firefox.

[bo...@mozilla.com]

On Wednesday, March 11, 2015 at 11:53:11 AM UTC, Brian Smith wrote:
> <bo...@mozilla.com> wrote:
> > Also, from what I can tell of the C++ features that gcc-4.8 enables (from [1]), none of them are available until MSVC 2015.
> > It seems likely that we'll be supporting MSVC 2013 until the next ESR, so I don't see that moving to 4.8 gives us any immediate benefits.
> >
> > [1] https://developer.mozilla.org/en-US/docs/Using_CXX_in_Mozilla_code
>
> ESR is also an incredibly wasteful drag on Mozilla development. When
> ESR was first proposed we agreed, as far as I understand, that we
> would NOT hold back improvements to the real Firefox for the sake of
> ESR. In fact, it is counterproductive to hold back changes like this
> for ESR's benefit because doing so makes it *harder* to backport
> changes to ESR. For example, imagine if you had updated the Chromium
> code after the next ESR branched, and then 12 months later a serious
> and hard-to-fix security bug in the old Chromium code was found*. It
> would be much less work to backport the patch is the minimum compiler
> version for ESR was as similar to the minimum compiler version for
> real Firefox.

Right, that's sort of the point I was trying (and probably failing) to make.
Given that we (and probably Chromium) would be unlikely to drop MSVC 2013 before our next ESR anyway, then a complex security fix that used new c++ features is less likely.
Unless the fix were in code only compiled with gcc, in which case we'd need to move to gcc-4.8 and look at alternatives for ESR.

Moving to 4.8 now gives us no new features and causes at least some work for maintainers and from what I can tell, work to upgrade some of our build systems.

If the policy is to not hold back improvements for the sake of ESR then the corollary to that should be that we don't push for changes in the ESR version just to protect it later.

I'll be happy to help move us to 4.8 or later when we are looking to make MSVC 2015 the minimum.

[Ehsan Akhgari]

On 2015-03-11 8:34 AM, bo...@mozilla.com wrote:
> Given that we (and probably Chromium) would be unlikely to drop MSVC 2013 before our next ESR anyway

That is not true. The compiler that we use to build ESR has nothing to
do with the minimum compiler requirements on mozilla-central. It's of
course convenient for backporting patches if they are the same, but it's
OK if they aren't.

[Ehsan Akhgari]

On 2015-03-11 7:35 AM, Brian Smith wrote:
> Mike Hommey <m...@glandium.org> wrote:
>> Brian Smith wrote:
>>> It is very inconvenient to have a minimum supported compiler version
>>> that we cannot even do test builds with using tryserver.
>>
>> Why this sudden requirement when our *current* minimum "supported"
>> version is 4.6 and 4.6 is nowhere close to that on try. That is also
>> true for older requirements we had for gcc. That is also true for clang
>> on OSX, and that was also true for the short period we had MSVC 2012 as
>> a minimum on Windows. I'm not saying this is an ideal situation, but I'd
>> like to understand why gcc needs to suddenly be treated differently.
>
> The current situation is very inconvenient. To improve it, all
> compilers should be treated the same: Code that builds on
> mozilla-inbound/central/tryserver is good enough to land, as far as
> supported compiler versions are concerned. So, for example, if clang
> 3.7 is what is used on the builders, then clang 3.6 would be
> unsupported. And the same with GCC and MSVC.

In my ideal world, instead of spending time debating what compilers we
should use, we would stop relying on the system compiler, and build on
all platforms with known good compilers of our choosing. That of course
requires some build system work to get the compilers installed etc and
unfortunately I don't think we have anyone lined up to do that work.
But if we ever got there, we could have total control over what
compilers are used to build our code, so we would not be affected by
external factors such as distros' compiler choice.

> Further, it is best to upgrade compiler versions as fast as possible,
> so that we can make more use of newer C++ features. I contributed many
> patches in bug 1119072 so that MSVC 2015 can become the minimum MSVC
> version ASAP. The same should happen with GCC and clang so that we can
> write better code using newer C++ features ASAP. (This also requires
> replacing STLPort with a reasonable C++ standard library
> implementation on Android/B2G.)

I don't think anybody disagrees with your assertion, but the problem as
always is finding people to do the actual work. For example, our clang
version that we use on builders has not changed for a couple of years or
so. Large scale changes such as changes to toolchains, replacing
libraries such as STLport with other libraries, etc. is an area that is
basically unowned.

Cheers,
Ehsan

[bo...@mozilla.com]

On Wednesday, March 11, 2015 at 5:14:39 PM UTC, Ehsan Akhgari wrote:
> On 2015-03-11 8:34 AM, bo...@mozilla.com wrote:
> > Given that we (and probably Chromium) would be unlikely to drop MSVC 2013 before our next ESR anyway
>
> That is not true. The compiler that we use to build ESR has nothing to
> do with the minimum compiler requirements on mozilla-central.

I wasn't suggesting that they are tied in some way.
I was just suggesting that there is a good chance that we won't have dropped MSVC 2013 by next March.

[Gregory Szorc]

On Wed, Mar 11, 2015 at 10:21 AM, Ehsan Akhgari <ehsan....@gmail.com>
wrote:

I would prefer the default build mode construct a chroot or Docker
environment [that is the same environment Mozilla uses in automation] and
we build in that. This build environment would be defined inside
mozilla-central in such a way that it is reproducible over time. e.g.
update your source repo to a commit from March 2015 and you automatically
get the build environment that was used in March 2015. Another aspect of
this switch is it gets us much closer to deterministic and reproducible
builds.

This is achievable today. Bug 1133877 tracks an aspect of it.

What I don't know is:

1) Who will write the code
2) Whether distros will use our official build environments.

I suspect distros will have concerns about using our build environment
(which is essentially a bare bones OS install) verbatim due to concerns
like dependency hell and other things that distros worry about. Those
reasons are valid.

So I guess we trend towards supporting 2 build modes: Mozilla's official
build environment via containers/chroots (preferred) or "host native" (for
the people who insist on using it). Only "host native" exists today and it
is a PITA.

[Joshua Cranmer 🐧]

On 3/11/2015 1:13 PM, Gregory Szorc wrote:
> So I guess we trend towards supporting 2 build modes: Mozilla's official
> build environment via containers/chroots (preferred) or "host native" (for
> the people who insist on using it). Only "host native" exists today and it
> is a PITA.

Using docker containers for most new contributor builds is likely to be
a poorer experience for contributors than not doing so--most
contributors are likely to want to run the built product on their local,
host side of things rather than within the container, and if the runtime
dependencies mismatch, the end result will be very painful.

--
Joshua Cranmer
Thunderbird and DXR developer
Source code archæologist

[Nicolas B. Pierron]

On 03/11/2015 07:13 PM, Gregory Szorc wrote:
> This build environment would be defined inside
> mozilla-central in such a way that it is reproducible over time. e.g.
> update your source repo to a commit from March 2015 and you automatically
> get the build environment that was used in March 2015. Another aspect of
> this switch is it gets us much closer to deterministic and reproducible
> builds.

Am I reading docker and reproducible in the same statement? Do we intend to
save one docker image for each modification of the script? How would we deal
with updates of the distribution? When should we update our dependencies?
Would we have to maintain a mirror for all versions of the the distribution?

When I complain about the build system of FirefoxOS being damn crazy because
you have to pull 106 repository over 108 that you only build once, this is
even crazier than that. I hope we are never going to claim that the safest
way of building Firefox is to download 8 GB of docker image.

> This is achievable today. Bug 1133877 tracks an aspect of it.

Yes, we could also use less invasive solutions. Something which will not
break users tools, while offering the option of a build environment which
can be controlled, reproduced, verified (reproducibility wise), signed by
Mozilla, shared between developers, and which works on Linux & Mac (&
Windows maybe one day …), such as what I started in Bug 1115107.

> What I don't know is:
>

> 2) Whether distros will use our official build environments.

I think answer would be something less polite than "no way!".

On the other hand, this sounds like a simple way to lose a minor part of our
user and the major part of our developers, if we are no longer going to
support anything else.

--
Nicolas B. Pierron

[bobow...@gmail.com]

This thread seems to have got de-railed somewhat.
Could someone help me with my questions above?
Thanks.

[Ehsan Akhgari]

On 2015-03-11 2:13 PM, Gregory Szorc wrote:
> On Wed, Mar 11, 2015 at 10:21 AM, Ehsan Akhgari <ehsan....@gmail.com
> <mailto:ehsan....@gmail.com>> wrote:
>
> On 2015-03-11 7:35 AM, Brian Smith wrote:
>

> and we build in that. This build environment would be defined inside

> mozilla-central in such a way that it is reproducible over time. e.g.
> update your source repo to a commit from March 2015 and you
> automatically get the build environment that was used in March 2015.
> Another aspect of this switch is it gets us much closer to deterministic
> and reproducible builds.

Docker will be a Linux specific solution though, right? Are there
similar plans for other platforms?

[Juan Gómez]

AFAIK, in B2G we are tied to the same host compiler version that our Android base platform needs. This is gcc-4.6 for ICS and JB devices (not really sure if KK/LL works with newer ones, but I'll bet they do).
Of course, we could fix any incompatibilty and create our patches/forks so we can bump up our host compiler version, but moving away from Android base requierments is something that we should take with care.

As Ryan stated earlier Gecko is being built with gcc-4.8 for all ICS/LL devices and gcc-4.7 for JB/KK (except Flatfish which is gcc-4.8 too), and I was going to start working on migrating to gcc-4.9 within the next coming days. When I finish, we could discuss if we should migrate _all_ B2G devices to the same toolchain version... but this is another story.

[Chris Peterson]

On 3/11/15 2:56 PM, Juan Gómez wrote:
> AFAIK, in B2G we are tied to the same host compiler version that our Android base platform needs. This is gcc-4.6 for ICS and JB devices (not really sure if KK/LL works with newer ones, but I'll bet they do).
> Of course, we could fix any incompatibilty and create our patches/forks so we can bump up our host compiler version, but moving away from Android base requierments is something that we should take with care.

How long do we need to continue building B2G trunk for ICS and JB? Do we
have partner commitments to ship future releases from B2G trunk on ICS
or JB devices?

[Gregory Szorc]

On Wed, Mar 11, 2015 at 11:50 AM, Joshua Cranmer 🐧 <Pidg...@gmail.com>
wrote:

> On 3/11/2015 1:13 PM, Gregory Szorc wrote:
>
>> So I guess we trend towards supporting 2 build modes: Mozilla's official
>> build environment via containers/chroots (preferred) or "host native" (for
>> the people who insist on using it). Only "host native" exists today and it
>> is a PITA.
>>
>
> Using docker containers for most new contributor builds is likely to be a
> poorer experience for contributors than not doing so--most contributors are
> likely to want to run the built product on their local, host side of things
> rather than within the container, and if the runtime dependencies mismatch,
> the end result will be very painful.

I think developers should be able to make the choice between:

a) build and develop in the exact same environment that Mozilla uses
officially [so that compiler errors, bugs, object code, etc are identical]
b) build for the host and accept some risk that behavior of the built
product may differ from what happens in automation

I'm pretty sure given the agony many of us have felt trying to reproduce
failures in Try or other parts of automation that "a" would be a reasonable
default for many scenarios. I think minimizing surprises would be good for
developer productivity. Now, getting this to "just work" does have its
challenges.

[Gregory Szorc]

On Wed, Mar 11, 2015 at 12:57 PM, Nicolas B. Pierron <
nicolas....@mozilla.com> wrote:

> On 03/11/2015 07:13 PM, Gregory Szorc wrote:
>
>> This build environment would be defined inside
>> mozilla-central in such a way that it is reproducible over time. e.g.
>> update your source repo to a commit from March 2015 and you automatically
>> get the build environment that was used in March 2015. Another aspect of
>> this switch is it gets us much closer to deterministic and reproducible
>> builds.
>>
>
> Am I reading docker and reproducible in the same statement? Do we intend
> to save one docker image for each modification of the script? How would we
> deal with updates of the distribution? When should we update our
> dependencies? Would we have to maintain a mirror for all versions of the
> the distribution?
>
> When I complain about the build system of FirefoxOS being damn crazy
> because you have to pull 106 repository over 108 that you only build once,
> this is even crazier than that. I hope we are never going to claim that
> the safest way of building Firefox is to download 8 GB of docker image.
>

I'm not sure what this will look like. In theory there will be a different
docker image for each modification of the script. That does conjure up
fears of having to download a massive new image with each new change.
First, the environment doesn't change that often. Second, I think we can do
better than pure Docker image distribution. There are more intelligent ways
to build and distribute images than stacking filesystem layers. I suspect
we can come up with something that doesn't waste bandwidth as much as
Docker images.

>
> This is achievable today. Bug 1133877 tracks an aspect of it.
>>
>
> Yes, we could also use less invasive solutions. Something which will not
> break users tools, while offering the option of a build environment which
> can be controlled, reproduced, verified (reproducibility wise), signed by
> Mozilla, shared between developers, and which works on Linux & Mac (&
> Windows maybe one day …), such as what I started in Bug 1115107.
>
> What I don't know is:
>>
>> 2) Whether distros will use our official build environments.
>>
>
> I think answer would be something less polite than "no way!".
>
> On the other hand, this sounds like a simple way to lose a minor part of
> our user and the major part of our developers, if we are no longer going to
> support anything else.
>

Yes, one of the benefits of doing "host builds" today is that we test on
many different platforms and distributions. We know pretty quickly when
Firefox breaks on a certain distro because we have Firefox developers
running that distro. Going hard into building from well-defined and
isolated environments runs the risk of delaying detection of breakage and
pushing the problem downstream. That's something that should be considered.

At the end of the day, I want to give developers a choice of what
environment to use. Many don't want to deal with things like breakages due
to incompatibilities with the host environment. If we can give them an
isolated environment that "just works" so they don't have to worry about
fire drills, I consider that a win for developer productivity.

[Gregory Szorc]

On Wed, Mar 11, 2015 at 2:30 PM, Ehsan Akhgari <ehsan....@gmail.com>
wrote:

>> and we build in that. This build environment would be defined inside

>> mozilla-central in such a way that it is reproducible over time. e.g.
>> update your source repo to a commit from March 2015 and you
>> automatically get the build environment that was used in March 2015.
>> Another aspect of this switch is it gets us much closer to deterministic
>> and reproducible builds.
>>
>

> Docker will be a Linux specific solution though, right? Are there similar
> plans for other platforms

In terms of system parity, Linux is the most difficult. Every distro is
different. Windows and OS X are more uniform. We get much more consistency
and predictability on these platforms, especially Windows, where
MozillaBuild is essentially a self-contained build environment (albeit with
the OS-level isolation that Docker or LXC provides). I'm pretty sure we
deal with more system incompatibilities issues from build system land with
Linux than we do OS X and Windows combined.

Supporting a well-defined and isolated build environment for Windows and
Mac is more difficult. We even have licensing issues complicating matters.
We do pretty well on both fronts today, even without the "strong" isolation
provided by Docker/LXC. Without Docker, I think we'd be fine with continued
use of MozillaBuild and something like a Mozilla-specific Homebrew/MacPorts
install or something.

[Ehsan Akhgari]

The emulator tests currently only run on ICS.

[Ehsan Akhgari]

On 2015-03-11 8:36 PM, Gregory Szorc wrote:
> On Wed, Mar 11, 2015 at 2:30 PM, Ehsan Akhgari <ehsan....@gmail.com

> <mailto:ehsan....@gmail.com>> wrote:
>
> On 2015-03-11 2:13 PM, Gregory Szorc wrote:
>
> On Wed, Mar 11, 2015 at 10:21 AM, Ehsan Akhgari
> <ehsan....@gmail.com <mailto:ehsan....@gmail.com>

> <mailto:ehsan.akhgari@gmail.__com

> <mailto:ehsan....@gmail.com>>> wrote:
>
> On 2015-03-11 7:35 AM, Brian Smith wrote:
>
> Mike Hommey <m...@glandium.org <mailto:m...@glandium.org>

> <mailto:m...@glandium.org <mailto:m...@glandium.org>>> wrote:
>
> Brian Smith wrote:
>
> It is very inconvenient to have a minimum supported
> compiler version
> that we cannot even do test builds with using
> tryserver.
>
>
> Why this sudden requirement when our *current* minimum
> "supported"
> version is 4.6 and 4.6 is nowhere close to that on
> try. That
> is also
> true for older requirements we had for gcc. That is
> also
> true for clang
> on OSX, and that was also true for the short period
> we had
> MSVC 2012 as
> a minimum on Windows. I'm not saying this is an ideal
> situation, but I'd
> like to understand why gcc needs to suddenly be treated
> differently.
>
>
> The current situation is very inconvenient. To improve
> it, all
> compilers should be treated the same: Code that builds on

> mozilla-inbound/central/__tryserver is good enough to

Hmm, why not do something much simpler then, which is downloading a
prebuilt zip file that contains the toolchain? The toolchains on Mac
and Linux are redistributable, and for Windows we can redistribute a
little python script that creates the toolchain from a Visual Studio
Commmunity Edition installation.

[Ehsan Akhgari]

On 2015-03-11 8:19 PM, Gregory Szorc wrote:
> On Wed, Mar 11, 2015 at 11:50 AM, Joshua Cranmer 🐧 <Pidg...@gmail.com>
> wrote:
>

>> On 3/11/2015 1:13 PM, Gregory Szorc wrote:
>>
>>> So I guess we trend towards supporting 2 build modes: Mozilla's official
>>> build environment via containers/chroots (preferred) or "host native" (for
>>> the people who insist on using it). Only "host native" exists today and it
>>> is a PITA.
>>>
>>
>> Using docker containers for most new contributor builds is likely to be a
>> poorer experience for contributors than not doing so--most contributors are
>> likely to want to run the built product on their local, host side of things
>> rather than within the container, and if the runtime dependencies mismatch,
>> the end result will be very painful.
>
>

> I think developers should be able to make the choice between:
>
> a) build and develop in the exact same environment that Mozilla uses
> officially [so that compiler errors, bugs, object code, etc are identical]
> b) build for the host and accept some risk that behavior of the built
> product may differ from what happens in automation
>
> I'm pretty sure given the agony many of us have felt trying to reproduce
> failures in Try or other parts of automation that "a" would be a reasonable
> default for many scenarios. I think minimizing surprises would be good for
> developer productivity. Now, getting this to "just work" does have its
> challenges.

Note that in practice, in 99.99% of cases it is the characteristics of
the test machine that cause us to be unable to reproduce try failures,
not the exact toolchain. [1] (a) is actually not that valuable for a
normal developer's workflow for reproducing the binaries that our
infrastructure produces. What burns *most* people is differences in the
compiler errors which occurs because of using different compiler (major)
versions, for which (a) sounds like overkill. I think a simpler
solution covers almost all of developers' needs.

[1] Anecdotally, I can probably count the number of times I have been
bitten by a bug that required the exact setup of our build infra to find
its way into the binaries we generate on one hand, and not even using
all 5 fingers. :-)

[bobow...@gmail.com]

I've just landed patches to change our minimum supported target GCC version to 4.7.

This does not affect the host compiler as mentioned earlier in this thread.
That check is being added in bug 1142352.
Once the blocking issue there has been sorted, it looks like the minimum version for the host compiler will be set to 4.6.

As far as I can tell the host and target versions of GCC on try are all 4.7 or later.
So, it may be that if you use features that are only available in 4.7 or later and these are compiled using the host compiler as well, then you will get build failures somewhere else in the release pipeline.
My (admittedly pretty incomplete) understanding is that this doesn't affect too many areas.
Some breakpad tools and nsinstall were mentioned on IRC, but there may be others.

Cheers,
Bob

[bo...@mozilla.com]

I have now updated the two MDN pages (that I know of) to reflect that gcc-4.7 or later is now required.

<https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions/Linux_Prerequisites>
<https://developer.mozilla.org/en-US/docs/Using_CXX_in_Mozilla_code>

This has made the following C++11 features available for use in Mozilla code:

* member initializers
* template aliases
* delegated constructors
* override and final as C++ keywords

Cheers,
Bob

[Mike Hommey]

On Wed, Mar 11, 2015 at 02:56:27PM -0700, Juan Gómez wrote:
> AFAIK, in B2G we are tied to the same host compiler version that our
> Android base platform needs. This is gcc-4.6 for ICS and JB devices
> (not really sure if KK/LL works with newer ones, but I'll bet they
> do).

I overlooked this when I (re)landed bug 1142352, and the host compiler
version we now require on central (well, inbound as of writing) is 4.7.
This didn't blow up B2G device images on treeherder so I guess we are
ok, OTOH, I don't know if Nexus 4 and Dolphin images are ICS/JB or KK/L.

We /could/ repatch to make 4.6 the minimum for host compiler, but not
having the same minimum version between host and target is dangerous
because some of the host code is using MFBT and may end up requiring 4.7
through MFBT anyways.

Mike

[Andreas Pehrson]

I think the gonk parts always use the provided AOSP toolchain for building,
while gecko uses the compiler provided by the system when building for the
host.

In any case I had a keon (ICS) build fail to compile libmar with gcc 4.6
due to the new requirement. Using gcc 4.8 on the system instead worked fine
for building from scratch, including an OTA package - which IIRC builds
some AOSP tooling for the host.

Dolphin is kitkat so not affected. Not sure about the Nexus.

*Andreas Pehrson *--- Software Engineer

(+47) 959 60 374 | peh...@telenordigital.com |
www.telenordigital.com

On Wed, Mar 25, 2015 at 3:39 PM, Mike Hommey <m...@glandium.org> wrote:

> On Wed, Mar 11, 2015 at 02:56:27PM -0700, Juan Gómez wrote:

> > AFAIK, in B2G we are tied to the same host compiler version that our
> > Android base platform needs. This is gcc-4.6 for ICS and JB devices
> > (not really sure if KK/LL works with newer ones, but I'll bet they
> > do).
>

> I overlooked this when I (re)landed bug 1142352, and the host compiler
> version we now require on central (well, inbound as of writing) is 4.7.
> This didn't blow up B2G device images on treeherder so I guess we are
> ok, OTOH, I don't know if Nexus 4 and Dolphin images are ICS/JB or KK/L.
>
> We /could/ repatch to make 4.6 the minimum for host compiler, but not
> having the same minimum version between host and target is dangerous
> because some of the host code is using MFBT and may end up requiring 4.7
> through MFBT anyways.
>
> Mike

> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>