Build system Q4 goals / meeting this week

[Gregory Szorc]

We will be having our regularly scheduled build system meeting this week
on Thursday, October 17 at 1430 PDT / 1730 EDT in Gregory Szorc's Vidyo
room. Backchannel is #build. Agenda at
https://etherpad.mozilla.org/build-config-meeting.

The big topic at this week's meeting will be discussing and formalizing
our Q4 goals.

As module owner, I will drive this discussion by first proposing a list
of Q4 goals.

The #1 priority for the build module should be making builds as fast as
possible while maintaining correctness. This includes all flavors of
builds - clobber and incremental - and where the build is performed -
local, release automation, Windows, Linux, OS X, etc.

While still important, goals related to build tools, developer
interaction with the build system, the sanity of the low-level build
config (read: whether things are in make files or moz.build files), and
integration with release automation are not as important as making the
build faster. I'm not saying we shouldn't do them - just that we should
be careful about prioritization. Also, many of these overlap with speed
wins, so they can be difficult to silo.

Before I begin, I want to introduce some new terms into our build
vocabulary: "light builds" and "heavy builds."

I tend to view the amount of work performed by the build system as a
spectrum with clobber builds and no-op builds on opposite ends. Clobber
builds perform all work imaginable. No-op builds perform as little work
as possible (ideally none). In between both extremes sit incremental
builds. These are builds where some subset of files has changed. But
incremental builds are a huge area of this spectrum. Behavior and
optimization strategy is different depending on whether an incremental
build is closer to a clobber or a no-op build. I therefore would like to
introduce the terms "light build" and "heavy build" to refer to an
incremental build that sits nearer the no-op and clobber extremes,
respectively. It is worth noting that "light builds" represent the
common developer workflow: change a small number of files and rebuild.
An example of a "heavy build" is your daily pull from mozilla-central
(or similar integration branch).

Now, on to the proposed goals list.

Build with GNU Make 4.0 on Windows
==================================

GNU Make 4.0 was released last week. It appears to fix the bugs that
prevented us from using GNU Make on Windows. And, it appears to be
significantly faster than pymake.

We have already landed support for building with GNU Make on Windows.
Provided it's stable, we should roll this out to all project branches
and ship a new MozillaBuild with the new binary. The remaining work is
largely outside the core build group, but this is a major improvement
and we need to track it.

A subgoal is to follow up and remove pymake from the tree.

Achieving this goal will make all flavors of builds faster for users
currently using pymake (Windows). It will go a long way to plugging the
gap between the excessively long build times on Windows, discouraging
fewer people from building on Windows and resulting in more Firefox
features written on the platform used by a majority of its users.

Minimize actions that run during light and no-op builds
=======================================================

We have a number of build actions that run unconditionally, either due
to clownshoes (like removing large parts of dist/) or due to bad build
rules. These build actions add excessive overhead to light builds. Light
builds are so slow that developers commonly perform a partial tree build
and hope they build everything they needed to. This is playing with fire
and this practice needs to be curtailed through us providing faster
light builds.

This goal is about minimizing the amount of unneccessary build actions
we perform.

Specific actions in this goal include:

* Tracking more files in install manifests
* Finish moving all mochitests to manifests
* Add JS files to manifests (likely via moz.build porting)
* Conditionally processing jar.mn files
* Handle headers from external projects better
* Handle XPCOM manifest files more efficiently (likely through some
moz.build powered mechansim)
* Proper dependencies for Preprocessor.py
* Capture INSTALL_TARGETS usage in moz.build

Progress towards this goal is measured by comparing the number of
non-make processes launched during no-op and light builds over time.
This number needs to go down. Drastically.

I have doubts that we will achieve all items in the list above during
this quarter. This goal may therefore bleed into Q1.

Minimize ICU impact
===================

ICU has been a world of pain for the build system. It makes builds
significantly slower and it is often the long pole, especially for
builds close to the no-op side of the spectrum.

We need to mitigate the impact of ICU on the build system. This might
mean loudly recommending people not build it (although more and more
features want to start using it, so likely not a long term solution).
This might mean someone contributing patches back to ICU to make its
build system suck less.

Cross-compile Mac builds from Linux
===================================

This is a request from Release Engineering. For capacity reasons, we
want to perform Mac builds on Linux. We should add support for this.

Generate IDE project files for Fennec
=====================================

The title should be self-explanatory. I imagine this will largely be
Nick Alexander's personal goal. But I wanted to capture it here.

Land an initial Tup backend
===========================

The long term goal is to rid ourselves of make. Of all the backends that
can replace it, we like Tup the best because of its robustness and
cross-platform support. We should land an initial Tup backend in the tree.

The initial landing doesn't need to do everything. If all it does is
compile (not even link) C++, that's a win, as something is better than
noting.

Since it will come up, my opinion on Ninja is that anything Ninja can do
Tup can do. Tup wins the tiebreaker. I would prefer we ignore Ninja and
focus on Tup.

Improve SpiderMonkey build system integration
=============================================

The way SpiderMonkey integrates with the build system is very hacky.
Nobody likes it - including the SpiderMonkey developers. Furthermore,
the way things are slows down builds. e.g. SpiderMonkey has its own
configure script, which is extremely similar to the one in m-c. This
slows down configure time, especially on Windows.

The goal here is to integrate SpiderMonkey into the main build system.
We'll remove the copying of files from config/ and build/ into js/src/.
We'll also provide mach command(s) for building just SpiderMonkey
(they've asked me for this).

(I am receptive to punting this to Q1.)

Improvements to mach
====================

Mach is being relied on more and more. It needs to grow some features to
meet demands.

There are many options being landed in mach commands that are
effectively user preferences. There is no good way to define sticky
defaults for those options. mach needs config files so people can tailor
their environments to their liking. We should add config file support to
mach.

Mach has become slower as more commands are added. We should look at
making it faster.

Support for detecting what commands are runnable at run-time has been
added recently. When you run |mach help| it even prunes the list to
commands that should work. We should hook up most of the commands to
this mechanism so people can't do things like attempt to run tests when
the tree isn't built.

Mach integration with comm-central and b2g is fragile. We should make
that work so we don't have to continue supporting legacy interfaces and
so developers there don't complain so much about not being able to use
mach. This is more important than ever since functionality like test
running is starting to require mach.

Many of the improvements to mach will make excellent bugs for new or
casual contributors.

Finish optimizing libxul building rules
=======================================

The MOZ_PSEUDO_RERECURSE work with the "binaries" make target has done a
lot to optimize the way that libxul is built. However, there are still a
few steps performed during the libs tier, including linking itself.
Furthermore, not all optimizations are enabled by default everywhere yet.

We should finish what we've started. We should enable the build system
to produce libxul without a slow/recursive libs tier traversal. This
build mode should be enabled on all release automation platforms.

Fix GYP dependencies
====================

Changes to .gypi files aren't picked up by the build config. We require
a clobber whenever some GYP projects are changed. An almost-working
patch has been written. We just need to put on the finishing touches.

Other goals
===========

There is no shortage of other things we could work on. If you have an
idea for an additional goal, please propose it. If you are already
working on something or have an idea that you plan to do in Q4, let's
capture it.

This list is in no way final. I anticipate parts of goals will get axed
and other goals may get split up.

I'm interested in having as much feedback as possible before the meeting
tomorrow so we don't have to waste time during the meeting.

What do people think? Is this a reasonable and achievable list?

Gregory

[Nick Alexander]

On 13-10-16 2:38 PM, Gregory Szorc wrote:
> We will be having our regularly scheduled build system meeting this week
> on Thursday, October 17 at 1430 PDT / 1730 EDT in Gregory Szorc's Vidyo
> room. Backchannel is #build. Agenda at
> https://etherpad.mozilla.org/build-config-meeting.
>
> The big topic at this week's meeting will be discussing and formalizing
> our Q4 goals.
>
> As module owner, I will drive this discussion by first proposing a list
> of Q4 goals.

<snip>

> Generate IDE project files for Fennec
> =====================================
>
> The title should be self-explanatory. I imagine this will largely be
> Nick Alexander's personal goal. But I wanted to capture it here.

A small point, but there is a cast of... ones... who are helping with
this effort, not least bnicholson, who is doing a lot of the tooling and
testing.

If you're interested in this work, I have been posting progress updates
in mobile-firefox-dev, and the meta bug is
https://bugzilla.mozilla.org/show_bug.cgi?id=924961.

> Other goals
> ===========
>
> There is no shortage of other things we could work on. If you have an
> idea for an additional goal, please propose it. If you are already
> working on something or have an idea that you plan to do in Q4, let's
> capture it.

I would like to consider "app tier only" builds. That is, we let devs
download the binaries (libxul, etc) needed to develop and just work on
front-end stuff. This makes sense in browser/ and in mobile/android,
but the concerns are particularly well separated in mobile/android. It
seems similar to gaia development; perhaps there is a cow-path in place
for this development mode? I have some ideas for driving this in
mobile/android, but I'm not sure how to work into a larger build system
mode.

Is this of general interest?

> What do people think? Is this a reasonable and achievable list?

Reasonable? Yes. Achievable? No, not all of it.

Best,
Nick

[Nicholas Nethercote]

On Wed, Oct 16, 2013 at 2:38 PM, Gregory Szorc <g...@mozilla.com> wrote:
>
> Before I begin, I want to introduce some new terms into our build
> vocabulary: "light builds" and "heavy builds."

I like it! And I wonder if these terms should be made more specific.
E.g. pick three specific light builds, something like:

- touch xpcom/glue/pldhash.cpp and rebuild
- touch nsprpub/lib/ds/plarena.cpp and rebuild
- touch jsapi.cpp and rebuild

(Those are just suggestions, files off the top of my head that stress
the build system in different ways. You can probably think of better
ones.)

Some ideas for specific heavy builds:

- touch mfbt/*.h and rebuild
- touch js/src/jsapi.h and rebuild
- touch xpcom/base/nsISupports.idl and rebuild
- touch configure.in and rebuild(?)

This would make the speed-up goals more precise, and avoid the problem
that no-op builds and clobber builds are the only two kinds of builds
that are precisely defined at the moment (and therefore are easy to
overfocus on).

> Minimize actions that run during light and no-op builds
> =======================================================

I'd remove the "and no-op" from this. Nobody runs no-op builds.
They're easy to measure, but their speed shouldn't be a primary goal.

Of course, analyzing no-op builds can give insight into the operation
of more interesting kinds of builds, and optimizations that speed up
other builds will probably speed up no-op builds. But I'll be sad if,
at the end of the quarter, "we sped up no-op builds by X%" is listed
as an achievement. In contrast, "we sped up <some specific non-no-op
build> by Y%" would be a worthy achievement.

> Improve SpiderMonkey build system integration
> =============================================

Wow, that would be nice.


Nick

[Gary Kwong]

> Improve SpiderMonkey build system integration
> =============================================
>
> The way SpiderMonkey integrates with the build system is very hacky.
> Nobody likes it - including the SpiderMonkey developers. Furthermore,
> the way things are slows down builds. e.g. SpiderMonkey has its own
> configure script, which is extremely similar to the one in m-c. This
> slows down configure time, especially on Windows.
>
> The goal here is to integrate SpiderMonkey into the main build system.
> We'll remove the copying of files from config/ and build/ into js/src/.
> We'll also provide mach command(s) for building just SpiderMonkey
> (they've asked me for this).

I like the way currently that I copy over the js/ directory and a small
subset of other directories e.g. build/ to compile only the js shell in
another folder. Please have a way for mach(?) to compile only the js
shell when only the js/ folder is present, this will be very nice.

-Gary

[Axel Hecht]

On 10/16/13 11:38 PM, Gregory Szorc wrote:
> Cross-compile Mac builds from Linux
> ===================================
>
> This is a request from Release Engineering. For capacity reasons, we
> want to perform Mac builds on Linux. We should add support for this.

This one's interesting also for repacks. Back in the days, they were
50/50 setup vs repack, and if we can do multiple platforms on linux, all
the per-locale setup is shared, like getting the repo, updating, running
l10n-merge.

IIRC, nsis also allows to generate windows installers from linux boxes,
so we could even do all platforms on just one run on linux.

Axel

[Gregory Szorc]

Supporting SpiderMonkey only builds and creating a standalone
SpiderMonkey source archive is a hard requirement.

[Gary Kwong]

> Supporting SpiderMonkey only builds and creating a standalone
> SpiderMonkey source archive is a hard requirement.

I'm not sure I follow. "hard" as in "difficult", or "solid, definite"?

[Gregory Szorc]

Definite.