We will be having our regularly scheduled build system meeting this week
on Thursday, October 17 at 1430 PDT / 1730 EDT in Gregory Szorc's Vidyo
room. Backchannel is #build. Agenda at
https://etherpad.mozilla.org/build-config-meeting.
The big topic at this week's meeting will be discussing and formalizing
our Q4 goals.
As module owner, I will drive this discussion by first proposing a list
of Q4 goals.
The #1 priority for the build module should be making builds as fast as
possible while maintaining correctness. This includes all flavors of
builds - clobber and incremental - and where the build is performed -
local, release automation, Windows, Linux, OS X, etc.
While still important, goals related to build tools, developer
interaction with the build system, the sanity of the low-level build
config (read: whether things are in make files or moz.build files), and
integration with release automation are not as important as making the
build faster. I'm not saying we shouldn't do them - just that we should
be careful about prioritization. Also, many of these overlap with speed
wins, so they can be difficult to silo.
Before I begin, I want to introduce some new terms into our build
vocabulary: "light builds" and "heavy builds."
I tend to view the amount of work performed by the build system as a
spectrum with clobber builds and no-op builds on opposite ends. Clobber
builds perform all work imaginable. No-op builds perform as little work
as possible (ideally none). In between both extremes sit incremental
builds. These are builds where some subset of files has changed. But
incremental builds are a huge area of this spectrum. Behavior and
optimization strategy is different depending on whether an incremental
build is closer to a clobber or a no-op build. I therefore would like to
introduce the terms "light build" and "heavy build" to refer to an
incremental build that sits nearer the no-op and clobber extremes,
respectively. It is worth noting that "light builds" represent the
common developer workflow: change a small number of files and rebuild.
An example of a "heavy build" is your daily pull from mozilla-central
(or similar integration branch).
Now, on to the proposed goals list.
Build with GNU Make 4.0 on Windows
==================================
GNU Make 4.0 was released last week. It appears to fix the bugs that
prevented us from using GNU Make on Windows. And, it appears to be
significantly faster than pymake.
We have already landed support for building with GNU Make on Windows.
Provided it's stable, we should roll this out to all project branches
and ship a new MozillaBuild with the new binary. The remaining work is
largely outside the core build group, but this is a major improvement
and we need to track it.
A subgoal is to follow up and remove pymake from the tree.
Achieving this goal will make all flavors of builds faster for users
currently using pymake (Windows). It will go a long way to plugging the
gap between the excessively long build times on Windows, discouraging
fewer people from building on Windows and resulting in more Firefox
features written on the platform used by a majority of its users.
Minimize actions that run during light and no-op builds
=======================================================
We have a number of build actions that run unconditionally, either due
to clownshoes (like removing large parts of dist/) or due to bad build
rules. These build actions add excessive overhead to light builds. Light
builds are so slow that developers commonly perform a partial tree build
and hope they build everything they needed to. This is playing with fire
and this practice needs to be curtailed through us providing faster
light builds.
This goal is about minimizing the amount of unneccessary build actions
we perform.
Specific actions in this goal include:
* Tracking more files in install manifests
* Finish moving all mochitests to manifests
* Add JS files to manifests (likely via moz.build porting)
* Conditionally processing
jar.mn files
* Handle headers from external projects better
* Handle XPCOM manifest files more efficiently (likely through some
moz.build powered mechansim)
* Proper dependencies for Preprocessor.py
* Capture INSTALL_TARGETS usage in moz.build
Progress towards this goal is measured by comparing the number of
non-make processes launched during no-op and light builds over time.
This number needs to go down. Drastically.
I have doubts that we will achieve all items in the list above during
this quarter. This goal may therefore bleed into Q1.
Minimize ICU impact
===================
ICU has been a world of pain for the build system. It makes builds
significantly slower and it is often the long pole, especially for
builds close to the no-op side of the spectrum.
We need to mitigate the impact of ICU on the build system. This might
mean loudly recommending people not build it (although more and more
features want to start using it, so likely not a long term solution).
This might mean someone contributing patches back to ICU to make its
build system suck less.
Cross-compile Mac builds from Linux
===================================
This is a request from Release Engineering. For capacity reasons, we
want to perform Mac builds on Linux. We should add support for this.
Generate IDE project files for Fennec
=====================================
The title should be self-explanatory. I imagine this will largely be
Nick Alexander's personal goal. But I wanted to capture it here.
Land an initial Tup backend
===========================
The long term goal is to rid ourselves of make. Of all the backends that
can replace it, we like Tup the best because of its robustness and
cross-platform support. We should land an initial Tup backend in the tree.
The initial landing doesn't need to do everything. If all it does is
compile (not even link) C++, that's a win, as something is better than
noting.
Since it will come up, my opinion on Ninja is that anything Ninja can do
Tup can do. Tup wins the tiebreaker. I would prefer we ignore Ninja and
focus on Tup.
Improve SpiderMonkey build system integration
=============================================
The way SpiderMonkey integrates with the build system is very hacky.
Nobody likes it - including the SpiderMonkey developers. Furthermore,
the way things are slows down builds. e.g. SpiderMonkey has its own
configure script, which is extremely similar to the one in m-c. This
slows down configure time, especially on Windows.
The goal here is to integrate SpiderMonkey into the main build system.
We'll remove the copying of files from config/ and build/ into js/src/.
We'll also provide mach command(s) for building just SpiderMonkey
(they've asked me for this).
(I am receptive to punting this to Q1.)
Improvements to mach
====================
Mach is being relied on more and more. It needs to grow some features to
meet demands.
There are many options being landed in mach commands that are
effectively user preferences. There is no good way to define sticky
defaults for those options. mach needs config files so people can tailor
their environments to their liking. We should add config file support to
mach.
Mach has become slower as more commands are added. We should look at
making it faster.
Support for detecting what commands are runnable at run-time has been
added recently. When you run |mach help| it even prunes the list to
commands that should work. We should hook up most of the commands to
this mechanism so people can't do things like attempt to run tests when
the tree isn't built.
Mach integration with comm-central and b2g is fragile. We should make
that work so we don't have to continue supporting legacy interfaces and
so developers there don't complain so much about not being able to use
mach. This is more important than ever since functionality like test
running is starting to require mach.
Many of the improvements to mach will make excellent bugs for new or
casual contributors.
Finish optimizing libxul building rules
=======================================
The MOZ_PSEUDO_RERECURSE work with the "binaries" make target has done a
lot to optimize the way that libxul is built. However, there are still a
few steps performed during the libs tier, including linking itself.
Furthermore, not all optimizations are enabled by default everywhere yet.
We should finish what we've started. We should enable the build system
to produce libxul without a slow/recursive libs tier traversal. This
build mode should be enabled on all release automation platforms.
Fix GYP dependencies
====================
Changes to .gypi files aren't picked up by the build config. We require
a clobber whenever some GYP projects are changed. An almost-working
patch has been written. We just need to put on the finishing touches.
Other goals
===========
There is no shortage of other things we could work on. If you have an
idea for an additional goal, please propose it. If you are already
working on something or have an idea that you plan to do in Q4, let's
capture it.
This list is in no way final. I anticipate parts of goals will get axed
and other goals may get split up.
I'm interested in having as much feedback as possible before the meeting
tomorrow so we don't have to waste time during the meeting.
What do people think? Is this a reasonable and achievable list?
Gregory