Short-term plan to provide fast builds for Firefox front-end developers

[Mike Hommey]

Hi,

Following is a proposal for short term wins for Firefox front end developers.

The starting point is that there are too many things to change in the
recursive make build system to allow for something like this to work
properly for many different use cases. This also starts from the fact
that, well, we've kept the recursive make build system use the same
model it's been using for, well, close to 20 years. That model is
flawed, and I thought it was about time to do something about it, while
addressing real needs at the same time.

My proposal is to start a new build backend from scratch, with the aim
to grow it into a complete solution. Build backend means I'm not talking
about replacing moz.build files with something new. I'm talking about
the build system they're munged into.

It would initially piggy back on the existing build system, because
there are things that aren't exposed in moz.build yet that are
necessary. It would also, initially, not care about the entire build,
and even less about things that are outside of Firefox (tests, addons,
etc.) or compilation.

My initial work is this patch:
https://pastebin.mozilla.org/8846539

(Note: it requires the patches from bug 1204712, bug 1204719 and bug
1204715, all landed on mozilla-inbound, as of writing)

What it allows is to:
- Run `mach configure` with a mozconfig with
ac_add_options --disable-compile-environment
- Download and unpack a nightly
- Use toolkit/mozapps/installer/unpack.py to explode its omni.ja files
- Move that unpacked nightly in $objdir/dist/bin (for mac, that involves
more fiddling, because dist/bin is a somewhat flattened version of the
.app directory)
- Ensure the files in $objdir/dist/bin are older than the source
files.
- Run `mach build-backend -b FasterMake`
- Run `mach build faster`

After the last command, $objdir/dist/bin should contain a bastardized
Firefox, with xul, js, etc. coming from the source tree, and the
remainder still being there from the original nightly.

`mach run` should work with that.

Note: removing files is not supported in this initial implementation, so
removing files will still require manual intervention. Editing files
#include'd from others should work properly.

>From there, changes to js, xul, images, etc. only require running `mach
build faster`, and on my machine, that takes something between 3 and 4
seconds. Future iterations should allow subsecond updates.

Other changes would require a full `mach build`, without
--disable-compile-environment.

Anyways, as you can see with the list of steps above, this requires a
lot of manual work at the moment, so my proposal extends to streamlining
this process.

- Extend mach artifact to handle any type of Firefox build (not only
Firefox for Android), and to unpack builds completely (not just .so
files), and doubly (by which I mean, unpacking omni.ja as well, with
the code used by toolkit/mozapps/installer/unpack.py)

- Extend the build system to allow to apply multiple backends at the
same time. `mach build-backend -b FasterMake` only spends 0.05s on my
machine generating its build system, but still spends 4s that are
already spent during configure to read and analyze moz.build files.
That's also true of other backends, and it would be better to allow
people to add their prefered extra backends to their mozconfig.
(and they all rely on the recursive make backend, so that one can be
kept always on for now). It would also help make some of those extra
backends enabled by default depending on what's built.

- Ensure --disable-compile-environment works with all the above, but I
don't think there is actually anything to do for this. As a matter
of fact, I've tested it, and it seems to work well enough (and that
lead me to recently fix bug 1203851)

- Make mac builds fill dist/Nightly.app directly instead of filling
dist/bin (or fill dist/bin as if it were dist/Nightly.app), because
having to run all of
https://dxr.mozilla.org/mozilla-central/source/browser/app/Makefile.in?offset=0#95-109
at the end of every build is a pain point. This would have benefits
for other kinds of builds, too.

With those four above adressed, we can have a better experience for
Firefox frontend developers in a very short time. The last point is
likely to take some time, so it might make sense to replace it with an
extension to the first point, where mach artifact would flatten the
build to make it look like dist/bin, and the FasterMake backend would
invoke the right rules from browser/app/Makefile.in.

The only downside I can see to this plan is that since the new backend
would not be exercized on automation, it would have chances of breaking.
Fair enough, but the same can be said of all the other backends.

On the other hand, I can see many benefits:
- The new backend would not be exercized on automation. While it can be
seen as a downside, it is also a *huge* benefit, because it breaks all
the limits that come from ensuring all of automation works. Not having
to care about l10n is what makes the PoC work so well.
- As mentioned above, short term win for frontend developers
- Other use cases can plug on top of it. We could quickly add tests, for
example, and this is where the piggy-backing happens: we can just
invoke small parts of the current build system when it makes sense.
- It sets things in motion so that a future with non-make based build
backends can actually happen (like, replacing make with ninja or tup
in this new backend would be fairly easy).
- It allowed me to spot some of the shortcomings of the current
moz.build model with the emitter and helped get a clearer vision of
where we should be going on the long term (we can discuss this later).
See for instance the horror show in FasterMakeBackend.consume_object.
6 slithly different ways to be notified of files to install or
preprocess, and that's only for things relevant to Firefox. The almost
1:1 mapping between moz.build variables and emitted objects is not
helpful.

I'm almost ready to send that patch to review. At this point, I just
want to add a few comments to fastermake.py before doing so.

Thoughts on the overall proposal?

Mike

[Mike Hommey]

On Wed, Sep 16, 2015 at 05:33:44PM +0900, Mike Hommey wrote:
> Hi,
>
> Following is a proposal for short term wins for Firefox front end developers.
>
> The starting point is that there are too many things to change in the
> recursive make build system to allow for something like this to work
> properly for many different use cases. This also starts from the fact
> that, well, we've kept the recursive make build system use the same
> model it's been using for, well, close to 20 years. That model is
> flawed, and I thought it was about time to do something about it, while
> addressing real needs at the same time.
>
> My proposal is to start a new build backend from scratch, with the aim
> to grow it into a complete solution. Build backend means I'm not talking
> about replacing moz.build files with something new. I'm talking about
> the build system they're munged into.
>
> It would initially piggy back on the existing build system, because
> there are things that aren't exposed in moz.build yet that are
> necessary. It would also, initially, not care about the entire build,
> and even less about things that are outside of Firefox (tests, addons,
> etc.) or compilation.
>
> My initial work is this patch:
> https://pastebin.mozilla.org/8846539
>
> (Note: it requires the patches from bug 1204712, bug 1204719 and bug
> 1204715, all landed on mozilla-inbound, as of writing)

Filed bug 1207882 for it.

> (...)

>
> - Extend mach artifact to handle any type of Firefox build (not only
> Firefox for Android), and to unpack builds completely (not just .so
> files), and doubly (by which I mean, unpacking omni.ja as well, with
> the code used by toolkit/mozapps/installer/unpack.py)

Filed bug 1207888 and bug 1207890 for this.

>
> - Extend the build system to allow to apply multiple backends at the
> same time. `mach build-backend -b FasterMake` only spends 0.05s on my
> machine generating its build system, but still spends 4s that are
> already spent during configure to read and analyze moz.build files.
> That's also true of other backends, and it would be better to allow
> people to add their prefered extra backends to their mozconfig.
> (and they all rely on the recursive make backend, so that one can be
> kept always on for now). It would also help make some of those extra
> backends enabled by default depending on what's built.

Filed bug 1207893 and bug 1207897 for this

> - Ensure --disable-compile-environment works with all the above, but I
> don't think there is actually anything to do for this. As a matter
> of fact, I've tested it, and it seems to work well enough (and that
> lead me to recently fix bug 1203851)
>
> - Make mac builds fill dist/Nightly.app directly instead of filling
> dist/bin (or fill dist/bin as if it were dist/Nightly.app), because
> having to run all of
> https://dxr.mozilla.org/mozilla-central/source/browser/app/Makefile.in?offset=0#95-109
> at the end of every build is a pain point. This would have benefits
> for other kinds of builds, too.

Let's repurpose bug 934070 for this.

Mike

[Gregory Szorc]

On Wed, Sep 16, 2015 at 1:33 AM, Mike Hommey <m...@glandium.org> wrote:

Hi,

Following is a proposal for short term wins for Firefox front end developers.

The starting point is that there are too many things to change in the
recursive make build system to allow for something like this to work
properly for many different use cases. This also starts from the fact
that, well, we've kept the recursive make build system use the same
model it's been using for, well, close to 20 years. That model is
flawed, and I thought it was about time to do something about it, while
addressing real needs at the same time.

My proposal is to start a new build backend from scratch, with the aim
to grow it into a complete solution. Build backend means I'm not talking
about replacing moz.build files with something new. I'm talking about
the build system they're munged into.

It would initially piggy back on the existing build system, because
there are things that aren't exposed in moz.build yet that are
necessary. It would also, initially, not care about the entire build,
and even less about things that are outside of Firefox (tests, addons,
etc.) or compilation.

My initial work is this patch:
  https://pastebin.mozilla.org/8846539

(Note: it requires the patches from bug 1204712, bug 1204719 and bug
1204715, all landed on mozilla-inbound, as of writing)

- Extend mach artifact to handle any type of Firefox build (not only
  Firefox for Android), and to unpack builds completely (not just .so
  files), and doubly (by which I mean, unpacking omni.ja as well, with
  the code used by toolkit/mozapps/installer/unpack.py)

- Extend the build system to allow to apply multiple backends at the
  same time. `mach build-backend -b FasterMake` only spends 0.05s on my
  machine generating its build system, but still spends 4s that are
  already spent during configure to read and analyze moz.build files.
  That's also true of other backends, and it would be better to allow
  people to add their prefered extra backends to their mozconfig.
  (and they all rely on the recursive make backend, so that one can be
  kept always on for now). It would also help make some of those extra
  backends enabled by default depending on what's built.

- Ensure --disable-compile-environment works with all the above, but I
  don't think there is actually anything to do for this. As a matter
  of fact, I've tested it, and it seems to work well enough (and that
  lead me to recently fix bug 1203851)

- Make mac builds fill dist/Nightly.app directly instead of filling
  dist/bin (or fill dist/bin as if it were dist/Nightly.app), because
  having to run all of
  https://dxr.mozilla.org/mozilla-central/source/browser/app/Makefile.in?offset=0#95-109
  at the end of every build is a pain point. This would have benefits
  for other kinds of builds, too.

I am generally thrilled to see these ideas and the code that's already started to hit my review queue!

One thing I am sensitive to is developer ergonomics and the intuitiveness of the solution. I want `mach build` to be sufficient to get the most ideal build possible. No `mach build binaries`. No `mach build faster`. Just a single command that "does the right thing." Existing solutions for Fennec and your proposal above seem to fall a little short of this, requiring people run special build backends and special, largely undocumented targets to `mach build`.

It's inevitable for the short term future that we won''t have a build backend that can evaluate the entire DAG as one unit and have it complete in <1s for no-op or light builds. So, build/make targets that perform a subset of the build (like "binaries" and "faster") will be necessary. However, I think there is room to provide a bit more intelligence.

One of the ideas that came out of the developer productivity meetup last week was the idea of a `mach config` or `mach create-environment` command that configures a new build environment. This is logically a tuple of (srcdir, objdir, build configuration). An interactive wizard would ask you what you would like to build and it would write out optimal build configuration files for you. The build tools (mach) would keep better state and would read these additional settings from the build configuration files. We could even throw in some filesystem watching to intelligently build the necessary subsystem. So e.g. `mach build` would automatically invoke `make binaries` if C++ files changed and `mach build` would invoke `make faster` if JS changed. These details are all flexible. The takeaway is the build system should be simple and fast without requiring additional pieces of knowledge like the "binaries" and "faster" targets.