Better incremental builds - Tup backend ready for early adopters on Linux

[Michael Shal]

Hi everyone,

Tup is a modern build system that enables fast and correct builds. I'm pleased to announce the availability of Tup for building Firefox on Linux. Compared to Make, building with Tup will result in faster incremental build times and reduce the need for clobber builds. See the in-tree docs for details [1].

We’re looking for some early adopters to try it out and let us know if it improves your workflow, or if you hit any blockers or barriers to adoption that keep you on the Make backend for now.

Quick howto:

You’ll need to install the Tup executable, as well as the nightly rust/cargo toolchain:

cd ~/.mozbuild && mach artifact toolchain --from-build linux64-tup
rustup install nightly
rustup default nightly

Your mozconfig needs to describe how to find the executable if it’s not in your PATH, and enable the Tup backend:

export TUP=~/.mozbuild/tup/tup
ac_add_options --enable-build-backends=Tup

If you have any issues, feel free to file a bug blocking “buildtup” [2], or contact mshal or chmanchester in #build on IRC.

[1] https://firefox-source-docs.mozilla.org/build/buildsystem/tup.html

[2] https://bugzilla.mozilla.org/show_bug.cgi?id=buildtup

Thanks!

-Mike

[Jonathan Watt]

This is great! A big thank you to all the folks who have worked for so long to
get us to this stage. I'm really looking forward to this work maturing and
extending to other platforms.

I've done a bit of testing on my workstation (14 core/28 thread i9, U.2
connected NVMe SSD, 64 GB DDR4 RAM) and got the following build times:

For -j28 (yes, -j28 is well into diminishing returns territory, but I wanted a
comparison of the best case scenarios for both backends):

| cold | top-level
| initial build | no-op rebuild
------------------------------------------
make | 8m 10s | 15s
tup | 6m 35s | 1.4s

And for -j8:

| cold | top-level
| initial build | no-op rebuild
------------------------------------------
make | 11m 33s | 15s
tup | 10m 52s | 1.4s

(To confirm: the rebuild numbers are the same for both -j28 and -j8.)

So a 20% reduction in build time on cold builds of a fresh tree with -j28, and
6% reduction with -j8. In addition to speeding up the build, Tup seems to
parallelize better which is great! Of course the highlight is the big reduction
in rebuild overhead which will make the edit-compile-run cycle a lot less
annoying (for me, 50s down to 37s after touching nsDocumentViewer.cpp,
rebuilding, and linking libxul, for both -j28 and -j8.)

These improvements, plus the promise of rarely having to clobber/figure out
which directories to rebuild in, put a big smile on my face. (Currently having
to turn off CARGO_INCREMENTAL, not being able to use sccache and other current
limitations, less so, but it's early days. :) )

A few notes for others trying this out (thanks cmanchester and mshal for the
help debugging some of these):

* Make sure you use:

--enable-build-backends=Tup

not:

--enable-build-backend=Tup

* You must be using an in-tree OBJDIR.

* MOZ_PARALLEL_BUILD isn't currently recognized, so you'll need to use
`mach build`s -j argument to control the parallel job count for now
( see https://bugzilla.mozilla.org/show_bug.cgi?id=1481441 )

mshal: I think you could cross-post this to dev-platform for wider testing. I
know there are a bunch of caveats (which people should be made aware of), but
since the initial teething issues I had were figured out I've so far had no
other problems building, or rebuilding with real code changes.

Jonathan

[Boris Zbarsky]

On 8/7/18 9:18 AM, Jonathan Watt wrote:
> * You must be using an in-tree OBJDIR.

Do you happen to know whether this is a temporary limitation or a
permanent one?

-Boris

[Jonathan Watt]

When I previously discussed that with mshal he said the following (edited to
replace locations specific to my setup):

"A little background about tup: by default it keeps track of everything in the
directory where 'tup init' gets run, and all subdirectories (if you look in your
top-level source directory, you should have a ".tup" directory there). So in the
case you use an out-of-tree $OBJDIR Tup doesn't see all the Tupfiles created in
that directory, since that is outside of where `tup init` gets run. I think it
might be possible to support this in the future, but that is likely a
non-trivial effort."

[Boris Zbarsky]

On 8/7/18 11:27 AM, Jonathan Watt wrote:
> When I previously discussed that with mshal he said the following (edited to
> replace locations specific to my setup):
>
> "A little background about tup: by default it keeps track of everything in the
> directory where 'tup init' gets run, and all subdirectories

Hmm. So if my objdirs are siblings of my srcdir, could I just run "tup
init" in the common parent?

Being able to have objdirs outside the srcdir is pretty important in my
case; it makes it a lot easier to search code without having the objdir
interfere with the searches...

-Boris

[Tom Prince]

On Tue, Aug 7, 2018 at 9:37 AM Boris Zbarsky <bzba...@mit.edu> wrote:

Being able to have objdirs outside the srcdir is pretty important in my
case; it makes it a lot easier to search code without having the objdir
interfere with the searches...

Depending on what tool you use, it should be possible to exclude objdir's from search anyway. I know ripgrep[1] (for example) will look at `.gitignore` in-tree and ignore any top-level directories that begin with `obj`. I guess pretty much any tool should be able to be configured to ignore object directories similarly.

-- Tom

[1] https://github.com/BurntSushi/ripgrep

[Boris Zbarsky]

On 8/7/18 11:49 AM, Tom Prince wrote:
> Depending on what tool you use, it should be possible to exclude
> objdir's from search anyway.

There are multiple tools, depending on what I'm doing. "find", "grep",
"mkid" (from idutils) are the most common ones.

"find" and "mkid" do have ways to exclude directories via command-line
options, but not a permanent configuration that I'm aware of.

-Boris

[Nicholas Nethercote]

>  * You must be using an in-tree OBJDIR.

Does it handle multiple in-tree objdirs? E.g. I have d64 (debug), o64 (opt), etc.

Nick

_______________________________________________
dev-builds mailing list
dev-b...@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-builds

[Michael Shal]

 Yes, that should work, though with a caveat - you'd probably want to make sure that the common parent is only used for one srcdir and objdir. So this works fine:

mydir/mozilla-central

mydir/objdir

mydir/.tup

So you would run 'tup init' manually in "mydir", and have a mozconfig like:

mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/../objdir
ac_add_options --build-backends=Tup

I've confirmed locally that this works, though you just get an innocuous warning when mozbuild tries to run 'tup init' inside the srcdir: "tup warning: database already exists in directory: .../mydir". We could improve support for this by trying to run 'tup init' automatically in the parent dir if the objdir is a sibling of the srcdir, which shouldn't be too hard to add.

For comparison, the "default" case of an objdir inside the srcdir looks like this:

mydir/mozilla-central

mydir/mozilla-central/objdir

mydir/mozilla-central/.tup

Without a file monitor, tup scans everything under where ".tup" exists, so all of "mydir" in the first example and "mydir/mozilla-central" in the default case.

The caveat applies in what I think jwatt's case is (jwatt, correct me if I'm wrong - I may have misunderstood your setup) where you have multiple trees and objdirs all as siblings. For example:

mydir/mozilla-central

mydir/mozilla-inbound

mydir/mozilla-beta

mydir/objdir-central-debug

mydir/objdir-central-opt

mydir/objdir-inbound-debug

mydir/objdir-inbound-opt

...

mydir/.tup

In this case, since we don't have the file monitor integrated yet, the initial scan would likely take way too long and you'd lose much of the benefit of switching backends in the first place. I think we could support this structure in the future, but it would require some changes in tup itself, and is likely not a quick fix.

So if you already have the srcdir/objdir pairs in an isolated parent, you should be good to go now with a manual 'tup init' in the parent, and we can support that workflow fairly easily. If it's the latter case where many trees are in a single common root, I don't have a good solution for that at the moment.

-Mike

[Michael Shal]

On Wed, Aug 8, 2018 at 3:03 AM, Nicholas Nethercote <n.neth...@gmail.com> wrote:

>  * You must be using an in-tree OBJDIR.

Does it handle multiple in-tree objdirs? E.g. I have d64 (debug), o64 (opt), etc.

That should work with a similar caveat that tup is going to scan everything under where ".tup" exists, which would include both objdirs. So there will be some additional startup overhead until we have the file monitor integrated -- I guess at the moment it's a question of how many objdirs you have and how much of an impact it adds to the startup overhead. I'll try it out locally myself, but I'd be interested to hear your experiences as well!

-Mike

[Boris Zbarsky]

On 8/8/18 10:29 AM, Michael Shal wrote:
>  Yes, that should work, though with a caveat - you'd probably want to
> make sure that the common parent is only used for one srcdir and objdir.

I definitely have multiple objdirs per srcdir. ;) opt and debug at the
very least; in some cases a few others.

> The caveat applies in what I think jwatt's case is (jwatt, correct me if
> I'm wrong - I may have misunderstood your setup) where you have multiple
> trees and objdirs all as siblings.

Right, that's not my situation. I have basically a bunch of dirs like
(names changed for clarity):

dir1/srctree
dir1/obj-debug
dir1/obj-opt
dir2/srctree
dir2/obj-debug
dir2/obj-opt
dir2/obj-opt-asan

etc. So all the srcdirs are siloed off from each other in my case.

Sounds like in my case tup would scan all the objdirs and the srcdir, right?

-Boris

[Michael Shal]

Yes, so it should work if you run 'tup init' manually in dir1 and again in dir2. I ran some tests on my machine, and it looks like each additional objdir only added about 100ms to the scan time. So I think with a handful of objdirs it's not likely you'll notice a huge difference as compared to a single objdir. I had 3 objdirs and a no-op 'mach build' was still under 2 seconds, but YMMV :)

-Mike

[Boris Zbarsky]

On 8/8/18 3:48 PM, Michael Shal wrote:
> Yes, so it should work if you run 'tup init' manually in dir1 and again
> in dir2. I ran some tests on my machine, and it looks like each
> additional objdir only added about 100ms to the scan time.

That's awesome. ;)

-Boris

[Myk Melez]

Michael Shal wrote on 2018-08-02 08:34:

Hi everyone,

Tup is a modern build system that enables fast and correct builds. I'm pleased to announce the availability of Tup for building Firefox on Linux. Compared to Make, building with Tup will result in faster incremental build times and reduce the need for clobber builds. See the in-tree docs for details [1].

Should Btup bustage trigger immediate backout? I appear to have busted it with a recent change [1], triggering an immediate backout, and it isn't obvious to me that this is the optimal policy for an experimental build backend.

(I'll investigate the bustage regardless. My question is more generally about the policy we want to enforce for this build task.)

-myk

[1] https://phabricator.services.mozilla.com/rMOZILLACENTRAL08fa47a24e89697e4e43177860b55cc28298bbff

[Mike Hommey]

On Wed, Aug 08, 2018 at 05:24:07PM -0700, Myk Melez wrote:
> Michael Shal wrote on 2018-08-02 08:34:
> >
> > Hi everyone,
> >
> >
> > Tup is a modern build system that enables fast and correct builds. I'm
> > pleased to announce the availability of Tup for building Firefox on
> > Linux. Compared to Make, building with Tup will result in faster
> > incremental build times and reduce the need for clobber builds. See the
> > in-tree docs for details [1].
> >
> Should Btup bustage trigger immediate backout? I appear to have busted it
> with a recent change [1], triggering an immediate backout, and it isn't
> obvious to me that this is the optimal policy for an experimental build
> backend.
>
> (I'll investigate the bustage regardless. My question is more generally
> about the policy we want to enforce for this build task.)

I think now that it has been advertized and that people may be using it,
it should be turned into tier 1.

Mike

[Michael Shal]

On Wed, Aug 8, 2018 at 8:24 PM, Myk Melez <m...@mykzilla.org> wrote:

Michael Shal wrote on 2018-08-02 08:34:

Hi everyone,

Tup is a modern build system that enables fast and correct builds. I'm pleased to announce the availability of Tup for building Firefox on Linux. Compared to Make, building with Tup will result in faster incremental build times and reduce the need for clobber builds. See the in-tree docs for details [1].

Should Btup bustage trigger immediate backout? I appear to have busted it with a recent change [1], triggering an immediate backout, and it isn't obvious to me that this is the optimal policy for an experimental build backend.

(I'll investigate the bustage regardless. My question is more generally about the policy we want to enforce for this build task.)

We created Btup as tier-2 so that any breaking it wouldn't prevent people from landing their other code, with the intention that chmanchester or myself would apply fixes to the tup backend in followup commits. I should've done a better job communicating with the sheriffs to enforce this - sorry about that! But as glandium suggests, it should probably be tier-1 now (or soon) anyway, so I filed bug 1482502 for that.

In any case, if anyone hits bustage in Btup when make-based builds are successful, feel free to grab one of us in #build to help out.

-Mike