Support for building outside of the unified environment

[Andi-Bogdan Postelnicu]

TL;DR: We’ve added support for building Firefox outside of the unified environment

More than 6 years ago we added support to build Firefox in an unified environment by bundling together more translation units into a single one and feeding it to our build pipeline. This approach proved to be very useful from two perspectives: it significantly decreased the building time and also we were providing the compiler, aka Clang, with more context that permitted a broader range of optimizations resulting in better results for perf tests like browsertime.

Unfortunately this also came with a cost: we have stopped supporting the non-unified building system, creating a large dependency between different translation units. 

Our entire build system for C/C++ relies on the LLVM toolchain, leveraging at the maximum potential the Clang tooling that is part of LLVM proved to be very challenging with the unified build system. Build time warnings, static-analysis (based on clang-tidy), IntelliSense (based on clangd) all are more difficult or suboptimal in a unified build context.

Not being able to leverage the Clang tooling toolsuite at its full potential was undesirable, so we’ve begun the task of adding support for building outside of the unified system. 

We’ve designed a hybrid build system that attempts to perform a build with as many files compiled individually as possible.

Due to the implicit dependency problem, not all modules are currently able to be compiled in a non-unified environment. To designate these for the hybrid build, the `REQUIRES_UNIFIED_BUILD` option can be set in their corresponding moz.build file.

To build in the hybrid mode, set the following flag in your mozconfig:
```
ac_add_options --disable-unified-build
```

If you want to benefit from better build time warnings, increased static analysis reliability and improved support for C++ language-specific IntelliSense for your modules, please help by making them buildable outside of the unified environment.

In order to prevent future regressions we also added 3 build tasks, targeting linux64, macos64 and win64, that run on autoland and build Firefox with the hybrid build method.

If you have any questions, concerns or you would like to know more about this, please feel free to reach out to us.

Thank you,

Compiler and Development Tools Team

[Tom Ritter]

When you refer to "build time warnings" - do you mean the traditional
compiler warnings we have[0] - or is this something different? I
guess I'm confused about how unified builds would negatively impact
those?

-tom

[0] https://searchfox.org/mozilla-central/source/build/moz.configure/warnings.configure

> --
> You received this message because you are subscribed to the Google Groups "dev-pl...@mozilla.org" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to dev-platform...@mozilla.org.
> To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-platform/AD041F24-F11F-4670-A8FC-36219B304F53%40mozilla.com.

[Daniel Holbert]

Not sure if this is what he'd meant, but yes - there are some traditional compiler warnings that get implicitly/automatically suppressed in unified builds, by virtue of the fact that all of the .cpp files end up being #included and hence the compiler thinks of them as headers.

One example is "Wunused-function" in clang++. Suppose you have foo.cpp and bar.cpp, with these contents:

** foo.cpp (2 lines long) **

static inline void do_stuff() {}
int main() { return 0; }

** bar.cpp (one line long, the "unified build" version of foo.cpp) **

#include "foo.cpp"

If you compile these two files with "clang++ -Wall <filename>", you'll get an unused_function warning when you directly compile foo.cpp, but *no such warning* when you compile bar.cpp, despite the fact that you're effectively compiling the same source code in both cases.

I think/assume this happens because "actual" .h file headers may be included by lots of .cpp files, and not all of those .cpp files will use all of the functions from the .h file.  So, for a given compilation unit, the compiler can't make a strong judgement about whether a given function in an included file is actually unused or not.

~Daniel

To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-platform/CADua4_uJ0jkorGG3T5a5U2wHNR5_SOd56jcQGEti0RiF-GM%3D-Q%40mail.gmail.com.

[Andi-Bogdan Postelnicu]

Thank you Daniel for being so fast with the reply.

Yes because of the way how we bundle up more translation units so warnings are discarded but in general we should still be nice to know about them.

On 10 Sep 2021, at 23:21, Daniel Holbert <dhol...@mozilla.com> wrote:



[Nika Layzell]

I've noticed that non-unified builds currently fail with `--enable-warnings-as-errors` (mostly due to unused `static`s which were previously undetected due to being in headers). Is the plan to eventually get them to the point where they'll pass with that configure flag set as well?

To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-platform/B6DADBF8-5F75-4208-8058-318CA06DE31E%40mozilla.com.

[Andi-Bogdan Postelnicu]

I guess we can do that in time, each developer that encounters this can fix it, as we do the rest of the modules. Currently the autoland tasks that build ff in the hybrid mode use this[1] kind of mozconfig that is based on nightly. I don’t think that the risk of regression from this perspective is that high.

Wdyt?

[1] https://searchfox.org/mozilla-central/source/browser/config/mozconfigs/linux64/hybrid