Using unity builds (jumbo) for Chromium

[Daniel Bratell]

Continuation of discussions in blink-dev:

The compilation time for chromium is a huge[1], increasing[2], problem for anyone without access to Google's internal compilation hardware and software and it prevents external parties from helping in the project.

My suggested improvement, initially for Blink, but in the continuation for every major code chunk, is to use the unity build technique (here called Jumbo for ease of conversations and fewer misunderstandings), which is to compile many cc/cpp files together instead of one at a time.

The speedup depends on a lot of factors but typically the average compilation time per file drops by 70-95% and there is no other known technique that can give that kind of speedup. Other techniques such as precompiled headers have been applied and have had a measurable but (too) limited effect. The jumbo speedup also speeds up the linking phase and in other tools that depend on object file contents.

What about goma? Full builds with goma also benefit, much more than I expected, but since those are quite fast already, 15% less time might not mean more than a couple of minutes.

Con of this is that code need to be in a state that prevents naming collisions. Bascially less copy-pasted code and more unique names. Recompiling a single class can also be slower (YMMV). 

Initial patch is in https://codereview.chromium.org/2821333003/ (not rebased in a while so may or may not work unchanged). We deployed this and some more internally at Opera a while back so it has been tested in production. Size is 100 lines globally shared .gni template, 50 lines .py script, 2 changed lines in core.gni.

First step is probably to setup a test build machine so that it can be phased in. It's on dpranke's list somewhere.

I have a document that explains all of this more in depth: https://docs.google.com/document/d/19jGsZxh7DX8jkAKbL1nYBa5rcByUL2EeidnYsoXfsYQ/edit#

Various images to illustrate. More than half of CPU usage for Blink core+modules+bindings disappeared:

Time to compile Opera for Mac on one particular build machine, from 30 minutes to 150 minutes in 2.5 years.

Since I'm not in blink-dev anymore, here is compile time data for content as well:

/Daniel

[1] Our 4 hour compilation timeouts have begun triggering.
[2] The compilation time has gradually gone up 400% in 1-2 years

--

/* Opera Software, Linköping, Sweden: CET (UTC+1) */

[Benoit Lize]

That's great!

Did you see any impact on performance?

AFAIK, we are not using LTO on Android, and this would be an approximation of LTO, so it could yield code size and/or perf improvements.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/op.y2hpdev9rbppqq%40cicero2.linkoping.osa.

[Daniel Bratell]

My initial intention is to have this for developer builds only (is_offical != true) to allow us to learn more about potential side effects before considering doing it for official builds. It is quite possible that jumbo might have positive effects on performance and binary size but I would treat that as a nice bonus in that case. It is also possible that jumbo hurts performance because the compiler optimizers run into internal limits, but I think that is less likely.

/Daniel

[Dirk Pranke]

For the record, I am *very* supportive of this work, and am planning to get this set up on bots Real Soon Now so that we can (a) prevent regressions for all of Daniel's work and (b) get solid numbers of the impact on compile times on our machines as well.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/op.y2hxzxtnrbppqq%40cicero2.linkoping.osa.

[bruce...@chromium.org]

This feels like very important work - build times are harming productivity and are continuing to grow.

One concern about jumbo builds is that the individual translation units can end up too large (as opposed to now when they are typically too small). This can reduce build parallelism, make incremental builds longer, and sometimes hit compiler limits. Could you share out .ninja_log files from full builds with and without this change so that we can see whether this is a concern? I'd be interested in seeing how the worst-case compile times compare.

[Daniel Bratell]

Sent some logs I made today (lucky timing, takes a day so I don't gather that data very often) in a private mail. If someone else wants to see some ninja logs, mail me off-list.

When you merge up to 200 (the current limit based on some testing and tuning) compilation units, they will indeed take longer to compile. The longest I've seen in the parts I've submitted is 61 seconds for one such jumbo file which is not ideal if you are one of the people that recompile that file all the time. I included a setting to "unjumbofy" your favourite target but I am actually not certain it is a win.

As it is right now, you might lose 10-20 seconds every time you compile a single cc file, but then you might save a couple of minutes or many minutes (or in the case of non-goma, an hour) for those times you touch a header that is included a lot. It really seems premature to even guess where the right balance will be.

Worth noting is that the linking step after compiling jumbo files is shorter too. I have not measured it but I have assumed it is because there is so much less duplicated debug/link data to merge. That might also regain the time possibly lost in the compilation phase.

/Daniel

[Daniel Bratell]

Update: Jumbo for Blink core landed yesterday (Monday) but is disabled. There is a mac-only problem ( https://crbug.com/716395 ) that might be worked around by the time you read this. Also, it didn't work for android+gcc last I tried (compiler warning that became an error that broke the build).

There is no trybot for this yet (dpranke is doing fireworks) so it might break without anyone noticing but assuming that has not happened, you can try this by setting

use_jumbo_build = true

in your args.gn and then you can compile as normal. If you compile locally, that should save about 15-25% of the compilation time (20-60 minutes?).

Next steps are:

* Get a try bot up.

* Make it default enabled for relevant configurations.

* Enable jumbo builds for other parts of the tree than third_party/WebKit/Source/core (for instance bindings, core unit_test, platforms, content, net, cc, base, chrome...).

* Tweak the code so that all files can be jumbo compiled.

* Port from template/python to native gn for better IDE support.

I can't say what the order of these are. They are happening kind of in parallel.

/Daniel

[Bruce Dawson]

Looks good! I did some tests on local goma (distributed) builds and it made them faster as well (about 15% less time).

[Daniel Bratell]

And then you have seen nothing yet!

Or seriously, it's landed for Blink/core production code which is the single largest consumer of time, but there is a lot more. Attaching an image from compilation times as measured locally on a branch that isn't even close to have landed on master (because of reasons). I would like to extrapolate from that but... well, we'll see.

All times in CPU minutes for  master chrome+blink_tests+content_shell Linux Debug.

Left column is mainline default.

Middle column is mainline with use_jumbo_build=true

Right column is my local branch where I've also converted base, content, cc, blink platform, modules, web and blink unit_tests.

/Daniel

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAE5mQiOL_C8KHzJm1gAYCLB4PtP7toXKtH4jic9J%3D0ikEN0q4Q%40mail.gmail.com.

[Yutaka Hirano]

Hi,

I saw https://chromium-review.googlesource.com/c/570018/ and am wondering file-local scope (static / unnamed namespace) is effectively disabled. Is that right?

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAE5mQiOL_C8KHzJm1gAYCLB4PtP7toXKtH4jic9J%3D0ikEN0q4Q%40mail.gmail.com.

--

/* Opera Software, Linköping, Sweden: CET (UTC+1) */

--

--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/op.y20mc2y1rbppqq%40cicero2.linkoping.osa.

[Shinya Kawanaka]

Hi,

From the goma point of view:

Pros:

- the number of parsing/compiling headers decrease, so compile can be faster

- this will decrease the number of objects, so link can be faster

Cons:

- goma is heavily caching compile result, however, this will increase cache miss ratio, so average build time might be worse.

- if a lot of sources are combined, goma's server might fail to compile because of out of memory.

So... I'm not so sure this will help average build time or not. So I believe we need to have a concrete number to argue.

# I agree compile is drastically increasing for years. When we improve goma to make build faster, someone always makes build slower while the build time is acceptable X(

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CABihn6GVYEWuKF%2B1%2B%3D3zBxD6amL1BACaHL_Ou60Tu1vNmxx-aQ%40mail.gmail.com.

[Daniel Bratell]

Yes, it will be interesting to see what the side effects will be. For someone from the outside goma seems to be extremely well optimized for the current work load. How it will react to a different work load I don't know. The currently checked in jumbo drops CPU usage requirements by 17% for a full build. That could mean that there are more available goma build nodes at any time. On the other hand, as you say, caches might get lower hit rates. On the third hand, the amount of data that has to be transferred will be much lower. On the fourth hand single files will take longer to compile, on the fifth hand linking times will drop.

Pros, cons and I have no idea where it will land for people with fast access to goma in the end.

Meanwhile, those of us that have multi-hour builds, we will probably be able to save many hours per week. At least for those that fetch master every day (which we have learned to not do because...). 

/Daniel

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAE5mQiOL_C8KHzJm1gAYCLB4PtP7toXKtH4jic9J%3D0ikEN0q4Q%40mail.gmail.com.

--

/* Opera Software, Linköping, Sweden: CET (UTC+1) */

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/op.y20mc2y1rbppqq%40cicero2.linkoping.osa.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CABihn6GVYEWuKF%2B1%2B%3D3zBxD6amL1BACaHL_Ou60Tu1vNmxx-aQ%40mail.gmail.com.

[Dirk Pranke]

By the way, I've finally gotten around to filing a bug to track the work for this project: crbug.com/745862, if anyone wants to follow along.

-- Dirk

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAE5mQiOL_C8KHzJm1gAYCLB4PtP7toXKtH4jic9J%3D0ikEN0q4Q%40mail.gmail.com.

--

/* Opera Software, Linköping, Sweden: CET (UTC+1) */

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/op.y20mc2y1rbppqq%40cicero2.linkoping.osa.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CABihn6GVYEWuKF%2B1%2B%3D3zBxD6amL1BACaHL_Ou60Tu1vNmxx-aQ%40mail.gmail.com.

--

/* Opera Software, Linköping, Sweden: CET (UTC+1) */

--

--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/op.y3k0m61hrbppqq%40cicero2.linkoping.osa.

[Daniel Bratell]

In jumbo builds the anonymous namespace becomes more of a "gn-target-local scope" than a "file scope". (Both terms are simplifications) So they still serve some purpose, both as informative to the reader and to avoid symbols escaping out to the linker.

/Daniel

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAE5mQiOL_C8KHzJm1gAYCLB4PtP7toXKtH4jic9J%3D0ikEN0q4Q%40mail.gmail.com.

--

/* Opera Software, Linköping, Sweden: CET (UTC+1) */

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/op.y20mc2y1rbppqq%40cicero2.linkoping.osa.