Compilation Time is Too Long

[Yizhuo Zhai]

Hi All:

I'm compiling Chromium using my modified clang, but it seems every round of build took 2 full days, which is longer than what I searched in the internet. Could anyone help me with the possible reason? I use a powerful server with 80 cores and 1T memory.

I use "gn args out/mybuild" to config the building arguments and use ninja -C out/mybuild -j72 to build the system, the arguments are showing as follows:

clang_base_path = "path/to/clang/build/"

clang_use_chrome_plugins = false

is_component_build = true

enable_nacl = false

is_debug = false

symbol_level = 0

use_lld = false

use_gold = true

treat_warnings_as_errors = false

[Lei Zhang]

Where is the time going? Is it by any chance stuck in the linking stage? If so, I would suggest leaving the 2 linker flags alone and stick with LLD as the linker.

When the build doesn't have a target, it tries to build all targets and that takes more time. Maybe just build the "chrome" target, if the main browser executable is what matters?

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/150347eb-d261-4b0b-bef7-aa2120391003n%40chromium.org.

[Yizhuo Zhai]

Hi Lei:

Thanks for your response, I use ninja -C out/mybuild chrome -j72 to build the system, and it shows 50K+ tasks in total.

I do find that the linking takes a lot of time, the linking flag is referring to this post (https://github.com/timmyyuan/build-whole-program-into-LLVM-IR/blob/master/chromium.md).

I can modify the ld flags and see if I have a shorter compilation time.

--

Kind Regards,

Yizhuo Zhai

Computer Science, Graduate Student

University of California, Riverside 

[Yizhuo Zhai]

Hi Lei:

I found that some CXX tasks can take 30min+ as well. The Clang I used is modified and add some sanitizers, I wonder if it's reasonable to take such a long time. 

[Lei Zhang]

Since this is with your modified Clang, only you can answer that question. Compare with the prebuilt Clang that Chromium normally uses.

[Yoshisato Yanagisawa]

Taking a long time to compile with sanitizer reminds me: https://bugs.chromium.org/p/chromium/issues/detail?id=420985.

It took more than 10 minutes to compile a file.  Just FYI.

2023年2月22日(水) 7:43 Lei Zhang <the...@chromium.org>:

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CACSHbcRoDyVPEzBPr4Wya9jG4Yy-wR6_ov%2BT-APyveaYPaxA5A%40mail.gmail.com.

[Bruce Dawson]

On an 80-core machine with -j 72 you should be able to build Chromium in maybe 40 minutes. On a 4 core laptop it probably takes six to seven hours.

No CXX task should take 30+ minutes, with the normal clang. If any compile takes longer than a minute then I start filing bugs asking for it to be fixed (no guarantees, but I wouldn't expect anything much longer than that).

If you set this environment variable:

set NINJA_SUMMARIZE_BUILD=1

and use autoninja then you'll get a post-build summary of where the time went. Or you can run the command explicitly to get a summary of your last build:

python3 c:\src\depot_tools\post_build_ninja_summary.py -C out\Default

If I were you I'd do a build of Chromium with all of the default tools (including lld-link) and get the timing summary of that, and then start swapping out tools, probably one at a time.

[Bruce Dawson]

What build target are you building? That makes a huge difference. I normally just build chrome, or a specific test binary. Building all targets takes a lot longer.

I just did a clean build of the chrome build target on my Windows laptop using goma (-j 640). Here is the build summary:

c:\src\chromium\src>python c:\src\depot_tools\post_build_ninja_summary.py -C out\release
    Longest build steps:
           1.8 weighted s to build gen/third_party/blink/public/mojom/frame/back_forward_cache_contr... (55.3 s elapsed time)
           1.8 weighted s to build gen/third_party/blink/public/mojom/frame/back_forward_cache_contr... (61.3 s elapsed time)
           1.9 weighted s to build irt_x64/obj/ppapi/proxy/ipc_sources/ppapi_messages.o (28.8 s elapsed time)
           2.0 weighted s to build gen/third_party/blink/public/mojom/ad_tagging/ad_evidence.mojom-f... (82.0 s elapsed time)
           2.3 weighted s to build gen/third_party/blink/public/mojom/ad_tagging/ad_evidence.mojom-b... (88.3 s elapsed time)
           2.6 weighted s to build irt_x64/obj/third_party/perfetto/src/tracing/core/service/tracing... (27.8 s elapsed time)
           4.0 weighted s to build v8_context_snapshot_generator.exe, v8_context_snapshot_generator.... (13.1 s elapsed time)
           6.1 weighted s to build irt_x64/nacl_irt.nexe, irt_x64/exe.unstripped/nacl_irt.nexe (11.7 s elapsed time)
           8.3 weighted s to build build.ninja.stamp (8.4 s elapsed time)
          22.9 weighted s to build chrome.dll, chrome.dll.lib, chrome.dll.pdb (22.9 s elapsed time)
    Time by build-step type:
          10.6 s weighted time to generate 330 .js files (475.7 s elapsed time sum)
          21.9 s weighted time to generate 706 .mojom-blink.obj files (13931.4 s elapsed time sum)
          27.2 s weighted time to generate 8313 .stamp files (3079.9 s elapsed time sum)
          28.9 s weighted time to generate 35 PEFile (linking) files (97.9 s elapsed time sum)
          29.5 s weighted time to generate 527 .pb.obj files (10267.8 s elapsed time sum)
          39.6 s weighted time to generate 1485 mojo files (2718.9 s elapsed time sum)
          43.7 s weighted time to generate 1026 .mojom.obj files (21986.2 s elapsed time sum)
          80.6 s weighted time to generate 1026 .mojom-shared.obj files (19730.7 s elapsed time sum)
          80.9 s weighted time to generate 1797 .o files (34299.6 s elapsed time sum)
        1255.8 s weighted time to generate 33993 .obj files (699776.6 s elapsed time sum)
    1703.5 s weighted time (815374.6 s elapsed time sum, 478.7x parallelism)
    57686 build steps completed, average of 33.86/s

It's a bit cryptic if you're not used to it but "weighted s" is the time for a build step divided by how many build steps were running at a time, and elapsed time is the total time for that build step. Everything is sorted by weighted time, because that is what matters, so I can't tell you what the longest compilation step was, but it probably wasn't much longer than 90 seconds. You could reconfigure this script to sort by elapsed time instead of weighted time if you wanted to.

Looking at the build times using ninjatracing (https://github.com/nico/ninjatracing) lets me visually explore the build process and that leads me to believe that the longest build step on my release build was probably about 105 s for some compilation steps.

Comparing goma builds to non-goma builds is a bit difficult, but you can compare builds on your machine with the normal compiler/linker to those with your custom compiler/linker.

FWIW.

--

Bruce Dawson, he/him

[Bruce Dawson]

I modified depot_tools\post_build_ninja_summary.py to sort by elapsed time instead of weighted time and got this summary:

    Longest build steps:
           0.1 weighted s to build obj/content/browser/browser/media_stream_dispatcher_host.obj (89.9 s elapsed time)
           0.1 weighted s to build obj/content/browser/browser/media_stream_manager.obj (90.2 s elapsed time)
           0.1 weighted s to build obj/content/browser/browser/render_frame_host_impl.obj (94.8 s elapsed time)
           1.0 weighted s to build obj/third_party/angle/translator_gl_d3d_only/OutputGLSLBase.obj (97.5 s elapsed time)
           1.0 weighted s to build obj/third_party/angle/translator_gl_d3d_only/TranslatorHLSL.obj (102.3 s elapsed time)
           1.0 weighted s to build obj/third_party/angle/translator_gl_d3d_only/StructureHLSL.obj (103.4 s elapsed time)
           1.0 weighted s to build obj/third_party/angle/translator_gl_d3d_only/ShaderStorageBlockOu... (103.8 s elapsed time)
           1.0 weighted s to build obj/third_party/angle/translator_gl_d3d_only/ResourcesHLSL.obj (104.4 s elapsed time)
           1.1 weighted s to build obj/third_party/angle/translator_gl_d3d_only/OutputHLSL.obj (105.1 s elapsed time)
           1.1 weighted s to build obj/third_party/angle/translator_gl_d3d_only/TranslatorGLSL.obj (109.2 s elapsed time)

So the longest build step is 109.s s when building with goma on my laptop, and there were only nine build steps that took longer than 90 seconds. This is the diff to get this summary:

@@ -273,7 +273,7 @@ def SummarizeEntries(entries, extra_step_types):

     # Print the slowest build steps (by weighted time).
     print('    Longest build steps:')
-    entries.sort(key=lambda x: x.WeightedDuration())
+    entries.sort(key=lambda x: x.Duration())

I might add this as a command-line option since it is occasionally useful.

--

Bruce Dawson, he/him

[Yizhuo Zhai]

Hi Bruce:

Thanks for being so informative and I appreciate your help a lot. I follow your instructions and compile the small binary "transport_security_state_generator". This is the time I got:

python ../depot_tools/post_build_ninja_summary.py -C out/reduced/

    Longest build steps:

          17.9 weighted s to build obj/third_party/perfetto/src/trace_processor/storage_minimal/trac... (376.3 s elapsed time)

          12.8 weighted s to build obj/third_party/boringssl/boringssl/bcm.o (410.8 s elapsed time)

          22.2 weighted s to build obj/third_party/perfetto/src/tracing/core/service/tracing_service... (514.7 s elapsed time)

          48.5 weighted s to build obj/third_party/protobuf/protobuf_full/descriptor.o (580.7 s elapsed time)

          27.1 weighted s to build obj/third_party/perfetto/src/trace_processor/storage/storage/trac... (655.9 s elapsed time)

          35.5 weighted s to build obj/third_party/perfetto/src/trace_processor/storage_minimal/trac... (940.7 s elapsed time)

         645.1 weighted s to build obj/third_party/perfetto/src/trace_processor/db/db/column.o (1961.4 s elapsed time)

        1133.4 weighted s to build obj/base/base/scoped_file_linux.o (2275.4 s elapsed time)

        1064.1 weighted s to build obj/third_party/perfetto/src/base/base/metatrace.o (2413.7 s elapsed time)

         826.1 weighted s to build obj/third_party/perfetto/src/tracing/core/service/metatrace_write... (2671.7 s elapsed time)

    Time by build-step type:

           3.1 s weighted time to generate 18 (no extension found) files (9.0 s elapsed time sum)

           3.4 s weighted time to generate 1 .instantiations.o files (109.5 s elapsed time sum)

           4.3 s weighted time to generate 15 .gen.cc files (107.0 s elapsed time sum)

           5.4 s weighted time to generate 9 .so (linking) files (15.2 s elapsed time sum)

           5.5 s weighted time to generate 2 .ipc.o files (115.0 s elapsed time sum)

           8.7 s weighted time to generate 29 .pbzero.cc files (216.1 s elapsed time sum)

          13.2 s weighted time to generate 148 .pbzero.o files (407.5 s elapsed time sum)

          37.4 s weighted time to generate 12 .pb.o files (1163.6 s elapsed time sum)

         116.4 s weighted time to generate 63 .gen.o files (3304.2 s elapsed time sum)

        6371.3 s weighted time to generate 1695 .o files (89213.4 s elapsed time sum)

    6570.9 s weighted time (94689.1 s elapsed time sum, 14.4x parallelism)

    2385 build steps completed, average of 0.36/s

The server is shared by others running a few qemu processes, maybe the slowdown also comes from the CPU switching.

[K. Moon]

In virtualized or shared environments, I generally would suspect adversarial CPU or I/O usage to be the main culprit here. You might also be running out of RAM.

Regardless, this doesn't look like a problem with the Chromium build, but more of a local system performance issue that you'll need to debug on your own (since none of us have access to the exact same environment, nor would this be a good use of other people's time).

Something that might help you is if you can reproduce the slowness by running the same command over and over. You can get Ninja to print out the exact command line, using "-t commands", and then just run that single command to see if you can reproduce your issue. If so, you can start looking if there are resource starvation issues at the system level.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CABvMjLRoaW6cEqYswF2z3SMigOv-DQfVd5L_6qSKBZnyNQ%2BBRw%40mail.gmail.com.

[K. Moon]

Also, given that it seems like you're a student, I wouldn't underestimate the value of asking your peers or faculty for help. :-) They're more likely to understand the details of the local environment than random people on the Internet.

[Yizhuo Zhai]

Hi Bruce:

No, it still slow, I found one file compiling for 11h on the server, the file is obj/third_party/vulkan-deps/glslang/src/glslang_lib_sources/doc.o

[Yizhuo Zhai]

Hi Moon:

Thanks for your message, I now believe it's more related to the system and my own code. I would do more work on my side. Thanks for the help from you all.

[Bruce Dawson]

I've uploaded a CL to add an option to post_build_ninja_summary.py to sort by elapsed time, FWIW. crrev.com/c/4285697

[Yizhuo Zhai]

Hi Dawson:

Thanks for the useful script and I appreciate your help a lot :) 

[Yizhuo Zhai]

Hi Vitali:

Thanks for the break down, I have figured it in my compilation case--it's about the compilers config and sanitizer. Chrome indeed takes a reasonable amount of time to compile.

On Fri, Nov 24, 2023, 8:30 PM Vitali Fridman <vitali....@gmail.com> wrote:

On a host with AMD 3990X CPU (64 cores), 256GB of RAM and Intel Optane SSDs with all compilation defaults it took 23 minutes to build from scratch. 

[1 processes, 54926/54926 @ 38.4/s : 1429.770s ] LINK ./chrome
metric count avg (us) total (ms)
.ninja parse 7790 1673.5 13036.5
.ninja_log load 1 50823.0 50.8
.ninja_deps load 1 168093.0 168.1
node stat 200124 1.6 314.3
node stat 434002 3.5 1501.8
depfile load 340 45.1 15.4
StartEdge 54926 4741.9 260452.7
FinishCommand 54926 1054.9 57939.4

path->node hash load 0.88 (308895 entries / 351061 buckets)
Longest build steps:
0.6 weighted s to build obj/third_party/blink/renderer/core/libblink_core.a (2.6 s elapsed time)
0.8 weighted s to build gen/third_party/devtools-frontend/src/test/unittests/front_end/pa... (2.4 s elapsed time)
0.8 weighted s to build gen/third_party/devtools-frontend/src/test/unittests/front_end/pa... (2.6 s elapsed time)
0.9 weighted s to build obj/v8/torque_generated_initializers/wasm-to-js-tq-csa.o (111.6 s elapsed time)
0.9 weighted s to build obj/chrome/browser/browser/render_view_context_menu.o (21.9 s elapsed time)
1.0 weighted s to build gen/content/browser/devtools/devtools_resources_grit.d.stamp, gen... (2.1 s elapsed time)
1.2 weighted s to build obj/chrome/browser/libbrowser.a (1.2 s elapsed time)
2.8 weighted s to build gen/third_party/devtools-frontend/src/front_end/Images/3d-center.... (5.6 s elapsed time)
6.3 weighted s to build v8_context_snapshot_generator (57.6 s elapsed time)
158.9 weighted s to build chrome (158.9 s elapsed time)
Time by build-step type:
3.2 s weighted time to generate 1704 .a files (87.9 s elapsed time sum)
10.3 s weighted time to generate 605 .pb.o files (1337.8 s elapsed time sum)
11.0 s weighted time to generate 267 .js files (896.3 s elapsed time sum)
13.1 s weighted time to generate 1164 mojo files (1205.5 s elapsed time sum)
15.7 s weighted time to generate 1096 .mojom-shared.o files (2039.0 s elapsed time sum)
24.1 s weighted time to generate 752 .mojom-blink.o files (3130.0 s elapsed time sum)
29.9 s weighted time to generate 803 .d.ts files (2074.2 s elapsed time sum)
34.6 s weighted time to generate 1096 .mojom.o files (4475.9 s elapsed time sum)
165.8 s weighted time to generate 32 (no extension found) files (263.8 s elapsed time sum)
1099.4 s weighted time to generate 34975 .o files (142022.8 s elapsed time sum)
1423.7 s weighted time (158864.0 s elapsed time sum, 111.6x parallelism)
54926 build steps completed, average of 38.58/s

real 23m50.918s
user 2388m53.144s
sys 213m55.978s