Build chromium on windows platform take longer time than before

[shaob...@intel.com]

Hi, all,

Recently, I find my machine build chromium on widows platform take longer time than before. Usually it took about half an hour to build the whole chrome, but now it took me more than 90 minutes to build. And I find that CPU, memory, disk are all under 50% usage(actually, most times they are under 10%). I highly suspect some windows update cause this regression. Do you have the same issue or any idea about this?

P.S. Here are infos of my machine 

OS : Windows 10 (OS build 15063.850)

CPU :  Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz (22 cores, 44 threads)

Memory : 64GB

 Here is my gn config

is_debug = false

is_component_build = true

symbol_level = 2

use_jumbo_build=true

[David Turner]

Spectre mitigation patches maybe ? You seem to have an Haswell-based CPU, which is more negatively affected by these when it comes to performance, as far as I understand.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev+unsubscribe@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/605cc3db-6e09-4c86-a674-f2372d23ca75%40chromium.org.

[Daniel Bratell]

Sounds like you have already checked that your machine isn't swapping extensively so it's strange. One of the resources CPU, disk or memory should be the bottle neck and be fully used.

I would suspect something local with your machine but there are some tools you could use to get a better idea of what is going on. https://github.com/nico/ninjatracing will visualize your out\Default\.ninja_log and could tell you if there is a strange lack of parallelism.

There is a phase in the middle when building v8 that the rest of the build will pause and wait for that to complete. Then parallelism will drop to near 0 in jumbo builds but that only lasts a minute or less.

In the jumbo work I compile all the time and build time keeps increasing (~1% per week) but not by 200% in a short time. I've not measured on Windows recently and clang was said to be up to 30% slower, but again, 30% isn't 200%.

There is also mspdbsrv in VS builds. That could be a bottleneck. Not sure what is going on with that in clang builds.

/Daniel

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/605cc3db-6e09-4c86-a674-f2372d23ca75%40chromium.org.

--

/* Opera Software, Linköping, Sweden: CET (UTC+1) */

[bruce...@chromium.org]

If your system is mostly idle then you are probably stuck in one of the link stages. However, before speculating about which build step is slow let me share a couple of tricks for investigating slow build times, from an internal email I sent earlier today. Let me know what this tells you. Note that the suggested NINJA_STATUS should let you see why your build is slow while it is running (sort of) while the NINJA_SUMMARIZE_BUILD/post_build_ninja_summary.py suggestion will let you analyze slow builds after the fact.

Why is my build slow?

With the latest depot_tools this question can more easily be answered with these three steps:

> set NINJA_STATUS=[%r processes, %f/%t @ %o/s : %es ] 

> set NINJA_SUMMARIZE_BUILD=1

> autoninja -C out\Default chrome

The NINJA_STATUS variable tells ninja to give extra perf-info while building. Highly recommended, available for years, see the bold text below.

The new feature is that when NINJA_SUMMARIZE_BUILD is set to 1 then autoninja.bat (created to set an appropriate -j value for goma/non-goma builds) will invoke a build-analysis summary script after every successful build. It shows the slowest build steps, and the slowest build types (PE files, obj files, etc.)

The "weighted time" in these reports is elapsed time divided by num-tasks-running and is good proxy for much a build step "matters" (how much it slows/serializes the build). This summary makes slow steps stand out and makes it clear how much time is spent on compilation, linking and .stamp file generation.

A subsequent patch will make this work with autoninja on Linux. You can also run post_build_ninja_summary.py manually using ninja syntax:

> python c:\src\depot_tools\post_build_ninja_summary.py -C out\debug_component

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/605cc3db-6e09-4c86-a674-f2372d23ca75%40chromium.org.

[shaob...@intel.com]

Hi, all

Thx for all of the tool your offered. And after some trying, It turns out to blame "Windows Defender" service.

After disable "Windows Defender" service, the build performance improve a lot. But in "task manager", "Windows Defender" charge for no CPU use or Memory, Disk use. It's weired.

P.S.Here are json files generated by ninjatracing to show the slow build and the fast build and here are screen shot about ninja summary about the slow build and fast build.

Thx again for all your helps

BR,

Yan, Shaobo

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/605cc3db-6e09-4c86-a674-f2372d23ca75%40chromium.org.

[Bruce Dawson]

Uggh. We hit problems with Windows Defender slowing down goma builds - the gomacc.exe compiler was taking 250 ms to be launched (normal is more like 4-5 ms). I've heard of the Rust compiler hitting this. Presumably you were hitting this with the clang-cl compiler. I wonder if signing the compiler would help at all, or if installing it in a read-only location would help.

Anti-virus is such a nuisance and this feels like something that we might keep hitting. Maybe ninja needs a warning whenever process creation is consistently slow?

You received this message because you are subscribed to a topic in the Google Groups "Chromium-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/a/chromium.org/d/topic/chromium-dev/PA08hu52-v8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to chromium-dev+unsubscribe@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/39e352a8-06fa-49a2-8f30-3ef8c9e7307f%40chromium.org.

[shaob...@intel.com]

I tried add exclusive rules for Windows Defender and test build performance. Either add ninja.exe and clang-cl.exe into exclusive lists or chrome folder into exclusive lists could improvement performance a lot.

To unsubscribe from this group and all its topics, send an email to chromium-dev...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/39e352a8-06fa-49a2-8f30-3ef8c9e7307f%40chromium.org.