VS 2015 build performance

[Bruce Dawson]

Chromium now builds with VS 2015* so I ran some tests over the weekend to see if the build-time improvements that I'd heard about had showed up. The answer is slightly complicated. I did spot tests of linking-speed for base_unittests.exe, chrome_child.dll, and unit_tests.exe. I also tested buildtype=Official linking speed (LTCG**) with base_unittests.exe and chrome_child.dll. Finally, I tested full build times (compilation and linking) of the 'chrome' target.

The summary is that some scenarios are slightly slower and some scenarios are significantly faster. Changing a few build settings may let us squeeze out more improvements on incremental builds.

• Linking speed: VS 2015 takes about 8% less time to link. However it uses 2.5-19.7% more CPU time, so on a busy machine it may end up being slower. Linking also typically uses ~20% more memory*** which can lead to severe build-time and machine-responsiveness issues.
  
• Linking speed with /debug:fastlink****: once I got this working (CL will land soon) linking took 40-60% less time than on VS 2013, and 45-85% less memory than VS 2013! This should help significantly with local builds. The generated code is identical, but the PDBs generated are much smaller (but machine-local).
  
• Even with /debug:fastlink the total build times (clean build of 'chrome' target) with VS 2015 are slightly slower - about 7% slower.
  
• With buildtype=Official (LTCG) and /debug:fastlink I found that link times were 4.5-13.0% shorter. This is good but I was hoping for much more speedup. Incremental LTCG (new in VS 2015) may help this dramatically but I have not tested it.
  
• Incremental linking is supposed to be much more effective with VS 2015.

Bruce Dawson

* Not all variations of Chromium necessarily build. In particular component builds need a little bit of work.

** LTCG is link-time-code-generation and incremental LTCG is documented here.

*** unittests.exe uses 7.8 GB to link with VS 2013, and 9.5 GB to link with VS 2015.

**** /debug:fastlink is 'documented' here. It doesn't work automatically because it is silently disabled by /PROFILE.

[Brett Wilson]

On Mon, Jul 27, 2015 at 11:38 AM, Bruce Dawson <bruce...@chromium.org> wrote:

Chromium now builds with VS 2015* so I ran some tests over the weekend to see if the build-time improvements that I'd heard about had showed up. The answer is slightly complicated. I did spot tests of linking-speed for base_unittests.exe, chrome_child.dll, and unit_tests.exe. I also tested buildtype=Official linking speed (LTCG**) with base_unittests.exe and chrome_child.dll. Finally, I tested full build times (compilation and linking) of the 'chrome' target.

The summary is that some scenarios are slightly slower and some scenarios are significantly faster. Changing a few build settings may let us squeeze out more improvements on incremental builds.

• Linking speed: VS 2015 takes about 8% less time to link. However it uses 2.5-19.7% more CPU time, so on a busy machine it may end up being slower. Linking also typically uses ~20% more memory*** which can lead to severe build-time and machine-responsiveness issues.
  
• Linking speed with /debug:fastlink****: once I got this working (CL will land soon) linking took 40-60% less time than on VS 2013, and 45-85% less memory than VS 2013! This should help significantly with local builds. The generated code is identical, but the PDBs generated are much smaller (but machine-local).

 I'm really excited about this one... ^^^

Brett

[Robert Iannucci]

Is there a way that we could post-process machine-local pdbs into generalized ones? I'm assuming the answer is 'not easily', but I also know we have some windows/msvc/pdb wizards on the team :)

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

[Zachary Turner]

I guess the question is what makes them machine local.  Has anyone investigated this yet?  Off the top of my head, the only thing I can think of is that it does not copy MSVCRT symbols (well, more accurately UCRT I guess) into the PDB, and instead just puts references to the debug info for these libraries into the PDB.  Normally what happens is the linker reads LIBCMT?.pdb (or similar corresponding to whatever version of the CRT you're using) and copies symbols into your program's PDB at link time.  But If I recall this is only for static CRT, and I'm not sure if Bruce's test was /MD or /MT.  For /MD I can't think of anything else in a PDB that would cause it to be machine local, and certainly not anything that would take up a significant amount of space in a PDB, so it's probably worth investigating.

FWIW I'm the owner of llvm-pdbdump which is designed to allow exploration, search, and dumping of PDB contents.  If anyone wants to send me a couple of PDBs (machine local and machine-agnostic) I can take a look and see if I can find anything.  Although I didn't write it with this use case in mind, so I'm not sure if it will turn anything up.

[Alex Vakulenko]

From http://blogs.msdn.com/b/vcblog/archive/2014/11/12/speeding-up-the-incremental-developer-scenario-with-visual-studio-2015.aspx

The Visual C++ linker for non LTCG builds spends majority of its time in generating program database file (PDB). Merging of type information, fixing up private symbol's type indexes, and generating global symbols are major time components in PDB generation. With /DEBUG:FASTLINK the linker produced PDB doesn't have any private symbol, and debug information is distributed among input object and library files and the linker generated PDB just serves as an indexing database.  DIA APIs have been modified to provide a seamless experience for debugging (only), and using this option provides much faster link times with little or no impact to overall debugging experience. To illustrate this further, notice the drop in full link times with the /DEBUG:FASTLINK switch thrown for a couple of benchmarks we have here in our labs. 

[Will Harris]

[resend, sorry] I suspect that breakpad dumpsyms (and thus, crash analysis) would have a really bad time with this new PDB format.

Will

On Mon, Jul 27, 2015 at 12:48 PM, Will Harris <w...@google.com> wrote:

I suspect that breakpad dumpsyms (and thus, crash analysis) would have a really bad time with this.

Will

[Zachary Turner]

Ahh, makes sense.  So basically this is kind of like a hybrid between /Z7 and /Zi.  Yea, you're not going to get this working on another machine without transferring all the obj files as well, and even then you'd have to make sure you transfer them to the location that the PDB is expecting.  And of course you're transferring much more than just debugging info at this point, you're transferring the entire generated code as well.

[Robert Iannucci]

Oh. Yeah OK that makes sense :)

[Bruce Dawson]

I'll suggest to Microsoft that they should ship a tool that converts a fastlink PDB to a regular PDB. That seems like it might occasionally be useful.

Meanwhile, any tool that consumes PDBs through msdia140 should just work, although I have not tested this at all.

I investigated working sets because I hit a severe paging storm last week while doing a full VS 2015 build. I'm not used to hitting paging problems on a machine with 64 GB of RAM. Since the linker memory consumption has only increased by ~20% in VS 2015 that suggests that VS 2013 builds have also been getting close to the edge. Currently ninja is configured to allow one linker instance per 4 GB of RAM. That should probably be increased to 5-6 GB of RAM, especially given the increased parallelism of linking in VS 2015. When using fastlink the number of parallel links could be increased, by using GYP_LINK_CONCURRENCY.

[Chris Hamilton]

This one would be incompatible with instrumenting the binary (ie: PGO builds), I presume, as /PROFILE is required for that, no?

Sounds like it would be great for developer builds, and even trybot builds, but it wouldn't speed up the official builds any.

[Bruce Dawson]

> even trybot builds

Well, those that don't need symbols probably have symbols mostly or completely disabled anyway.

> it wouldn't speed up the official builds any.

Correct. /DEBUG:FASTLINK won't help official builds. It looks like official builds will be slightly slower with VS 2015, although incremental LTCG could help with buildtype=Official builds on the buildbots.

[Will Harris]

With this change, would we need to update tools\win\copy-installer.bat to copy all the obj files across if debugging is to work in a VM?

Will

On Mon, Jul 27, 2015 at 2:02 PM, Chris Hamilton <chr...@chromium.org> wrote:

[Bruce Dawson]

I think that if you plan to debug on a VM you should not use /DEBUG:FASTLINK. The time it takes to copy over the .obj files (and the effort to figure out which .obj or .lib files need to be copied) would probably erase all of the benefits.

This is a scenario where a PDB-de-fastlink tool would be very helpful. I'll suggest that scenario to Microsoft. Now that they update VS more frequently we might even get what we ask for.

[Paweł Hajdan, Jr.]

Curious: have you tried measuring compile/link time difference with goma?

Paweł

[Shinya]

Sorry, Visual Studio 2015 is not working with goma yet.

I'm working on it. I still have a few problems to make it work with goma.

[Sunny Sachanandani]

Please let us know if the CL to enable /debug:fastlink lands and what gyp flags enable that. I would like my local debug builds to be faster.

Thanks!

- Sunny

[Zachary Turner]

On Mon, Jul 27, 2015 at 2:06 PM Will Harris <w...@chromium.org> wrote:

With this change, would we need to update tools\win\copy-installer.bat to copy all the obj files across if debugging is to work in a VM?

Will

Can't you just share a folder from your host to your guest? 

[Bruce Dawson]

The /debug:fastlink CL landed yesterday -- see https://codereview.chromium.org/1254723012. It requires VS 2015 which is still bleeding edge, so caveat codor.

Instructions are in the CL but it's basically GYP_DEFINES=win_fastlink=1. For gn builds it's is_win_fastlink = true but I'm not sure that gn builds work with VS 2015.

Sharing a folder from your host to your guest should work if the drive letters are the same. Or maybe the paths are relative -- I don't know. I think that if you need to debug on a VM you shouldn't use /debug:fastlink.

Note that if you switch win_fastlink on/off you have to relink but not recompile, so it's expensive but not tragically expensive.

Note also that if you switch win_fastlink on then your PDBs won't get smaller. PDBs never get smaller unless you delete them. To save disk space when switching to win_fastlink you might want to delete your PDBs.

[Zachary Turner]

On Wed, Jul 29, 2015 at 11:13 AM Bruce Dawson <bruce...@chromium.org> wrote:

The /debug:fastlink CL landed yesterday -- see https://codereview.chromium.org/1254723012. It requires VS 2015 which is still bleeding edge, so caveat codor.

Instructions are in the CL but it's basically GYP_DEFINES=win_fastlink=1. For gn builds it's is_win_fastlink = true but I'm not sure that gn builds work with VS 2015.

Sharing a folder from your host to your guest should work if the drive letters are the same. Or maybe the paths are relative -- I don't know. I think that if you need to debug on a VM you shouldn't use /debug:fastlink.

I imagine the paths are absolute.  But you can always use some combination of subst or directory symlinks to achieve the same effect.  Obviously this is probably totally unsupported, but I *think* it would work.  YMMV 

[Marc-Antoine Ruel]

[Back from vacation]

I think enabling fastlink by default for devs is probably a good idea but it must never be used on the infrastructure, as it won't work with isolated testing.

Thanks,

M-A

--