Windows build linking with lld by default

361 views
Skip to first unread message

Hans Wennborg

unread,
Apr 17, 2018, 1:35:27 PM4/17/18
to blink-dev, chromium-dev, Nico Weber
Hello everyone,

With the commit of https://chromium-review.googlesource.com/c/chromium/src/+/983632, the Windows build now links with lld (LLVM's linker) instead of link.exe (MSVC's linker) by default.

At this point, the idea is mainly to gather data and find out about unknown unknowns, so it may or may not stick.

lld is significantly faster than link.exe (see Benchmarks at http://blog.llvm.org/2018/01/improving-link-time-on-windows-with.html), and it also understands LLVM's internal code representation, which will enable us to use link-time optimization and control-flow integrity checks with Clang.

However, lld unlike link.exe does not support incremental linking, so if you're only touching a few files and re-linking a large executable, link times may be slower.

Also, the PDB files produced by lld haven't seen much real-world use yet, so we're interested to hear about any debugging issues with this switch. If you find issues, please file bugs on https://crbug.com/ and cc hans, rnk, thakis and zturner.

Cheers,
Hans

Nico Weber

unread,
Apr 17, 2018, 1:55:47 PM4/17/18
to Hans Wennborg, blink-dev, chromium-dev
We found an unknown unknown (hooray); reverting at https://chromium-review.googlesource.com/c/chromium/src/+/1015141 . We'll be back!

Daniel Bratell

unread,
Apr 19, 2018, 8:58:47 AM4/19/18
to blink-dev, chromium-dev, Hans Wennborg, Nico Weber
When you say that lld is faster than link.exe, that seems to refer to the case when link.exe does not use mspdbsrv.exe. That comparion makes sense since many use goma which by necessity disables mspdbsrv, but the rest of us is still using it. What is the expected performance change for us?

(No good access to my build environment right now or I would have checked myself)

/Daniel
--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAB8jPhf4EEUDq2q9UMkFKx1X26HYTTED-qe5o2Nn31DbGLsPww%40mail.gmail.com.



--
/* Opera Software, Linköping, Sweden: CEST (UTC+2) */

Nico Weber

unread,
Apr 19, 2018, 10:03:22 AM4/19/18
to Daniel Bratell, blink-dev, Chromium-dev, Hans Wennborg
Clang never uses mspdb, so we never hit that case in a chromium build.

Reid Kleckner

unread,
Apr 19, 2018, 12:26:10 PM4/19/18
to Nico Weber, Daniel Bratell, blink-dev, chromium-dev, Hans Wennborg
LLD can read type server PDBs produced by cl /Zi and mspdbsrv just fine, so it will do the same amount of work as link.exe in a non-incremental link. Overall, LLD should be faster than link.exe, so non-incremental links should be faster.

As Nico says, Clang doesn't implement type server PDBs (/Zi maps to /Z7), so this may not be very relevant going forward. However, if you do use cl.exe /Zi with LLD, that may be faster than clang-cl.exe /Z7 with LLD, because type server PDBs do effectively shift type merging work to the compilation phase.

I would expect type server PDBs to improve link.exe's /incremental link performance, so the relative slowdown from link.exe to LLD for incremental links may be higher. The absolute time should be acceptable (a few seconds per DLL), though, since it will at least as fast the link times Chromium gets with goma.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAMGbLiHTru3BEpOA9aL%2BxGr0bj7-68QbrWQmfMcW61KzfPTb1Q%40mail.gmail.com.

Hans Wennborg

unread,
Apr 25, 2018, 8:25:26 PM4/25/18
to Nico Weber, blink-dev, chromium-dev
We have fixed the issue and enabled lld by default again:
https://chromium-review.googlesource.com/c/chromium/src/+/1028374

Zachary Turner

unread,
Apr 25, 2018, 9:09:24 PM4/25/18
to r...@google.com, Daniel Bratell, Hans Wennborg, Nico Weber, blink-dev, chromium-dev
(Resending again, this time from an email address that has permissions to post to chromium-dev).

LLD is faster for many reasons, *one of which is /DEBUG:GHASH*.  So you will still see a large performance gain over link.exe even when linking Zi object files produced by cl.  Unfortunately I can’t effectively profile link.exe since obviously I don’t have the source, so all i can do is make guesses as to why (which i do have some theories about, fwiw).



Zachary Turner

unread,
Apr 25, 2018, 10:32:41 PM4/25/18
to Yoichi Osato, Daniel Bratell, Hans Wennborg, Nico Weber, blink-dev, chromium-dev, r...@google.com
That is correct, incremental linking is the one thing we consistently are slower on.  (LLD has no notion of an incremental link, every link is a full link).  So This is working as intended.  Note that a full link of this same target with link.ece would probably be several minutes.

My own measurements show us to be about 2x slower on incremental links and it sounds like you’re seeing a similar slowdown.

You can add use_ghash=true to your gn args for a bit faster link.  I’m guessing it will get you down to the 23-25s range.  This option is experimental though, so please let us know if you encounter any issues.

On Wed, Apr 25, 2018 at 7:12 PM Yoichi Osato <yoi...@chromium.org> wrote:
For incremental build by editing few lines, LLD is not better.
I'm using following gn to debug:

target_cpu = "x86"

enable_nacl = false

is_component_build = true

is_debug = true

is_win_fastlink = true

use_lld = false

win_console_app = true

It takes 13 seconds to build content_shell after "touch Node.cc"

If I turns use_lld true, it takes 30 seconds.

2018年4月26日(木) 10:07 Zachary Turner <ztu...@chromium.org>:

Hans Wennborg

unread,
Apr 25, 2018, 11:18:13 PM4/25/18
to Nico Weber, blink-dev, chromium-dev
... and we found new problems (crbug.com/837090), so reverted again.
Sorry for the back and forth.

Daniel Bratell

unread,
Apr 26, 2018, 9:43:21 AM4/26/18
to Yoichi Osato, Zachary Turner, Hans Wennborg, Nico Weber, blink-dev, chromium-dev, r...@google.com
So in linking speed from fastest to slowest

FAST
--------
1. Incremental link.exe with /Zi
2. Incremental link.exe with /Z7
3. link.exe with /Zi
4. lld.exe with new hashing
5. lld.exe
6. link.exe with /Z7 
--------
SLOW

And for full builds this is an improvement (jumping from pos 6 to pos 4/5 above) and for incremental builds it may not be (jumping from pos 2 to pos 4/5 above). 

/Zi can't be used (is disabled) for clang builds, and was previously also disabled for goma builds so link.exe with /Zi is not an option.

In summary:

Pro vs now
----------------
Faster full builds (minutes)
Support for some clang-only analysis/optimization features

Con vs now
----------------
Slower incremental builds (tens of seconds)
(Relatively untested)

I'd prefer something that was better in every aspect, but it's never that easy, is it?

/Daniel

Nico Weber

unread,
Apr 26, 2018, 9:47:47 AM4/26/18
to Daniel Bratell, Yoichi Osato, Zachary Turner, Hans Wennborg, blink-dev, Chromium-dev, Reid Kleckner
Also keep in mind that Z7 with link.exe means you either pass /fastlink (which makes the msvc debugger crashy) or you don't and your links are even slower.

Zachary Turner

unread,
Apr 26, 2018, 12:59:36 PM4/26/18
to Daniel Bratell, Yoichi Osato, Hans Wennborg, Nico Weber, blink-dev, Chromium-dev, Reid Kleckner
I'd be surprised if #3 is faster than 4, 5, or 6. But I don't think I've
measured it. What kind of numbers are you seeing?
https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAAErz9jjPOtjNCwtPiHSSTwJ8LWKq69enNoQv6RXO4jO917f4g%40mail.gmail.com
.

Daniel Bratell

unread,
Apr 26, 2018, 1:47:34 PM4/26/18
to Zachary Turner, Yoichi Osato, Hans Wennborg, Nico Weber, blink-dev, Chromium-dev, Reid Kleckner
I also don't have any numbers here, but it's the impression I get from
what has been written in the blog post and other places. The problem with
link.exe+/Zi does not seem to be linking speed, but that it creates a
bottleneck in the compiling phase, especially if you have many (dozens) of
cores that all need to communicate with mspdbsrv. And that it can't be
distributed.

We used link.exe with /Zi up until the switch to clang on Windows and on
our machines, full linking time on Windows was not a problem most of the
time (Mac on the other hand...). If it turns out that lld for full builds
is even faster than link.exe with /Zi then ... yay.

/Daniel

On Thu, 26 Apr 2018 18:57:50 +0200, Zachary Turner <ztu...@chromium.org>

Zachary Turner

unread,
Apr 26, 2018, 4:13:52 PM4/26/18
to Daniel Bratell, Yoichi Osato, Hans Wennborg, Nico Weber, blink-dev, Chromium-dev, Reid Kleckner
Definitely try it, I'd be interested in seeing those measurements. I
honestly still think LLD is going to be noticeably faster (minus the
incremental link case). Our type merging is blazing fast now, but we're
still faster on many other aspects of linking as well, such as the way we
use the file system. On links where the total size of all input files is
large, or where the size of the output PDB is large, the file I/O alone is
a big area in which we come out ahead.

If I had to wager a guess, I'd re-order your list 1, 2, 4, 5, 3, 6. But if
you find out I'm wrong, let me know so I can figure out why and fix the
glitch ;-)

bruce...@chromium.org

unread,
Apr 27, 2018, 6:41:09 AM4/27/18
to Chromium-dev, bra...@opera.com, yoi...@chromium.org, ha...@chromium.org, tha...@chromium.org, blin...@chromium.org, r...@google.com
> Also keep in mind that Z7 with link.exe means you either pass /fastlink (which makes the msvc debugger crashy) or you don't and your links are even slower.

+++ to this point. Fastlink was supposed to save the world but it has caused me significant grief and wasted time. When calculating the relative advantages of link.exe and lld-link.exe it is important to either compare without fastlink, or to use link.exe /debug:fastlink and then factor in all the time spent restarting the VS debugger after it crashes, relinking after realizing you can't enumerate symbols, relinking after realizing that windbg and fastlink get along poorly, etc.

Yes, I'd love incremental linking but the slight increase in link times for incremental builds is a small price to pay for "It Just Works (tm)"
> > send an email to blink-dev+unsubscribe@chromium.org.
> >>>>>>>> To view this discussion on the web visit
> >
https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAB8jPhf4EEUDq2q9UMkFKx1X26HYTTED-qe5o2Nn31DbGLsPww%40mail.gmail.com
> > .
> >
> >
> >
> >
> >>>>>>>> --
> >>>>>>>> /* Opera Software, Linköping, Sweden: CEST (UTC+2) */
> >
> >>>>>>> --
> >>>>>>> --
> >>>>>>> Chromium Developers mailing list: chromi...@chromium.org
> >>>>>>> View archives, change email options, or unsubscribe:
> >>>>>>> http://groups.google.com/a/chromium.org/group/chromium-dev
> >>>>>>> ---
> >>>>>>> You received this message because you are subscribed to the Google
> > Groups "Chromium-dev" group.
> >
> >>>>>>> To unsubscribe from this group and stop receiving emails from it,
> >
> >
> >>>>>>> To view this discussion on the web visit
> >
https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAMGbLiHTru3BEpOA9aL%2BxGr0bj7-68QbrWQmfMcW61KzfPTb1Q%40mail.gmail.com
> > .
> >
> >>>>>> --
> >>>>>> --
> >>>>>> Chromium Developers mailing list: chromi...@chromium.org
> >>>>>> View archives, change email options, or unsubscribe:
> >>>>>> http://groups.google.com/a/chromium.org/group/chromium-dev
> >>>>>> ---
> >>>>>> You received this message because you are subscribed to the Google
> > Groups "Chromium-dev" group.
> >>>>>> To view this discussion on the web visit
> >
https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CACs%3DtyLRVR-NFuq0CAL1SD%3DWh5L%3D0kbKQH3pWNRrjBnR_RcrAw%40mail.gmail.com
> > .
> >
> >>>> --
> >>>> You received this message because you are subscribed to the Google
> > Groups "blink-dev" group.
> >>>> To unsubscribe from this group and stop receiving emails from it,
send

Nico Weber

unread,
May 14, 2018, 10:17:38 PM5/14/18
to Hans Wennborg, blink-dev, chromium-dev
We relanded the switch a few hours ago once more, in https://chromium-review.googlesource.com/c/chromium/src/+/1057767

On Tue, Apr 17, 2018 at 1:33 PM, Hans Wennborg <ha...@chromium.org> wrote:

Nico Weber

unread,
Jun 20, 2018, 9:42:22 PM6/20/18
to mohammad....@gmail.com, blink-dev, Hans Wennborg, Chromium-dev
Can you post your exact build config? Are you talking about incremental links or full links?

Most users are on Android and Windows; Windows certainly isn't a second-class citizen.

On Wed, Jun 20, 2018 at 7:49 PM <mohammad....@gmail.com> wrote:
I just tried the webrtc checkout with lld vs link.exe on windows 10 and using the vs2017. link.exe is faster. I think windows is a second class citizen for the webrtc community, so it's best to use whatever MS tools available because that's their main business and will be updated more frequently. My 2 cents.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

Nico Weber

unread,
Aug 29, 2018, 6:43:39 PM8/29/18
to ash...@gmail.com, blink-dev, Mohammad Abuomar, Hans Wennborg, Chromium-dev
Hi,

the lld build defaults to thin static libraries, meaning it just refers to obj files on disk. This is configured here: https://cs.chromium.org/chromium/src/build/toolchain/win/BUILD.gn?type=cs&q=%5C/lib%5C++file:%5C.gn&sq=package:chromium&g=0&l=111

On Linux, we have a way to opt out of similar behavior there, see https://cs.chromium.org/chromium/src/build/config/compiler/BUILD.gn?type=cs&q=thin_archive+file:%5C.gn&sq=package:chromium&g=0&l=1690 If there's demand, we could have an opt-out mechanism on Windows too. (But the default for chromium builds shouldn't change, since writing 35MB is way faster than writing 400MB as you can imagine.)

lld output in general can be read by link.exe, but link.exe doesn't support thin archives. So those can't be read by link.exe.

> Going forward will the msvc compiler and linker be supported?

They are community-supported, meaning we have no bot coverage for them. We do accept patches to keep them going though.



On Wed, Aug 29, 2018 at 3:14 PM <ash...@gmail.com> wrote:
When building webrtc m69 branch (coming from m68 branch) the produced webrtc.lib file is very small now. Earlier it was ~400 mb, now it's ~35 mb. Is this an expected behavior with lld? Also I can no longer use the produced lib files in a visual studio project. I'm assuming lld output cannot be read by link? Going forward will the msvc compiler and linker be supported?
Reply all
Reply to author
Forward
0 new messages