Using clang-cl to ship Windows builds

[David Major]

Bug 1443590 is switching our official Windows builds to use clang-cl
as the compiler.

Please keep an eye out for regressions and file a blocking bug for
anything that might be fallout from this change. I'm especially
interested in hearing about the quality of the debugging experience.

It's possible that the patch may bounce and we'll go back and forth to
MSVC for a while. You can check your build's compiler at
`about:buildconfig`. Treeherder is running an additional set of MSVC
jobs on mozilla-central to make sure we can fall back to a green MSVC
if needed.

Watch for more toolchain changes to come. The next steps after this
will be to switch to lld-link and enable ThinLTO. That will open the
door to a cross-language LTO that could inline calls between Rust and
C++. In the longer term we can look into cross-compiling from Linux.

But for now, shipping our most-used platform with an open-source
compiler is a huge milestone in and of itself. Big thanks to everyone
who has contributed to this effort on the Mozilla side, and also big
thanks to the developers of LLVM and Chromium who helped make clang on
Windows a realistic possibility.

[Chris Peterson]

How does the performance of clang-cl builds compare to MSVC builds on
benchmarks like Speedometer?

> _______________________________________________
> firefox-dev mailing list
> firef...@mozilla.org
> https://mail.mozilla.org/listinfo/firefox-dev

[Gregory Szorc]

On Tue, Jul 10, 2018 at 1:29 PM, David Major <dma...@mozilla.com> wrote:

> Bug 1443590 is switching our official Windows builds to use clang-cl
> as the compiler.
>
> Please keep an eye out for regressions and file a blocking bug for
> anything that might be fallout from this change. I'm especially
> interested in hearing about the quality of the debugging experience.
>
> It's possible that the patch may bounce and we'll go back and forth to
> MSVC for a while. You can check your build's compiler at
> `about:buildconfig`. Treeherder is running an additional set of MSVC
> jobs on mozilla-central to make sure we can fall back to a green MSVC
> if needed.
>
> Watch for more toolchain changes to come. The next steps after this
> will be to switch to lld-link and enable ThinLTO. That will open the
> door to a cross-language LTO that could inline calls between Rust and
> C++. In the longer term we can look into cross-compiling from Linux.
>
> But for now, shipping our most-used platform with an open-source
> compiler is a huge milestone in and of itself. Big thanks to everyone
> who has contributed to this effort on the Mozilla side, and also big
> thanks to the developers of LLVM and Chromium who helped make clang on
> Windows a realistic possibility.
>

A lot of people have wanted to see this day for years (for various reasons
- an open source toolchain, potential for cross compiling, unified
toolchains across platforms, etc). This is a *major* milestone. And while
the transition will likely have a few bumps, the payoff should be well
worth it.

Thank you to everyone who helped us get here. From the initial work to
support Clang on Windows a few years ago. To the upstream work that was put
into LLVM and Clang (especially by Google/Chromium). To those that worked
through all the issues to transition the compiler today. This was a
significant effort by a lot of people. We should all be ecstatic we're
finally crossing this bridge. I know I am! Congratulations!

[David Major]

At the moment, performance is a mixed bag. Some tests are up and some
are down. In particular I believe Speedometer is down a few percent.

Note however that clang-cl is punching above its weight. These builds
currently have neither LTO nor PGO, while our MSVC builds use both of
those. Any regressions that we're seeing ought to be short-lived. Once
we enable LTO and PGO, I expect clang to be a clear performance win.

If the short-term regressions end up being unacceptable, we can revert
to MSVC later in the release, but at this early point in the cycle
that shouldn't prevent us from collecting data on Nightly.

On Tue, Jul 10, 2018 at 4:31 PM Chris Peterson <cpet...@mozilla.com> wrote:
>
> How does the performance of clang-cl builds compare to MSVC builds on
> benchmarks like Speedometer?
>
>

> On 2018-07-10 1:29 PM, David Major wrote:
> > Bug 1443590 is switching our official Windows builds to use clang-cl
> > as the compiler.
> >
> > Please keep an eye out for regressions and file a blocking bug for
> > anything that might be fallout from this change. I'm especially
> > interested in hearing about the quality of the debugging experience.
> >
> > It's possible that the patch may bounce and we'll go back and forth to
> > MSVC for a while. You can check your build's compiler at
> > `about:buildconfig`. Treeherder is running an additional set of MSVC
> > jobs on mozilla-central to make sure we can fall back to a green MSVC
> > if needed.
> >
> > Watch for more toolchain changes to come. The next steps after this
> > will be to switch to lld-link and enable ThinLTO. That will open the
> > door to a cross-language LTO that could inline calls between Rust and
> > C++. In the longer term we can look into cross-compiling from Linux.
> >
> > But for now, shipping our most-used platform with an open-source
> > compiler is a huge milestone in and of itself. Big thanks to everyone
> > who has contributed to this effort on the Mozilla side, and also big
> > thanks to the developers of LLVM and Chromium who helped make clang on
> > Windows a realistic possibility.

[Bobby Holley]

+1. This is really fantastic news, and frankly happened way faster than I
would have thought possible. Thanks to everyone involved!

On Tue, Jul 10, 2018 at 1:51 PM Gregory Szorc <g...@mozilla.com> wrote:

> On Tue, Jul 10, 2018 at 1:29 PM, David Major <dma...@mozilla.com> wrote:
>
>> Bug 1443590 is switching our official Windows builds to use clang-cl
>> as the compiler.
>>
>> Please keep an eye out for regressions and file a blocking bug for
>> anything that might be fallout from this change. I'm especially
>> interested in hearing about the quality of the debugging experience.
>>
>> It's possible that the patch may bounce and we'll go back and forth to
>> MSVC for a while. You can check your build's compiler at
>> `about:buildconfig`. Treeherder is running an additional set of MSVC
>> jobs on mozilla-central to make sure we can fall back to a green MSVC
>> if needed.
>>
>> Watch for more toolchain changes to come. The next steps after this
>> will be to switch to lld-link and enable ThinLTO. That will open the
>> door to a cross-language LTO that could inline calls between Rust and
>> C++. In the longer term we can look into cross-compiling from Linux.
>>
>> But for now, shipping our most-used platform with an open-source
>> compiler is a huge milestone in and of itself. Big thanks to everyone
>> who has contributed to this effort on the Mozilla side, and also big
>> thanks to the developers of LLVM and Chromium who helped make clang on
>> Windows a realistic possibility.
>>
>

> A lot of people have wanted to see this day for years (for various reasons
> - an open source toolchain, potential for cross compiling, unified
> toolchains across platforms, etc). This is a *major* milestone. And while
> the transition will likely have a few bumps, the payoff should be well
> worth it.
>
> Thank you to everyone who helped us get here. From the initial work to
> support Clang on Windows a few years ago. To the upstream work that was put
> into LLVM and Clang (especially by Google/Chromium). To those that worked
> through all the issues to transition the compiler today. This was a
> significant effort by a lot of people. We should all be ecstatic we're
> finally crossing this bridge. I know I am! Congratulations!

[pedro....@gmail.com]

Is there a link to the required changes? I'm curious.

[Sylvestre Ledru]

Le 11/07/2018 à 14:01, pedro....@gmail.com a écrit :
> Is there a link to the required changes? I'm curious.

One of the meta bug is:

https://bugzilla.mozilla.org/show_bug.cgi?id=752004

Look at the list of "Depends on" to see what have been required!
(not mentioning other related work like the Linux and Mac support).

Sylvestre

[Thomas Daede]

On 07/10/2018 01:29 PM, David Major wrote:
> Bug 1443590 is switching our official Windows builds to use clang-cl
> as the compiler.

Another great effect of this change is that it finally fixes the issue
of constantly running out of virtual address space when linking Win32
builds: https://bugzilla.mozilla.org/show_bug.cgi?id=709193

This was, for example, blocking AV1 support on Win32 until today. Thanks
for the great work!

[Mike Hommey]

That will still block it, until clang-cl actually becomes the compiler
we use to ship. At the moment, it's not certain it will stay that way,
so we need to keep MSVC builds working, including PGO.

Mike

[halivi...@gmail.com]

I hope that both Firefox and Chrome continue to keep the build and tests running on MSVC. It would suck if for example we can't build Firefox with MSVC.

Will the Firefox team publish builds of Firefox from both MSVC and Clang with symbols so we can profile ourselves and compare which is faster for the webpages we use?

[Anthony Jones]

On Thursday, 12 July 2018 15:50:40 UTC+12, halivi...@gmail.com wrote:
> I hope that both Firefox and Chrome continue to keep the build and tests running on MSVC. It would suck if for example we can't build Firefox with MSVC.

I can't comment on Chrome.

> Will the Firefox team publish builds of Firefox from both MSVC and Clang with symbols so we can profile ourselves and compare which is faster for the webpages we use?

The MSVC nightly builds will likely continue until we fully commit to clang-cl. It is expensive to maintain MSVC workarounds and given that cross-language LTO[1] is compelling for Firefox, it is unlikely we'd return to MSVC.

Anthony

[1] https://github.com/rust-lang/rust/issues/49879

[Mike Hommey]

Actually, I don't think we do produce MSVC nightly builds. We *do* MSVC
build on automation, but they're not exactly nightlies. You won't find
them along other nightlies. And until bug 1474756 lands, those builds
aren't even with PGO enabled.

Mike

[Jörg Knobloch]

On 10/07/2018 22:29, David Major wrote:
> Bug 1443590 is switching our official Windows builds to use clang-cl
> as the compiler.
>

> Please keep an eye out for regressions and file a blocking bug for
> anything that might be fallout from this change. I'm especially
> interested in hearing about the quality of the debugging experience.

Just out of interest a question from the de-facto Thunderbird maintainer:

Does clang-cl give an executable that does the same thing as the
executable created by MS VS C++?

After switching to clang-cl, one of our Window10-only test failures
magically disappeared
(https://bugzilla.mozilla.org/show_bug.cgi?id=1469188) and another one
magically appeared on Windows only
(https://bugzilla.mozilla.org/show_bug.cgi?id=1475166).

I'm not in a position to debug any of that, it's just an observation.

Jörg.

[Anthony Jones]

Changing the compiler will change all the timing characteristics. This can reliably produce different outcomes on race conditions. We've seen some of that. You're likely to be seeing the same thing.

[David Major]

A quick update about performance: ThinLTO and PGO were enabled for our
clang-cl builds over the last few weeks. These optimizations have
cancelled the previous regressions and brought clang well ahead of
MSVC: https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=2adb3cb404ad&newProject=try&newRevision=5336df94a9ce&framework=1
To call out a few: Speedometer scores increased by 6% on Win64 and 12%
on Win32, and the most dramatic improvement was displaylist_mutate at
27%!

As far as I'm aware, no blockers have come up and clang-cl is still
looking good for the 63 train.

On Tue, Jul 10, 2018 at 4:29 PM David Major <dma...@mozilla.com> wrote:
>
> Bug 1443590 is switching our official Windows builds to use clang-cl
> as the compiler.
>
> Please keep an eye out for regressions and file a blocking bug for
> anything that might be fallout from this change. I'm especially
> interested in hearing about the quality of the debugging experience.
>

[Stuart Philp]

That is fantastic, lots of nice wins!