Dropping support for compiling with MSVC in the near future

[Ted Mielczarek]

Hello,

In light of the fact that we've switched to clang-cl for our Windows builds[1], we are planning to drop support for compiling Firefox with MSVC in the near future[2]. Our estimate is that this will happen sometime in Q1. Supporting more than one compiler is a maintenance burden and we've already seen developers spend considerable time getting their patches that work with clang-cl to build with MSVC. We are currently blocked by the fact that our aarch64-windows builds are still using MSVC and we are waiting on upstream clang-cl work to switch those builds to clang-cl. Once that takes place we no longer have a compelling reason to continue supporting MSVC.

To preempt the question--when this happens we intend to make MSVC error in configure, and not just move MSVC to Tier 3 "patches welcome" status. Our reasoning is that Tier 3 configurations still create work: developers spend time building in those configurations, and lack of CI coverage means that when they inevitably break they waste time trying to fix things. Bugs get filed, and other developers waste time trying to help or reviewing patches to fix things. Explicitly unsupporting MSVC is the best way for us to convey the fact that developers should not be using it and we will not accept patches to fix issues that only affect MSVC.

If you have specific reasons for continuing to use MSVC please let us know. If there are deficiencies in clang-cl compared to MSVC we should get them filed and fixed.

Thanks,
-Ted

1. https://bugzilla.mozilla.org/show_bug.cgi?id=1443590
2. https://bugzilla.mozilla.org/show_bug.cgi?id=1512504

[Kris Maglione]

\o/

[Jonathan Kew]

On 06/12/2018 15:00, Ted Mielczarek wrote:
> Hello,
>
> In light of the fact that we've switched to clang-cl for our Windows builds[1], we are planning to drop support for compiling Firefox with MSVC in the near future[2]. Our estimate is that this will happen sometime in Q1. Supporting more than one compiler is a maintenance burden and we've already seen developers spend considerable time getting their patches that work with clang-cl to build with MSVC. We are currently blocked by the fact that our aarch64-windows builds are still using MSVC and we are waiting on upstream clang-cl work to switch those builds to clang-cl. Once that takes place we no longer have a compelling reason to continue supporting MSVC.

While I sympathize with the concern that "supporting more than one
compiler is a maintenance burden", this still leaves me feeling a little
uneasy. Ensuring that our code builds successfully with multiple
compilers is a useful way to keep us from becoming dependent on quirks
of a particular tool, and different compilers may provide different
(valid, useful) diagnostics that we should not simply ignore.

Our C/C++ code should (IMO) be standard and portable enough to build
with any modern, mainstream C++ compiler; writing code for a clang
monoculture feels wrong, a bit like building websites for a Blink
monoculture...

JK

>
> To preempt the question--when this happens we intend to make MSVC error in configure, and not just move MSVC to Tier 3 "patches welcome" status. Our reasoning is that Tier 3 configurations still create work: developers spend time building in those configurations, and lack of CI coverage means that when they inevitably break they waste time trying to fix things. Bugs get filed, and other developers waste time trying to help or reviewing patches to fix things. Explicitly unsupporting MSVC is the best way for us to convey the fact that developers should not be using it and we will not accept patches to fix issues that only affect MSVC.
>
> If you have specific reasons for continuing to use MSVC please let us know. If there are deficiencies in clang-cl compared to MSVC we should get them filed and fixed.
>
> Thanks,
> -Ted
>

> 1. https://bugzilla.mozilla.org/show_bug.cgi?id=1443590
> 2. https://bugzilla.mozilla.org/show_bug.cgi?id=1512504

> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>

[Kris Maglione]

On Thu, Dec 06, 2018 at 03:34:59PM -0500, Jonathan Kew wrote:
>While I sympathize with the concern that "supporting more than one
>compiler is a maintenance burden", this still leaves me feeling a
>little uneasy. Ensuring that our code builds successfully with
>multiple compilers is a useful way to keep us from becoming dependent
>on quirks of a particular tool, and different compilers may provide
>different (valid, useful) diagnostics that we should not simply
>ignore.
>
>Our C/C++ code should (IMO) be standard and portable enough to build
>with any modern, mainstream C++ compiler; writing code for a clang
>monoculture feels wrong, a bit like building websites for a Blink
>monoculture...

As it stands, we need to remain compatible with at least GCC and
Clang, because some of our static analysis code still depends on
GCC plugins. But supporting MSVC is a much larger burden. It
tends to be the slowest compiler to adopt major new features,
and the fact that it's closed source means that we're completely
dependent on Microsoft to fix bugs which are trivial for us to
fix in our open source tool chains.

I'm all for not chaining ourselves so tightly to Clang, though.

[Gijs Kruitbosch]

Can someone elaborate on what this means for debugging on Windows, and
for our onboarding story on Windows?

I don't do this very often, but I ran into this today and the story is
already very poor. I'm currently trying to work out why a patch I have
crashes on startup. I tried using google, and found
https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/Debugging_on_Windows,
which is hopelessly out of date (like, how old is MSVS6? Why does it
talk about "Mozilla" as the executable? etc.). After struggling and
eventually finding the correct path for devenv (the executable name for
MSVS), and using `./mach run --debugger=devenv`, it opens MSVS, clicking
"Start" in it runs Firefox, but it doesn't break on
exceptions/segfaults/whatever, and so it just stops running again
shortly afterwards, with cryptic console output indicating a non-0 exit
status. No crash stack, no minidump, nothing. This may well be a
coincidence and nothing to do with clang itself, but it seems worth
pointing out.

It also seems worth bearing in mind that despite the vast majority of
our users being on Windows, the developing experience on Windows is
basically horrible (when compared to Linux/Mac). We're already making
people install MSVS to get the relevant Windows SDKs (manually, not
supported via ./mach bootstrap, and hopefully ticking the right boxes in
the installer or they have to do it again until they do win at
checkbox-golfing), and now we're telling them that although we just made
them download multiple gigs of stuff and install a pile of MS C++
compiler infrastructure on their machine, we can't actually use that and
they need to download *another* C++ compiler to actually build/debug
Firefox? That seems pretty terrible in terms of getting/encouraging
contributions from people that don't work on Firefox fulltime.

~ Gijs

[Nathan Froyd]

On Thu, Dec 6, 2018 at 6:10 PM Gijs Kruitbosch <gijskru...@gmail.com> wrote:
> Can someone elaborate on what this means for debugging on Windows, and
> for our onboarding story on Windows?

At least in terms of stepping through, examining variables, etc.,
clang-cl is on par with MSVC. If there are specific, stop-the-presses
cases that MSVC handles better than clang-cl...that's part of this
thread's reason for existence: for people to speak up about issues.

I can't speak to your debugging experience today, though; perhaps
somebody with more experience on Windows can chime in. And the docs
should be modernized, as you note. e10s and the relative inability of
debuggers to handle multi-process debugging well means the debugging
experience has gotten worse everywhere, and it would be worth thinking
about ways that we could address e10s issues as well.

> We're already making
> people install MSVS to get the relevant Windows SDKs (manually, not
> supported via ./mach bootstrap, and hopefully ticking the right boxes in
> the installer or they have to do it again until they do win at
> checkbox-golfing), and now we're telling them that although we just made
> them download multiple gigs of stuff and install a pile of MS C++
> compiler infrastructure on their machine, we can't actually use that and
> they need to download *another* C++ compiler to actually build/debug
> Firefox?

clang-cl is installed as part of `mach boostrap`, and configure will
automatically find clang-cl in the location bootstrap places it,
without any fuss on the user's part.

-Nathan

[Honza Bambas]

On 2018-12-07 04:23, Nathan Froyd wrote:
> On Thu, Dec 6, 2018 at 6:10 PM Gijs Kruitbosch <gijskru...@gmail.com> wrote:

>> Can someone elaborate on what this means for debugging on Windows, and
>> for our onboarding story on Windows?

> At least in terms of stepping through, examining variables, etc.,
> clang-cl is on par with MSVC. If there are specific, stop-the-presses
> cases that MSVC handles better than clang-cl...that's part of this
> thread's reason for existence: for people to speak up about issues.
>
> I can't speak to your debugging experience today, though; perhaps
> somebody with more experience on Windows can chime in. And the docs
> should be modernized, as you note. e10s and the relative inability of
> debuggers to handle multi-process debugging well means the debugging
> experience has gotten worse everywhere, and it would be worth thinking
> about ways that we could address e10s issues as well.

So far, debugging clang builds with MSVS (2017 community) works pretty
well with the patch for bug 1490743 applied.

I have mozilla.sln file of my own making with my own created projects
and one "startup" mozilla.vcproj that has the final firefox.exe as a run
path (with few args like -profile c:\testing-profile -no-remote).  When
you F5 in VS then it will start and automatically attach.

You can use the solution/project files we generate at build time with
mach.  I believe it's possible to run the final exe the same simple way
using the generated solution.

To automatically attach to child processes I use following MSVS extensions:
- Microsoft Child Process Debugging Power Tool (needs to be manually
enabled after installation)
- when there are issues (not all processes attached, e.g. after `build
binaries` and not a full build, as I've recently figured) I enable also
Spawned Process Catcher X (needs to be repacked for installation in
VS2017, I can provide a link)

That's it.  MSVS is a good editor and debugger with a relatively well
working intelli-sense and symbol search, text search, refactoring.

Hope this helps.

-hb-

>
>> We're already making
>> people install MSVS to get the relevant Windows SDKs (manually, not
>> supported via ./mach bootstrap, and hopefully ticking the right boxes in
>> the installer or they have to do it again until they do win at
>> checkbox-golfing), and now we're telling them that although we just made
>> them download multiple gigs of stuff and install a pile of MS C++
>> compiler infrastructure on their machine, we can't actually use that and
>> they need to download *another* C++ compiler to actually build/debug
>> Firefox?

> clang-cl is installed as part of `mach boostrap`, and configure will
> automatically find clang-cl in the location bootstrap places it,
> without any fuss on the user's part.
>
> -Nathan

[Simon Sapin]

On 07/12/2018 00:08, Gijs Kruitbosch wrote:
> We're already making people install MSVS to get the relevant Windows
> SDKs (manually, not supported via ./mach bootstrap, and hopefully
> ticking the right boxes in the installer or they have to do it again
> until they do win at checkbox-golfing)

We should fix that.

Microsoft provides "Visual Studio Build Tools" which allows installing
components such as those needed to build Firefox, without installing the
whole IDE.

It can be configured with the same command line arguments as "full" MSVC
[1], in particular to select the components (checkboxes) and/or to run
unattended (see "--passive" or "--quiet"). Component IDs are documented
at [2].

I’ve used this for unattended install in a system image used for Servo
CI [3]. Could `./mach bootstrap` do something similar?


[1]:
https://docs.microsoft.com/en-us/visualstudio/install/use-command-line-parameters-to-install-visual-studio?view=vs-2017

[2]:
https://docs.microsoft.com/en-us/visualstudio/install/workload-component-id-vs-build-tools?view=vs-2017

[3]:
https://github.com/servo/servo/blob/3c19cd49e/etc/taskcluster/windows/first-boot.ps1#L71-L77

--
Simon Sapin

[Gabriele Svelto]

On 06/12/18 15:39, Kris Maglione wrote:
> As it stands, we need to remain compatible with at least GCC and Clang,
> because some of our static analysis code still depends on GCC plugins.

Some Linux distros will keep building Firefox with GCC so there's going
to be at least some external users of that particular combination. MSVC
is a different story since nobody else would be using it.

Gabriele

[Ted Mielczarek]

On Fri, Dec 7, 2018, at 8:37 AM, Simon Sapin wrote:
> On 07/12/2018 00:08, Gijs Kruitbosch wrote:

> > We're already making people install MSVS to get the relevant Windows
> > SDKs (manually, not supported via ./mach bootstrap, and hopefully
> > ticking the right boxes in the installer or they have to do it again

> > until they do win at checkbox-golfing)
> We should fix that.

Agreed. We've discussed this[1] but just haven't found time to do the work.

> Microsoft provides "Visual Studio Build Tools" which allows installing
> components such as those needed to build Firefox, without installing the
> whole IDE.
>
> It can be configured with the same command line arguments as "full" MSVC
> [1], in particular to select the components (checkboxes) and/or to run
> unattended (see "--passive" or "--quiet"). Component IDs are documented
> at [2].
>
> I’ve used this for unattended install in a system image used for Servo
> CI [3]. Could `./mach bootstrap` do something similar?

Yes, I did some exploratory work around this[2] a while ago. I'm quite sure we could make this work if we get someone to spend time on it.

-Ted

1. https://bugzilla.mozilla.org/show_bug.cgi?id=1123821
2 .https://bugzilla.mozilla.org/show_bug.cgi?id=1270667#c9

[Ted Mielczarek]

On Thu, Dec 6, 2018, at 6:08 PM, Gijs Kruitbosch wrote:
> Can someone elaborate on what this means for debugging on Windows, and
> for our onboarding story on Windows?

For debugging: as others have pointed out, nothing should change. For onboarding we will continue to require users to install Visual C++ and then `mach bootstrap` will install clang-cl and other things. We can and should improve that to make getting started on Windows simpler.

> I don't do this very often, but I ran into this today and the story is
> already very poor.

This sucks, I'm sorry, but it's tangential to the discussion here. If you'd like to file some bugs on the docs and such we can obviously get those fixed. Ideally `mach run --debug` should do the right thing here. If it doesn't we should fix that.

> It also seems worth bearing in mind that despite the vast majority of
> our users being on Windows, the developing experience on Windows is
> basically horrible (when compared to Linux/Mac).

We're aware of this and would like to make more improvements here.

> We're already making
> people install MSVS to get the relevant Windows SDKs (manually, not
> supported via ./mach bootstrap, and hopefully ticking the right boxes in
> the installer or they have to do it again until they do win at
> checkbox-golfing), and now we're telling them that although we just made
> them download multiple gigs of stuff and install a pile of MS C++
> compiler infrastructure on their machine, we can't actually use that and
> they need to download *another* C++ compiler to actually build/debug
> Firefox? That seems pretty terrible in terms of getting/encouraging
> contributions from people that don't work on Firefox fulltime.

I brought this up in discussions because I agree that it feels bad to require installing a compiler that we're not going to use. Unfortunately we don't have a good alternative AFAIK. We might be able to install the Windows SDK separately via `mach bootstrap` (if Microsoft distributes some sort of standalone installer or package for it) but we will continue to require the MSVC standard library headers and import libraries so we can build Firefox against Microsoft's CRT. Microsoft does not distribute those outside of the MSVC installer. The only alternative we would have would be to use mingw-w64 and I don't know if that would be suitable for our shipped binaries.

-Ted

[Ted Mielczarek]

On Thu, Dec 6, 2018, at 3:34 PM, Jonathan Kew wrote:
> While I sympathize with the concern that "supporting more than one
> compiler is a maintenance burden", this still leaves me feeling a little
> uneasy. Ensuring that our code builds successfully with multiple
> compilers is a useful way to keep us from becoming dependent on quirks
> of a particular tool, and different compilers may provide different
> (valid, useful) diagnostics that we should not simply ignore.
>
> Our C/C++ code should (IMO) be standard and portable enough to build
> with any modern, mainstream C++ compiler; writing code for a clang
> monoculture feels wrong, a bit like building websites for a Blink
> monoculture...

I understand your point of view but I don't think that ensuring that we continue to support a closed-source commercial compiler from Microsoft purely for these reasons is something we should expend resources on. We don't support building with Intel's C compiler and that has not historically been something we've worried about.

-Ted

[Jim Mathies]

On Thursday, December 6, 2018 at 5:08:53 PM UTC-6, Gijs Kruitbosch wrote:
> Can someone elaborate on what this means for debugging on Windows, and
> for our onboarding story on Windows?

I just checked to be sure this was all working, and it is. I have VS 2017 and the 'Install Microsoft Child Process Debugging tool' installed.

1) open mozilla build console
2) get your repo and such set up
3) run mach bootstrap setup
4) export your .mozconifg location info
5) mach build
46:53.92 We know it took a while, but your build finally finished successfully!
6) mach build-backend
Generated Visual Studio solution at (obj dire path)\msvc\mozilla.sln
7) Open Visual Studio 2017
8) Open the generated mozilla sln file
9) Set firefox.exe as the default startup application
10) F5

Full source code annotations, variable info, breakpoints, everything works. This was definitely a clang build too.

Jim

[Kris Maglione]

I'm not sure how long that's going to be viable, at least for branded
builds. GCC's optimizers already generate significantly slower code than
clang's, and the disadvantage is going to get much worse when we have
cross-language inlining in our clang builds, which GCC will never be
able to support.

If they want to continue shipping GCC builds with their own branding, I
suppose that's fine, but I don't think that would be a good reason for
us to continue investing effort into supporting it.

[Kartikaya Gupta]

This is sort of tangential, but what's the linking story currently?
Are we still linking with MSVC, or with lld? I discovered recently
that the rust toolchain we use in automation on windows tries to use
link.exe for linking. I'm in the process of trying to get standalone
webrender tests running on taskcluster/windows, and to do that I had
to ensure that the worker has MSVC installed (I couldn't get it to
play nice with lld-link.exe from the clang-cl toolchain, but maybe I
didn't do it right). If we remove support for MSVC generally, are we
going to remove the MSVC tarballs from tooltool as well?

On Thu, Dec 6, 2018 at 3:00 PM Ted Mielczarek <t...@mielczarek.org> wrote:
>
> Hello,
>
> In light of the fact that we've switched to clang-cl for our Windows builds[1], we are planning to drop support for compiling Firefox with MSVC in the near future[2]. Our estimate is that this will happen sometime in Q1. Supporting more than one compiler is a maintenance burden and we've already seen developers spend considerable time getting their patches that work with clang-cl to build with MSVC. We are currently blocked by the fact that our aarch64-windows builds are still using MSVC and we are waiting on upstream clang-cl work to switch those builds to clang-cl. Once that takes place we no longer have a compelling reason to continue supporting MSVC.
>
> To preempt the question--when this happens we intend to make MSVC error in configure, and not just move MSVC to Tier 3 "patches welcome" status. Our reasoning is that Tier 3 configurations still create work: developers spend time building in those configurations, and lack of CI coverage means that when they inevitably break they waste time trying to fix things. Bugs get filed, and other developers waste time trying to help or reviewing patches to fix things. Explicitly unsupporting MSVC is the best way for us to convey the fact that developers should not be using it and we will not accept patches to fix issues that only affect MSVC.
>
> If you have specific reasons for continuing to use MSVC please let us know. If there are deficiencies in clang-cl compared to MSVC we should get them filed and fixed.
>
> Thanks,
> -Ted
>
> 1. https://bugzilla.mozilla.org/show_bug.cgi?id=1443590
> 2. https://bugzilla.mozilla.org/show_bug.cgi?id=1512504

[Ted Mielczarek]

On Mon, Dec 10, 2018, at 8:29 PM, Kartikaya Gupta wrote:
> This is sort of tangential, but what's the linking story currently?
> Are we still linking with MSVC, or with lld?

We're using lld-link for Windows builds in CI:
https://dxr.mozilla.org/mozilla-central/rev/c2593a3058afdfeaac5c990e18794ee8257afe99/browser/config/mozconfigs/win64/common-opt#23
https://dxr.mozilla.org/mozilla-central/rev/c2593a3058afdfeaac5c990e18794ee8257afe99/build/win64/mozconfig.vs-latest#3
https://dxr.mozilla.org/mozilla-central/rev/c2593a3058afdfeaac5c990e18794ee8257afe99/build/mozconfig.lld-link#8

> I discovered recently
> that the rust toolchain we use in automation on windows tries to use
> link.exe for linking.

We use binaries repackaged from Rust releases, so this is just the way rustc works on Windows. We do pass the linker we use for non-Windows platforms down to cargo in the Firefox build:
https://dxr.mozilla.org/mozilla-central/rev/c2593a3058afdfeaac5c990e18794ee8257afe99/config/rules.mk#1009

and we pass down a host linker on Win32 builds because we use a 64-bit rustc:
https://dxr.mozilla.org/mozilla-central/rev/c2593a3058afdfeaac5c990e18794ee8257afe99/build/moz.configure/rust.configure#343
https://dxr.mozilla.org/mozilla-central/rev/c2593a3058afdfeaac5c990e18794ee8257afe99/.cargo/config.in#23

> I'm in the process of trying to get standalone
> webrender tests running on taskcluster/windows, and to do that I had
> to ensure that the worker has MSVC installed (I couldn't get it to
> play nice with lld-link.exe from the clang-cl toolchain, but maybe I
> didn't do it right). If we remove support for MSVC generally, are we
> going to remove the MSVC tarballs from tooltool as well?

Due to some other reasons we will still require the MSVC tarballs for building in CI. We need the standard library headers + libs as well as the Windows SDK even when using clang-cl.

-Ted

[Kartikaya Gupta]

This makes sense, thanks!

[Mike Hommey]

Hi,

Bug 1512504 has now landed on autoland, meaning that compiling Firefox
with MSVC is now not supported anymore. MSVC is however still necessary
as a build requirement for its headers and libraries (as well as its
assembler on aarch64 and the preprocessor for midl ; and maybe a few other
things)

Mike