Windows launcher process enabled by default on Nightly

[Aaron Klotz]

Hi everybody,

Yesterday evening bug 1488554 [1] merged to mozilla-central, thus
enabling the launcher process by default on Windows Nightly builds. This
change is at the build config level.

What is the launcher process?
=============================
Simply put, on Windows builds, the launcher process is an initial
process hosted by firefox.exe that is responsible for configuring and
starting the "real" browser process. Such configurations include early
initialization of the DLL blocklist, as well as the setting of other
Windows process mitigations.

How does this affect startup perf?
==================================

A new test has been added to the xperf-e10s suite in Talos to track
regressions in total elapsed startup time of both the launcher and
browser processes. Previous tests on try have shown the perf impact to
be less than 1%, but that is on a small data set, hence the interest in
gathering more data from Nightly.

On Windows 10 devices with magnetic hard disks, users may actually see a
performance *improvement* due to an altered search path for DLL loading
that results in reduced HDD seeking.

As a developer, how does this affect me?
========================================

If you are not running Windows, you are unaffected! \o/

If you are running Windows, there are two possible issues:

1. Automation
-------------

If you are doing some kind of automation that waits for the Firefox
process to finish, you'll find that the initial Firefox process exits
sooner than the browser, appearing to your script that Firefox is
already done when in fact it is still running. This is because, by
default, the launcher process only lives long enough to stand up the
browser process and transfer foreground to the new browser's window.

In order to make the launcher wait for the browser to complete, pass the
--wait-for-browser flag on the command line. The launcher will wait for
the entire life of the browser process, and will also propagate the
browser process's exit code as its own. The launcher also accepts the
presence of the MOZ_AUTOMATION environment variable as an implicit
--wait-for-browser request.

I have already fixed up our automation to supply this parameter when
necessary, so I expect everything in Mozilla's CI system to already Just
Work.

|mach run| and |mach gtest| also automagically add --wait-for-browser
when necessary.

mozregression updates are being tracked by bug 1494398 [2].

2. Debugging the browser process
--------------------------------

If you launch Firefox via a debugger, the initial process will now be
the launcher process instead of the browser process. To debug the
browser process, here are your options:

* You can attach your debugger to the browser process once it has
started. Obviously this solution is less than ideal if you want to debug
something during startup;
* If you use WinDbg, you may start your debugging session from the
command line with the "-o" option to ensure that the debugger attaches
to all child processes. If you prefer opening your executable via the
GUI, you may select the "Debug child processes also" checkbox within the
"Open Executable" file picker.
* If you use Visual Studio, you may install the Child Process Debugging
Power Tool [3] which, once configured, will make the VS debugger
automagically attach to the launcher's child processes.

How to Help
===========

If you encounter any bugs, please file them blocking bug 1481546 [4] and
needinfo me. As always, feel free to reach out to me if you have any
questions/concerns/comments.

Thanks,

Aaron


[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1488554
[2] https://bugzilla.mozilla.org/show_bug.cgi?id=1494398
[3]
https://blogs.msdn.microsoft.com/devops/2014/11/24/introducing-the-child-process-debugging-power-tool/

[4] https://bugzilla.mozilla.org/show_bug.cgi?id=1481546

[Ted Mielczarek]

Hi Aaron,

Great work getting this landed, and thanks for doing the work to minimize the impact on everyone!

On Thu, Sep 27, 2018, at 11:19 AM, Aaron Klotz wrote:
> 2. Debugging the browser process
> --------------------------------
>
> If you launch Firefox via a debugger, the initial process will now be
> the launcher process instead of the browser process. To debug the
> browser process, here are your options:
>
> * You can attach your debugger to the browser process once it has
> started. Obviously this solution is less than ideal if you want to debug
> something during startup;

We have `MOZ_DEBUG_CHILD_PROCESS` for e10s content processes[1]. Would something similar (or maybe just supporting that?) be useful for the browser process as well?

> * If you use WinDbg, you may start your debugging session from the
> command line with the "-o" option to ensure that the debugger attaches
> to all child processes. If you prefer opening your executable via the
> GUI, you may select the "Debug child processes also" checkbox within the
> "Open Executable" file picker.
> * If you use Visual Studio, you may install the Child Process Debugging
> Power Tool [3] which, once configured, will make the VS debugger
> automagically attach to the launcher's child processes.

Can we make these work automatically via mozdebug so that `mach run --debug` works directly?

-Ted

1. https://mikeconley.ca/blog/2014/04/25/electrolysis-debugging-child-processes-of-content-for-make-benefit-glorious-browser-of-firefox/

[Honza Bambas]

On 2018-09-27 17:19, Aaron Klotz wrote:
> * If you use Visual Studio, you may install the Child Process
> Debugging Power Tool [3] which, once configured, will make the VS
> debugger automagically attach to the launcher's child processes.

I'm using "Spawned Process Catcher X" that works better, but has to be
manually repacked for VS2017, you can download it here:
https://janbambas.cz/moz/SpawnedProcessCatcher-2017.vsix

-hb-

[David Teller]

Hi Aaron,

It sounds cool, but I'm trying to understand what it means :) Do I
understand correctly that the main benefit is security?

Cheers,
David

[Aaron Klotz]

On 9/27/2018 9:47 AM, Ted Mielczarek wrote:
> We have `MOZ_DEBUG_CHILD_PROCESS` for e10s content processes[1]. Would something similar (or maybe just supporting that?) be useful for the browser process as well?

Sure, I have filed bug 1495039.

>
>> * If you use WinDbg, you may start your debugging session from the
>> command line with the "-o" option to ensure that the debugger attaches
>> to all child processes. If you prefer opening your executable via the
>> GUI, you may select the "Debug child processes also" checkbox within the
>> "Open Executable" file picker.

>> * If you use Visual Studio, you may install the Child Process Debugging
>> Power Tool [3] which, once configured, will make the VS debugger
>> automagically attach to the launcher's child processes.

> Can we make these work automatically via mozdebug so that `mach run --debug` works directly?

Sure. I have filed bug 1495040.

[Ehsan Akhgari]

On Thu, Sep 27, 2018 at 11:19 AM Aaron Klotz <akl...@mozilla.com> wrote:

> 1. Automation
> -------------
>
> If you are doing some kind of automation that waits for the Firefox
> process to finish, you'll find that the initial Firefox process exits
> sooner than the browser, appearing to your script that Firefox is
> already done when in fact it is still running. This is because, by
> default, the launcher process only lives long enough to stand up the
> browser process and transfer foreground to the new browser's window.
>
> In order to make the launcher wait for the browser to complete, pass the
> --wait-for-browser flag on the command line. The launcher will wait for
> the entire life of the browser process, and will also propagate the
> browser process's exit code as its own. The launcher also accepts the
> presence of the MOZ_AUTOMATION environment variable as an implicit
> --wait-for-browser request.
>
> I have already fixed up our automation to supply this parameter when
> necessary, so I expect everything in Mozilla's CI system to already Just
> Work.
>

Is the --wait-for-browser flag the default in headless mode, since that
mode is mostly for automation

Thanks,
--
Ehsan

[Aaron Klotz]

On 9/27/2018 1:53 PM, David Teller wrote:
> It sounds cool, but I'm trying to understand what it means :) Do I
> understand correctly that the main benefit is security?
>

In its current form, there are currently two primary security benefits
that the launcher process provides:

* When the launcher process detects that it has been started with
administrator privileges, it creates the browser process with a
non-administrative integrity level.
* When run on Windows 10 Anniversary Update or newer, we activate a
kernel mitigation that alters the dynamic linker's search path such that
it always attempts to load DLLs from the Windows system32 directory
first, before attempting other directories.

Stability benefits:

As for the stability benefits, we cannot realize most of them yet. Once
we have validated the performance and functionality of the launcher
process and the basic implementation reaches release, we will be able to
do much more interesting things on the stability front. Feel free to
reach out to me off-list if you'd like to make further inquiries about
our future plans for this.

Aaron

[Aaron Klotz]

On 9/28/2018 12:10 PM, Ehsan Akhgari wrote:
>
> Is the --wait-for-browser flag the default in headless mode, since
> that mode is mostly for automation
>
>

Good question. We already do that for --marionette, but not for
headless. I have filed bug 1495049 to add this.

Aaron

[Jean-Yves Avenard]

Hi

On 27/09/2018 5:19 PM, Aaron Klotz wrote:
> Hi everybody,
>
> Yesterday evening bug 1488554 [1] merged to mozilla-central, thus
> enabling the launcher process by default on Windows Nightly builds.
> This change is at the build config level.

Can we have something to entirely disable that new feature such as
./mach run --disable-e10s that won't make firefox spawn another process?

affects development/debugging process otherwise, not everyone use Visual
Studio nor WinDbg for debugging purposes. Having to manually attach a
process each time is rather unpleasant.

Thanks
JY

[Aaron Klotz]

On 10/1/2018 3:21 AM, Jean-Yves Avenard wrote:
> Can we have something to entirely disable that new feature such as
> ./mach run --disable-e10s that won't make firefox spawn another process?
>
> affects development/debugging process otherwise, not everyone use
> Visual Studio nor WinDbg for debugging purposes. Having to manually
> attach a process each time is rather unpleasant.

For various reasons we don't want to put escape hatches into any builds
that we ship.

For local builds, if it would ease developer concerns over this feature,
we can look into it. I have filed bug 1495628 for that purpose.

[Aaron Klotz]

On 9/28/2018 12:07 PM, Aaron Klotz wrote:
>
> On 9/27/2018 9:47 AM, Ted Mielczarek wrote:
>> We have `MOZ_DEBUG_CHILD_PROCESS` for e10s content processes[1].
>> Would something similar (or maybe just supporting that?) be useful
>> for the browser process as well?
> Sure, I have filed bug 1495039.
>

The patch for this bug is now on autoland and should be merging to
central shortly. The MOZ_DEBUG_BROWSER_PROCESS and
MOZ_DEBUG_BROWSER_PAUSE environment variables are analogous to the
MOZ_DEBUG_CHILD_* equivalents.

[Jean-Yves Avenard]

Hi

On 2/10/2018 5:05 AM, Aaron Klotz wrote:
>
> For various reasons we don't want to put escape hatches into any
> builds that we ship.
>
> For local builds, if it would ease developer concerns over this
> feature, we can look into it. I have filed bug 1495628 for that purpose.

Seems that we can build with --disable-launcher-process to get around
the issue. That seems good enough to me (though a dynamic pref/option
would do better of course)