Lack of browser mochitests in non-e10s configuration and support for turning off e10s on desktop going forward

[Gijs Kruitbosch]

Hello,

Today it came to my attention that there are no 1proc (non-e10s) browser
mochitests running anymore.

It appears they were disabled in
https://bugzilla.mozilla.org/show_bug.cgi?id=1433276 in early 2018,
which is somewhat odd because it looks like the bug talks about linux32,
but removed the win7 non-e10s browser chrome tests. At the time,
linux64-jsdcov was still non-e10s, but that was changed in
https://bugzilla.mozilla.org/show_bug.cgi?id=1451849, a bit later in 2018 .

This was a surprise to me because there is still a bunch of in-tree
desktop browser frontend code that is supposed to work if e10s is turned
off using the relevant pref. But we apparently have no automated test
coverage for this [so one assumes that some of it does not, in fact,
work anymore...]. The last discussion about e10s test support that I'm
aware of dates back to 2017. I do not recall there being public
discussion about turning off these tests when it did happen (though of
course, being human, it's possible I missed or forgot about it).

I'm aware we still turn off e10s on esr60 in some circumstances, and
that on other channels the hidden pref can be used to do the same.

Open questions that I'd like to ask:
1. do we still consider running desktop Firefox with e10s disabled a
supported configuration?
2. Will we need to turn it off on esr68 in the same circumstances where
that happens on esr60?
3. If the answer to either of the previous 2 questions is 'yes', do we
think it's acceptable not to run desktop tests on the configuration?
4. If the answer to both (1) and (2) is 'no', can we remove support for
the pref and running desktop Firefox without e10s ? Some of the
codepaths are not unified and so there is effectively dead code that
we're lugging around for what would be no reason. (Note: a significant
proportion isn't dead because even in e10s, we load some
browser-provided content in parent process, ie a tab's browser is not
always remote/non-same-process -- but even so, there's a bunch of stuff
that keys off gMultiProcessBrowser that could be removed.)

Thanks,
Gijs

[Bobby Holley]

If we're not testing it, we shouldn't be shipping it to users. It would be
great if someone familiar with the esr60 situation could confirm that we
don't plan to repeat it for esr68. It would also be great if someone could
explain the rationale for running some, but not all of the suites in 1proc
mode.

Separately, I know some engineers disable e10s locally as a hack to
simplify debugging (e.g [1]). The 1proc mochitest+xpcshell+reftest jobs
currently on automation are probably sufficient to continue support for
this use-case, but if we turn those off, we should consider this workflow
and how much we're willing to do to preserve it.

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1530977#c0

On Wed, Apr 17, 2019 at 5:40 AM Gijs Kruitbosch <gijskru...@gmail.com>
wrote:

> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>

[Joel Maher]

Here is where we initially turned on non-e10s tests for win7:
https://bugzilla.mozilla.org/show_bug.cgi?id=1391371

and then moved to linux32:
https://bugzilla.mozilla.org/show_bug.cgi?id=1433276

Currently mochitest-chrome and mochitest-a11y run as 1proc in-tree- these
run this way as they do not work with e10s=true. I suspect the
mochitest-chrome is by design and a11y is a bug.

-Joel

On Thu, Apr 18, 2019 at 5:47 PM Bobby Holley <bobby...@gmail.com> wrote:

> If we're not testing it, we shouldn't be shipping it to users. It would be
> great if someone familiar with the esr60 situation could confirm that we
> don't plan to repeat it for esr68. It would also be great if someone could
> explain the rationale for running some, but not all of the suites in 1proc
> mode.
>
> Separately, I know some engineers disable e10s locally as a hack to
> simplify debugging (e.g [1]). The 1proc mochitest+xpcshell+reftest jobs
> currently on automation are probably sufficient to continue support for
> this use-case, but if we turn those off, we should consider this workflow
> and how much we're willing to do to preserve it.
>
> [1] https://bugzilla.mozilla.org/show_bug.cgi?id=1530977#c0
>
> On Wed, Apr 17, 2019 at 5:40 AM Gijs Kruitbosch <gijskru...@gmail.com>
> wrote:
>

[Dave Townsend]

Is there still a configuration (ignoring hidden prefs) that can cause a
user to end up using non-e10s? If so we should turn these tests back on.

[Mike Conley]

Hey folks,

For Desktop, I don't believe there are any normal conditions under which
our users will have e10s disabled. [1] is where the decision gets made, and
it looks like these days, the only thing that will disable e10s is if the
user sets `browser.tabs.remote.autostart` to false themselves, or if a
MOZ_FORCE_DISABLE_E10S environment variable is set. I don't think either of
those are ever set by Mozilla these days.

There was a case a few months back where e10s was disabled for users with
certain screen readers back for Firefox 60. Since that time, those screen
readers have updated to become more stable with e10s enabled.
Unfortunately, I don't have a reference to the bug(s) where that occurred,
but I spoke to yzen in #accessibility, and Firefox 60 ESR is the last
supported version where this e10s-disabling occurs on Desktop.

So, to sum, I'm reasonably confident that, outside of Firefox 60 ESR,
e10s-disabled is not a mode that we ship to any of our users. We can
trigger it by pref flips or environment variables, but that's it.

Mobile is another story - according to the fine folks in #mobile, Fennec
still runs Gecko in non-e10s mode.

To circle back to Gijs's questions:

1. do we still consider running desktop Firefox with e10s disabled a
> supported configuration?
>

Outside of Firefox 60 ESR, no, I don't believe so.

2. Will we need to turn it off on esr68 in the same circumstances where
> that happens on esr60?
>

According to yzen from the Accessibility team, no, we won't.

3. If the answer to either of the previous 2 questions is 'yes', do we
> think it's acceptable not to run desktop tests on the configuration?
>

Answers are no.

4. If the answer to both (1) and (2) is 'no', can we remove support for
> the pref and running desktop Firefox without e10s ? Some of the
> codepaths are not unified and so there is effectively dead code that
> we're lugging around for what would be no reason. (Note: a significant
> proportion isn't dead because even in e10s, we load some
> browser-provided content in parent process, ie a tab's browser is not
> always remote/non-same-process -- but even so, there's a bunch of stuff
> that keys off gMultiProcessBrowser that could be removed.)
>

Yes, I believe that stuff is probably safe to remove at this point, as long
those changes don't assume e10s support on Fennec.

-Mike

[1]:
https://searchfox.org/mozilla-central/rev/ec489aa170b6486891cf3625717d6fa12bcd11c1/toolkit/xre/nsAppRunner.cpp#4964-5002

[Mike Conley]

Just a quick follow-up -

yzen got back to me - here's the bug that tracked disabling e10s in Firefox
60 for older screen readers:

https://bugzilla.mozilla.org/show_bug.cgi?id=1489605

On Tue, 23 Apr 2019 at 16:30, Mike Conley <mco...@mozilla.com> wrote:

> Hey folks,
>
> For Desktop, I don't believe there are any normal conditions under which
> our users will have e10s disabled. [1] is where the decision gets made, and
> it looks like these days, the only thing that will disable e10s is if the
> user sets `browser.tabs.remote.autostart` to false themselves, or if a
> MOZ_FORCE_DISABLE_E10S environment variable is set. I don't think either of
> those are ever set by Mozilla these days.
>
> There was a case a few months back where e10s was disabled for users with
> certain screen readers back for Firefox 60. Since that time, those screen
> readers have updated to become more stable with e10s enabled.
> Unfortunately, I don't have a reference to the bug(s) where that occurred,
> but I spoke to yzen in #accessibility, and Firefox 60 ESR is the last
> supported version where this e10s-disabling occurs on Desktop.
>
> So, to sum, I'm reasonably confident that, outside of Firefox 60 ESR,
> e10s-disabled is not a mode that we ship to any of our users. We can
> trigger it by pref flips or environment variables, but that's it.
>
> Mobile is another story - according to the fine folks in #mobile, Fennec
> still runs Gecko in non-e10s mode.
>
> To circle back to Gijs's questions:
>

> 1. do we still consider running desktop Firefox with e10s disabled a
>> supported configuration?
>>
>

> Outside of Firefox 60 ESR, no, I don't believe so.
>

> 2. Will we need to turn it off on esr68 in the same circumstances where
>> that happens on esr60?
>>
>

> According to yzen from the Accessibility team, no, we won't.
>

> 3. If the answer to either of the previous 2 questions is 'yes', do we
>> think it's acceptable not to run desktop tests on the configuration?
>>
>

> Answers are no.

>
> 4. If the answer to both (1) and (2) is 'no', can we remove support for
>> the pref and running desktop Firefox without e10s ? Some of the
>> codepaths are not unified and so there is effectively dead code that
>> we're lugging around for what would be no reason. (Note: a significant
>> proportion isn't dead because even in e10s, we load some
>> browser-provided content in parent process, ie a tab's browser is not
>> always remote/non-same-process -- but even so, there's a bunch of stuff
>> that keys off gMultiProcessBrowser that could be removed.)
>>
>

[Bobby Holley]

Thanks Mike!

So Fennec is the last remaining non-e10s configuration we ship to users.
Given that Fennec test coverage is somewhat incomplete, we probably want to
keep running desktop 1proc tests until Fennec EOL. We don't strictly need
the browser-chrome tests, but we should probably avoid intentionally
breaking non-e10s browser-chrome as long as engineers may still need to
debug non-e10s test failures.

After Fennec EOL, we basically have two options: a clean break where we rip
out non-e10s support entirely, or a more muddled approach where we
halfheartedly keep it working as a handy engineering hack as long as is
practical. I think we should do the former.

I don't want to downplay the handiness - being able to test and debug the
browser in a single process can eliminate complexity and non-determinism,
and save a lot of time. But at some point there's going to be a piece of
core functionality that's too much work to keep supporting under non-e10s -
and agonizing over that threshold on an ongoing basis is not a good use of
anyone's time.

Other opinions?

On Tue, Apr 23, 2019 at 1:30 PM Mike Conley <mco...@mozilla.com> wrote:

> Hey folks,
>
> For Desktop, I don't believe there are any normal conditions under which
> our users will have e10s disabled. [1] is where the decision gets made, and
> it looks like these days, the only thing that will disable e10s is if the
> user sets `browser.tabs.remote.autostart` to false themselves, or if a
> MOZ_FORCE_DISABLE_E10S environment variable is set. I don't think either of
> those are ever set by Mozilla these days.
>
> There was a case a few months back where e10s was disabled for users with
> certain screen readers back for Firefox 60. Since that time, those screen
> readers have updated to become more stable with e10s enabled.
> Unfortunately, I don't have a reference to the bug(s) where that occurred,
> but I spoke to yzen in #accessibility, and Firefox 60 ESR is the last
> supported version where this e10s-disabling occurs on Desktop.
>
> So, to sum, I'm reasonably confident that, outside of Firefox 60 ESR,
> e10s-disabled is not a mode that we ship to any of our users. We can
> trigger it by pref flips or environment variables, but that's it.
>
> Mobile is another story - according to the fine folks in #mobile, Fennec
> still runs Gecko in non-e10s mode.
>
> To circle back to Gijs's questions:
>

> 1. do we still consider running desktop Firefox with e10s disabled a
>> supported configuration?
>>
>

> Outside of Firefox 60 ESR, no, I don't believe so.
>

> 2. Will we need to turn it off on esr68 in the same circumstances where
>> that happens on esr60?
>>
>

> According to yzen from the Accessibility team, no, we won't.
>

> 3. If the answer to either of the previous 2 questions is 'yes', do we
>> think it's acceptable not to run desktop tests on the configuration?
>>
>

> Answers are no.

>
> 4. If the answer to both (1) and (2) is 'no', can we remove support for
>> the pref and running desktop Firefox without e10s ? Some of the
>> codepaths are not unified and so there is effectively dead code that
>> we're lugging around for what would be no reason. (Note: a significant
>> proportion isn't dead because even in e10s, we load some
>> browser-provided content in parent process, ie a tab's browser is not
>> always remote/non-same-process -- but even so, there's a bunch of stuff
>> that keys off gMultiProcessBrowser that could be removed.)
>>
>

[David Major]

On Wed, Apr 24, 2019 at 1:39 PM Bobby Holley <bobby...@gmail.com> wrote:
>
> Thanks Mike!
>
> So Fennec is the last remaining non-e10s configuration we ship to users.
> Given that Fennec test coverage is somewhat incomplete, we probably want to
> keep running desktop 1proc tests until Fennec EOL. We don't strictly need
> the browser-chrome tests, but we should probably avoid intentionally
> breaking non-e10s browser-chrome as long as engineers may still need to
> debug non-e10s test failures.
>
> After Fennec EOL, we basically have two options: a clean break where we rip
> out non-e10s support entirely, or a more muddled approach where we
> halfheartedly keep it working as a handy engineering hack as long as is
> practical. I think we should do the former.
>
> I don't want to downplay the handiness - being able to test and debug the
> browser in a single process can eliminate complexity and non-determinism,
> and save a lot of time. But at some point there's going to be a piece of
> core functionality that's too much work to keep supporting under non-e10s -
> and agonizing over that threshold on an ongoing basis is not a good use of
> anyone's time.

Why not have the conversation when such a piece of core functionality
arises? It's a much more convincing argument when you can say
"non-e10s needs to go away in order to support X".

In the meantime, single process debugging is a tremendous saver of
time and hassle, and we'd need a great reason to disable it. I'm not
convinced that one currently exists.

> Other opinions?
>
> On Tue, Apr 23, 2019 at 1:30 PM Mike Conley <mco...@mozilla.com> wrote:
>
> > Hey folks,
> >
> > For Desktop, I don't believe there are any normal conditions under which
> > our users will have e10s disabled. [1] is where the decision gets made, and
> > it looks like these days, the only thing that will disable e10s is if the
> > user sets `browser.tabs.remote.autostart` to false themselves, or if a
> > MOZ_FORCE_DISABLE_E10S environment variable is set. I don't think either of
> > those are ever set by Mozilla these days.
> >
> > There was a case a few months back where e10s was disabled for users with
> > certain screen readers back for Firefox 60. Since that time, those screen
> > readers have updated to become more stable with e10s enabled.
> > Unfortunately, I don't have a reference to the bug(s) where that occurred,
> > but I spoke to yzen in #accessibility, and Firefox 60 ESR is the last
> > supported version where this e10s-disabling occurs on Desktop.
> >
> > So, to sum, I'm reasonably confident that, outside of Firefox 60 ESR,
> > e10s-disabled is not a mode that we ship to any of our users. We can
> > trigger it by pref flips or environment variables, but that's it.
> >
> > Mobile is another story - according to the fine folks in #mobile, Fennec
> > still runs Gecko in non-e10s mode.
> >
> > To circle back to Gijs's questions:
> >

> > 1. do we still consider running desktop Firefox with e10s disabled a
> >> supported configuration?
> >>
> >

> > Outside of Firefox 60 ESR, no, I don't believe so.
> >

> > 2. Will we need to turn it off on esr68 in the same circumstances where
> >> that happens on esr60?
> >>
> >

> > According to yzen from the Accessibility team, no, we won't.
> >

> > 3. If the answer to either of the previous 2 questions is 'yes', do we
> >> think it's acceptable not to run desktop tests on the configuration?
> >>
> >

> > Answers are no.

> >
> > 4. If the answer to both (1) and (2) is 'no', can we remove support for
> >> the pref and running desktop Firefox without e10s ? Some of the
> >> codepaths are not unified and so there is effectively dead code that
> >> we're lugging around for what would be no reason. (Note: a significant
> >> proportion isn't dead because even in e10s, we load some
> >> browser-provided content in parent process, ie a tab's browser is not
> >> always remote/non-same-process -- but even so, there's a bunch of stuff
> >> that keys off gMultiProcessBrowser that could be removed.)
> >>
> >

[Bobby Holley]

I'm concerned about the impact of a gradual degradation of the e10s
experience and an ambiguous bar as to exactly how much work developers are
expected to do to support it. It's effectively going to be a Tier-3
platform with no owner.

>
> In the meantime, single process debugging is a tremendous saver of
> time and hassle, and we'd need a great reason to disable it. I'm not
> convinced that one currently exists.
>

I think the tradeoff boils down to (a) how many developers are using
non-e10s debugging, with what frequency, versus (b) how much ongoing
maintenance work is required across various components to keep non-e10s
working. We all have intuition about these things, but I doubt we have hard
data.

I'm open to the argument that my proposal is too aggressive. We could
certainly try the muddle approach for a while and see how it goes, and how
often 1proc breaks in practice. I don't think we can justify continuing to
run the full suite of 1proc tests as tier-1, but we could potentially run a
few smoketests, which might keep the builds usable enough for debugging.

If anyone is chomping at the bit to remove 1proc support from their module,
please speak up.

> > > 1. do we still consider running desktop Firefox with e10s disabled a
> > >> supported configuration?
> > >>
> > >

> > > Outside of Firefox 60 ESR, no, I don't believe so.
> > >

> > > 2. Will we need to turn it off on esr68 in the same circumstances where
> > >> that happens on esr60?
> > >>
> > >

> > > According to yzen from the Accessibility team, no, we won't.
> > >

> > > 3. If the answer to either of the previous 2 questions is 'yes', do we
> > >> think it's acceptable not to run desktop tests on the configuration?
> > >>
> > >

> > > Answers are no.

> > >
> > > 4. If the answer to both (1) and (2) is 'no', can we remove support for
> > >> the pref and running desktop Firefox without e10s ? Some of the
> > >> codepaths are not unified and so there is effectively dead code that
> > >> we're lugging around for what would be no reason. (Note: a significant
> > >> proportion isn't dead because even in e10s, we load some
> > >> browser-provided content in parent process, ie a tab's browser is not
> > >> always remote/non-same-process -- but even so, there's a bunch of
> stuff
> > >> that keys off gMultiProcessBrowser that could be removed.)
> > >>
> > >

[Mike Hommey]

I'm not versed on the details of non-e10s, but is it so different from
e10s that it could easily break? Or are we more concerned, long term,
that non-e10s would block us from making some architectural changes to
e10s?

Relatedly, do we have a history of having broken non-e10s with e10s
changes?

Mike

[Bobby Holley]

Probably not "so different" in relative terms, since we try to abstract
over the distinction wherever possible. But browsers are big and
abstractions are rarely perfect, so there's a long tail of special-cases
that do different things depending on the process model. If we're not
testing it, it will eventually break.

> Or are we more concerned, long term,
> that non-e10s would block us from making some architectural changes to
> e10s?
>

If a big architectural change gets blocked (which will probably happen
eventually), that will simplify the discussion. So I think the question
here is how much fiddling we should do to keep it working in the interim.

> Relatedly, do we have a history of having broken non-e10s with e10s
> changes?
>

I doubt there's a comprehensive list, but bug 1530977 is the example that
recently came across my desk. That bug is specific to hardware acceleration
on Windows, which is why it wasn't caught by the linux 1proc tests. Once we
those off, the regression surface will widen from platform-specific code to
all of Gecko. I suspect that will result in an increased rate in
regressions, but it's hard to guess exactly how much.


>
> Mike
>

[Jean-Yves Avenard]

> On 25 Apr 2019, at 8:49 am, Bobby Holley <bobby...@gmail.com> wrote.

>>
>
> I think the tradeoff boils down to (a) how many developers are using
> non-e10s debugging, with what frequency, versus (b) how much ongoing
> maintenance work is required across various components to keep non-e10s
> working. We all have intuition about these things, but I doubt we have hard
> data.
>
> I'm open to the argument that my proposal is too aggressive. We could
> certainly try the muddle approach for a while and see how it goes, and how
> often 1proc breaks in practice. I don't think we can justify continuing to
> run the full suite of 1proc tests as tier-1, but we could potentially run a
> few smoketests, which might keep the builds usable enough for debugging.
>
> If anyone is chomping at the bit to remove 1proc support from their module,
> please speak up.

I am one of those developers that find non-e10s essential to debug core features.

Debugging in e10s adds a big overhead in the effort required to determine on the why something isn't working as it should.

When debugging HW specialised features (for media those are running in the GPU process), in e10s you always hit timeouts that cause a cascade of failures elsewhere.

Of course we can do without, but at the expense of time and convenience. Before non-e10s support is removed, I'd love to see better development/debugging tools, particularly on Windows added to help our workflow.

I knew this day would come, tbh I'm surprise non-e10s has lived for so long. However, I don't believe my workflow is unique to whomever is working on gecko.

Jean-Yves

[Gijs Kruitbosch]

On 24/04/2019 18:39, Bobby Holley wrote:
> Thanks Mike!
>
> So Fennec is the last remaining non-e10s configuration we ship to users.

Well, or anything where a user (or malicious actor) has turned off e10s
using the pref which is now hidden (but wasn't in the past, and for
which we actively prompted older-version-of-JAWS users on esr60, AIUI).
We used to have telemetry for this (E10S_STATUS) but AFAICT we do not
anymore, so we don't know how big this user group is.

> Given that Fennec test coverage is somewhat incomplete, we probably want to
> keep running desktop 1proc tests until Fennec EOL. We don't strictly need
> the browser-chrome tests, but we should probably avoid intentionally
> breaking non-e10s browser-chrome as long as engineers may still need to
> debug non-e10s test failures.

I've still been too busy to check by actually trying it, but I expect
that if we ran the full suite of browser chrome tests in non-e10s today
there would be substantial amounts of orange.

mochitest-chrome (not to be confused with mochitest-browser-chrome) and
mochitest-plain and mochitest-a11y continuing to run in non-e10s should
be sufficient to prevent completely breaking debugging workflows. If the
browser doesn't start or content doesn't load, those tests will break,
so we won't be able to break things completely.

However, I think even before fennec EOL, it would be a good idea to
remove the desktop pref (to avoid people unwittingly running unsandboxed
Firefox to connect to the web), moving all those users to e10s. We can
keep the debugging case working using the env var and the extant ./mach
and test switches. I would also intend to slowly remove all the
Firefox-desktop-frontend switches and other dead code that cater for
this usecase, inasmuch as doing so wouldn't break the debugging / other
tests.

Ideally, I think the debugging usecase would be better served by a more
barebones "here's a docshell that will load arbitrary content off our
trusted localhost server in 1 process" type app than by making desktop
firefox run these tests, but perhaps it's not worth our time to make
such a thing...

Does removal of the pref for 68 (assuming we keep the tests + mach
switches working) sound OK to folks? If so, I will file a bug and get
that work going so we remove this footgun in time for this ESR round.

Gijs

[Cameron McCormack]

On Thu, Apr 25, 2019, at 7:55 PM, Gijs Kruitbosch wrote:
> Ideally, I think the debugging usecase would be better served by a more
> barebones "here's a docshell that will load arbitrary content off our
> trusted localhost server in 1 process" type app than by making desktop
> firefox run these tests, but perhaps it's not worth our time to make
> such a thing...

We have `mach run -layoutdebug <url>` which will load a document from any host in the content process. I'm not sure how easy it would be to be able to load mochitests in this environment (although you could probably kick off a real mochitest single test run, leave it open, then run the layout debugger pointing to mochi.test), but for most of my single process debugging needs the layout debugger works well.

[Joel Maher]

On Wed, Apr 24, 2019 at 1:39 PM Bobby Holley <bobby...@gmail.com> wrote:

> >
>> > Thanks Mike!
>> >
>> > So Fennec is the last remaining non-e10s configuration we ship to users.

>> > Given that Fennec test coverage is somewhat incomplete, we probably
>> want to
>> > keep running desktop 1proc tests until Fennec EOL.
>>
>

Fennec runs the full set of tests, there is no need to run non-e10s tests
on desktop to support Fennec.

>From reading this thread we are keeping a lot of code around and tests
around to support local debugging. As we already run mochitest-chrome on
all platforms in non-e10s, is this not enough coverage to ensure everything
works for debugging? If not, could we add a simple test to
mochitest-chrome in order to ensure our local debugging scenarios work as
needed?

[Bobby Holley]

On Thu, Apr 25, 2019 at 3:36 AM Joel Maher <jma...@mozilla.com> wrote:

>
>
> On Wed, Apr 24, 2019 at 1:39 PM Bobby Holley <bobby...@gmail.com>
> wrote:
>
>> >
>>> > Thanks Mike!
>>> >
>>> > So Fennec is the last remaining non-e10s configuration we ship to
>>> users.
>>> > Given that Fennec test coverage is somewhat incomplete, we probably
>>> want to
>>> > keep running desktop 1proc tests until Fennec EOL.
>>>
>>
> Fennec runs the full set of tests, there is no need to run non-e10s tests
> on desktop to support Fennec.
>

I had the impression that we had a fair number of tests disabled under
Fennec, but maybe not. In any case - insofar as non-e10s is a supported
platform, it's useful to be able to hit those failures directly on Desktop
tests rather than armv7 emulators - so there's still value in keeping 1proc
enabled until Fennec EOL.

> From reading this thread we are keeping a lot of code around and tests
> around to support local debugging. As we already run mochitest-chrome on
> all platforms in non-e10s, is this not enough coverage to ensure everything
> works for debugging? If not, could we add a simple test to
> mochitest-chrome in order to ensure our local debugging scenarios work as
> needed?
>

Yeah, I agree that running a full 1proc suite in perpetuity is too
expensive, both in machine time and maintenance. I think a more limited
subset of smoketests is probably the way to go.

[James Graham]

On 25/04/2019 17:12, Bobby Holley wrote:
> On Thu, Apr 25, 2019 at 3:36 AM Joel Maher <jma...@mozilla.com> wrote:
>
>>
>>
>> On Wed, Apr 24, 2019 at 1:39 PM Bobby Holley <bobby...@gmail.com>
>> wrote:
>>
>>>>
>>>>> Thanks Mike!
>>>>>
>>>>> So Fennec is the last remaining non-e10s configuration we ship to
>>>> users.
>>>>> Given that Fennec test coverage is somewhat incomplete, we probably
>>>> want to
>>>>> keep running desktop 1proc tests until Fennec EOL.
>>>>
>>>
>> Fennec runs the full set of tests, there is no need to run non-e10s tests
>> on desktop to support Fennec.
>>
>
> I had the impression that we had a fair number of tests disabled under
> Fennec, but maybe not. In any case - insofar as non-e10s is a supported
> platform, it's useful to be able to hit those failures directly on Desktop
> tests rather than armv7 emulators - so there's still value in keeping 1proc
> enabled until Fennec EOL.

Note that Android testing has been moving to x86[_64]. I don't know if
moving more of the tests to that platform would make a meaningful
difference to this decision (or indeed how practical it is to do so),
but I think the performance improvement is substantial, which might
solve some of the issue with local debugging.

[Joel Maher]

On Thu, Apr 25, 2019 at 12:12 PM Bobby Holley <bobby...@gmail.com> wrote:

> On Thu, Apr 25, 2019 at 3:36 AM Joel Maher <jma...@mozilla.com> wrote:
>
>>
>>
>> On Wed, Apr 24, 2019 at 1:39 PM Bobby Holley <bobby...@gmail.com>
>> wrote:
>>
>>> >
>>>> > Thanks Mike!
>>>> >
>>>> > So Fennec is the last remaining non-e10s configuration we ship to
>>>> users.
>>>> > Given that Fennec test coverage is somewhat incomplete, we probably
>>>> want to
>>>> > keep running desktop 1proc tests until Fennec EOL.
>>>>
>>>
>> Fennec runs the full set of tests, there is no need to run non-e10s tests
>> on desktop to support Fennec.
>>
>
> I had the impression that we had a fair number of tests disabled under
> Fennec, but maybe not. In any case - insofar as non-e10s is a supported
> platform, it's useful to be able to hit those failures directly on Desktop
> tests rather than armv7 emulators - so there's still value in keeping 1proc
> enabled until Fennec EOL.
>
>

While tests are disabled under Fennec most have been reenabled if possible-
many tests are not relevant on Fennec. Desktop is much faster than those
android emulators. When we switch to Firefox 69 (May 13th) the plan is to
leave android 4.3 emulators around as tier-2 (mozilla-central/try only)
until we get all the tests ported over to the new android 7.0 x86_64
emulators which are just as fast as desktop, run geckoview, and are multi
process. We are about halfway done with getting tests running on there.
For test suites that are running successfully on the new emulators, we will
turn them off on android 4.3, the plan is probably early Q3 to be done.
Fennec will ship off the 68 branch as that will live on ESR for a while.

[Bobby Holley]

That all sounds sensible to me.

As long as we're certain that we won't ship Fennec past ESR68, 1proc will
cease to be a user-accessible configuration once Nightly moves to 69 in 2.5
weeks.

Once that happens, I propose the following steps, in order:
(1) Remove support for the e10s pref, per Gijs' suggestion above.
(2) Define and launch a small "1proc-smoketest" job, based on
mochitest-plain, with a handful of tests across various components to
verify that things mostly work.
(3) Turn off the existing 1proc jobs.

Does this seem like a reasonable path forward?

[Joel Maher]

this sounds reasonable. We don't have to do things exactly on May 13th, if
it is a week or two later, that seems reasonable to me.

mochitest-a11y and mochitest-chrome will continue to run as 1proc, and then
sometime next month the new mochitest-1proc suite we will define.

Glad to have a plan, if others disagree or have other thoughts, it would be
good to know.

[Botond Ballo]

On Thu, Apr 25, 2019 at 4:58 PM Bobby Holley <bobby...@gmail.com> wrote:
> we won't ship Fennec past ESR68

That's news to me. Was this announced somewhere? Is there a discussion
you could point me to?

Thanks,
Botond

[Joel Maher]

This was announced last night:
https://groups.google.com/forum/#!topic/mozilla.dev.platform/k-irJtmCcqg

[Gijs Kruitbosch]

On 26/04/2019 18:05, Joel Maher wrote:
> This was announced last night:
> https://groups.google.com/forum/#!topic/mozilla.dev.platform/k-irJtmCcqg

This seems to suggest we shouldn't make fennec-related changes until the
71 cycle and that we're not 100% certain that we won't ship it past esr68.

On 25/04/2019 21:57, Bobby Holley wrote:
> As long as we're certain that we won't ship Fennec past ESR68, 1proc will
> cease to be a user-accessible configuration once Nightly moves to 69
in 2.5
> weeks.
>
> Once that happens, I propose the following steps, in order:
> (1) Remove support for the e10s pref, per Gijs' suggestion above.
> (2) Define and launch a small "1proc-smoketest" job, based on
> mochitest-plain, with a handful of tests across various components to
> verify that things mostly work.
> (3) Turn off the existing 1proc jobs.
>
> Does this seem like a reasonable path forward?

So I'm not sure how this jives with the Fennec plans as per the above.

Even if it's fine to make such changes against a reduced-test-coverage
fennec on 69, I'd like to push and ask if we can reasonably switch the
enabling/disabling of e10s to an env var (and force e10s off via a build
flag for fennec) on the 68 cycle still? (We'd keep `--disable-e10s`
working on mach for developer use as it does today.)

I think from a security perspective we should avoid continuing to ship
something ("even" on esr) where users can either still (relatively)
easily manually disable e10s or get grandfathered into having it
disabled on desktop. We renamed some stuff to "allow viruses to take
over your computer" in the past - I think in 2019, turning off e10s
and/or sandboxing on desktop is similarly problematic.

Gijs

[Daniel Veditz]

On Thu, Apr 25, 2019 at 1:58 PM Bobby Holley <bobby...@gmail.com> wrote:

> As long as we're certain that we won't ship Fennec past ESR68,
>

The timeline was left vague. Ideally I assume we'd like to migrate Fennec
folks to Fenix before ESR68 EOL, but if it's not ready there's no reason we
have to stop shipping a 68-based Fennec when we stop supporting ESR68 on
Desktop. It will be increasingly difficult to support with security fixes
but we've supported branches longer than a year in the past when we've had
to. Hopefully it won't come to that.

-Dan Veditz

[Randell Jesup]

Jean-Yves writes:
>> If anyone is chomping at the bit to remove 1proc support from their module,
>> please speak up.
>
>I am one of those developers that find non-e10s essential to debug core features.

I've depended on using non-e10s for ages as well. Most of my debugging
can in theory be done in e10s, but whenever I have to have multiple GDBs
running, and deal with breaking in one causing breaks in the other (or
timeouts, etc), it's really painful, and especially for any work that
crosses the Content/master boundary. Also: how will Fission affect our
debugging workflows? I imagine that will make things much more
complicated without better tools.

>Of course we can do without, but at the expense of time and
>convenience. Before non-e10s support is removed, I'd love to see better
>development/debugging tools, particularly on Windows added to help our
>workflow.

+100 -- we need better tools for e10s debugging, especially with Fission coming.
We can get away without them, and without non-e10s - but it will be more
friction that lots of us will have on a daily basis.

[Part of the problem is that lots of debugging tools don't deal with, or
deal poorly with, multi-process apps. No revelation here, of course.]
--
Randell Jesup, Mozilla Corp
remove "news" for personal email

[Jonathan Kew]

On 01/05/2019 17:29, Randell Jesup wrote:
> Jean-Yves writes:
>>> If anyone is chomping at the bit to remove 1proc support from their module,
>>> please speak up.
>>
>> I am one of those developers that find non-e10s essential to debug core features.
>
> I've depended on using non-e10s for ages as well. Most of my debugging
> can in theory be done in e10s, but whenever I have to have multiple GDBs
> running, and deal with breaking in one causing breaks in the other (or
> timeouts, etc), it's really painful, and especially for any work that
> crosses the Content/master boundary.

Just chiming in to agree strongly here. Most bugs I investigate are in
code that doesn't care whether the browser is running in e10s or
non-e10s mode, and so I routinely use --disable-e10s to make my
debugging sessions simpler. Losing that option would hurt.

I realize the time may come when it's impractical to keep non-e10s
working, but let's not kill it until there's a compelling reason to make
the sacrifice.

JK

[Bobby Holley]

On Wed, May 1, 2019 at 10:27 AM Jonathan Kew <jfkt...@gmail.com> wrote:

> On 01/05/2019 17:29, Randell Jesup wrote:

> > Jean-Yves writes:
> >>> If anyone is chomping at the bit to remove 1proc support from their
> module,
> >>> please speak up.
> >>
> >> I am one of those developers that find non-e10s essential to debug core
> features.
> >
> > I've depended on using non-e10s for ages as well. Most of my debugging
> > can in theory be done in e10s, but whenever I have to have multiple GDBs
> > running, and deal with breaking in one causing breaks in the other (or
> > timeouts, etc), it's really painful, and especially for any work that
> > crosses the Content/master boundary.
>

> Just chiming in to agree strongly here. Most bugs I investigate are in
> code that doesn't care whether the browser is running in e10s or
> non-e10s mode, and so I routinely use --disable-e10s to make my
> debugging sessions simpler. Losing that option would hurt.
>
> I realize the time may come when it's impractical to keep non-e10s
> working, but let's not kill it until there's a compelling reason to make
> the sacrifice.
>

This thread is getting long, so here's a quick recap:
* It it well-understood that a number of developers use --disable-e10s as a
debugging aid, so more +1s to that effect aren't needed.
* The "please speak up" is aimed at developers who want to _remove_ 1proc
support.
* The compromise plan of record is to keep a small subset of 1proc tests as
tier-1. So basic functionality should continue to work, but stuff around
the edges may break.
* Alternative proposals to the above are welcome, but please be concrete
and weigh the trade-offs.
* If there's specific functionality you want to keep testing in 1proc mode,
please mention it in bug 1548035.

To add a bit more motivation for the proposed plan: Continuing to run the
entire test suite in 1proc mode costs money and developer time. The quality
bar of builds that ship to users is _much_ higher than the bar for builds
used as debugging hacks. If a single test starts failing in 1proc mode
only, the right answer is probably to turn the test off rather than spend
developer time chasing it. What we want to prevent are the sort of
across-the-board breakages that make the browser unusable for debugging -
and we can detect those with a much smaller subset of tests than what we're
presently running.

bholley



> JK

[Aaron Klotz]

On Wed, Apr 24, 2019 at 11:40 PM Jean-Yves Avenard <jyav...@mozilla.com>
wrote:

>
> Before non-e10s support is removed, I'd love to see better
> development/debugging tools, particularly on Windows added to help our
> workflow.

Was there anything in particular that you had in mind?

- Aaron

>
>

[Gijs Kruitbosch]

On 01/05/2019 19:01, Bobby Holley wrote:
> This thread is getting long, so here's a quick recap:
> * It it well-understood that a number of developers use --disable-e10s as a
> debugging aid, so more +1s to that effect aren't needed.
> * The "please speak up" is aimed at developers who want to _remove_ 1proc
> support.
> * The compromise plan of record is to keep a small subset of 1proc tests as
> tier-1. So basic functionality should continue to work, but stuff around
> the edges may break.
> * Alternative proposals to the above are welcome, but please be concrete
> and weigh the trade-offs.
> * If there's specific functionality you want to keep testing in 1proc mode,
> please mention it in bug 1548035.
>
> To add a bit more motivation for the proposed plan: Continuing to run the
> entire test suite in 1proc mode costs money and developer time. The quality
> bar of builds that ship to users is _much_ higher than the bar for builds
> used as debugging hacks. If a single test starts failing in 1proc mode
> only, the right answer is probably to turn the test off rather than spend
> developer time chasing it. What we want to prevent are the sort of
> across-the-board breakages that make the browser unusable for debugging -
> and we can detect those with a much smaller subset of tests than what we're
> presently running.

I think narrowing test coverage for 1proc on 69 is fine, but I'd like to
do something for 68 still. Specifically, I strongly feel we should not
continue to support the "turn off e10s with a pref" scenario when it's
so loosely tested - as you said, the bar for builds that ship to users
is much higher. On nightly, about:newtab/home (activity stream) is
completely broken and the bug to fix it was marked wontfix due to us not
supporting e10s (this bustage relates to the privileged content process
which is held to nightly, so won't ride the trains with 68... but no
idea what other bustage there might be that will [ride the trains]).

So, although this is in addition to what you're suggesting, a concrete
proposal: I've put up a patch at
https://phabricator.services.mozilla.com/D29892 /
https://bugzilla.mozilla.org/show_bug.cgi?id=1548941 .

This continues supporting the pref for:

- non-Firefox (hi TB/Fennec/SM/...)
- automation
- non-official builds (like local developer builds)

In official builds, the pref will be ignored and e10s is always turned
on. The only way to turn it off there is the environment variable
MOZ_FORCE_DISABLE_E10S which we already supported (so if you desperately
needed to debug something in non-e10s on an official build, you still
could).

For 69, I think we could remove the pref entirely and transition
everything into the env var (incl. test suites, where 1proc ones
currently just flip the pref), to further reduce confusion and
maintenance burdens, and I'd plan to remove various bits of browser
chrome code specific to (non-critical) bits of non-e10s.

Gijs