disabled non-e10s tests on trunk

[jma...@mozilla.com]

As Firefox 57 is on trunk, we are shipping e10s by default. This means that our primary support is for e10s. As part of this, there is little to no need to run duplicated tests in non-e10s and e10s mode.

In bug 1386689, we have turned them off. There was some surprise in doing this and some valid concerns expressed in comments in the bug. Given that, I thought we should bring this information to a wider audience on dev.platform so more developers are aware of the change.

While we get some advantages to not running duplicated tests (faster try results, less backlogs, fewer intermittent failures) there might be compelling reasons to run some tests in e10s based on specific coverage. With that said, I would like to say that any tests we turn on as non-e10s must have a clearly marked end date- as in this is only a temporary measure while we schedule work in to gain this coverage fully with e10s tests.

Also keep in mind that most if not all of the CI/scheduling/admin benefits of turning off the tests are already accounted for with the new stylo tests we are running in parallel on osx and windows.

[Ben Kelly]

On Tue, Aug 8, 2017 at 5:12 PM, <jma...@mozilla.com> wrote:

> As Firefox 57 is on trunk, we are shipping e10s by default. This means
> that our primary support is for e10s. As part of this, there is little to
> no need to run duplicated tests in non-e10s and e10s mode.
>

We still run android in non-e10s mode, right?

> In bug 1386689, we have turned them off. There was some surprise in doing
> this and some valid concerns expressed in comments in the bug. Given that,
> I thought we should bring this information to a wider audience on
> dev.platform so more developers are aware of the change.
>
> While we get some advantages to not running duplicated tests (faster try
> results, less backlogs, fewer intermittent failures) there might be
> compelling reasons to run some tests in e10s based on specific coverage.
> With that said, I would like to say that any tests we turn on as non-e10s
> must have a clearly marked end date- as in this is only a temporary measure
> while we schedule work in to gain this coverage fully with e10s tests.
>

If we have an android test disabled (a lot I think), then we should
consider continuing to run the test in non-e10s on desktop linux. Having
more real devices to run android tests on would probably reduce the number
of tests that are disabled. The emulator is extremely slow and not
representative of real hardware.

Ben

[Ben Kelly]

On Tue, Aug 8, 2017 at 5:18 PM, Ben Kelly <bke...@mozilla.com> wrote:

> On Tue, Aug 8, 2017 at 5:12 PM, <jma...@mozilla.com> wrote:
>

>> While we get some advantages to not running duplicated tests (faster try
>> results, less backlogs, fewer intermittent failures) there might be
>> compelling reasons to run some tests in e10s based on specific coverage.
>> With that said, I would like to say that any tests we turn on as non-e10s
>> must have a clearly marked end date- as in this is only a temporary measure
>> while we schedule work in to gain this coverage fully with e10s tests.
>>
>

> If we have an android test disabled (a lot I think), then we should
> consider continuing to run the test in non-e10s on desktop linux. Having
> more real devices to run android tests on would probably reduce the number
> of tests that are disabled. The emulator is extremely slow and not
> representative of real hardware.
>

Sorry to self-reply, but the best solution would of course be to move
android to a single content process e10s model. Then we could truly remove
non-e10s. Obviously that is probably not easy to do, though...

Ben

[Boris Zbarsky]

On 8/8/17 5:12 PM, jma...@mozilla.com wrote:
> In bug 1386689, we have turned them off. There was some surprise in doing this and some valid concerns expressed in comments in the bug.

Indeed. Given how often non-e10s mode needs to get used for debugging,
it's a little concerning to see the "yeah, it's fine if we just totally
break non-e10s mode" comments in that bug.

Something as simple as "debug something that happens during pageload" is
currently fairly rocket science to do in e10s mode; doubly so in
e10s-multi. I haven't seen any concrete proposals for improving this....

-Boris

[L. David Baron]

On Tuesday 2017-08-08 14:12 -0700, jma...@mozilla.com wrote:
> As Firefox 57 is on trunk, we are shipping e10s by default. This means that our primary support is for e10s. As part of this, there is little to no need to run duplicated tests in non-e10s and e10s mode.
>
> In bug 1386689, we have turned them off. There was some surprise in doing this and some valid concerns expressed in comments in the bug. Given that, I thought we should bring this information to a wider audience on dev.platform so more developers are aware of the change.

Has there been any additional effort to deal with tests that have
been disabled under e10s? This change means we've essentially
turned off a decent number of tests.

Some of these tests show up in these searches:
https://searchfox.org/mozilla-central/search?q=skip-if+%3D+e10s
https://searchfox.org/mozilla-central/search?q=skip-if%28browserIsRemote
although that search isn't exhaustive, and contains some tests that
we still run under some conditions.

-David

--
𝄞 L. David Baron http://dbaron.org/ 𝄂
𝄢 Mozilla https://www.mozilla.org/ 𝄂
Before I built a wall I'd ask to know
What I was walling in or walling out,
And to whom I was like to give offense.
- Robert Frost, Mending Wall (1914)

[Robert O'Callahan]

On Wed, Aug 9, 2017 at 9:31 AM, Boris Zbarsky <bzba...@mit.edu> wrote:

Something as simple as "debug something that happens during pageload" is
> currently fairly rocket science to do in e10s mode; doubly so in
> e10s-multi. I haven't seen any concrete proposals for improving this....

rr makes it fairly easy on Linux. On Mac and Windows ... yeah.

Rob
--
lbir ye,ea yer.tnietoehr rdn rdsme,anea lurpr edna e hnysnenh hhe uresyf
toD
selthor stor edna siewaoeodm or v sstvr esBa kbvted,t
rdsme,aoreseoouoto
o l euetiuruewFa kbn e hnystoivateweh uresyf tulsa rehr rdm or rnea
lurpr
.a war hsrer holsa rodvted,t nenh hneireseoouot.tniesiewaoeivatewt sstvr
esn

[Blake Kaplan]

Boris Zbarsky <bzba...@mit.edu> wrote:
> On 8/8/17 5:12 PM, jma...@mozilla.com wrote:
>> In bug 1386689, we have turned them off. There was some surprise in doing this and some valid concerns expressed in comments in the bug.

> Something as simple as "debug something that happens during pageload" is
> currently fairly rocket science to do in e10s mode; doubly so in
> e10s-multi. I haven't seen any concrete proposals for improving this....

What part of the current set-up is rocket science? In multi, there's
definitely a problem figuring out which process is the active one (though
tooltips when hovering over tabs can help). That being said, once you have a
remote tab, you can load pages in that tab and we'll stay in the same process.
That means that loading about:blank explicitly, attaching to that tab's
process, setting breakpoints, and loading the page should do the right thing.
--
Blake Kaplan

[Blake Kaplan]

L. David Baron <dba...@dbaron.org> wrote:
> Has there been any additional effort to deal with tests that have
> been disabled under e10s? This change means we've essentially
> turned off a decent number of tests.

IIRC, the last time we talked about this, there was interest in either running
tests explicitly disabled in e10s in a non-e10s browser or doing another
e10s-style run-through and sign-off of the remaining tests to move them to
suites that won't be turned off or to verify that we don't lose test coverage
of things we care about. Felipe was going to use the scripts that we used in
the original e10s rollout to generate a new list of tests to verify.
--
Blake Kaplan

[Ehsan Akhgari]

Was this done before the tests were turned off though? From what I read
from the bug, it seems like the answer is no. It seems to me that the
above should have been a prerequisite for turning these tests off,
especially at a time like this leading to the release of Firefox 57 when
we should all we can to reduce the risk of the release... Losing test
coverage on many tests here and there, and also on Android for the tests
that we run on desktop in lieu seems to me as potential factors to
increase risk. :-(

[Boris Zbarsky]

On 8/8/17 6:40 PM, Blake Kaplan wrote:
> What part of the current set-up is rocket science?

Debugging pageload. Especially pageload with no session history.

> In multi, there's
> definitely a problem figuring out which process is the active one (though
> tooltips when hovering over tabs can help).

That doesn't help when you want a no-history load. You might get an
entirely new process. Or one of your existing processes, who knows.

> remote tab, you can load pages in that tab and we'll stay in the same process.

Right, but then you have to worry about the interactions of the unload
and new load...

> That means that loading about:blank explicitly, attaching to that tab's
> process, setting breakpoints, and loading the page should do the right thing.

Hmm. Do we load about:blank from the url bar in a content process?

-Boris

[Gabor Krizsanits]

On Wed, Aug 9, 2017 at 3:36 AM, Boris Zbarsky <bzba...@mit.edu> wrote:

>
> Hmm. Do we load about:blank from the url bar in a content process?
>
>

Yes.

I agree, I find it annoying too that we have to rely on MOZ_DEBUG_CHILD_PROCESS
or MOZ_DEBUG_CHILD_PAUSE and that I have to be clever all the time about
how to hit the right process at the right time with the debugger. I never
switched back to non-e10s though since I don't trust that everything will
work the same and I don't think that should be the solution. Switching back
to single content process for debugging should come with less side effects
though... Also, this is not just an e10s/e10s-multi related issues we're
adding all kind of processes (extension/gpu/plugin/etc.).

I didn't file a bug about this but I've been trying to find a decent
solution for it, but it seems like it's not trivial in any debugger (msvc,
gdb, lldb). Or maybe I was just hoping to find something better than what
seems to be achievable. Anyway, let's start with the bug: Bug 1388693.

Gabor

[Felipe G]

I ran some scripts that I had to find out tests that are *fully* disabled
on e10s, and posted the results to
https://bugzilla.mozilla.org/show_bug.cgi?id=1376934

In summary:
mochitest-plain: 49 tests
browser-chrome: 15 tests
devtools: 86 tests

Note that the script evaluates the skip-if condition for each test, so it's
able to not count tests such as "skip=if e10s && debug" as fully disabled.

On Wed, Aug 9, 2017 at 7:35 AM, Gabor Krizsanits <gkriz...@mozilla.com>
wrote:

> On Wed, Aug 9, 2017 at 3:36 AM, Boris Zbarsky <bzba...@mit.edu> wrote:
>
> >

> > Hmm. Do we load about:blank from the url bar in a content process?
> >
> >

> Yes.
>
> I agree, I find it annoying too that we have to rely on
> MOZ_DEBUG_CHILD_PROCESS
> or MOZ_DEBUG_CHILD_PAUSE and that I have to be clever all the time about
> how to hit the right process at the right time with the debugger. I never
> switched back to non-e10s though since I don't trust that everything will
> work the same and I don't think that should be the solution. Switching back
> to single content process for debugging should come with less side effects
> though... Also, this is not just an e10s/e10s-multi related issues we're
> adding all kind of processes (extension/gpu/plugin/etc.).
>
> I didn't file a bug about this but I've been trying to find a decent
> solution for it, but it seems like it's not trivial in any debugger (msvc,
> gdb, lldb). Or maybe I was just hoping to find something better than what
> seems to be achievable. Anyway, let's start with the bug: Bug 1388693.
>
> Gabor

> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>

[Ben Kelly]

On Tue, Aug 8, 2017 at 5:18 PM, Ben Kelly <bke...@mozilla.com> wrote:

> On Tue, Aug 8, 2017 at 5:12 PM, <jma...@mozilla.com> wrote:
>
>> In bug 1386689, we have turned them off. There was some surprise in
>> doing this and some valid concerns expressed in comments in the bug. Given
>> that, I thought we should bring this information to a wider audience on
>> dev.platform so more developers are aware of the change.
>>
>> While we get some advantages to not running duplicated tests (faster try
>> results, less backlogs, fewer intermittent failures) there might be
>> compelling reasons to run some tests in e10s based on specific coverage.
>> With that said, I would like to say that any tests we turn on as non-e10s
>> must have a clearly marked end date- as in this is only a temporary measure
>> while we schedule work in to gain this coverage fully with e10s tests.
>>
>

> If we have an android test disabled (a lot I think), then we should
> consider continuing to run the test in non-e10s on desktop linux. Having
> more real devices to run android tests on would probably reduce the number
> of tests that are disabled. The emulator is extremely slow and not
> representative of real hardware.
>

BTW, we have a large corpus of tests that don't run on android at all:
WPT. Increasingly over time features are only covered by WPT tests.

I think we should keep at least non-e10s running on linux or linux64 for
now until we can improve our android test coverage or move android to e10s.

Ben

[Andrew Halberstadt]

We now have the ability to set prefs from a mochitest manifest (see bug
1328830 <https://bugzilla.mozilla.org/show_bug.cgi?id=1328830> and my
recent newsgroup post). We could refactor these tests into a special
non-e10s manifest that sets browser.tabs.remote.autostart=false and keep
running them as part of the normal test run.

On Wed, Aug 9, 2017 at 5:28 PM Felipe G <fel...@gmail.com> wrote:

> I ran some scripts that I had to find out tests that are *fully* disabled
> on e10s, and posted the results to
> https://bugzilla.mozilla.org/show_bug.cgi?id=1376934
>
> In summary:
> mochitest-plain: 49 tests
> browser-chrome: 15 tests
> devtools: 86 tests
>
> Note that the script evaluates the skip-if condition for each test, so it's
> able to not count tests such as "skip=if e10s && debug" as fully disabled.
>
> On Wed, Aug 9, 2017 at 7:35 AM, Gabor Krizsanits <gkriz...@mozilla.com>
> wrote:
>
> > On Wed, Aug 9, 2017 at 3:36 AM, Boris Zbarsky <bzba...@mit.edu> wrote:
> >
> > >

> > > Hmm. Do we load about:blank from the url bar in a content process?
> > >
> > >

[jma...@mozilla.com]

Thanks everyone for commenting on this thread. As a note, we run many tests in non-e10s mode:
* android mochitest, reftest, crashtest, marionette,
* mochitest-chrome
* xpcshell
* gtest/cpptest/jittest
* mochitest-a11y
* mochitest-jetpack (very few tests remain)

While there are many tests which individually are disabled or lacking coverage, these test suites have no non-e10s coverage:
* web-platform-tests
* browser-chrome
* devtools
* jsreftests
* mochitest-webgl, mochitest-gpu, mochitest-media
* reftest un-accel

I would propose running these above tests on windows7-opt (we don't have these running yet in windows 10, although we are close), and only running specific tests which are not run in e10s mode, turning them off December 29th, 2017.

Keep in mind we have had many years to get all our tests running in e10s mode and we have known since last year that Firefox 57 would be the first release that we ship e10s by default to our users- my proposal is a 4 month temporary measure to let test owners finish ensuring their tests are running in e10s mode.

[Ben Kelly]

On Tue, Aug 15, 2017 at 4:29 PM, <jma...@mozilla.com> wrote:

> While there are many tests which individually are disabled or lacking
> coverage, these test suites have no non-e10s coverage:
> * web-platform-tests
> * browser-chrome
> * devtools
> * jsreftests
> * mochitest-webgl, mochitest-gpu, mochitest-media
> * reftest un-accel
>
> I would propose running these above tests on windows7-opt (we don't have
> these running yet in windows 10, although we are close), and only running
> specific tests which are not run in e10s mode, turning them off December
> 29th, 2017.
>
> Keep in mind we have had many years to get all our tests running in e10s
> mode and we have known since last year that Firefox 57 would be the first
> release that we ship e10s by default to our users- my proposal is a 4 month
> temporary measure to let test owners finish ensuring their tests are
> running in e10s mode.
>

WPT tests do not run on android. Unless we can get WPT tests running on
android, I don't think dropping all non-e10s coverage for them is
reasonable. I don't know of any plan to move android to e10s by Dec 29.

[Joel Maher]

All of the above mentioned tests are not run on Android (well
mochitest-media is to some degree). Is 4 months unreasonable to fix the
related tests that do not run in e10s? Is there another time frame that
seems more reasonable?

On Tue, Aug 15, 2017 at 4:34 PM, Ben Kelly <bke...@mozilla.com> wrote:

> On Tue, Aug 15, 2017 at 4:29 PM, <jma...@mozilla.com> wrote:
>

>> While there are many tests which individually are disabled or lacking
>> coverage, these test suites have no non-e10s coverage:
>> * web-platform-tests
>> * browser-chrome
>> * devtools
>> * jsreftests
>> * mochitest-webgl, mochitest-gpu, mochitest-media
>> * reftest un-accel
>>
>> I would propose running these above tests on windows7-opt (we don't have
>> these running yet in windows 10, although we are close), and only running
>> specific tests which are not run in e10s mode, turning them off December
>> 29th, 2017.
>>
>> Keep in mind we have had many years to get all our tests running in e10s
>> mode and we have known since last year that Firefox 57 would be the first
>> release that we ship e10s by default to our users- my proposal is a 4 month
>> temporary measure to let test owners finish ensuring their tests are
>> running in e10s mode.
>>
>

[Ben Kelly]

On Tue, Aug 15, 2017 at 4:37 PM, Joel Maher <jma...@mozilla.com> wrote:

> All of the above mentioned tests are not run on Android (well
> mochitest-media is to some degree). Is 4 months unreasonable to fix the
> related tests that do not run in e10s? Is there another time frame that
> seems more reasonable?
>

Last I checked it was your team that told me WPT on android was not an
immediate priority. The WPT harness itself does not run there.

[Ben Kelly]

On Tue, Aug 15, 2017 at 4:43 PM, Joel Maher <jma...@mozilla.com> wrote:

> This is a discussion about tests in e10s mode, not WPT on Android.
>

Yes. And android is our last tier 1 platform that requires non-e10s. I
think that makes it relevant to the discussion.


>
> What specific coverage are we missing by not running WPT in non-e10s mode
> on desktop? When can we ensure we have that coverage in e10s mode? I
> assume this is just making sure the tests that we have disabled on e10s
> mode need to get fixed.
>

Some features are implemented completely differently in e10s vs non-e10s
mode. The main one I am aware of is service workers. If you turn on
non-e10s WPT tests there will be regressions.

Don't get me wrong. I'd prefer not to deal with non-e10s either. But its
*absolutely false* to say we don't need to support it right now because its
required for a tier 1 platform.

[J. Ryan Stinnett]

I think Ben's argument has merit:

1. Even after Firefox 57, we will still be shipping a product in non-e10s
mode: Firefox for Android
2. If WPT (and potentially other suites) aren't being run in non-e10s mode,
it increases risk because we are shipping untested code paths to our users

Someone might add code that assumes e10s is the only mode and land it
successfully, only later to hear that it fails or crashes on Android, since
e10s doesn't exist there today.

- Ryan

On Tue, Aug 15, 2017 at 3:43 PM, Joel Maher <jma...@mozilla.com> wrote:

> This is a discussion about tests in e10s mode, not WPT on Android.
>

> What specific coverage are we missing by not running WPT in non-e10s mode
> on desktop? When can we ensure we have that coverage in e10s mode? I
> assume this is just making sure the tests that we have disabled on e10s
> mode need to get fixed.
>

[Nils Ohlmeier]

I guess not a lot of people are aware of it, but for WebRTC we still have two distinct implementations for the networking code.
So if I understand the impact here right we just lost test coverage for probably a couple of thousand lines of code.

And yes you can potentially blame it on WebRTC networking for not having unified it’s networking code for e10s and non-e10s.
But when e10s got turned on I asked around if we could delete the non-e10s code soon, I was told no.
So I assumed both technologies would stick around for some more time.

I’m not sure how others do it, but our low level C++ unit tests don’t have an e10s mode at all.
Therefore we can’t simply delete the non-e10s WebRTC networking code either (without loosing a ton of test coverage).

With this being said I think the right thing here would be to turn the non-e10s mochitests back on for WebRTC until we have a better solution.

Nils

> On Aug 15, 2017, at 13:54, J. Ryan Stinnett <jry...@gmail.com> wrote:
>
> I think Ben's argument has merit:
>
> 1. Even after Firefox 57, we will still be shipping a product in non-e10s
> mode: Firefox for Android
> 2. If WPT (and potentially other suites) aren't being run in non-e10s mode,
> it increases risk because we are shipping untested code paths to our users
>
> Someone might add code that assumes e10s is the only mode and land it
> successfully, only later to hear that it fails or crashes on Android, since
> e10s doesn't exist there today.
>
> - Ryan
>
> On Tue, Aug 15, 2017 at 3:43 PM, Joel Maher <jma...@mozilla.com> wrote:
>
>> This is a discussion about tests in e10s mode, not WPT on Android.
>>
>> What specific coverage are we missing by not running WPT in non-e10s mode
>> on desktop? When can we ensure we have that coverage in e10s mode? I
>> assume this is just making sure the tests that we have disabled on e10s
>> mode need to get fixed.
>>
>> On Tue, Aug 15, 2017 at 4:39 PM, Ben Kelly <bke...@mozilla.com> wrote:
>>
>>> On Tue, Aug 15, 2017 at 4:37 PM, Joel Maher <jma...@mozilla.com> wrote:
>>>
>>>> All of the above mentioned tests are not run on Android (well
>>>> mochitest-media is to some degree). Is 4 months unreasonable to fix the
>>>> related tests that do not run in e10s? Is there another time frame that
>>>> seems more reasonable?
>>>>
>>>
>>> Last I checked it was your team that told me WPT on android was not an
>>> immediate priority. The WPT harness itself does not run there.
>>>

[James Graham]

On 16/08/17 01:26, Nils Ohlmeier wrote:
> I guess not a lot of people are aware of it, but for WebRTC we still have two distinct implementations for the networking code.
> So if I understand the impact here right we just lost test coverage for probably a couple of thousand lines of code.

[…]

> I’m not sure how others do it, but our low level C++ unit tests don’t have an e10s mode at all.
> Therefore we can’t simply delete the non-e10s WebRTC networking code either (without loosing a ton of test coverage).

If the networking code is only covered by C++ unit tests, there is
separate code for non-e10s vs e10s, and the unit tests don't work in
e10s mode doesn't that mean we currently don't have any test coverage
for our shipping configuration on desktop? What am I missing?

[James Graham]

On 15/08/17 21:39, Ben Kelly wrote:
> On Tue, Aug 15, 2017 at 4:37 PM, Joel Maher <jma...@mozilla.com> wrote:
>
>> All of the above mentioned tests are not run on Android (well
>> mochitest-media is to some degree). Is 4 months unreasonable to fix the
>> related tests that do not run in e10s? Is there another time frame that
>> seems more reasonable?
>>
>
> Last I checked it was your team that told me WPT on android was not an
> immediate priority. The WPT harness itself does not run there.

FWIW the story with wpt/Android is that originally Android didn't
support the kind of remote control required to run wpt tests (i.e.
marionette). That has subsequently been fixed, and it's believed to be
possible incorporate Android into the wpt harness without a significant
refactoring, but after that there is substantial, time consuming, work
required to get from "it runs" to "this is a thing we can run in
production".

I doubt that the work required to implement an Android backend, sort out
issues with the relative slowness of the android emulator, and green up
the tests, would be less than one person's work for a quarter. Even once
this is done there would likely be ongoing problems updating the
metadata on android for syncs from upstream simply from the additional
slowness of try runs and probable additional intermittency.

So far there haven't been enough people working on wpt to make this a
priority relative to other goals. Given the situation with e10s, it
seems reasonable to reassess this when planning future work, but I
wouldn't bet on such work being complete by Dec. 29th this year.

[Ehsan Akhgari]

On 08/15/2017 04:29 PM, jma...@mozilla.com wrote:
> I would propose running these above tests on windows7-opt (we don't have these running yet in windows 10, although we are close), and only running specific tests which are not run in e10s mode, turning them off December 29th, 2017.
>
> Keep in mind we have had many years to get all our tests running in e10s mode and we have known since last year that Firefox 57 would be the first release that we ship e10s by default to our users- my proposal is a 4 month temporary measure to let test owners finish ensuring their tests are running in e10s mode.

Hi Joel,

I think Ben is right on here. Please note that it's not just service
workers that have a different implementation in e10s vs. non-e10s, many
other super important parts of the browser are in the same boat. For
example, our entire HTTP(S) stack has different code paths in e10s vs.
non-e10s. Furthermore, even in places where we share a lot of code in
the two modes, there are often subtle differences in the behavior in
between the two modes. I hope this is convincing in terms of the
necessity of maintaining the test coverage across the two modes.

To slightly readjust what you stated above from the perspective of the
Firefox developers, we have known since last years that Firefox 57 would
be the first release that we ship e10s by default for our *desktop*
users, and we have also known that Firefox 57 for Android will be
shipped without e10s for our *mobile* users, so it shouldn't be any
surprise why no large scale effort has been put into porting all of our
tests to run into e10s mode. Also please note that some of the test
frameworks you have listed above like browser-chrome aren't relevant to
Android, and some like jsreftests aren't relevant to e10s, so we should
probably have a more detailed conversation about the remaining ones.

I think a better way to think about this problem is perhaps how to get
to a point where we never end up losing test coverage on code that we
ship on our tier 1 platforms. Thinking about this in terms of
date-based deadlines where we don't have the human power to do the work
necessary isn't particularly helpful, and would ultimately result us in
losing invaluable test coverage. If past history is any lesson for us,
regressions will creep in as a result, and especially due to the fact
that this is mostly affecting Firefox for Android where we have the
lowest pre-release testing population among our tier1 products (AFAIK)
makes that extremely risky.

Therefore, I think we should:

* start running all of the non-e10s tests that can affect Android
again immediately.
* work out a realistic plan between engineering and the automation
team to address the existing gaps that prevent us from turning off
non-e10s tests without losing coverage on Android, and execute that plan.
* turn non-e10s tests off when the above has been done.

Please let me know what you think!

Thanks,
Ehsan

[Joel Maher]

Thanks everyone for chiming in here. I see this isn't as simple as a
binary decision and to simplify things, I think turning on all non-e10s
tests that were running for windows7-debug would give us reasonable
coverage and ensure that users on our most popular OS (and focus for 57)
have a stable experience if they choose to go without e10s.

Instead of a date to turn things off, I would like to just focus on each
framework one at a time and ideally in a few months we would have a clear
set of requirements (in the form of bugs) to resolve before turning off the
specific non-e10s tests.

If there is a different configuration other than windows7-debug I would
like to hear about it.

Joel


On Wed, Aug 16, 2017 at 12:53 PM, Ehsan Akhgari <ehsan....@gmail.com>
wrote:

> On 08/15/2017 04:29 PM, jma...@mozilla.com wrote:
>

>> I would propose running these above tests on windows7-opt (we don't have
>> these running yet in windows 10, although we are close), and only running
>> specific tests which are not run in e10s mode, turning them off December
>> 29th, 2017.
>>
>> Keep in mind we have had many years to get all our tests running in e10s
>> mode and we have known since last year that Firefox 57 would be the first
>> release that we ship e10s by default to our users- my proposal is a 4 month
>> temporary measure to let test owners finish ensuring their tests are
>> running in e10s mode.
>>

[Ben Kelly]

On Wed, Aug 16, 2017 at 2:32 PM, Joel Maher <jma...@mozilla.com> wrote:

> Thanks everyone for chiming in here. I see this isn't as simple as a
> binary decision and to simplify things, I think turning on all non-e10s
> tests that were running for windows7-debug would give us reasonable
> coverage and ensure that users on our most popular OS (and focus for 57)
> have a stable experience if they choose to go without e10s.
>
> Instead of a date to turn things off, I would like to just focus on each
> framework one at a time and ideally in a few months we would have a clear
> set of requirements (in the form of bugs) to resolve before turning off the
> specific non-e10s tests.
>
> If there is a different configuration other than windows7-debug I would
> like to hear about it.
>

Thank you Joel.

My only thought about windows7-debug is that android is a variant of
linux. Running a linux platform might be closer to android behavior. But
I don't have a known specific difference in mind.

It would also be nice to opt and debug since things do differ between them,
but I'll take what I can get.

Thanks.

Ben

[James Graham]

On 16/08/17 19:36, Ben Kelly wrote:
> My only thought about windows7-debug is that android is a variant of
> linux. Running a linux platform might be closer to android behavior. But
> I don't have a known specific difference in mind.

Right it seems like there are two use cases here:

1) Tests that are really aimed at Desktop or are cross-product but don't
run on e10s for (reasons).

2) Tests for features that are run in e10s on Desktop, but exercise
functional differences in non-e10s and don't run in Android.

For 1) running on Windows makes some sense because that's where most
users are. For 2) it makes no sense because it's the most different from
Android. For those cases running on Linux(64) makes more sense (and is
also usually where we have most capacity, so that helps with infra issues).

[Nils Ohlmeier]

So we have mochitest-media which works as kind of integration test on a higher level. They execute high level JS API tests, but also try to ensure that the lower level networking pieces (the once which are exposed through JS) match the expectations.
The mochitest-media got executed for e10s and non-e10s and therefore covered both implementations.

And then we have C++ unit tests, which cover a lot more corner cases of different scenarios for networking. And yes these only work with non-e10s right now. It would be a lot of work to create the same amount of tests with a higher level test suite like mochitest to get the e10s coverage. Plus these tests would probably take a lot execution time.

Technically that leaves us with a somewhat blind spot for the coverage of networking corner cases under e10s. I guess if there is a high demand for turning off all non-e10s tests we need to look at how to get our C++ unit tests working with something like e10s.
But until we can get to that I think we really should keep running mochitest-media with e10s and without it.

Best
Nils Ohlmeier

[jma...@mozilla.com]

As a note, C++ tests will continue to run in non-e10s mode (cppunit, gtest)- there are no plans to run these as e10s; this is mostly referring to: mochitest*, web-platform-test*, reftest*, marionette*

[jma...@mozilla.com]

Yesterday I landed bug 1391371 which enabled non-e10s unittests on windows 7 debug. A few tests had to be disabled in order to do this. To keep track of what we did and each of the test suites to evaluate, I have filed bug 1391350.

As a note we now have the following non-e10s tests running:
windows7-debug: all trunk branches
android opt/debug: all trunk branches (existing media, plain, reftest)
linux64-jsdcov: mozilla-central (mochitest-plain/browser-chrome/devtools)
** this is a linux64 opt build and we use the jsdebugger to collect code coverage metrics- but we specifically run this in non-e10s mode.

Please let me know if there are large areas that we have overlooked.

Thanks!

[Ben Kelly]

Joel,

Is there an easy way for me to run non-e10s tests on linux? We often use
"t-style" try pushes where we only run tests on one platform. Restricting
non-e10s to win7-debug means I either need to run tests on multiple
platforms or use windows for the "t-style". I don't want to use windows
because it builds/runs much slower than linux.

Thanks.

Ben

[Joel Maher]

you could edit the taskcluster/ci/test/tests.yml file and add
|linux64/debug: both| in the places we have |windows7-32/debug: both|, as
seen here:
http://searchfox.org/mozilla-central/source/taskcluster/ci/test/tests.yml#138

If there are concerns with windows build times, please ask for those to be
faster- it will make it better for all of us :)

On Fri, Sep 8, 2017 at 5:40 PM, Ben Kelly <bke...@mozilla.com> wrote:

> Joel,
>
> Is there an easy way for me to run non-e10s tests on linux? We often use
> "t-style" try pushes where we only run tests on one platform. Restricting
> non-e10s to win7-debug means I either need to run tests on multiple
> platforms or use windows for the "t-style". I don't want to use windows
> because it builds/runs much slower than linux.
>
> Thanks.
>
> Ben
>
> On Fri, Aug 18, 2017 at 4:15 PM, <jma...@mozilla.com> wrote:
>

[Randell Jesup]

>you could edit the taskcluster/ci/test/tests.yml file and add
>|linux64/debug: both| in the places we have |windows7-32/debug: both|, as
>seen here:
>http://searchfox.org/mozilla-central/source/taskcluster/ci/test/tests.yml#138
>
>If there are concerns with windows build times, please ask for those to be
>faster- it will make it better for all of us :)

Where's my magic wand...? if it was easy to make them faster I presume
that would have been done long ago.

I frequently do (only) linux Trys unless I suspect platform-specific
issues. Saves resources... Can't "mochitest-media" (and other non-e10s
versions) be not-run by default, unless you specify it explicitly in -u ?

--
Randell Jesup, Mozilla Corp
remove "news" for personal email