Running mochitests faster

[Florian Quèze]

Did you know that until 2 weeks ago, very many browser-chrome mochitests included 300ms waits?

Whenever `BrowserTestUtils.switchTab` was called (either directly, or indirectly from helpers like `openNewForegroundTab` or `withNewTab`), it waited for a `TabSwitchDone` event which is only dispatched after a 300ms delay controlled by the `browser.tabs.remote.unloadDelayMs` preference.

This wastes time, but even more importantly, it makes the tests fail intermittently whenever the thing the test should have actually been waiting for starts taking slightly longer than 300ms. See https://firefox-source-docs.mozilla.org/testing/intermittent/index.html#using-magical-timeouts-to-cause-delays for more information about why timers should be avoided in tests.

In https://bugzilla.mozilla.org/show_bug.cgi?id=1702637, I changed `BrowserTestUtils` to instead wait for the `TabSwitched` event, which is dispatched immediately after the new foreground tab becomes visible.

Consequences of this change include:

- Many browser-chrome mochitests now run much faster locally.

- The average bc job time on CI has decreased by about a minute.

- A few tests became perma-fail, and many low frequency intermittent failures became frequent. To green up the tree, I introduced a “test.wait300msAfterTabSwitch” preference that reverts to the previous behavior, and I had to set this pref in a bit more than 350 tests (that’s about 4.5% of all browser chrome mochitests).

Please avoid setting the “test.wait300msAfterTabSwitch” preference in new tests and instead await the thing the test actually needs. Debugging existing tests in your components to remove the preference would also be useful, but I understand that other tasks are higher priority. Debugging this kind of issue might be a good way to get familiar with using the Firefox Profiler to debug tests failures (https://firefox-source-docs.mozilla.org/testing/mochitest-plain/index.html#debug-using-the-firefox-profiler ).

Separately, in https://bugzilla.mozilla.org/show_bug.cgi?id=1968587 Joel Maher removed 1s timers that delayed the execution of other flavors of mochitests. This should make running these kinds of mochitests locally much more efficient.

--

Florian Quèze

[ISHIKAWA,chiaki]

Sounds great.
I hope this also is plus for C-C TB mochitests as well. :-)

I run them locally and on try-comm-central and often I see intermittent errors.

Chiaki

--
You received this message because you are subscribed to the Google Groups "dev-pl...@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-platform...@mozilla.org.
To view this discussion visit https://groups.google.com/a/mozilla.org/d/msgid/dev-platform/CAFu36SrPcP32Uv9yV0jw7pYfjtJTdOBC-Sj_kzxusY_8%2Be-JZg%40mail.gmail.com.