--verify option added to mochitest, reftest, xpcshell test harnesses

[Geoffrey Brown]

The mochitest, reftest, and xpcshell test harnesses now support a
--verify option. For example:

mach mochitest
docshell/test/test_anchor_scroll_after_document_open.html --verify

In verify mode, the requested test is run multiple times, in various
"modes", in hopes of quickly finding any intermittent failures. Once
tests are complete, a summary of results is printed:

:::
::: Test verification summary for:
:::
::: docshell/test/test_anchor_scroll_after_document_open.html
:::
::: 1. Run each test 10 times in one browser. : Pass
::: 2. Run each test 5 times in a new browser each time. : Pass
::: 3. Run each test 10 times in one browser, in chaos mode. : Pass
::: 4. Run each test 5 times in a new browser each time, in chaos mode. : Pass
:::
::: Test verification PASSED
:::

There's no flexibility in the number of times the test is run in each
mode and that's by design: I wanted a simple, standard way of checking
"Is this test likely to produce a frequent intermittent failure"?

Verify mode was developed for the test-verify task (announcement
coming soon!) but it may also be a convenient addition to your local
testing.

More info at [1]. Bug and enhancement requests welcomed: please file
bugs blocking bug 1357513.

[1] https://developer.mozilla.org/en-US/docs/Test_Verification

[Mike Conley]

This is super cool! Thanks Geoffrey!

> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>

[Andrew Halberstadt]

This is really great Geoff! Hopefully it can cut down the number of new
intermittents we introduce to the tree. Do you know if orangefactor or
ActiveData can track the rate of new incoming intermittents? Would be neat
to see how much of an impact this tool has on that.

[Geoffrey Brown]

On Tue, Oct 3, 2017 at 1:05 PM, Andrew Halberstadt
<ahalbe...@mozilla.com> wrote:
> This is really great Geoff! Hopefully it can cut down the number of new
> intermittents we introduce to the tree. Do you know if orangefactor or
> ActiveData can track the rate of new incoming intermittents? Would be neat
> to see how much of an impact this tool has on that.

Thanks Andrew.

I think that data is probably available in ActiveData, but querying
for "new" might be tricky...I wouldn't know where to begin.

Bugzilla might be the way to go: count bugs with keyword
intermittent-failure created today, vs yesterday vs the day before
...?

I suspect those numbers would be dominated by low frequency
intermittent failures, which might not change. In fact I'm actually
hoping to catch only mid- to high-frequency failures with TV, so that
it doesn't end up wasting people's time fixing intermittents that we
would hardly notice otherwise.

- Geoff