test-verify now running as tier 2

[Geoffrey Brown]

Today the test-verify test task will start running as a tier 2 job.
Look for the "TV" symbol on treeherder, on linux-64 test platforms.

TV is intended as an "early warning system" for identifying the
introduction of intermittent test failures. When a mochitest, reftest,
or xpcshell test file is modified on a push, TV runs that particular
test over and over until it fails (orange job, standard failure
messages), or until max iterations are achieved (green job, all's
well), or until TV runs out of time (green job, maybe all's well?). As
a consequence, when a new test is added or a test is modified and an
intermittent failure is introduced, TV will usually be the first job
to fail, and it will fail on the push that modified the test, making
it (usually) simple to identify where the intermittent was introduced.

In future I hope to run TV on more platforms, apply it to more test
suites, and refine the --verify implementation to find intermittent
failures more efficiently. As a tier 2 task, TV failures will be
starred but will not cause backouts. I hope to move to tier 1 once TV
is proven to be effective.

More info at [1]. Bug and enhancement requests welcomed: please file
bugs blocking bug 1357513.

[1] https://developer.mozilla.org/en-US/docs/Test_Verification

[Chris Peterson]

This is very cool, Geoff! People have been talking about this idea for a
long, so it is great to see it actually running. I'm glad to see chaos
mode being tested, too.

[Chris Peterson]

This is very cool, Geoff! People have been talking about this idea for a
long, so it is great to see it actually running. I'm glad to see chaos
mode being tested, too.


On 2017-10-02 10:11 AM, Geoffrey Brown wrote:

[gbr...@mozilla.com]

Now test-verify runs on all platforms - even Android!

And beginning today, there's a new test task, test-verify-wpt, symbol TVw on treeherder, which is just like test-verify, but for web-platform tests (including web-platform reftests and wdspec tests).

Both TV and TVw run as tier 2, so bugs are filed for failures, but failures do not result in backouts. These do not run as tier 1 because many tests currently fail verification; I don't want verification to cause a backout when a trivial change triggers verification of a test with an existing fault. Verifying and fixing all our tests is a big project...maybe one day! For now, if your push triggers a verification failure, please have a good look at it and consider fixing the issue right away. TV/TVw failures are generally an indication of a "bad" test that is likely to fail intermittently in the future.