mochitest failures in CI are now retried within the same job

[Florian Quèze]

A recent patch (bug 2032038) changed how mochitest failures are handled in CI:
- Failed mochitests are now retried within the same job
- The job fails only if the test fails twice

This brings mochitest behavior in line with xpcshell. Retrying mochitests used to expose issues caused by dependencies between tests, but we now have mochitest-standalone jobs that enforce mochitests are and remain independent from each other, making retries safe.


Impact:

These retries reduced the intermittent failure rate of browser-chrome mochitest jobs from over 10% to under 5%:
https://tests.firefox.dev/#job-failures

This should make try push results significantly less noisy and save engineering time.


Will tests become flakier as a result?

Not really, because:

- Dashboards are better than painful oranges on Treeherder for tracking and fixing flakiness.
Mochitest issues (including failures that passed on retry) can be monitored:
  - Per component https://tests.firefox.dev/issues.html?kind=mochitest#date=21days
  - Per failure message https://tests.firefox.dev/failures.html?kind=mochitest#date=21days
  - Per individual test https://tests.firefox.dev/test.html

- We are working on making test-verify (TV) jobs tier 1, which will make it much harder to land new flaky tests. When testing changes to tests on try, triggering TV jobs and ensuring they are green is recommended.

Feedback welcome!

Thanks,

--

Florian Quèze