shifting the way we track intermittent failures

[Joel Maher]

For many years we have followed the same process where we file a bug for each failure where the summary is the exact error message (a few exceptions).

This results in a many:one relationship between bugs and testcases as testcases can fail for different reasons.

This current process makes it harder to:

1) understand how often the test case is really failing (which indicates noise or need for attention)

2) automatically annotate known failures (sheriffs/try pushes/etc.)

The change we will be implementing is to have a single meta bug for each test case (file in mozilla-central).  An example is:

https://bugzilla.mozilla.org/show_bug.cgi?id=1757074

If there is an investigation to fix the testcase or disable it on a given platform, then we can create a blocking bug on our meta bug for tracking purposes. 

We will start off small, here are the restrictions we will use:

1) only test failures, not crashes || timeouts || leaks || infra

2) only for xpcshell

The plan is to roll this out (in the next 2 weeks) and monitor it for a few weeks before expanding to other test suites.

Feel free to ask questions in Matrix: #test  (firefox-testing).