New Treeherder Feature on Try - NEW failures have a flag

[Joel Maher]

For the last 6 months the code sheriffs have had a NEW flag on autoland/mozilla-central to aid in their sheriffing. As of April 8th, this feature is also available on try.

What is this "NEW flag"?

 - when clicking on a failed job there is a list of failure lines- there will be an orange background NEW word if the failure is new [1].   This means we haven't seen this failure line in the last 3 weeks on autoland/central.

    - NOTE: if the failure is slightly different than an existing failure it will show as NEW, or if the previous failure is 22 days old it will show as NEW

    - NOTE: with this change we will not track failures from try server, only from autoland/central.

How effective is this on Autoland [2]?

90% of regressions are identified with NEW (top edge cases here: regression seen originally on m-c, backed out multiple times, intermittent turns perma fail)

96% of new bugs filed by the sheriffs have the NEW flag (edge cases: creating new bugs related to root cause, existing errors on other tasks but in a longer list and we only annotate the first failure)

15% of failures marked as existing intermittent had a NEW flag - this would be NOISE

How will this help on Try?

If you click through the failures and see a NEW flag in the error summary, it is probably worth looking at, retrigger, etc.

If you do not see a NEW flag, the chance of introducing a regression by ignoring this is pretty low.

There are some ideas we have to filter out failures without the NEW flag as well as making try server help find more regressions and be less confusing.  If you want to provide feedback or communicate more on this, please fill out this survey:
https://forms.gle/edae2f1EASqUEwqs6

Happy try pushing!

[1] https://treeherder.mozilla.org/jobs?repo=mozilla-central&revision=ba377571e5cd05957790422128fd6657ac65601d&searchStr=web-platform&selectedTaskRun=Tx-T2HheQSms-58hCccmrw.0

[2] https://docs.google.com/document/d/1mrtr620rp-X1clU1z6NF1fl7RfKYaIbNqN3d89KaHu0/edit?usp=sharing

[Andrew Sutherland]

This is super awesome and sure to be a huge help to evaluating my try pushes in conjunction with the also very awesome Push Health mechanism!  Many thanks to you and anyone/everyone involved!

Andrew

--
You received this message because you are subscribed to the Google Groups "dev-pl...@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-platform...@mozilla.org.
To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-platform/06525400-bfef-4a02-9029-6dbd28113893n%40mozilla.org.