Fwd: Enabling automated back-filling for few hours on Tuesday

[Armen Zambrano G.]

For some reason the first email did not go through to the Google Groups.
Trying again by email the mailing list directly.


-------- Forwarded Message --------
Subject: Enabling automated back-filling for few hours on Tuesday
Date: Mon, 27 Jul 2015 13:26:18 -0400
From: Armen Zambrano G. <arm...@mozilla.com>
Organization: Mozilla Corporation
CC: Alice Scarpa <alice...@gmail.com>, Sheriffs <sher...@mozilla.org>
Newsgroups:
mozilla.tools,mozilla.dev.tree-management,mozilla.release.engineering



tl;dr; we're going to enable automated back-filling tomorrow Tuesday for
few hours on m-i.

We aimed for Monday but pushed it to Tuesday to help publicize this more.

If on Wednesday there are no fall-outs we will leave it on m-i for a
week before enabling it on other places.

> Hello all,
>
> We are planning to turn on a service that automatically backfills
> failed test jobs on m-i. If there are no concerns, we would like to
> turn this on experimentally for a couple of hours on [Tuesday]. We
> hope this will make it easier to identify which revision broke a
> test. Suggestions are welcome.
>
> The backfilling works like this: - It triggers the job that failed
> one extra time - Then it looks for a successful run of the job on the
> previous 5 revisions. If a good run is found, it triggers the job
> only on revisions up to this good run. If not, it triggers the job on
> every one of the previous 5 revisions. Previous jobs will be
> triggered one time.
>
> The tracking bug is:
> https://bugzilla.mozilla.org/show_bug.cgi?id=1180732
>
> Best, Alice



--
Zambrano Gasparnian, Armen
Automation & Tools Engineer
http://armenzg.blogspot.ca

[Armen Zambrano G.]

We need to take time to tackle this again in the future once we build
safe guards for preventing automatic backfiling from causing unnecessary
load for some cases.

tracker: https://bugzilla.mozilla.org/show_bug.cgi?id=1180732

Bug 1195809 - consider more recent pushes that say "backout"
Bug 1195821 - follow some rules to decide when not to backfill
Bug 1195824 - deal better with perma failures
Bug 1195851 - not trigger jobs for a pool when it is heavely back logged
Bug 1197223 - respect hidden jobs