Tasks that get hidden on mozilla-central but not try

[Andrew Halberstadt]

I think most developers have wasted at least some of their time debugging
failures that have shown up in their try pushes, not realizing that the
failures are known and those same tasks have been hidden on
mozilla-central. Ideally whenever a task is hidden on central, it should
also be hidden on try. But it's easy to forget to do this, so this is a
problem that should be solved by tools.

There are currently two ways to hide a task:
1. By marking it excluded (via treeherder exclusion profile)
2. By marking it Tier-3 (via treeherder exclusion profile or in-tree task
config)

Now I have a feeling sheriffs won't like this proposal, but hear me out :).
What if we moved the exclusion profile out of treeherder and into the tree?
This has a few benefits:

1. Exclusions automatically propagate everywhere they're supposed to
2. Easier for developers unfamiliar with treeherder to inspect/modify
3. Task's exclusion history gets built-in to version control

There are also a couple of downsides:

1. Requires taskcluster scheduling (though this should include everything
by ~Nov)
2. Adds overhead (especially for sheriffs)

The second downside can be mitigated with tools. Any commits that only
touch the exclusion profile can safely use DONTBUILD. So we could make a
mach command that automatically modifies the exclusion profile and pushes
it with DONTBUILD.

At the end of the day, the exclusion profiles for central and try need to
stay in sync. If there are other ways of accomplishing this, I'd be equally
thrilled if those could be implemented instead. So let me know if you have
other ideas to do this.

What do people think?

Thanks!
Andrew

[Mike Hommey]

On Thu, Jul 20, 2017 at 09:13:14PM +0000, Andrew Halberstadt wrote:
> I think most developers have wasted at least some of their time debugging
> failures that have shown up in their try pushes, not realizing that the
> failures are known and those same tasks have been hidden on
> mozilla-central. Ideally whenever a task is hidden on central, it should
> also be hidden on try. But it's easy to forget to do this, so this is a
> problem that should be solved by tools.
>
> There are currently two ways to hide a task:
> 1. By marking it excluded (via treeherder exclusion profile)
> 2. By marking it Tier-3 (via treeherder exclusion profile or in-tree task
> config)
>
> Now I have a feeling sheriffs won't like this proposal, but hear me out :).
> What if we moved the exclusion profile out of treeherder and into the tree?
> This has a few benefits:
>
> 1. Exclusions automatically propagate everywhere they're supposed to
> 2. Easier for developers unfamiliar with treeherder to inspect/modify
> 3. Task's exclusion history gets built-in to version control
>
> There are also a couple of downsides:
>
> 1. Requires taskcluster scheduling (though this should include everything
> by ~Nov)
> 2. Adds overhead (especially for sheriffs)

Another downside is that the exclusion list will only apply to
changesets that have the changes. Meaning, an excluded job will still
show up for old changesets that don't have it exluded. Consequently,
they will still show up for people pushing old trees to try (and they
do).

The most obvious way to handle this would be for treeherder to use the
central list of excluded tasks for try...

Mike

[Andrew Halberstadt]

On Thu, Jul 20, 2017 at 5:25 PM Mike Hommey <m...@glandium.org> wrote:

> Another downside is that the exclusion list will only apply to
> changesets that have the changes. Meaning, an excluded job will still
> show up for old changesets that don't have it exluded. Consequently,
> they will still show up for people pushing old trees to try (and they
> do).
>

Is that really a downside though? Presumably if a task was unhidden in the
past, that's because it was stable at that point in time. And if it was
super flaky but for some reason still unhidden, then that's no different
than updating to an old commit that happens to have bustage on it. It's
always a risk.

> The most obvious way to handle this would be for treeherder to use the
> central list of excluded tasks for try...
>

Heh, true. I'm not sure how exclusion profiles work in treeherder, but if
that's a possibility, that would be a good solution.


>
> Mike
>