Revisions to the Job Visibility Policy

[Ryan VanderMeulen]

For a little over 2 years, our Job Visibility Policy has been in place to
identify all the requirements necessary for a build or test suite to be
visible in our infrastructure.
https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy

However, little has changed with the policy since TBPL was still our
primary CI dashboard and buildbot our only infra capable of reporting into
it. Since then, Treeherder has made it possible to have jobs reporting from
various data sources and harnesses. Also, our automated testing needs have
grown as new features have landed, some of which have pushed the boundaries
of what falls under the existing policy.

As a result, the sheriffing team has been working to update the policy to
better encompass the current state of our CI infrastructure and testing
needs. The latest version of the revised policy is visible below:
https://wiki.mozilla.org/User:RyanVM/Job_Visibility_Policy_Updates

The tl;dr:
Instead of only having binary "visible" or "hidden" states, we're adding a
middle ground. A Tier 2 option is being added that allows for jobs that
meet most but not all of the visibility requirements to be visible by
default on Treeherder, but with reduced sheriffing expectations. Treeherder
already supports this mode, albeit with some bugs at the moment that are
being worked on.

Please take a minute to look at the revised policy and let me know of any
questions/concerns that need addressing before it is finalized. I'm hoping
to do so within the next few days - end of the week at the latest.

Thanks,
Ryan