PSA: Deprioritizing excessively large try pushes

[Andrew Halberstadt]

Hey folks, just wanted to warn everyone that a change to `./mach try` is about to land on central that will deprioritize excessively large pushes.

What changed?

Pushes that exceed 1000 tasks (excluding test chunks), will now have their tasks scheduled at the "lowest" priority. This means they will always be at the end of the queue relative to other tasks on try. It's much more likely tasks in these pushes will be delayed until overnight hours PST, and possibly might even exceed their deadline before they get picked up (i.e never end up running at all).

For these pushes, a warning will be printed and you will be prompted whether you want to continue, or abort and select fewer tasks.

Why?

While many worker pools in CI use cloud VM instances and can grow workers indefinitely, there are many others that are hardware constrained. You've likely felt this if you ever had to wait for MacOS tests in a try push. A single excessively large try push can delay other people's results for hours or even days in some cases.

This change is intended to:

1. Ensure excessively large pushes don't disrupt others.
2. Provide an incentive to be a bit more thoughtful about try usage.

What if I need a high priority push?

If you are working on an incident or some other high priority item, you can use the `--chemspill` flag to disable this behaviour (e.g `./mach try fuzzy --chemspill`). Using this flag will bump your priority above everyone else's, so please use your discretion and avoid using this flag except for incidents and other highly time sensitive needs.

How can I avoid being penalized?

When iterating on a patch, make sure your pushes drill down onto the single task you are iterating on. If there are many failures in your push, there's often no need to run everything on each successive iteration of your patch. Focus on a single task, then re-run the full set again only after it is green.

If you're using a preset, you can use the `-xi` flags to interactively filter down the preset's starting labels. E.g `./mach try fuzzy --preset android-geckoview -xi` will pop you into the fuzzy finder interface with only that preset's tasks pre-populated. You can then pinpoint the task(s) you want as normal.

What is an "excessively large push"?

Currently this is defined as 1000 tasks. Due to an implementation detail, we're unable to know how many test chunks will end up being selected, so these are omitted for now. E.g if a test suite would end up with 5 chunks, it would still only count as a single task for the purposes of determining a large push. In the future we might lower the threshold, or change the algorithm such that test chunks will be estimated. But for now we'll see how this goes.

If you have any questions or concerns, please feel free to reach out to me directly, or ask in #firefox-ci on Matrix.

Thanks for your understanding,

Andrew

[Andrew Halberstadt]

Just to answer some questions I should have included in the original e-mail:

1. This only applies to try selectors where you explicitly select labels (e.g fuzzy and chooser). `./mach try auto` and other selectors that do things automatically are exempt.

2. Like test chunks, dependencies that get pulled in to support the tasks you selected are also excluded from the threshold calculation.