[PSA] Firefox tasks starting to clone from Github

[Andrew Halberstadt]

Hey Firefox devs!

Migration of Firefox's CI to Github is officially underway. If you inspect the logs of some tier 2 lint tasks on autoland (e.g), you may notice they're cloning from https://github.com/mozilla-firefox/firefox instead of https://hg.mozilla.org (HGMO). Over the coming weeks and months, we'll be switching more and more tasks to clone from Github. This process should largely be transparent to you the developer, but there is one caveat that you should be aware of. When you make a try push, your changes exist only on your local machine and HGMO. There's no revision anywhere in Github for CI to clone from! To solve this you have two options:

1. Do nothing

When you push to Try (or project branches), CI will continue to clone from HGMO like it used to. This is simple but comes with a downside. For a growing subset of tasks, autoland / main will only be exercising the Git clone path. This leaves the door open for the Hg clone path to regress undetected. In other words, you could pull the latest main, make a new Try push and hit Hg bustage that's unrelated to your patch. In practice, I suspect the odds of a task regressing due to how it was cloned will be very small. The vast majority of tasks just need a clone and don't interact with the vcs in any way! But the possibility is there.

If this happens please ping #firefox-ci on Matrix or reach out to myself directly. We'll work to sort out the bustage as quickly as possible. But you have an alternative!

2. Use the Git-backing repo

If you want your Try pushes to match what runs on autoland as closely as possible you can instead enable the "Git-backing" repo. Add the following to `~/.mozbuild/machrc`:
```

[try]

gitbacking = true
```

Or set the `GIT_BACKING_ENABLED` env:
```

$ GIT_BACKING_ENABLED=1 ./mach try fuzzy
```

When set, `mach try` does a few extra things:

1. Authenticates with Taskcluster (only required once) to fetch an SSH deploy key stored in a secret

2. Use that SSH key to push your changes to https://github.com/mozilla-releng/git-backing (no CI runs on this repo, it's just a container to hold your commits)

3. Use Lando to push to https://hg.mozilla.org/try like normal, except this time CI knows how to clone your changes from `mozilla-releng/git-backing` on Github

Using this method, your Try pushes will behave the same as if the tasks were running on autoland. But there's a side benefit as well! Cloning from Github is faster than cloning from HGMO. You'll see modest improvements for Linux and Mac tasks, but massive improvements (16min down to 4min) on Windows.

So while enabling the Git-backing repo does increase your push times (you're now pushing to two different places), it can mean getting your results faster. If the Git-backing repo ends up working well at scale, this might become the default option in the future.

Note that this option is not available when pushing to project branches at this time. Tasks running off a project branch will continue to fallback to HGMO regardless of whether you have the Git-backing repo enabled or not.

Future Plans

No matter which option you choose, if you see anything not working like it should, please either ping #firefox-ci in Matrix or myself directly. We'll help get you sorted out. I expect this state of affairs to be only temporary for the duration of the CI migration. Once all tasks have been switched over, we can make CI trigger off of the Github events themselves, at which point the Mercurial repos and Git-backing repo can all be de-commissioned. This might take several months however. If you're curious, you can follow along with a hacky little dashboard here:
https://fxci.quick.mozilla.cloud/gecko2github.html

Finally if you find yourself writing a task that *does* interact with the vcs, or you're making changes to `python/mozversioncontrol`.. I ask that you please test your changes with both `GIT_BACKING_ENABLED=0` and `GIT_BACKING_ENABLED=1`. This way we can avoid regressions for everyone.

Thanks for your understanding!

Andrew