Upgrading repositories on hg.mozilla.org

[Gregory Szorc]

Literally thousands of repositories on hg.mozilla.org are using an older
Mercurial storage convention that makes repositories larger and slower than
they could be.

Several months ago, we converted the mission critical repositories to a
newer and more optimal storage format.

Earlier today, I started kicking off processes to mass upgrade the
remaining repositories. Those processes are actively running and all
non-user repositories should be upgraded by end of day Wednesday.

When a repository is being upgraded, it is read-only. If `hg push` fails
due to timing out waiting for a lock, this is likely due to the repository
being upgraded. A repository upgrade can take multiple hours.

We can cancel the upgrade at any time and restore write access to a
repository immediately without risk to the integrity of the repository. If
you need this done, make noise in #vcs or #moc and request the following:

1. Run `ps faux` on on hgssh4 and find the `upgrade-hg-repos.py` process
2. Find the child process mentioning the repository you want access
restored to.
3. `kill <pid>` that process

Also, pushes to hg.mozilla.org may be a bit slower due to the CPU and I/O
required to upgrade repositories. If it becomes problematic, we can reduce
the load and prioritization of the upgrade processes. Make noise if it
becomes problematic. CI should generally not be impacted, as they use the
https:// endpoint and that is serviced by a different pool of servers.

Thank you for your understanding.