Mercurial 4.4 Sprint Notes

[Gregory Szorc]

As earlier promised, here is a recap of the Mercurial 4.4 developer sprint,
which was held September 29-October 1 in Dublin. (I've been away on holiday
hence the report delay.) Full notes can be found at
https://www.mercurial-scm.org/wiki/4.4sprint.

State of the Project and its Automation/Tooling

* The Mercurial project is relatively healthy. Patches continue to flow in
at a significant rate from a reasonable number of contributors. The project
didn't implode when founder and long-time maintainer - mpm - retired from
the project last year.
* There are some reviewer bandwidth concerns.
* Phabricator is starting to be used for code review. Feedback is mixed.
* Automated CI coverage and capacity is lacking - especially on Windows and
MacOS. (They think TaskCluster is cool and would love to experiment with
it.)
* The issue tracker is under-loved. Not being used as much as it should.
Contemplated enabling Phabricator's issue tracker to aid usability, provide
better coherence with Phabricator-based code review.

State of Mercurial at Facebook

* Switching to tree manifests made commits 2x faster, sped up diff
exporting 5x. Big perf wins for large repos.
* "restack" workflow is deployed to their user base (thousands of engineers
I reckon). They are using obsolescence with restack. Custom command set of
{restack, next --rebase, hide, unhide, split, fold, uncommit, unamend}.
* Starting to roll out Mercurial on Windows. (Expect to see some perf work
from Facebook that targets Windows.)
* Various FB authored extensions have been upstreamed: copytrace,
directaccess, uncommit, unamend, pushvars, commitextras, morestatus.
* Want to upstream remotenames, pushrebase, infinitepush. (I also care
about infinitepush since it can be leveraged to make Try and review repos
scale. It may also help eliminate the need for most "user" repos on hg.mo.)
* Work on the "Mononoke" Mercurial server written in Rust continues.
* Working on Eden - a FUSE filesystem - with 1st class Mercurial support.
* Continuing to invest in Mercurial perf
* They feel they are reaching the long tail of the CPU perf pile.
* Previous work was about making operations go from 10+ -> 1s. Current
work is on 1s -> 0s.
* Python (particularly process startup overhead) is a bottleneck to ~0s
operations.
* Stateful chg will lower overhead.
* Need more in-memory operations (e.g. in-memory merge) to avoid I/O.
* Demo of Mercurial integration in Nuclide. Drag and drop to rebase. Other
cool UI candy.

State of Mercurial at Mozilla

* Firefox repo has grown significantly in past year.
* Full clone times becoming a larger issue.
* Windows performance is really becoming an issue, especially for larger
working directory.
* Using sparse checkouts in CI for ~2 months. Helps drastically with
checkout times.
* Default Mercurial config not appropriate for our scale or power user base.
* No mechanism to centrally configure machines (unlike other large
Mercurial users). So difficult to ensure developers are getting an optimal
Mercurial experience.
* People with optimally configured Mercurial have a significantly different
opinion from people with a vanilla install.
* People who use evolve seem to love it. Not officially recommended to
users because of UX complexities.
* As repo grows and perf suffers, tension grows between maintaining a
monorepo and splitting into microrepos. Want to make the monorepo feel
small to alleviate these tensions and to lower cognitive overhead of
interacting with it. Need narrow and shallow clones and sparse checkouts.

State of Mercurial at Google

* 2.5% of commits to Google's monorepo are done via Mercurial. (Context:
Google is switching their main internal repository to use Mercurial.)
* Using narrowhg, remotefilelog, and evolve
* Repo is linear. No merges.
* Not encouraging bookmarks, named branches, topics, any other names.
* Using `hg xl` (another `hg smartlog` / `hg show work`) repo visualization
command
* Using a virtual filesystem to host working directories.

Rust

* Mercurial *really* wants to transition from C to Rust (for most of the
typical reasons that I shouldn't have to explain to Mozillians who've
witnessed "oxidation").
* Desire to transition some Python components to Rust as well. (Namely
lower-level things like storage.)
* Desire for `hg` to be a Rust binary. This would remove binary startup
overhead and allow certain Mercurial commands to be implemented in Rust.
Python interpreter could be initialized (with startup overhead of course)
when needed.
* Problem: Mercurial currently uses Python 2.7 and 2.7 is built against
MSVC 2008. There are a handful of binary and runtime compatibility issues
at play here. Making `hg` Rust may have to wait for Mercurial to support
Python 3.5+.
* Desire to reimplement Python C extensions in Rust. Bridging Python and
Rust still not great. Lose a lot of perf instantiating the Python objects
and bridging between languages. Path forward is likely cffi into a Rust
library. More investigation is needed here.
* We'll need to double down on interfaces and strong API boundaries to
facilitate moving Python code to Rust and having swappable implementations
of various components (dirstate, storage, etc).
* Concerns over transition to Rust losing ability for extensions to
monkeypatch Mercurial. Mitigation: well-defined interfaces and hook points.
The project is also OK with some very low-level components (like storage)
not being monkeypatchable from Python. Plug in a new impleentation if you
want to experiment with new functionality.
* Contemplating a CLA for new Rust code to facilitate relicensing and
sharing of code between various projects (e.g. libhg, Mononoke).

Restack and Visibility

* Proposing a variation of evolve related to hiding and unhiding changesets.
* Removes the need for strip by default (which is a huge source of perf
loss especially on large repos)
* Conclusion: will ship a) obsolescence markers enabled by default b)
hide/unhide commands c) obsolescence marker cycle support required for new
visibility model

Annotate/Blame Feature Improvement

* Mercurial currently has an experimental flag to `hg annotate` to skip
revisions. Takes a revset.
* hgweb in 4.4 has checkboxes to ignore various whitespaces changes for
annotate views.
* `hg log --line-range` (or similar) being proposed to easily look at
log/annotate of specific range of lines.
* "Don't reformat code because it breaks annotate" is indication of tool
deficiencies. Goal: don't allow this justification to be used with
Mercurial.
* `hg annotate` in 4.4 displays asterisk on lines that have changes that
have been skipped over.
* Will add a file(path/to/file) or similar revset to resolve revisions from
lines in a file. This will enable `hg annotate --skip` to load lists of
revisions to skip from a file [checked into the repo]. Repos can then
self-define revisions to skip.
* Idea: establish a well-named file like .hgannotateskiprevs to make
revision skipping easier to use?

Mononoke

* Mercurial server written in Rust by Facebook.
* Not: a replacement for Mercurial, a reimplementation of Mercurial, or a
reimplementation of `hg`.
* Supports geographic redundancy, automatic failover. Think having a mirror
in all offices around the globe.
* Server implements both Mercurial wire protocol and an API for services. I
think they support Thirft today. Plans for GraphQL.
* Infinitepush support means all commits are automatically pushed to
server. All local changes backed up automatically and accessible from other
machines immediately.
* Pluggable backend support. Facebook uses MySQL (and other FB tech by the
sounds of it).
* Various Rust code written for Mononoke could be imported into and used by
core Mercurial.
* No server-side extension support for Python. Server-side "extensions"
written in Lua instead.
* Desire: Mercurial test harness should abstract `hg serve` so core test
suite can be run against Mononoke.

Repo Self-Configuration

* hg out of the box can be very lacking in specialized environments.
* Problem: how to get users to apply Mercurial best practices. e.g. enable
config option or install custom extension to provide better experience.
* Some companies can manage end-user machines and push out custom configs.
So this problem is "solved" via machine management.
* This doesn't work in open source or for companies that don't have this
capability (like Mozilla)
* Desire: repo best practices would be defined in the repo itself - either
via checked in content or by server-side setting.
* Problem: trust and security issues galore. Mercurial can't run arbitrary
code without user's permission. The most compelling potential features are
also the scariest from a security perspective.
* Idea: "meta repo" associated with a repo. This repo would contain
extensions, config supplements, etc.
* Idea: ability for server to send a "welcome message" during clone. Would
allow server operators to nudge users into documentation, running a setup
command, etc.
* Idea: define a data format for making config changes and support in core
or via extension. Then we can hash out automagical integration for this
self-configuration feature later.

Sparse Checkout / Narrow Clone / Lazy Changelog

* Sparse checkout support now in core as experimental feature.
* Need to figure out UX for sparse before it can be non-experimental.
Various ideas were proposed. There are paths forward.
* There are some wonky issues related to dirstate, purge, sparse, and
virtual filesystems that need to be hashed out. Google and Facebook
engineers seem to have a grasp on it.
* Idea: `hg clone --interactive` where server advertises what
clone/checkout "profiles" are available and user can choose.

Shell Prompts

* Idea: `hg` command to emit shell code to configure a shell prompt. e.g.
`eval $(hg shellprompt)`.
* Idea: async zsh prompts leveraging watchman for real-time updating

Workspaces

* Concept is a way to provide bookkeeping for something you are working on.
Relevant changesets. Uncommitted/pending file changes. Metadata like open
tabs. Build artifacts.
* Attempts to have the VCS solve the "restore the state I was in when
working on X" problem. Drastically lowers the barrier to context switching.
* (I'm not sure what the definitive plans for workspaces are.)

Topics

* Topics experiment (essentially an alternative to bookmarks) continues to
progress.
* Idea: randomly generate topic name for new draft head so all changesets
have a name (not just a hash).

hg show

* Various people like `hg show work` and `hg show stack`. Especially those
not using names.
* Decision: implement sub-command support so we can add arguments to `hg
show`'s individual views. Will also enable cleaning up various other
extensions.
* I think there is buy-in to move `hg show` from an extension to a core
command.

[David Burns]

Thanks for the update Greg!

David

> _______________________________________________
> dev-version-control mailing list
> dev-versi...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-version-control
>

[Andrew Halberstadt]

Yes great write-up, thanks! Guess I'll have to give topics a try at some
point soon.