Mercurial 4.8 Developer Sprint Meetup

[Gregory Szorc]

Connor Sheehan and I represented Mozilla at the Mercurial 4.8 Developer
Meetup (or "Sprint") in Stockholm, Sweden on October 12-14. This email is a
summary of that meetup. Topics are covered in no particular order. More
info about the sprint and a link to the raw notes can be found at
https://www.mercurial-scm.org/wiki/4.8sprint.

The sprints can typically be divided into talks/discussions and
hacking/coding.

On the hacking side, a lot of effort was put into Python 3 porting and
Oxidation/Rust.

Large areas of Mercurial are now Python 3 compatible. Mercurial now appears
to be in the long tail of Python 3 porting work. Most day-to-day commands
seem to work. An alpha or possibly beta quality Python 3 release in 2019H1
is likely.

A few exciting things have happened with Rust and Mercurial recently!

First, a port of the `chg` program from C to Rust is well underway. The
`chg` program is a drop-in replacement for the `hg` executable that spawns
a persistent Mercurial/Python process then connects to it over a pipe to
run commands. This avoids startup overhead in Python processes. The Rust
port is nearly feature complete. Fun fact: when a mostly straight port of
the C code to Rust was performed, the Rust compiler actually flushed out a
few bugs in the C code! I think it's crazy (in a good way) that "sprinkling
Rust over your code base" can tease out bugs like that!

Second, the first Rust-code-called-from-Python was landed in the Mercurial
repository! This was a big step because it required us to cross a number of
bridges the first time, including how to integrate Rust into the build
process. Things are still evolving here. But a lot of the initial hurdles
are behind us and it sounds like there is general consensus on how to move
forward with more formal/stable Rust integration into Mercurial. The Rust
code is "shipping" in Mercurial 4.8, but it isn't enabled by default.
Hopefully we'll get it enabled by default (at least in some Mercurial
distributions) in 2019.

During the sprint, Google upstreamed their patches overhauling the help
system. If you run `hg help` in Mercurial 4.8, help topics are grouped by
category, hopefully making it a lot easier to find what you are looking for.

Speaking of Google, their Mercurial rollout on the massive Google monorepo
continues. Apparently their users are *very* pleased with Mercurial - so
much so that they initially thought their user sentiment numbers were wrong
because they were so high! Google's contribution approach with Mercurial is
to upstream as many of their modifications and custom extensions as
possible: they seem to want to run a vanilla Mercurial out-of-the-box as
possible. Their feature contributions so far have been very well received
upstream and they've been contributing a number of performance improvements
as well. Their contributions should translate to a better Mercurial
experience for all.

There was a long discussion about a generic undo/rewind functionality for
Mercurial. This is something the 3rd party evolve extension is starting to
dabble in. It's a hard technical problem. But it is a commonly requested
feature ("I want to go back to the state of my repo/patches at X"). Look
for the evolve extension to start landing some features in this space.

There was another big discussion about how to handle obsolescence markers /
changeset visibility and exchange of this metadata. This has been a
long-running discussion because there are some entrenched opinions that are
incompatible with each other. There was a decision to introduce a new
"internal" phase to be used for changeset visibility instead of using
obsolescence markers. I'll spare the details, but this will allow
strip-based history rewrites in the default Mercurial install to be a thing
of the past. Strip-based history rewriting is extremely slow on large
repositories (like Firefox) and is a big reason why the evolve extension
makes operations on Firefox repos much faster. The goal is to transition to
the new mode in Mercurial 4.9.

Octobus gave a presentation about integrating Mercurial into GitLab.
Apparently it is closer to reality than one may realize! Read more at
https://octobus.net/blog/2018-09-18-heptapod-announce.html.

There was some discussion about turning on ui.tweakdefaults by default.
This config option is a meta config option that activates a lot of modern
and useful Mercurial features. It isn't on by default for backwards
compatibility. This discussion stemmed from a larger discussion on "making
the default install more useful by default." The answer to that issue is
apparently "put useful things behind tweakdefaults first."

There was talk of introducing a more unified way to programmatically carry
out complex history rewriting operations. Essentially, a lot of extensions
and 3rd party tools (like code review submission tools) want to perform
rewriting operations programmatically. Today, they must author an extension
(fast to run, slow to implement/maintain) or invoke N `hg` commands (slow
to run, fast to implement/maintain). The idea was to introduce some kind of
"protocol" to perform complex history rewriting operations. Apparently
https://www.mercurial-scm.org/wiki/PlanHistoryRewritePlan exists for this
but is vaporware. People seemed to want it. But it is a complex technical
problem and there is no obvious party willing to implement at this time.
However, a good first step is to consolidate all the history rewriting APIs
into core Mercurial (away from extensions) and there is some work planned
there.

There was talk of changing Mercurial's time-based release plan to avoid a
long feature freeze at the end of each 3 month release cycle. As an
experiment, there was no feature freeze after the 4.8 RC was released and
numerous patches for 4.9 landed. I don't think there's been a formal
discussion around the effects that had.

Octobus is doing some interesting things with computing "stable ranges" of
DAG segments to facilitate groupings for data transfer. By doing this,
client-server interaction can be optimized to reduce time to discover
"missing" data and caching servers can yield higher cache hit rates. A very
promising area for future data exchange efficiency wins.

On the project governance front, there was a discussion about establishing
a code of conduct and procedures for resolving code of conduct issues. The
general consensus is that the project wants to establish a code of conduct.
However, no decisions about specifics were made because we wanted the
discussion to occur in the open on a mailing list instead of in a room with
the few people who attended the sprint in person.

There was a discussion on improving the website and user documentation for
Mercurial. I think everyone recognizes the potential for improvements here.
One significant project contributor is willing to mentor contributions to
these areas.

Finally, I gave a presentation on the work I'm doing to improve the
Mercurial wire protocol and local storage layers. I'll spare the details
for another post. But the slides from my presentation are available at
https://docs.google.com/presentation/d/1DltJOrFndBvVuFDvMaqe9LHfJqErbynEcXs3si3-72k/edit?usp=sharing
and much can be learned by reading them.

And this is only a slice of what was discussed and hacked on.

If you have any questions, hopefully Connor or I can answer them.

Gregory

[Jonathan Watt]

No questions, but I wanted to say that I appreciate these updates and the work
that's been put into making Mercurial awesome. I moved away to Git for a while
but Evolution and other improvements brought me back. I'm now also very much
appreciating `hg absorb`!

-Jonathan