More reviewers needed and general project update

[Georges Racinet]

Hi everyone,

Thanks to the tremendous efforts of Dan Villiom Podlaski Christiansen,
the hg-git project is in an much better shape than it was a year ago,
but we are still too constrained : we need more reviewers and we are
willing to appoint new maintainers among the contributors that will step
up. In the mean time we are willing to relax some rules (details below).

As you may know, I am the maintainer of Heptapod [1], and that doesn't
leave me much time for anything else. 

These last few months, Dan did almost all the implementation work in
hg-git, including maintaining compatibility with current Mercurial
versions and then had to wait way too long for my review. This is enough
to keep hg-git afloat, but it is obviously unhealthy in the long run.

From the Bitbucket era, we've inherited a long list of issues [2] and
patch series, which were sometimes duplicate efforts to solve given
problems. We've made progress sorting all that out, but there's still a
long way to go.

I should add that we made good progress on quality, notably with a CI
setup that tests all contributions to hg-git with many Mercurial and
Python versions. In short, everything is in place, we just need more
people to participate.


-- Code Review and new fast-tracking rule --

  One does not need to have the Maintainer role to help with code review!

  Everything happens nowadays as Merge Requests (MR) on
foss.heptapod.net [3]. Any logged-in user can comment, and there's an
"Approve" button to hit.

  It should be easy to checkout the changes and test them [4]. I
personally prefer helping people doing that, whether as direct
interaction or as improvements in the Heptapod UI than to test features
and environments that I'm not comfortable with.

  As a special case, we will stop making review mandatory after a couple
of days for MRs

    - that fix compatibility with the forthcoming Mercurial version. It
really does not help when hg-git becomes incompatible with Mercurial
because review was stalled. The only thing it accomplishes is masking
potential subtle bugs behing API breakage. Also, it is quite a
complication for distributions, including TortoiseHg.

    - that fix bugs with no ambiguity about the expected behavior.
Today's example:
https://foss.heptapod.net/mercurial/hg-git/-/merge_requests/108

  What remains mandatory however to fast-track such fixes:

    - they come with the tests that were previously missing

    - CI passes

-- Issues --

  We need help sorting out which issues are still reproducible,
identifying duplicates etc.

  At some point, just for the sake of clarity, we will have to close all
long standing issues with a friendly message asking to reopen if still
current. Still it would be better if that was only for a handful leftovers.

  Cross-referencing Merge Requests (!108) and Issues (#360) can be done
in all comments, this can be a great tool for clarity.


Thank you for your attention, I am looking forward to your comments!

Regards,


[1] https://heptapod.net

   Heptapod is itself using hg-git, but that entails a limited subset of
features only, essentially what `hg gexport` does.

[2] https://foss.heptapod.net/mercurial/hg-git/-/issues

[3] https://foss.heptapod.net/mercurial/hg-git/-/merge_requests

[4] The evolve and topic Mercurial extensions are needed, see for
instance https://heptapod.net/pages/quick-start-guide.html#quick-start-guide

--
Georges Racinet
https://octobus.net, https://about.heptapod.host, https://heptapod.net
GPG: BF5456F4DC625443849B6E58EE20CA44EF691D39, sur serveurs publics

[Dan Villiom Podlaski Christiansen]

Hi again,

So, since last time a lot of things happened, but we still have a
problem with a review policy that just doesn't match the resources
available. It would be nice if every single change could get a thorough
review, but that requires at least two active contributors. In reality,
we have one — me — and helpful people who try to keep up, but just don't
have the time. I don't mean to show any disrespect or ingratitude, and I
know people have good reasons to be busy, but that's just the way it is.

So, I'd like to propose a change to allow the project to move forward:

- Make me the sole maintainer.

This might sound a bit drastic, but it's worth pointing out that our
main outside dependency, Dulwich, has worked this way for years. And we
just don't have a team of contributors available, sadly. We also have a
rather long list of maintainers, some of whom have simply gone silent.
For about a year, I've been primary contributor of code. I hope people
would consider me worthy of the responsibility!

We have a rather thorough test suite that should catch the most obvious
mistakes, but what about changes to the user experience? In order to
strike a compromise between getting good changes in while stopping bad
changes, I'd like propose a policy of a one-month beta prior to any
feature release. This way, people have a reasonable timeframe for
complaining, should I make a change that's too wild or invasive 🙂
Review needn't disappear completely, it'd just be another tool in the
box when needed.

Now, what's the motivation for this?

This is not only about getting reviews unstuck, but also about something
deeper: One challenge with hg-git is that the codebase is rather
unwieldy, with most complicated code contained within one very large
file, `git_handler.py`. This Python module has a class that both handles
state and storage, conversion details, interacting with both Dulwich and
Mercurial. In short: it does waaaay too much. The code is ripe for a
refactoring. We've done some of this, but more is needed.

We have outstanding pull requests that add many useful things, for example:

- Allow creating obsolescence markers on pull.
- Transfer Git metadata between Mercurial clones.
- Allow using branches or topics instead of bookmarks.

Some of these are relatively straightforward, but just add yet more
complexity to an already overcomplicated codebase. Others run into
issues with where we store files. In particular, where we store our
metadata necessitates that we lock the working directory on both pull
and push. That's not only annoying, but also a problem when transferring
it. In order to address this, and allow some form of upgrading, we need
some abstraction or decoupling.

Currently, I'd rather not start that refactor, as the effort might very
well just stall. Again.

So, if there's agreement on this, my plan is as follows:

- First, merge the least invasive changes into default, along with some
other tweaks to make pushing safer and more consistent with Mercurial.
- Cut a beta release, and await feedback.
- Start thinking about how to refactor the code.

(For one thing, this would allow me to answer a question recently posted
on our issue tracker: When is the next release?)

Now, this isn't the only way forward, of course. One possible
alternative is to significantly lower the standard of review for most
changes.

Thoughts?

- Dan

[Uwe Brauer]

>>> "DVPC" == Dan Villiom Podlaski Christiansen <dan...@gmail.com> writes:

Hi

> Hi again,
> So, since last time a lot of things happened, but we still have a
> problem with a review policy that just doesn't match the resources
> available. It would be nice if every single change could get a thorough
> review, but that requires at least two active contributors. In reality,
> we have one — me — and helpful people who try to keep up, but just don't
> have the time. I don't mean to show any disrespect or ingratitude, and I
> know people have good reasons to be busy, but that's just the way it is.

I don't contribute, unfortunately, to the code base, and I only did in
the past testing on two changesets (bc1841341adb and e8045105764b
provided by Manuel Jacob), which have been largely forgotten[1], although I
am using them on a daily base when I deal with git repositories and need
to use named-branches.

Since I have to deal with at least the git converted repositories I
would be very interested in beta testing.

> So, I'd like to propose a change to allow the project to move forward:

> - Make me the sole maintainer.

I would also like to add that I really appreciate Dan's effort in the
last month and it is sad to notice that he is in fact, as far as I can
tell using «hg log» and «hg churn», the main contributer and the only
contributer in the last time.

That is why his proposal makes a lot of sense to me, although I don't
have a say there

> We have outstanding pull requests that add many useful things, for example:

> - Allow creating obsolescence markers on pull.
> - Transfer Git metadata between Mercurial clones.
> - Allow using branches or topics instead of bookmarks.

I find the last issue the most interesting and important for my
workflow, I know that there is already a topic branch on it, and I hope
to give it a try next week (I am still struggling using a python3
compatible mercurial version since I use very much the artemis extension
(for issues) which only works with python 2.7 😓 )


Regards

Uwe Brauer

Footnotes:
[1] it seems to me that Manuel has not been very active lately.

[Georges Racinet]

Hi Dan,

On 9/18/21 4:36 PM, Dan Villiom Podlaski Christiansen wrote:
> Hi again,
>
> So, since last time a lot of things happened, but we still have a
> problem with a review policy that just doesn't match the resources
> available. It would be nice if every single change could get a thorough
> review, but that requires at least two active contributors. In reality,
> we have one — me — and helpful people who try to keep up, but just don't
> have the time. I don't mean to show any disrespect or ingratitude, and I
> know people have good reasons to be busy, but that's just the way it is.
>
> So, I'd like to propose a change to allow the project to move forward:
>
> - Make me the sole maintainer.
>
> This might sound a bit drastic, but it's worth pointing out that our
> main outside dependency, Dulwich, has worked this way for years. And we
> just don't have a team of contributors available, sadly. We also have a
> rather long list of maintainers, some of whom have simply gone silent.
> For about a year, I've been primary contributor of code. I hope people
> would consider me worthy of the responsibility!

I certainly would consider that you are worthy and also that you've
earned it, obviously. I'd like to see what other people would think of
it, though.

But what would be the consequence in terms of process, especially
(quoting you out-of-order) compared to:

> Now, this isn't the only way forward, of course. One possible
> alternative is to significantly lower the standard of review for most
> changes.

Would it be simply that you'd allow yourself to merge changes and that
you'd be the only one with that kind of power?

> We have a rather thorough test suite that should catch the most obvious
> mistakes, but what about changes to the user experience? In order to
> strike a compromise between getting good changes in while stopping bad
> changes, I'd like propose a policy of a one-month beta prior to any
> feature release. This way, people have a reasonable timeframe for
> complaining, should I make a change that's too wild or invasive 🙂
> Review needn't disappear completely, it'd just be another tool in the
> box when needed.
>
> Now, what's the motivation for this?
>
> This is not only about getting reviews unstuck, but also about something
> deeper: One challenge with hg-git is that the codebase is rather
> unwieldy, with most complicated code contained within one very large
> file, `git_handler.py`. This Python module has a class that both handles
> state and storage, conversion details, interacting with both Dulwich and
> Mercurial. In short: it does waaaay too much. The code is ripe for a
> refactoring. We've done some of this, but more is needed.

This is putting it in a polite way.

I'm a bit wary of the consequences on the (parts of) Heptapod that are
downstream of hg-git (otherwise I'd done it a long time ago), but I
agree this has to be done eventually, so…

Note that in general review is great, but I don't believe in cutting big
and invasing refactorings in small changesets to make them more
reviewable. Too much of the intent gets lost, and it can make amendments
just too expensive both for the author and the reviewers. Of course,
this is just a personal opinion, but I like to practice what I preach…
On that particular topic, I'd suggest that you come up with something
you find appropriate, and then I'll tell you if it creates problems
downstream and what would be needed to overcome them. This would be way
easier, precisely because the current code base is hard to understand
and I believe yours would be much clearer. You can be sure I'd listen if
you explain the plan in broad strokes before committing serious time to it.

This is assuming hg-git should care about Heptapod, but since it is
hosted on foss.heptapod.net, I suppose it should.

>
> We have outstanding pull requests that add many useful things, for example:
>
> - Allow creating obsolescence markers on pull.
> - Transfer Git metadata between Mercurial clones.
> - Allow using branches or topics instead of bookmarks.
>
> Some of these are relatively straightforward, but just add yet more
> complexity to an already overcomplicated codebase. Others run into
> issues with where we store files. In particular, where we store our
> metadata necessitates that we lock the working directory on both pull
> and push. That's not only annoying, but also a problem when transferring
> it. In order to address this, and allow some form of upgrading, we need
> some abstraction or decoupling.
>
> Currently, I'd rather not start that refactor, as the effort might very
> well just stall. Again.
>
> So, if there's agreement on this, my plan is as follows:
>
> - First, merge the least invasive changes into default, along with some
> other tweaks to make pushing safer and more consistent with Mercurial.
> - Cut a beta release, and await feedback.
> - Start thinking about how to refactor the code.
>
> (For one thing, this would allow me to answer a question recently posted
> on our issue tracker: When is the next release?)

Whether hg-git ends up with you as a single maintainer or not, I'm +1 on
that plan.


Best,

[Dan Villiom Podlaski Christiansen]

Apologies for the late response; I haven't had much time to do hg-git development lately.

Sure, yeah! The end result is mostly the same, so that's fine 🙂

Well, Heptapod is pretty nice in my opinion, so I for one do care 🙂 After thinking about this a bit more, the closest thing to I plan is to approach incoming first. The current “discovery” phase in hg-git is quite limited. On a high level, `hg incoming` should have everything available to it: bookmark, phase, branch, and so on. Everything except file data and Mercurial node IDs, essentially. I think attacking that before state is easiest.

This change seems likely to break Heptapod, but I don't think there's a way around that.

>> We have outstanding pull requests that add many useful things, for example:
>>
>> - Allow creating obsolescence markers on pull.
>> - Transfer Git metadata between Mercurial clones.
>> - Allow using branches or topics instead of bookmarks.
>>
>> Some of these are relatively straightforward, but just add yet more
>> complexity to an already overcomplicated codebase. Others run into
>> issues with where we store files. In particular, where we store our
>> metadata necessitates that we lock the working directory on both pull
>> and push. That's not only annoying, but also a problem when transferring
>> it. In order to address this, and allow some form of upgrading, we need
>> some abstraction or decoupling.
>>
>> Currently, I'd rather not start that refactor, as the effort might very
>> well just stall. Again.
>>
>> So, if there's agreement on this, my plan is as follows:
>>
>> - First, merge the least invasive changes into default, along with some
>> other tweaks to make pushing safer and more consistent with Mercurial.
>> - Cut a beta release, and await feedback.
>> - Start thinking about how to refactor the code.
>>
>> (For one thing, this would allow me to answer a question recently posted
>> on our issue tracker: When is the next release?)
>
> Whether hg-git ends up with you as a single maintainer or not, I'm +1 on
> that plan.

In the short term, I'd like to merge some of the topics/MRs that go into the `testing` branch. I'd say these are safe:

* incremental-phases
* initial-clone
* less-caching
* more-helpful-output

I think these are a bit more controversial:

* branches-and-stuff (i.e. support for branches & topics)
* replacements (rebase detection)
* transfer-git-state (i.e. mercurial-to-mercurial metadata)

I'm a bit torn about whether the UI changes are to invasive and unstable with these. What do you think?

--

Dan Villiom Podlaski Christiansen
dan...@gmail.com — +45 2728 9771

[Pierre-Yves David]

I am definitely in the group of "people who tries to follow but are
lacking the time". The second half of 2021 have been definitely crazy on
my side. Sorry for the long delay.

I definitely support the idea of simplifying the review process and
allow you to push change without waiting for other review. I am trusting
you for doing the right things in the right situation. The main thing
hg-git needs is forward-progress, but over-process.

Feel free to keep pinging people for part where you need another pairs
of eyes. I should be more available than in the past handful of month.

I am especially interested in the phase and rebase MRs and should be
able to get back to them soon, hoping to land them by the end of the week.

Cheers,

--
Pierre-Yves David

[Dan Villiom Podlaski Christiansen]

Thank you for replying. I've merged almost all the MRs I want in the next release. One of the outstanding ones reformats the codebase with Black — would that cause problems for you?

Anyways, I plan on cutting a beta release either once I've resolved the remaining issues. I've settled on calling it 1.0b1.

Regarding the other MRs I mentioned below — support for branches, topics and transferring state — that sounds like something that'd make it 2.0 to me 🙂