How to work with Hg Branches and hg-git?

[John Volkar]

Our standard Hg workflow heavily relies on true Hg Branches (not Bookmarks).  We do not allow multiple heads, we do one head per branch.   We do not use Hg's 'default' branch at all, we start our repos with the first commit creating a 'dev' branch.   Short quick fixes take place on 'dev' directly, bigger work that might take a while gets a branch named with the ticket number.   'dev'elop, 'int'egration, 'rel'ease, and 'prod' are our base branch names.  You can imagine the workflow is self explanatory.  We use one hosted Hg server instance as our 'central publishing repo' and all developers point at it.  (And yes, I will likely have a "why are all hg-git changesets 'draft' phase question in the future.)

Here is an example of branches that we would have in our Hg repo:

- dev

- dev/JIRA-123

- rel

- rel/JIRA-456

- prod

For the life of me, I do not know how to map out how to do this sensibly with hg-git.  I do not want to introduce Hg Bookmarks into our scheme, we are happy with Hg Branches just being 'real" branches, not this bookmarks stuff.

Is there any practical advice that anyone can offer?  Anything written anywhere to show me how I should think about this?   

If full roundtripping of branches (Hg/Git/Hg/Git/...) is not possible, I would be happy with just one Hg Branch (say 'dev') being sync'd with GitHub's 'main' and letting that be the only thing pushed/pulled.   Heck, I would be fine with a way for me to just manually map hg-branch-name = git-branch-name for a subset.

I have been tinkering and I cannot figure out how to get a workable scheme going (Hg 'default' keeps appearing) and ugh, this feel so close but yet so far there is no joy.

[Uwe Brauer]

Hm I am not entirely sure I understand you workflow.

So you start with a hg repository which posses named branches, and you
want to push that to a git repository?
That is maybe off topic, but why not try to use Helix or heptapod?

Anyhow, do all users that push/pull use mercurial or some use git?

That is important to know, because I have some experience with this
workflow, if

1. The repository was created with hg,

2. All users use mercurial and hg-git

In that case a patch[1] provided by Manuel Jacob might be helpful
namely change set 6942fb0a8c9d and cd663e6379f0

@ changeset: 1616:6942fb0a8c9d
│ Branch: default
│ Author: Manuel Jacob <m...@manueljacob.de>
│ Date: Sat, 01 Jun 2019 16:54:53 +0200
│ Topic: export-additional-refs
│ Phase: draft
│ Summary: git_handler: add mechanism to specify revsets to be exported to Git refs according to a template
│
○ changeset: 1615:cd663e6379f0
│ Branch: default
│ Author: Manuel Jacob <m...@manueljacob.de>
│ Date: Sat, 01 Jun 2019 16:55:10 +0200
│ Topic: same-ref-for-multiple-changesets
│ Phase: draft
│ Summary: git_handler: make sure that the same ref is not used for multiple changesets
│

You need to configure that carefully. If you need some advice I think I
could help a but, however, be warned that is more than 2 years old.

There is another more promising approach provided by
Dan Villiom Podlaski Christiansen

1874:4b51ff13e8f5

changeset: 1874:4b51ff13e8f5
│ Branch: default
│ Author: Dan Villiom Podlaski Christiansen <dan...@gmail.com>
│ Date: Sat, 08 May 2021 19:45:01 +0200
│ Topic: branches-and-stuff
│ Phase: draft
│ Summary: modes: add support for topics
│
○ changeset: 1873:e3fde5eb86e8
│ Branch: default
│ Author: Dan Villiom Podlaski Christiansen <dan...@gmail.com>
│ Date: Fri, 09 Jul 2021 20:49:31 +0200
│ Topic: branches-and-stuff
│ Phase: draft
│ Summary: modes: use an enum
│
○ changeset: 1872:cf5377553bb3
│ Branch: default
│ Author: Dan Villiom Podlaski Christiansen <dan...@gmail.com>
│ Date: Sat, 08 May 2021 16:45:51 +0200
│ Topic: branches-and-stuff
│ Phase: draft
│ Summary: pull: add support for branch mode


I, regrettably, had no time to test it, but it is on my TODO list.

Regards

Uwe Brauer

Footnotes:
[1] that never was merged into default

[John Volkar]

>  So you start with a hg repository which posses named branches,
> and you want to push that to a git repository?

Yes.

> do all users that push/pull use mercurial or some use git?

All users will use Hg as the common practice, but it is possible that there will be occasional isolated changes made in the Git repo that would need to come back out.  (No new branches created in Git, no significant development in Git.)

> why not try to use Helix or heptapod?

We have a new corporate imposed GitHub Enterprise environment to which we will be have to maintain/publish our product software.  So Helix or Heptapod are not an option. (There will be security scanning, secrets management, and a variety of corporate workflow actions, etc.) 

I have a need for a "reasonably narrow" integration with a git repo, I was hoping that hg-git was the ticket, and it seems like it should be, if I can figure out how to get a semi-reasonable branch handling figured out.  I'd be happy with a single named branch in Hg to a single named branch in Git that round trips.  

I guess I could try one of the patches?  I am a bit hesitant go down this route, but I don't think I have many options to try.  (I guess I can make it an interns job to manually do the Hg/Git/Hg/Git dance forever, but ick.)

Aside from hg-git, there is the other 'git' extension, I didn't look at it much as hg-git sounded more full featured.   As always, suggestions are welcome.

Regards

[Uwe Brauer]

>>> "JV" == John Volkar <john....@gmail.com> writes:

>> So you start with a hg repository which posses named branches,
>> and you want to push that to a git repository?
> Yes.

>> do all users that push/pull use mercurial or some use git?
> All users will use Hg as the common practice, but it is possible that there
> will be occasional isolated changes made in the Git repo that would need to
> come back out. (No new branches created in Git, no significant development
> in Git.)

Good and bad. See below

> I have a need for a "reasonably narrow" integration with a git repo, I was
> hoping that hg-git was the ticket, and it seems like it should be, if I can
> figure out how to get a semi-reasonable branch handling figured out. I'd
> be happy with a single named branch in Hg to a single named branch in Git
> that round trips.

> I guess I could try one of the patches? I am a bit hesitant go down this
> route, but I don't think I have many options to try. (I guess I can make
> it an interns job to manually do the Hg/Git/Hg/Git dance forever, but ick.)

I think the more recent patch (topic/branch) by Dan is the better
choice, since he is quite an active developer. Manuel Jacob seems too
busy to maintain his patches.

That being said, some comments on Manuel Jacob's patches

I now use them only for git repositories (I used them for a while in
bitbucket when bitbucket, regrettably discontinued its mercurial
support, but then I found helix and stopped using bitbucket).

GNU emacs and most packages are using git, Xemacs is a notable exception
(and was one of the reasons for me to get acquainted with HG), but
Xemacs is almost dead, so that does not count much.

Here is my experience with Manuel's patches

1. They work quite well if *everybody* uses mercurial.

a. A named branch with some changesets that is pushed to a Github
repository and then pulled from another hg user, comes trough
smoothly.

b. One word of warning: you said you almost never use the default
branch, in that case you need to configure the functionality
provided by that patch quite carefully (because the default
branch is mapped to the master branch in git). That setting can
be changed but you need to run some tests

2. The problem starts if git users are involved.

a. If the users creates a new git branch, that branches ends up
in a mercurial repository as a bookmark on the default branch.

b. If the user commits a change to a named branch, then I am not
entirely sure, I think it also ends up as a bookmark on the
default branch. In my use cases (matlab-emacs) I tell every
git user best not to commit to my branches.

3. I think Dan's branch allows to pull git branches into named
branches, I am not sure though.

4. Phases. Secret phases at least in hg-git I use do not work, so
you should be careful with them, especially with Manuel patch. In
my understanding secret phases will be added soon to hg-git in
the stable branch.

> Aside from hg-git, there is the other 'git' extension, I didn't look at it
> much as hg-git sounded more full featured. As always, suggestions are
> welcome.

There was one but as far as I remember it was given up in favor of hg-git.
There are converters of course but you want it as a two way extension
and that is much harder, so in a nutshell there is only hg-git

Regards

Uwe

[Dan Villiom Podlaski Christiansen]

First off let me respond to the easiest question: Can you mirror a fixed set of Mercurial branches to Github, with the released hg-git?

Yes, but with a bit of configuration and manual setup. The first step is to set up a repository and server somewhere to do manual mirroring, either adding a `post-pull` hook to force the relevant bookmarks, or do everything in a script. Then, you'd add a regular cronjob to pull from Mercurial and push to Git. I use the following script for mirroring[1] hg-git itself to Github:

>>> hg book -q | xargs hg book -d
>>> hg branches -q | xargs -I X hg book -r X X-branch
>>> hg book --rename default-branch master-branch
>>> hg push --config git.branch_bookmark_suffix=-branch --force git
>>> hg book -q | xargs hg book -d


One thing you will eventually run into: Git stores refs as files and directories, so it is _inherently_ incapable of having both `dev` and `dev/whatever`. You might want to either rename your branches now, only mirror certain branches or do further processing in the script.

--

Concerning longer term options, I've been toying with adding two new “modes” to hg-git, where rather than bookmarks, the source of refs would either be topics or branches. In addition to that, there's also another merge request that adds the ability to transfer the relationship to Git when pushing and pulling from Mercurial repositories. Both sets of changes are highly experimental and untested, but they just might work for you. I haven't decided on whether they are safe for inclusion in the next release, though. That depends in part on whether they actually work in practice, which is a bit of a chicken-and-egg problem.

As a final note, please note that hg-git does have some support for phases: In particular, you can set certain refs to be protected, or remote HEAD and tags.

For your testing, please consider using the default branch here from Heptapod, or a topic/MR that's based on it. Quite a lot has changed since the last release 🙂

[1] https://github.com/danchr/hg-git — please note that the script runs on FreeBSD.

--

Dan Villiom Podlaski Christiansen
dan...@gmail.com — +45 2728 9771

[John Volkar]

> Yes, but with a bit of configuration and manual setup.

I would rather not have to have a separate mirroring server (with hooks or chron job) but I suppose that can be done if it is the simplest/best/safest way that 100% works and is fairly idiot proof? 

> Git stores refs as files and directories, so it is _inherently_ incapable

Okay, thank you for the heads up.  Sigh, so "dev/JIRA-123" was a 8y old poor naming convention then.  (All of our branches are named like that, and the repo is yes, 8 years old.)  This is going to be a problem, I guess I'll need to hack

I am willing to be a test monkey for the chicken and egg problem.  I need to think how deep I want to get into this.  I don't want to have to learn the guts of hg-git though, I just want to be a user (sorry!).  I cannot really take the time to learn the codebase and get into writing and debugging (simply too much work on my plate and this need to push to corporate GitHub is just one minor annoyance that I am trying to proactively solve in my "spare time".)

So... I take the 'branch/default' branch from Heptapod (last touched 3mo ago?), try to apply !101 (Allow converting branches or topics rather than bookmarks) as well as ...what?  I guess I am a bit lost as to what to try taking for a spin.

John

[Uwe Brauer]

>>> "JV" == John Volkar <john....@gmail.com> writes:

>> Yes, but with a bit of configuration and manual setup.
> I would rather not have to have a separate mirroring server (with hooks or
> chron job) but I suppose that can be done if it is the simplest/best/safest
> way that 100% works and is fairly idiot proof?

>> Git stores refs as files and directories, so it is _inherently_ incapable
> Okay, thank you for the heads up. Sigh, so "dev/JIRA-123" was a 8y old
> poor naming convention then. (All of our branches are named like that, and
> the repo is yes, 8 years old.) This is going to be a problem, I guess I'll
> need to hack

Maybe you know this already, but if you use the evolve extension, you
can «rename named branches»

Here is an example

hg init
echo "First" > test.org
hg add test.org
hg commit -u "Bernhard Riemann <bernhard...@gmail.com>" -m "First"
echo "Second" >> test.org
hg commit -u "Bernhard Riemann <bernhard...@gmail.com>" -m "Second"
echo "Third" >> test.org
hg commit -u "Bernhard Riemann <bernhard...@gmail.com>" -m "Third"
echo "Forth" >> test.org
hg commit -m "Fourth"
hg up 0
hg branch feature
hg amend
hg evolve

[John Volkar]

Yes, when I said "I'll need to hack" I mostly meant "I'll have to fuss and bother".  Our branch naming pattern is written into documentation, including formal Work Instructions and Procedures/Policies.  (I work in a regulated industry and the admin/management/governance side of things often feels like coding with handcuffs on.)  I was aware that Hg evolve existed but had never used it.   Good to know that the "\" in branch names can be remediated though.

I now am also letting my brain process the impact of Phases now and it is making my head hurt.   We (a group of ~30) push to a "central" Hg publishing repo that sets all commits to phase = public.  Given that only a handful of people would interact with GitHub via hg-git, they will have public commits that need to go to GitHub... where I guess no one had better try to rebase or touch history... Ugh.  

I think I need to block out a chunk of time, plan out a set of mini test repos and workflows and just setup and try it all.

[Dan Villiom Podlaski Christiansen]

On 4 Nov 2021, at 14.49, John Volkar <john....@gmail.com> wrote:

> Yes, when I said "I'll need to hack" I mostly meant "I'll have to fuss and bother". Our branch naming pattern is written into documentation, including formal Work Instructions and Procedures/Policies. (I work in a regulated industry and the admin/management/governance side of things often feels like coding with handcuffs on.) I was aware that Hg evolve existed but had never used it. Good to know that the "\" in branch names can be remediated though.

Please note that you'd only need to _rename_ the branches, i.e. create new ones and close the old. That's one advantage, of sorts, to Git's simplistic refs: Closing a “branch” destroys all knowledge that a commit used to belong to it. So the branch mode will simply never tell Git about closed ones.

As an example you could do change `dev` to `development`, or similar, and keep all the other branches using the same name.

> I now am also letting my brain process the impact of Phases now and it is making my head hurt. We (a group of ~30) push to a "central" Hg publishing repo that sets all commits to phase = public. Given that only a handful of people would interact with GitHub via hg-git, they will have public commits that need to go to GitHub... where I guess no one had better try to rebase or touch history... Ugh.

Git itself has no notion of “public” vs. “draft”, or protected changesets, for that matter. But Github does — a branch protection rule can prevent force pushing, deletion or mandate Pull Requests and approvals. Hg-Git has no knowledge of such rules, and currently doesn't publish anything by default.[1] (That might change in the future, as I'm contemplating changing the defaults to enabling publishing of remote HEAD and tags when pushing over the network. Even in this case, you could disable phases entirely.)

In the off chance that someone actually does something weird in Git, you can force push from Mercurial and overwrite it.

> I think I need to block out a chunk of time, plan out a set of mini test repos and workflows and just setup and try it all.

Generally speaking, there are two levels of difficulty here:

1) Just mirror your current repositories to Github.
2) Allow your developers to actively interact with other people using Git/Github.

The first one is reasonably easy. The second one — especially in a branch based workflow — is where all the edge cases come into play, and I'm not certain that it's possible at the moment. Actually, I'd consider enabling it a long term goal of the project 🙂

> --
> You received this message because you are subscribed to the Google Groups "hg-git" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to hg-git+un...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/hg-git/102c09ac-dcdb-4231-8562-37af5e45bdfen%40googlegroups.com.

[John Volkar]

I wrote a long message, thought a bit, deleted it, and now am thinking somewhat differently about my direction.   

I am now thinking that I'll just install git and have both a .hg and .git in my local project directory.  I will use Hg (and TortoiseHg) as I do normally.  I will use command line git normally.  I will 98% work in Hg with branches as I like, and very occasionally, use git to push/pull one branch and handle "re-comitting" any inbound changes in Hg.   This feels simple and with less ceremony for the basic stuff I want to do.  I don't see any reason that my local working dir cannot have two repos rooted in it.  I have to manually maintain both, but for my very limited need to push stuff into GitHub for some corporate integrations, this seems simple and brute force.

Any advice to the contrary here?  (Sorry, I wish I could use hg-git but it seems like more effort than I can spend to get it to where I would need to be.  Parallel use of hg and git in one directory is more manual, but also feels simpler?)   As always the devil is in the details, and I think this weekends experiment is to see if this works out.  If it doesn't, I'll turn back to hg-git and my original email draft.

John

[Dan Villiom Podlaski Christiansen]

It seems to me that you're essentially duplicating what hg-git does,
only in a more manual and error-prone way. If you're going to manage the
Mercurial->Git conversion manually, why not just use a bookmark? After
all, you'd simply:

1) Create one or more bookmarks
2) Push to Git

You could either push everything or just some of the bookmarks with
-r/--rev.

The features I talked about are part of a long-term plan of mine; I was
hoping you'd be interested, but if you just want something simple that
works _now_, I still think hg-git might be your best bet.

>> dan...@gmail.com — +45 2728 9771 <+45%2027%2028%2097%2071>
>>
>>
>