hg-git performance on large git repositories.

[Uwe Brauer]

Hi

I am starting to notice a problem when I access large git repositories.



1. I am one of the maintainers of matlab-mode. That is,
unfortunately a git repository, that I access via hg-git. It has
around 700 changesets and I have no problems with the performance
using hg-git

2. The same occurs with auctex. A git repository with around 7800
change sets, again no problem.

3. GNU emacs however (that was converted from CVS to Bazzarr to git
to hg) has 264610 changesets. There I do notice a siginificant
delay in almost every operation:
not only

a. hg branches but also

b. hg ci etc


I have also the evolve extension enabled, I disabled it, but it did not make any difference.


Does or did anybody notice something similar?

regards

Uwe Brauer

[Dan Villiom Podlaski Christiansen]

Den 19 Sep 2021 kl. 08.38 skrev Uwe Brauer <o...@mat.ucm.es>:

> Hi
>
> I am starting to notice a problem when I access large git repositories.

[snip]

> 3. GNU emacs however (that was converted from CVS to Bazzarr to git
> to hg) has 264610 changesets. There I do notice a siginificant
> delay in almost every operation:
> not only
>
> a. hg branches but also
>
> b. hg ci etc

hg-git shouldn't be doing anything particular w.r.t. committing. One possible cause is that you have _a_lot_ of draft commits. Try publishing `master` and see if it helps:

$ hg phase -p master

Alternatively, you can enable the `hggit.usephases` configuration setting and pull.

Which version of hg-git, Mercurial and Python are you running?

> I have also the evolve extension enabled, I disabled it, but it did not make any difference.
>
>
> Does or did anybody notice something similar?

I regularly use Mercurial with a repository[1] with ~188k commits and a ~44k files in total. I haven't noticed any particular slowness, apart from status and so on — which is to be expected with that many files.

[1] https://github.com/macports/macports-ports

--

Dan Villiom Podlaski Christiansen
dan...@gmail.com — +45 2728 9771

[Uwe Brauer]

>>> "DVPC" == Dan Villiom Podlaski Christiansen <dan...@gmail.com> writes: > >

> Den 19 Sep 2021 kl. 08.38 skrev Uwe Brauer <o...@mat.ucm.es>:
>> Hi
>>
>> I am starting to notice a problem when I access large git repositories.

> [snip]

>> 3. GNU emacs however (that was converted from CVS to Bazzarr to git
>> to hg) has 264610 changesets. There I do notice a siginificant
>> delay in almost every operation:
>> not only
>>
>> a. hg branches but also
>>
>> b. hg ci etc

> hg-git shouldn't be doing anything particular w.r.t. committing. One
> possible cause is that you have _a_lot_ of draft commits. Try
> publishing `master` and see if it helps:

> $ hg phase -p master

I did this,

hg status

Is faster but that is all. I add an empty line to a file, and
run hg up -C needed 10 sec.

I tried it again running

hg ci

Also 10 sec.

> Alternatively, you can enable the `hggit.usephases` configuration setting and pull.

Well I think I have that
because my setting is

[hggit]
usephases = True

> Which version of hg-git, Mercurial and Python are you running?

Well aehm (since I need the artemis extension which requires python 2.7)

Hg:
I use version 5.2rc0+20200125 (on Ubuntu 16.04)

Hg-git (I use a very particular change set committed by Manuel Jacob,
that allows named branches to be pushed as git branches, it does work
the other way around. We talked about it and I am keen to give your
implementation a try. I hoped to port artemis to python 3.X but I have
not very much time, and frankly don't use python that much.

So hg-git change set acc4280e2ab7
topic export-additional-refs

And of course also 3ad09bbcbe3e
which has a different topic namely
topic same-ref-for-multiple-changesets

Well that is I know more than 2 years old

That might be one reason.....

>> I have also the evolve extension enabled, I disabled it, but it did not make any difference.
>>
>>
>> Does or did anybody notice something similar?

> I regularly use Mercurial with a repository[1] with ~188k commits and
> a ~44k files in total. I haven't noticed any particular slowness,
> apart from status and so on — which is to be expected with that many
> files.

> [1] https://github.com/macports/macports-ports

Oh you are a mac user. If you are bored and have a lot of free time, I
would be very interested if you could clone the GNU emacs repository
(however that took I think 2 hours ahem) and tell me what is your
experience

git://git.savannah.gnu.org/emacs.git


Regards

Uwe

[Dan Villiom Podlaski Christiansen]

On 20 Sep 2021, at 14.24, Uwe Brauer <o...@mat.ucm.es> wrote:

>> Which version of hg-git, Mercurial and Python are you running?
>
> Well aehm (since I need the artemis extension which requires python 2.7)
>
> Hg:
> I use version 5.2rc0+20200125 (on Ubuntu 16.04)
>
> Hg-git (I use a very particular change set committed by Manuel Jacob,
> that allows named branches to be pushed as git branches, it does work
> the other way around. We talked about it and I am keen to give your
> implementation a try. I hoped to port artemis to python 3.X but I have
> not very much time, and frankly don't use python that much.
>
> So hg-git change set acc4280e2ab7
> topic export-additional-refs
>
> And of course also 3ad09bbcbe3e
> which has a different topic namely
> topic same-ref-for-multiple-changesets
>
> Well that is I know more than 2 years old
>
> That might be one reason.....

Well, all of those are either very old, unsupported, or both…

>
>>> I have also the evolve extension enabled, I disabled it, but it did not make any difference.
>>>
>>>
>>> Does or did anybody notice something similar?
>
>> I regularly use Mercurial with a repository[1] with ~188k commits and
>> a ~44k files in total. I haven't noticed any particular slowness,
>> apart from status and so on — which is to be expected with that many
>> files.
>
>> [1] https://github.com/macports/macports-ports
>
> Oh you are a mac user. If you are bored and have a lot of free time, I
> would be very interested if you could clone the GNU emacs repository
> (however that took I think 2 hours ahem) and tell me what is your
> experience
>
> git://git.savannah.gnu.org/emacs.git

On my Mac,[1] updating from `null` to `master` takes about 2 seconds with that repository.[2] Please note that stuff like updating depends on how many files and directories exist in the repository, even if untracked. Also note that you can check whether it's an issue with an extension by either:

1) Disabling extensions one-by-one using either something like `--config extensions.hggit=!` or editing hgrc.
2) Bypassing configuration entirely, using e.g. `HGRCPATH=/dev/null HGRCSKIPREPO=1 hg status`.

[1] A 2 GHz quad-core MBP from 2020.
[2] Mirrored at <https://villiom.dk/emacs> — although the server is somewhat underpowered and unstable.

--

Dan Villiom Podlaski Christiansen
dan...@gmail.com — +45 2728 9771

[Uwe Brauer]

>>> "DVPC" == Dan Villiom Podlaski Christiansen <dan...@gmail.com> writes:

Hi Dan

> On 20 Sep 2021, at 14.24, Uwe Brauer <o...@mat.ucm.es> wrote:
>>> Which version of hg-git, Mercurial and Python are you running?
>>
>> Well aehm (since I need the artemis extension which requires python 2.7)
>>
>> Hg:
>> I use version 5.2rc0+20200125 (on Ubuntu 16.04)
>>
>> Hg-git (I use a very particular change set committed by Manuel Jacob,
>> that allows named branches to be pushed as git branches, it does work
>> the other way around. We talked about it and I am keen to give your
>> implementation a try. I hoped to port artemis to python 3.X but I have
>> not very much time, and frankly don't use python that much.
>>
>> So hg-git change set acc4280e2ab7
>> topic export-additional-refs
>>
>> And of course also 3ad09bbcbe3e
>> which has a different topic namely
>> topic same-ref-for-multiple-changesets
>>
>> Well that is I know more than 2 years old
>>
>> That might be one reason.....

> Well, all of those are either very old, unsupported, or both…

> On my Mac,[1] updating from `null` to `master` takes about 2 seconds
> with that repository.[2] Please note that stuff like updating depends
> on how many files and directories exist in the repository, even if
> untracked. Also note that you can check whether it's an issue with an
> extension by either:

Aha I compiled the repo and therefore had a lot of untracked files I
deleted them and now navigation it is much faster!
However a commit still need 10 sec to come up

> 1) Disabling extensions one-by-one using either something like `--config extensions.hggit=!` or editing hgrc.

I am not sure I understand that

Do you mean

hg --config extensions.hggit=! status

That was a bit faster


Now

hg --config extensions.hggit=! status

Was still slow 8 sec.

hg --config extensions.hggit=! ci


(Emacs coming up) is 8 sec

hg --config extensions.hggit=! ci -m "Test"

Also 9 sec

hg --config extensions.hggit=! uncommit -a

8 sec

hg --config extensions.hggit=! prune .

1.2 sec

But hg uncommit -a (8sec)
hg prune . (1.2)

So no difference.

Do I understand you correctly that this implies the problem is caused by
my extensions but either by hg-git or mercurial?

BTW I only have

hggit = /home/oub/ALLES/src/hg-git-heptapod/hggit

In my .hgrc

I did not byte compile or anything like this.

> 2) Bypassing configuration entirely, using e.g. `HGRCPATH=/dev/null HGRCSKIPREPO=1 hg status`.

> [1] A 2 GHz quad-core MBP from 2020.

Ok I have a thinkpad X1 4th gen 4 quad-core

Dual core Intel Core i5-6200U (-HT-MCP-) speed/max: 499/2800 MHz

> [2] Mirrored at <https://villiom.dk/emacs> — although the server is somewhat underpowered and unstable.

I will try later on to upgrade mercurial and hg-git and also try out you
pull request for Artemisis (I finally figured out you to clone it)

Thanks for your help

Uwe