Performance issue with cloning

[sandro...@gmail.com]

Hi there

I have a number of rather large repos and would like to improve their cloning performance.

Why? Because our build servers clone the repos for every build. And I have to wait for the build results...

I tried cloning a repo with a 400MB .git folder on several windows clients using msysgit 1.8.1.2 and it took around 5 to 6 minutes. The thing is that a) cloning the same repo (from the same server) with a mac client takes around 30 seconds and b) both CPU and network load on the windows clients are below 1% while cloning. That makes me say that there must be some sort of bottle neck and probably it can be solved?

I tried cloning on different disc types (NTFS / FAT32 / exFAT) but that didn't make much difference. Instead I'm guessing it could have something to do with SSH but my knowledge with that is somewhat limited so I'd like to ask if anyone out there might have an idea what could cause this behavior or any suggestions how to perform in order to narrow down the problem?

Thx & kind regards

[sandro...@gmail.com]

I found this solution: http://darrell.mozingo.net/2011/09/29/painfully-slow-clone-speeds-with-msysgit-gitextensions/

It helps bringing cloning times down to around 1:30 instead of 6:00. But:

a) I still don't see what is limiting the cloning - neither CPU nor network bandwith nor disc usage is coming to 100%. Can I improve my cloning times beyond 1:30?

b) Could someone improve MsysGit to use an SSH version with the same bandwidth as the one from CygWin? I don't see the point in having everyone do the replacement described above.

[Konstantin Khomoutov]

On Mon, 29 Apr 2013 00:32:04 -0700 (PDT)
sandro...@gmail.com wrote:

> I found this solution:
> http://darrell.mozingo.net/2011/09/29/painfully-slow-clone-speeds-with-msysgit-gitextensions/
>
> It helps bringing cloning times down to around 1:30 instead of 6:00.
>
> But:
>
> a) I still don't see what is limiting the cloning - neither CPU nor
> network bandwith nor disc usage is coming to 100%. Can I improve my
> cloning times beyond 1:30?

This might relate to [2].

> b) Could someone improve MsysGit to use an SSH version with the same
> bandwidth as the one from CygWin? I don't see the point in having
> everyone do the replacement described above.

Yes, you could try to update msysGit to fetch and build that newer
OpenSSH version (and probably the HPN patchset [2] as well) and open a
pull request (or send a patch to this mailing list).
See [1] for a starter.

1. https://github.com/msysgit/msysgit/wiki#how-to-participate
2. http://www.psc.edu/index.php/hpn-ssh

[Ciaran]

Also, please see the work that we've tried to do here: 

 https://github.com/msysgit/msysgit/issues/31

but I ran into plenty of problems that I was lacking in enough skill to resolve :( 

- Cj 

[Konstantin Khomoutov]

On Mon, 29 Apr 2013 12:42:46 +0100
Ciaran <cia...@gmail.com> wrote:

[...]

> Also, please see the work that we've tried to do here:
>
> https://github.com/msysgit/msysgit/issues/31
>
> but I ran into plenty of problems that I was lacking in enough skill
> to resolve :(

[1] looks interesting in particular. The OP might try to build Git for
Windows using MingwDevEnv to solve the issue at hand (?)

1. https://github.com/msysgit/msysgit/issues/31#issuecomment-13537020

[sandro...@gmail.com]

I have found this pull request from the Git Dev Env repository that seems to incorporate a new package using OpenSSH6.1 with the HPN patches. I'm a bit confused about GitDevEnv and Git and where what package is supposed to go. Is this what we need or not? Or is it at least a good template?

I have been struggling around with building OpenSSH6.1 from scratch myself but this seems to be an endless journey from one compiler error to the next without warranty that it'll work in the end. But I'm completely new to all of this (living in the .NET world myself) so maybe it's just me not having enough knowhow.

[sandro...@gmail.com]

Sorry forgot to include the link:

https://github.com/sschuberth/mingwGitDevEnv/pull/5

[Sebastian Schuberth]

On 01.05.2013 10:26, sandro...@gmail.com wrote:

> I have found this pull request from the Git Dev Env repository that
> seems to incorporate a new package using OpenSSH6.1 with the HPN
> patches. I'm a bit confused about GitDevEnv and Git and where what
> package is supposed to go. Is this what we need or not? Or is it at
> least a good template?

The mingwGitDevEnv project provides the environment to build the Git for
Windows installer. Inside that environment we already have OpenSSH 6.1
incl. HP patches (as well as severl other updated MSYS packages) as we
use the new mgwport packaging system of upstream MSYS. However, there is
not Git for Windows installer based on these new developments yet.

--
Sebastian Schuberth