Git 10x slower on Windows 7 versus Ubuntu Linux
David K
running a Windows 7 host OS and an Ubuntu 10 guest VM. I have a git
repository with 85000 files.
Under Windows 7, git status takes 15s.
Under the Ubuntu VM, git status takes 1s.
Why is it so much slower under Windows 7. Is this a known issue? Is
there a planned fixed?
Thanks.
Sebastian Schuberth
> Why is it so much slower under Windows 7. Is this a known issue? Is
> there a planned fixed?
It's a known issue. Git was written against the Linux / Posix API (and
uses several Shell and Perl scripts). So some emulation is required to
make (almost) the same code compile and run on Windows. This emulation
is provided by MSYS, a Cygwin fork. Routing all sorts of functions calls
through the emulation layer is costly.
Moreover, Git is heavily tuned towards filesystems that can deal with
touching a lot of small files quickly. But unfortunately, Windows' NTFS
performs very badly when touching many small files.
--
Sebastian Schuberth
Johannes Schindelin
Note that there have been ideas floating about to help both the dependence
on POSIX APIs (and shell scripts, i.e. starting quite a bunch of processes
where in theory only one would be needed) and the problem with NTFS' not
being tuned to access timestamps of a lot of files.
But it takes somebody who needs to see the issues resolved to implement
the ideas. It is just too big a task for developers who do not have the
problem and want to be nice to others.
Ciao,
Dscho
Erik Faye-Lund
<sschu...@gmail.com> wrote:
> On 07.03.2011 21:23, David K wrote:
>
>> Why is it so much slower under Windows 7. Is this a known issue? Is
>> there a planned fixed?
>
> It's a known issue. Git was written against the Linux / Posix API (and uses
> several Shell and Perl scripts). So some emulation is required to make
> (almost) the same code compile and run on Windows. This emulation is
> provided by MSYS, a Cygwin fork. Routing all sorts of functions calls
> through the emulation layer is costly.
>
git status is a built-in, which does not use the msys.dll. So this
part most likely isn't the issue at hand here.
> Moreover, Git is heavily tuned towards filesystems that can deal with
> touching a lot of small files quickly. But unfortunately, Windows' NTFS
> performs very badly when touching many small files.
This however, is a real problem. I don't know how much we can do about
this, though.
But it seems the problem for David is UAC Virtualization (he responded
that disabling it entirely helped a lot on the performance). I've
written a patch for that, and if it helps we'll probably apply some
version of it.
Johannes Schindelin
On Wed, 9 Mar 2011, Erik Faye-Lund wrote:
> On Wed, Mar 9, 2011 at 9:01 AM, Sebastian Schuberth
> <sschu...@gmail.com> wrote:
> > On 07.03.2011 21:23, David K wrote:
> >
> >> Why is it so much slower under Windows 7. Is this a known issue? Is
> >> there a planned fixed?
> >
> > It's a known issue. Git was written against the Linux / Posix API (and
> > uses several Shell and Perl scripts). So some emulation is required to
> > make (almost) the same code compile and run on Windows. This emulation
> > is provided by MSYS, a Cygwin fork. Routing all sorts of functions
> > calls through the emulation layer is costly.
>
> git status is a built-in, which does not use the msys.dll. So this part
> most likely isn't the issue at hand here.
In a sense, it is, since we have a couple of functions implementing
lacking POSIX functionality in compat/mingw.[ch] (such as a fast stat()
emulation).
> > Moreover, Git is heavily tuned towards filesystems that can deal with
> > touching a lot of small files quickly. But unfortunately, Windows'
> > NTFS performs very badly when touching many small files.
>
> This however, is a real problem. I don't know how much we can do about
> this, though.
There was the idea to have a file monitor implemented as a service that
keeps track of modified files efficiently (i.e. as a callback to the
kernel rather than testing the timestamps and inodes and device numbers
all the time).
> But it seems the problem for David is UAC Virtualization (he responded
> that disabling it entirely helped a lot on the performance). I've
> written a patch for that, and if it helps we'll probably apply some
> version of it.
IIRC there were others who did not have UAC enabled but a metric ton of
files, and the only thing that helped was assume-unchanged.
Ciao,
Dscho
Erik Faye-Lund
<Johannes....@gmx.de> wrote:
> Hi,
>
> On Wed, 9 Mar 2011, Erik Faye-Lund wrote:
>
>> On Wed, Mar 9, 2011 at 9:01 AM, Sebastian Schuberth
>> <sschu...@gmail.com> wrote:
>> > On 07.03.2011 21:23, David K wrote:
>> >
>> >> Why is it so much slower under Windows 7. Is this a known issue? Is
>> >> there a planned fixed?
>> >
>> > It's a known issue. Git was written against the Linux / Posix API (and
>> > uses several Shell and Perl scripts). So some emulation is required to
>> > make (almost) the same code compile and run on Windows. This emulation
>> > is provided by MSYS, a Cygwin fork. Routing all sorts of functions
>> > calls through the emulation layer is costly.
>>
>> git status is a built-in, which does not use the msys.dll. So this part
>> most likely isn't the issue at hand here.
>
> In a sense, it is, since we have a couple of functions implementing
> lacking POSIX functionality in compat/mingw.[ch] (such as a fast stat()
> emulation).
>
He was quite clearly talking about the msys.dll. Yes, there's some
overhead in emulating some POSIX-functions, but the emulation-layer
itself isn't really the costly part here, it's the underlying
functionality. Windows is simply slower at some things than Linux (and
faster at some other things, but with Linus on board the Git team has
been good at avoiding those).
>> > Moreover, Git is heavily tuned towards filesystems that can deal with
>> > touching a lot of small files quickly. But unfortunately, Windows'
>> > NTFS performs very badly when touching many small files.
>>
>> This however, is a real problem. I don't know how much we can do about
>> this, though.
>
> There was the idea to have a file monitor implemented as a service that
> keeps track of modified files efficiently (i.e. as a callback to the
> kernel rather than testing the timestamps and inodes and device numbers
> all the time).
>
Hmm, interesting. Did anyone ever do any work on this?
>> But it seems the problem for David is UAC Virtualization (he responded
>> that disabling it entirely helped a lot on the performance). I've
>> written a patch for that, and if it helps we'll probably apply some
>> version of it.
>
> IIRC there were others who did not have UAC enabled but a metric ton of
> files, and the only thing that helped was assume-unchanged.
>
I'm less than convinced that disabling UAC also disables UAC
Virtualization. I don't know what reports you're talking about, so I
don't know if they actually tried to explicitly disable UAC
Virtualization or not. But if they didn't, I think it would be
interesting to see if it helped.
But it seems like UAC Virtualization is only a problem on some type of
set-ups. I have two different Win7 setups, one x64 and one x86, both
with UAC enabled. Neither of them have any performance issues like
what being reported. Git is just as fast as on my Vista x64 setup
(with UAC disabled, Vista doesn't have UAC Virtualization).
On the other hand, UAC Virtualization might turn out to be a red
herring. It's difficult for me to know without having access to a
system with the problem.
Sakari Jokinen
>>> > touching a lot of small files quickly. But unfortunately, Windows'
>>> > NTFS performs very badly when touching many small files.
Why mercurial status is faster then (60K files, Win 7, i7, ~1.3s vs. ~2.3s ) ?
I seem to remember some discussion, perhaps on this mailing list, about
using findFirstFile for stat. Did anything come out of it?
>> There was the idea to have a file monitor implemented as a service that
>> keeps track of modified files efficiently
> Hmm, interesting. Did anyone ever do any work on this?
Somebody tried using inotify on linux for monitoring the file system
with less than stellar outcome
http://thread.gmane.org/gmane.comp.version-control.git/151934
The inotify equivalent for windows would be I guess ReadDirectoryChangesW [1]
but I don't know whether this has been discussed for example on this
mailing list.
[1] http://msdn.microsoft.com/en-us/library/aa365465(v=VS.85).aspx
Sakari