git-status performance
karste...@dcon.de
I saw the recent discussion on the list about slow git status, and with
unicode support finally done, I thought I might warm up a project that
I've been playing around with some time ago.
Basic idea is a cache that sits transparently below our lstat and readdir
implementations, but always reads entire directories (instead of
lstat'ting single files). The cache is kept up to date via
ReadDirectoryChangesW (there is no *A version, so I needed unicode support
first...).
The performance improvements of the cached version are quite impressive
for git status, even with core.preloadindex turned on. The numbers below
are for git status on the WebKit repo, normal git v1.7.9 vs. the cached
version, for different core.preloadindex and --untracked-files settings:
preload | -u | normal | cached | gain
--------+-----+--------+--------+------
false | all | 25.144 | 3.055 | 8.2
false | no | 22.822 | 1.748 | 12.8
true | all | 9.234 | 2.179 | 4.2
true | no | 6.833 | 0.955 | 7.2
Drawback of this approach is that git operations that modify the working
copy are actually slower with the cache, because e.g. git checkout
typically lstats every file immediately after creating it (so the cache is
always dirty). It may be better to activate the cache just for readonly
operations.
In case you wanna try it out, I've pushed the code to:
https://github.com/kblees/git/tree/kb/fscache-v0
However, beware that this actually predates the unicode patches: it is
pretty ugly and quite sparse on comments and error handling (but seems to
work mostly, at least the test suite doesn't complain that much...). I
just pulled the code to devel, fixed compile errors and added thread
synchronization to test with core.preloadindex=true.
Unless I hear violent objections from the list, I plan to polish this code
a bit and provide some patches that are probably worthy of inclusion...(no
promises on the time frame, though :-).
Cheers,
Karsten
Johannes Schindelin
On Tue, 14 Feb 2012, karste...@dcon.de wrote:
> Basic idea is a cache that sits transparently below our lstat and
> readdir implementations, but always reads entire directories (instead of
> lstat'ting single files). The cache is kept up to date via
> ReadDirectoryChangesW (there is no *A version, so I needed unicode
> support first...).
>
> The performance improvements of the cached version are quite impressive
> for git status, even with core.preloadindex turned on. The numbers below
> are for git status on the WebKit repo, normal git v1.7.9 vs. the cached
> version, for different core.preloadindex and --untracked-files settings:
>
> preload | -u | normal | cached | gain
> --------+-----+--------+--------+------
> false | all | 25.144 | 3.055 | 8.2
> false | no | 22.822 | 1.748 | 12.8
> true | all | 9.234 | 2.179 | 4.2
> true | no | 6.833 | 0.955 | 7.2
Those are impressive numbers!
> Drawback of this approach is that git operations that modify the working
> copy are actually slower with the cache, because e.g. git checkout
> typically lstats every file immediately after creating it (so the cache
> is always dirty). It may be better to activate the cache just for
> readonly operations.
In the alternative, we could add code hints that turn off the cache in
certain code-blocks (althoug we have to be careful to do this
thread-locally) in a later commit. That is, if I understand correctly that
updating the cache is only worth it when the ratio between file updates vs
lstats calls is small?
> In case you wanna try it out, I've pushed the code to:
>
> https://github.com/kblees/git/tree/kb/fscache-v0
>
> However, beware that this actually predates the unicode patches: it is
> pretty ugly and quite sparse on comments and error handling (but seems
> to work mostly, at least the test suite doesn't complain that much...).
> I just pulled the code to devel, fixed compile errors and added thread
> synchronization to test with core.preloadindex=true.
>
> Unless I hear violent objections from the list, I plan to polish this
> code a bit and provide some patches that are probably worthy of
> inclusion...(no promises on the time frame, though :-).
Sounds like a fun project, and worthwhile, too, what with Git being
optimized speed-wise for Linux.
Will try to find time to play with it tomorrow...
Ciao,
Dscho
Scott Graham
The performance improvements of the cached version are quite impressive
for git status, even with core.preloadindex turned on. The numbers below
are for git status on the WebKit repo, normal git v1.7.9 vs. the cached
version, for different core.preloadindex and --untracked-files settings:
preload | -u | normal | cached | gain
--------+-----+--------+--------+------
false | all | 25.144 | 3.055 | 8.2
false | no | 22.822 | 1.748 | 12.8
true | all | 9.234 | 2.179 | 4.2
true | no | 6.833 | 0.955 | 7.2
karste...@dcon.de
03:27:20:[...]
> > Drawback of this approach is that git operations that modify the
working
> > copy are actually slower with the cache, because e.g. git checkout
> > typically lstats every file immediately after creating it (so the
cache
> > is always dirty). It may be better to activate the cache just for
> > readonly operations.
>
> In the alternative, we could add code hints that turn off the cache in
> certain code-blocks (althoug we have to be careful to do this
> thread-locally) in a later commit. That is, if I understand correctly
that
> updating the cache is only worth it when the ratio between file updates
vs
> lstats calls is small?
>
Yes, I've implemented (very) simple heuristics in fsentry.modcnt, so that
a cache entry is only replaced if there have been several consecutive
reads without a change. Lstat for dirty cache entries is redirected to
mingw_lstat.
However, the heuristics code and the ReadDirectoryChangesW stuff is quite
complex. Additionally, the documentation of ReadDirectoryChangesW states
that it may actually return the short file name (e.g. PROGRA~1 instead of
Program Files), which is useless to find the cache entry.
Implementing a read-only cache activated on demand might be a much more
robust solution, and give us decent improvement without all the trouble.
I'm thinking of git-status / wt_status_collect and read_index_preload
(AFAICT the latter is used by practically every command that operates on
the working copy).
> > In case you wanna try it out, I've pushed the code to:
> >
> > https://github.com/kblees/git/tree/kb/fscache-v0
> >
> > However, beware that this actually predates the unicode patches: it is
> > pretty ugly and quite sparse on comments and error handling (but seems
> > to work mostly, at least the test suite doesn't complain that
much...).
> > I just pulled the code to devel, fixed compile errors and added thread
> > synchronization to test with core.preloadindex=true.
> >
> > Unless I hear violent objections from the list, I plan to polish this
> > code a bit and provide some patches that are probably worthy of
> > inclusion...(no promises on the time frame, though :-).
>
> Sounds like a fun project, and worthwhile, too, what with Git being
> optimized speed-wise for Linux.
>
Ok, then I'll put some time into this...looking forward to a lot of
enlightening discussions :-)
Bye,
Karsten
Pat Thoyts
> Ok, then I'll put some time into this...looking forward to a lot of
> enlightening discussions :-)
>
> Bye,
> Karsten
>
Not so long ago I looked at using the ReadDirectoryChanges functions
via Tcl in git-gui to see about having refresh itself in a more timely
fashion than it does now. I've a small extension to Tcl that can
notify the app when required and it just needs some refining to only
pay attention to files that are actually part of the git project and
not any old rubbish. However, I wonder if there is scope for some hook
calling in your cache updating. You'd want to just send a message to
something so you don't clobber the performance gains you are chasing
but some sort of "the status changed" message might fall out of this
usefully.
karste...@dcon.de
I don't think so. Currently I'm using ReadDirectoryChanges synchronously
in the context of the calling thread, to be absolutely certain the cache
is up to date. Besides, the cache sits below lstat / opendir, it would be
hard to process pathspecs or .gitignore there, and you wouldn't be
interested in e.g. .o / .exe changes...
I think what you're looking for is some asynchronous notification
mechanism, something like a daemonized git process, using a
background-thread to process changes and send notification messages,
right? Lets say a new command git-monitor-status [<pathspec>] that tracks
file system changes of relevant files and prints them to stdout? It could
use ReadDirectoryChanges on Windows and inotify on Linux...
That would surely be an interesting project as well, but not what I had in
mind with the file system cache.
Albert
How persistent is the cache? Will it live between reboots on Widows?
Your numbers thrash what I was able to get by using the USN Journal.
Congrats!
(I was doing a lot of native system calls from a managed programming language which is always a recipe for slooooow)
Johannes Schindelin
On Tue, 14 Feb 2012, Albert wrote:
> How persistent is the cache?
Karsten posted a link to the Git branch. Look for yourself.
> Will it live between reboots on Widows?
Assuming you refer to Windows, I am pretty certain that it does not.
> *** Please reply-to-all at all times ***
> *** (do not pretend to know who is subscribed and who is not) ***
>
> *** Please avoid top-posting. ***
Thank you so much,
Johannes
karste...@dcon.de
Btw., switching branches with a one-liner change in the WebKit Makefile (i.e. with minimal working-copy changes) is reduced from 4.2s to 2.1s if core.preloadindex is on.
Current version is here: https://github.com/kblees/git/tree/kb/fscache-v1
Installer: https://docs.google.com/uc?id=0BxXUoUg2r8ftMThiNTJlMzgtNzkzNC00YzlmLWFlMGYtMjk2OTM2YjIzMjZk&export=download
Ciao,
Karsten
Karsten Blees
> Wow those are some amazing performance numbers!
>
> How persistent is the cache? Will it live between reboots on Widows?
>
process). Keeping persistent caches up to date is difficult, and as
the cache sits below lstat, we need 100% reliable information here
(i.e. something as 'current' as Tortoise's icon overlays with their
TGitCache/TSvnCache background process is just not good enough :-) ).
Albert
I've been running your fscache-v1 code for the past few days in my day-to-day work. It's amazingly fast. Thank you for your work on it.
I seem to have run across a issue where git status has issues picking up unversion files in new folders.
- If you add a new file to a folder that already has version controlled files git status works as expected.
- If you create a new folder and add unversioned files (and folders) into it git status only picks up the top level folder as being unversiond and nothing below it.
Has anybody else come across such behaviour?
Albert
P.S. Sorry if I missed anybody on the CC list, using the google groups web interface.
karste...@dcon.de
Albert <pro-...@optusnet.com.au> wrote on 07.03.2012 22:44:55:
> Hi Karsten,
>
> I've been running your fscache-v1 code for the past few days in my
> day-to-day work. It's amazingly fast. Thank you for your work on it.
>
> I seem to have run across a issue where git status has issues
> picking up unversion files in new folders.
> - If you add a new file to a folder that already has version
> controlled files git status works as expected.
> - If you create a new folder and add unversioned files (and folders)
> into it git status only picks up the top level folder as being
> unversiond and nothing below it.
>
> Has anybody else come across such behaviour?
>
Albert Krawczyk
--untracked-files=all" or "git config [--global] status.showUntrackedFiles
all".
Seems I have spent too much time in the TortoiseGit land, and had assumed
that git command line behaved the same way.
Again thanks for the awesome code. Hope it makes its way into msysGit soon.
Albert
Albert
Current version is here: https://github.com/kblees/git/tree/kb/fscache-v1
Ciao,
Karsten
Hi Karsten,
I realise this is an old thread to dig up, and realise you may have discontinued working on this branch, but hope you don't mind me providing some feedback. I've continued to use your fscache-v1 code and it's been working very well though all the versions of git that have been released.
I have however recently noticed some odd behaviour with preloadindex = true.
It seems that with preloadindex set to true, after a successful checkout the index thinks it's dirty, even though doing a diff there are actually no differences between the files. With preloadindex = false everything works as required, and blazing fast.
The commands to reproduce the error:
albert@pc /d/GitRepos
$ mkdir test
albert@pc /d/GitRepos
$ cd test/
albert@pc /d/GitRepos/test
$ git init
Initialized empty Git repository in d:/GitRepos/test/.git
albert@pc /d/GitRepos/test (master)
$ echo "hello" > file.txt
albert@pc /d/GitRepos/test (master)
$ git add file.txt
albert@pc /d/GitRepos/test (master)
$ git commit -m "First commit"
[master (root-commit) aff3cd5] First commit
1 file changed, 1 insertion(+)
create mode 100644 file.txt
albert@pc /d/GitRepos/test (master)
$ git checkout -b devel
Switched to a new branch 'devel'
albert@pc /d/GitRepos/test (devel)
$ echo "yellow" > file.txt
albert@pc /d/GitRepos/test (devel)
$ git add file.txt
albert@pc /d/GitRepos/test (devel)
$ git commit -m "Second commit"
[devel 598ac47] Second commit
1 file changed, 1 insertion(+), 1 deletion(-)
albert@pc /d/GitRepos/test (devel)
$ git status -s
albert@pc /d/GitRepos/test (devel)
$ git checkout master
Switched to branch 'master'
albert@pc /d/GitRepos/test (master)
$ git status -s
albert@pc /d/GitRepos/test (master)
$ git checkout devel
Switched to branch 'devel'
albert@pc /d/GitRepos/test (devel)
$ git status -s
albert@pc /d/GitRepos/test (devel)
$ git config core.preloadindex true
albert@pc /d/GitRepos/test (devel)
$ git status -s
albert@pc /d/GitRepos/test (devel)
$ git checkout master
Switched to branch 'master'
albert@pc /d/GitRepos/test (master)
$ git status -s
M file.txt
albert@pc /d/GitRepos/test (master)
$ git config core.preloadindex false
albert@pc /d/GitRepos/test (master)
$ git status -s
M file.txt
albert@pc /d/GitRepos/test (master)
$ git reset --hard head
HEAD is now at aff3cd5 First commit
albert@pc /d/GitRepos/test (master)
$ git status -s
albert@pc /d/GitRepos/test (master)
$ git checkout devel
Switched to branch 'devel'
albert@pc /d/GitRepos/test (devel)
$ git status -s
albert@pc /d/GitRepos/test (devel)
$ git config core.preloadindex true
albert@pc /d/GitRepos/test (devel)
$ git status -s
albert@pc /d/GitRepos/test (devel)
$ git checkout master
Switched to branch 'master'
albert@pc /d/GitRepos/test (master)
$ git status -s
M file.txt
albert@pc /d/GitRepos/test (master)
$ git diff file.txt
albert@pc /d/GitRepos/test (master)
$
Again apologies for brining up this old message, but wanted to provide my experience on the code, and this is the best place I could find.
Also apologies for any horrible formatting in the message, using the Goggle Groups online interface
Albert
karste...@dcon.de
>
> Hi Karsten,
>
> I realise this is an old thread to dig up, and realise you may have
> discontinued working on this branch, but hope you don't mind me
> providing some feedback. I've continued to use your fscache-v1 code
> and it's been working very well though all the versions of git that
> have been released.
>
The hashtable part (first patch) is now as clean as it may get, but the
rest has to wait a bit longer... Thanks for the feedback, though, good to
know that this stuff is actually useful to someone.
> I have however recently noticed some odd behaviour with preloadindex =
true.
>
> It seems that with preloadindex set to true, after a successful
> checkout the index thinks it's dirty, even though doing a diff there
> are actually no differences between the files. With preloadindex =
> false everything works as required, and blazing fast.
>
Yes, the preload_index function has an early return for repositories <
1.000 files, which prevents the cache from beeing disabled. Thus after
checkout, git stores the cached lstat data (from before the checkout) in
the index...
I've pushed a fixed version to
https://github.com/kblees/git/tree/kb/fscache-v2
Hope that helps,
Karsten
diff --git a/preload-index.c b/preload-index.c
index 355fa74..a67d340 100644
--- a/preload-index.c
+++ b/preload-index.c
@@ -74,18 +74,19 @@ static void preload_index(struct index_state *index,
const char **pathspec)
if (!core_preload_index)
return;
-#ifdef USE_WINFSCACHE
- fscache_enable(1);
-#endif
threads = index->cache_nr / THREAD_COST;
if (threads < 2)
return;
if (threads > MAX_PARALLEL)
threads = MAX_PARALLEL;
offset = 0;
work = DIV_ROUND_UP(index->cache_nr, threads);
+
+#ifdef USE_WINFSCACHE
+ fscache_enable(1);
+#endif
for (i = 0; i < threads; i++) {
struct thread_data *p = data+i;
p->index = index;
p->pathspec = pathspec;
wwa...@gmail.com
rupert thurner
Can confirm your version is much faster. Mind providing a fscache-v2 installer too?
Yes, the preload_index function has an early return for repositories <
1.000 files, which prevents the cache from beeing disabled. Thus after
checkout, git stores the cached lstat data (from before the checkout) in
the index...
I've pushed a fixed version to
https://github.com/kblees/git/tree/kb/fscache-v2
anybody of you could upload a compiled version for reuse by others please?
rupert.
Johannes Schindelin
On Sat, 29 Mar 2014, rupert thurner wrote:
> anybody of you could upload a compiled version for reuse by others please?
;-)
Seriously again, I did not spend hundreds of hours to make the net
installer difficult to use. And I did not spend hundreds of hours to make
the process of building a Git for Windows installer difficult. In fact,
you can make your own installer of your own, personalized Git for Windows
simply by downloading and running the net installer and then running a
single command-line:
https://github.com/msysgit/msysgit/wiki/InstallMSysGit#How_to_remake_the_installer
So if somebody would upload a ready-compiled version for you, we would
spoil you even more.
Ciao,
Dscho
rupert THURNER
Haha, i love you, this project, and the tone on the list. The question was more directed to persons already using this. And thanks for the hints!
Already spoiled,
Rupert :)
pro-logic
Haha, i love you, this project, and the tone on the list. The question was more directed to persons already using this. And thanks for the hints!
Johannes Schindelin
On Sun, 30 Mar 2014, pro-logic wrote:
> Personally I reckon they should be defaults in the Windows version, but
> maybe there is a reason it's not yet.
archives and commit messages.
Sorry to be so succinct.
Ciao,
Johannes
Paul Betts
On Fri, Jul 27, 2012 at 11:21:42PM +0200, karste...@dcon.de wrote:
> Thanks for the feedback, though, good to know that this stuff is actually
> useful to someone.
internally, then put this out in the version of msysgit shipped with GitHub
for Windows (https://github.com/github/msysgit). Thanks for the great work
that you guys do, this patch is killer. Let us know if we can help at all
(we'll be watching for weird edge cases in support and we'll try to file good
bugs, or better yet, good bugs with patches attached)
--
Paul Betts <pa...@paulbetts.org>
pro-logic
You reckon correctly, and the reasoning is all in the mailing list
archives and commit messages.
Sorry to be so succinct.
Konstantin Khomoutov
rupert THURNER <rupert....@gmail.com> wrote:
[...]
> please?
Here we go [1]. I've also heard some people got speedups on Windows
with core.preloadindex set to true (this is a vanilla Git option so it
works on any OS).
Please note that this might be a good idea to subscribe to this list to
read release announces or at least read release notes after installing
the next Git for Windows version. It's not like the devs keep the
results of their work in secret.
1. https://groups.google.com/d/msg/msysgit/8_I7X-Bzsq0/m6e_if2DCQgJ
rupert thurner
what is the easiest way to install two versions side-by-side? i tried to call the installer, but it overwrites the original. moving the resulting dir aside and install the other version seems not to work:
Welcome to Git (version 1.8.0-preview20121022)
Run 'git help git' to display the help index.
Run 'git help <command>' to display help for specific commands.
$ git --version
git version 1.9.0.msysgit.0
rupert.
Johannes Schindelin
On Tue, 1 Apr 2014, rupert thurner wrote:
> what is the easiest way to install two versions side-by-side?
portable version for one of the both, but you will only be able to install
one version into the Start menu.
Ciao,
Johannes
rupert thurner
https://github.com/msysgit/git/commit/64d63240762df22e92b287b145d75a0d68a66988
on a laptop penalized by a virus scanner the gain is quite impressive. the repo size is >100'000 files, >35'000 directories. the inital "git add" took 86m, the commit 22m, and the variation for git status was between 1m16 and 7m11. tinkering with the parameters gave a speed up from 1m38 to 0m32. for linux this takes 0m08 (but faster hardware). the details:
$ time git ls-files | wc -l
102223
real 0m0.954s
$ time git status
On branch master
It took 24.57 seconds to enumerate untracked files. 'status -uno'
may speed it up, but you have to be careful not to forget to add
new files yourself (see 'git help status').
nothing to commit, working directory clean
real 1m37.894s
user 0m0.000s
sys 0m0.047s
$ git config --global core.fscache true
$ time git status
On branch master
It took 10.43 seconds to enumerate untracked files. 'status -uno'
may speed it up, but you have to be careful not to forget to add
new files yourself (see 'git help status').
nothing to commit, working directory clean
real 0m48.474s
user 0m0.000s
sys 0m0.046s
$ git config --global core.preloadindex true
$ time git status; date
On branch master
It took 3.46 seconds to enumerate untracked files. 'status -uno'
may speed it up, but you have to be careful not to forget to add
new files yourself (see 'git help status').
nothing to commit, working directory clean
real 0m36.453s
user 0m0.015s
sys 0m0.015s
$ time git status -uno
On branch master
nothing to commit (use -u to show untracked files)
real 0m32.987s
user 0m0.000s
sys 0m0.015s
the same thing on linux, using jgit (http://www.eclipse.org/jgit/download/, jgit.sh):
$ time git status
real 0m8.424s
user 0m7.905s
sys 0m3.837s
$ time find . -type d | wc -l
31731
real 0m0.671s
user 0m0.163s
sys 0m0.480s
rupert