Issue 867 in tortoisegit: Add should not run a new Git process for each file

[torto...@googlecode.com]

Status: New
Owner: ----
Labels: Type-Defect Priority-Medium

New issue 867 by yves.goe...@gmail.com: Add should not run a new Git
process for each file
http://code.google.com/p/tortoisegit/issues/detail?id=867

What steps will reproduce the problem?
1. Select a whole directory with many files to add

What is the expected output? What do you see instead?
git-add accepts a list of files to add. That should be done, calling the
git process a single time. Instead, a new git process is started for each
file. This takes forever and causes incredible system load.

What version of the product are you using? On what operating system?
TortoiseGit 1.6.5.0 (1.7.2.0 does not work at all) on Windows XP.

[torto...@googlecode.com]

Comment #1 on issue 867 by yves.goe...@gmail.com: Add should not run a new

Git process for each file
http://code.google.com/p/tortoisegit/issues/detail?id=867

A similar issue exists when opening the Add dialogue for multiple files
from one context menu. So select many files, right-click on them, select
TortoiseGit/Add and then see many git processes started which takes a long
time. This seems to be a common problem of TortoiseGit. Git is fast, the
tortoise is slow.

[torto...@googlecode.com]

Comment #2 on issue 867 by yves.goe...@gmail.com: Add should not run a new

Git process for each file
http://code.google.com/p/tortoisegit/issues/detail?id=867

Same for commit of ~800 files (initial commit)... still waiting...

[torto...@googlecode.com]

Comment #3 on issue 867 by sstrickr...@googlemail.com: Add should not run a

new Git process for each file
http://code.google.com/p/tortoisegit/issues/detail?id=867

I've a nice proof of concept for this issue until we do not have any direct
access to the git index:

https://github.com/csware/TortoiseGit/tree/massive-git-task

or

git://github.com/csware/TortoiseGit.git massive-git-task

Frank, what do you think?

[torto...@googlecode.com]

Comment #5 on issue 867 by lzn...@gmail.com: Add should not run a new Git

process for each file
http://code.google.com/p/tortoisegit/issues/detail?id=867

I review your code. which can't resolve problem 100%. But it will a little
improve.

I suggest export "add" function at gitdll. So Tgit can call git function
directly without launch git.exe.

[torto...@googlecode.com]

Comment #6 on issue 867 by sstrickr...@googlemail.com: Add should not run a

new Git process for each file
http://code.google.com/p/tortoisegit/issues/detail?id=867

Doing it directly would be the best way (and I hope libgit2 is capable of
some more tasks soon), but atm it is not just the add-method which is
called very often. I tested my code and it speeds things up a lot and can
also be used for a lot of more git-commands like revert, update-index, ...

I tested my code with "add" for ~1500 files: 3 sec vs. 30+ sec (aborted),
same for commit.

[torto...@googlecode.com]

Comment #7 on issue 867 by sstrickr...@gmail.com: Add should not run a new

Git process for each file
http://code.google.com/p/tortoisegit/issues/detail?id=867

Issue 940 has been merged into this issue.

[torto...@googlecode.com]

Comment #8 on issue 867 by sstrickr...@gmail.com: Add should not run a new

Git process for each file
http://code.google.com/p/tortoisegit/issues/detail?id=867

"git add" is implemented using libgit2 (revision
65d2888f8eaffe82eaa4825ef51f9fce4bb9ead2).

[torto...@googlecode.com]

Updates:
Status: Fixed

Comment #9 on issue 867 by sstrickr...@gmail.com: Add should not run a new

Git process for each file
http://code.google.com/p/tortoisegit/issues/detail?id=867

This issue is closed for add. Other optimizations like commit are stimm on
our todo list, but libgit2 doesn't support all required calls atm.

[torto...@googlecode.com]

Comment #10 on issue 867 by pro-lo...@optusnet.com.au: Add should not run a

new Git process for each file
http://code.google.com/p/tortoisegit/issues/detail?id=867

Has this been reverted in 1.7.6? I noticed 1.7.6 starts new process for
each file to add.

[torto...@googlecode.com]

Comment #11 on issue 867 by sstrickr...@gmail.com: Add should not run a new

Git process for each file
http://code.google.com/p/tortoisegit/issues/detail?id=867

Not it has not been reverted. But you might have seen this in the commit
process (we csannot switch to libgit2 there). Also in 1.7.6+ we use "git
add" if we know that libgit2 cannot work (separate git dir).

[torto...@googlecode.com]

Comment #12 on issue 867 by pro-lo...@optusnet.com.au: Add should not run a

new Git process for each file
http://code.google.com/p/tortoisegit/issues/detail?id=867

Ok, I found the problem.

If I select a list of files and do
Right click -> Tortoise Git -> Add it will use libgit2 to do the addition
to index, it's VERY fast.

If I do
Right click -> Commit -> Select List of Files -> Right Click -> Add it will
launch a git process for each file.

I've attached a test repo where you can compare the performance of Right
Click -> TortoiseGit -> Add vs Right Click -> Commit -> Right Click -> Add.

I also suggest you
1. Extract the .git into a folder, navigate to it in windows explorer, with
explorer still open extract the *.txt's you'll see a massive performance
lockup.

Attachments:
libgit2addtest.7z 38.0 KB

[torto...@googlecode.com]

Updates:
Status: Discuss

Comment #13 on issue 867 by pro-lo...@optusnet.com.au: Add should not run a

new Git process for each file
http://code.google.com/p/tortoisegit/issues/detail?id=867

(No comment was entered for this change.)

[torto...@googlecode.com]

Comment #14 on issue 867 by sstrickr...@gmail.com: Add should not run a new

Git process for each file
http://code.google.com/p/tortoisegit/issues/detail?id=867

I suggest to revert to use libgit2 for adding files in 1.7.7:
- issue #1051: Commit doesn't normalize line endings
- issue #1044: Add file to index with special character (ei – "dash") or
accent fail without any message
- issue #1015: when I "Git add", it then says add failed
- issue #1022: Cannot add files to submodules

Opinions?

[torto...@googlecode.com]

Comment #15 on issue 867 by sstrickr...@gmail.com: Add should not run a new

Git process for each file
http://code.google.com/p/tortoisegit/issues/detail?id=867

or at least add a registry flag to explicitely enable it, but disable it by
default.

[torto...@googlecode.com]

Comment #16 on issue 867 by pro-lo...@optusnet.com.au: Add should not run a

new Git process for each file
http://code.google.com/p/tortoisegit/issues/detail?id=867

You mean revert to use of git.exe to add files? Given the number of issue
caused by libgit2 it's probably a good idea. (We should probably report all
these issues to libgit2, at the end of the day we want to use it since it's
much faster and easier to work with than git.exe)

My suggestion would be to pass many file names to each git process at a
time, as opposed to just 1, this will at least speed it up a bit. A single
command can be at most 8191 characters in windows XP[1].

Eg: 'git.exe add "file1" "folder with space in it\file 2"' is 52
characters, so 8191 should fit a fair few files to the add parameter.

My guess would be, have an array of files to add, and start formatting
the "add" string, keep adding files in "" and relative posix paths until
the array of files to add is empty, or the string exceeds 8191 characters.
If there are no more files to add to the sting, execute it, if it exceeds
8191 remove the last string, execute the add, and start formatting a new
add string.

This way we save a lot of time for the process start/terminate cycle.

[1] http://support.microsoft.com/kb/830473

[torto...@googlecode.com]

Comment #17 on issue 867 by sstrickr...@gmail.com: Add should not run a new

Git process for each file
http://code.google.com/p/tortoisegit/issues/detail?id=867

pro-logic: I already had this idea: See my massive-git-task branch.

[torto...@googlecode.com]

Comment #18 on issue 867 by pro-lo...@optusnet.com.au: Add should not run a

new Git process for each file
http://code.google.com/p/tortoisegit/issues/detail?id=867

I like the idea your massive-git-task :) I personally think it should be
the default for all operations that a list of files. Like add / revert /
remove etc etc the CPU time spent formatting the string is insignificant
compared to the CPU time of the git process starting up.

The only issue I see with it (from a very quick read) is you have
MAX_COMMANDLINE_LENGTH = 30000 where it has to be 8191

My idea - which i think you have - is esentially

string baseCommand = "git.exe add"
string command = baseCommand;

foreach fileYouWantToAdd
if(command.length + fileYouWantToAdd.length + 3 < 8191)
command += " " + "\"" + fileYouWantToAdd.path + "\"
else
command.execute
command = baseCommand
command += " " + "\"" + fileYouWantToAdd.path + "\"
end foreach
command.execute

[torto...@googlecode.com]

Comment #19 on issue 867 by sstrickr...@gmail.com: Add should not run a new

Git process for each file
http://code.google.com/p/tortoisegit/issues/detail?id=867

I suppose 8191 is only for parameters of the cmd.exe command and not a
general limitation.

"CreateProcess function": "The command line to be executed. The maximum
length of this string is 32,768 characters" (for W2K and newer)

Taken from:
http://msdn.microsoft.com/en-us/library/windows/desktop/ms682425%28v=vs.85%29.aspx

[torto...@googlecode.com]

Comment #20 on issue 867 by pro-lo...@optusnet.com.au: Add should not run a

new Git process for each file
http://code.google.com/p/tortoisegit/issues/detail?id=867

Any plans to include this in 1.7.8 :) Happy to beta test ;)

[torto...@googlecode.com]

Comment #21 on issue 867 by sstrickr...@gmail.com: Add should not run a new

Git process for each file
http://code.google.com/p/tortoisegit/issues/detail?id=867

it's already in 1.7.7

[torto...@googlecode.com]

Comment #22 on issue 867 by pro-lo...@optusnet.com.au: Add should not run a

new Git process for each file
http://code.google.com/p/tortoisegit/issues/detail?id=867

Do you just use it for a specific list of operations?

I just tested doing an add of 1200 files (using the repo from comment 12),
and it's FAAAST.

But selecting those same 1200 files and doing a right click "revert" seems
to start a few git processes as opposed to just 1 for the start.

[torto...@googlecode.com]

Comment #23 on issue 867 by sstrickr...@gmail.com: Add should not run a new

Git process for each file
http://code.google.com/p/tortoisegit/issues/detail?id=867

Revert isn't that easy (interaction with windows recycle bin and "checkout"
vs. delete file), so it doesn't use this approach atm.

[torto...@googlecode.com]

Comment #24 on issue 867 by ch3co...@gmail.com: Add should not run a new

Git process for each file
http://code.google.com/p/tortoisegit/issues/detail?id=867

For newly added files, current TortoiseGit calls this for each file
git add "file"

For hundreds / thousands of new files, it takes long time.

I suggest to add "assume-all" option, which calls this to boost performance
git add .

assume-all means the user will not change files in working copy from the
time he launch the commit dialog to the time the commit process finishes.
All non-ignored files are assumed to be committed, regardless whether the
files appeared in GUI.

[torto...@googlecode.com]

Comment #25 on issue 867 by sstrickr...@gmail.com: Add should not run a new

Git process for each file
http://code.google.com/p/tortoisegit/issues/detail?id=867

The add dialog uses libgit2 in 1.7.11.5+ again.

[torto...@googlecode.com]

Comment #27 on issue 867 by ch3co...@gmail.com: Add should not run a new

Git process for each file
http://code.google.com/p/tortoisegit/issues/detail?id=867

Although "add dialog" is improved, how many of you use it often instead of
directly using "commit dialog"? When should we use "add dialog"?

[torto...@googlecode.com]

Comment #28 on issue 867 by sstrickr...@gmail.com: Add should not run a new

Git process for each file
http://code.google.com/p/tortoisegit/issues/detail?id=867

Please open a new issue (however, it's planned for 1.7.13).