Issue 867 in tortoisegit: Add should not run a new Git process for each file

54 views
Skip to first unread message

torto...@googlecode.com

unread,
Aug 18, 2011, 4:44:24 PM8/18/11
to tortois...@googlegroups.com
Status: New
Owner: ----
Labels: Type-Defect Priority-Medium

New issue 867 by yves.goe...@gmail.com: Add should not run a new Git
process for each file
http://code.google.com/p/tortoisegit/issues/detail?id=867

What steps will reproduce the problem?
1. Select a whole directory with many files to add

What is the expected output? What do you see instead?
git-add accepts a list of files to add. That should be done, calling the
git process a single time. Instead, a new git process is started for each
file. This takes forever and causes incredible system load.

What version of the product are you using? On what operating system?
TortoiseGit 1.6.5.0 (1.7.2.0 does not work at all) on Windows XP.

torto...@googlecode.com

unread,
Aug 18, 2011, 4:48:26 PM8/18/11
to tortois...@googlegroups.com

Comment #1 on issue 867 by yves.goe...@gmail.com: Add should not run a new

A similar issue exists when opening the Add dialogue for multiple files
from one context menu. So select many files, right-click on them, select
TortoiseGit/Add and then see many git processes started which takes a long
time. This seems to be a common problem of TortoiseGit. Git is fast, the
tortoise is slow.

torto...@googlecode.com

unread,
Aug 18, 2011, 4:56:39 PM8/18/11
to tortois...@googlegroups.com

Comment #2 on issue 867 by yves.goe...@gmail.com: Add should not run a new

Same for commit of ~800 files (initial commit)... still waiting...

torto...@googlecode.com

unread,
Oct 2, 2011, 11:06:45 PM10/2/11
to tortois...@googlegroups.com

Comment #3 on issue 867 by sstrickr...@googlemail.com: Add should not run a

I've a nice proof of concept for this issue until we do not have any direct
access to the git index:

https://github.com/csware/TortoiseGit/tree/massive-git-task

or

git://github.com/csware/TortoiseGit.git massive-git-task

Frank, what do you think?

torto...@googlecode.com

unread,
Oct 3, 2011, 8:39:04 PM10/3/11
to tortois...@googlegroups.com

Comment #5 on issue 867 by lzn...@gmail.com: Add should not run a new Git

I review your code. which can't resolve problem 100%. But it will a little
improve.

I suggest export "add" function at gitdll. So Tgit can call git function
directly without launch git.exe.

torto...@googlecode.com

unread,
Oct 3, 2011, 10:43:23 PM10/3/11
to tortois...@googlegroups.com

Comment #6 on issue 867 by sstrickr...@googlemail.com: Add should not run a

Doing it directly would be the best way (and I hope libgit2 is capable of
some more tasks soon), but atm it is not just the add-method which is
called very often. I tested my code and it speeds things up a lot and can
also be used for a lot of more git-commands like revert, update-index, ...

I tested my code with "add" for ~1500 files: 3 sec vs. 30+ sec (aborted),
same for commit.

torto...@googlecode.com

unread,
Oct 24, 2011, 8:14:23 PM10/24/11
to tortois...@googlegroups.com

Comment #7 on issue 867 by sstrickr...@gmail.com: Add should not run a new

Issue 940 has been merged into this issue.

torto...@googlecode.com

unread,
Nov 19, 2011, 6:11:03 PM11/19/11
to tortois...@googlegroups.com

Comment #8 on issue 867 by sstrickr...@gmail.com: Add should not run a new

"git add" is implemented using libgit2 (revision
65d2888f8eaffe82eaa4825ef51f9fce4bb9ead2).

torto...@googlecode.com

unread,
Nov 20, 2011, 6:43:58 PM11/20/11
to tortois...@googlegroups.com
Updates:
Status: Fixed

Comment #9 on issue 867 by sstrickr...@gmail.com: Add should not run a new

This issue is closed for add. Other optimizations like commit are stimm on
our todo list, but libgit2 doesn't support all required calls atm.

torto...@googlecode.com

unread,
Jan 17, 2012, 12:41:41 AM1/17/12
to tortois...@googlegroups.com

Comment #10 on issue 867 by pro-lo...@optusnet.com.au: Add should not run a

Has this been reverted in 1.7.6? I noticed 1.7.6 starts new process for
each file to add.

torto...@googlecode.com

unread,
Jan 17, 2012, 4:45:29 AM1/17/12
to tortois...@googlegroups.com

Comment #11 on issue 867 by sstrickr...@gmail.com: Add should not run a new

Not it has not been reverted. But you might have seen this in the commit
process (we csannot switch to libgit2 there). Also in 1.7.6+ we use "git
add" if we know that libgit2 cannot work (separate git dir).

torto...@googlecode.com

unread,
Jan 19, 2012, 7:30:37 PM1/19/12
to tortois...@googlegroups.com

Comment #12 on issue 867 by pro-lo...@optusnet.com.au: Add should not run a

Ok, I found the problem.

If I select a list of files and do
Right click -> Tortoise Git -> Add it will use libgit2 to do the addition
to index, it's VERY fast.

If I do
Right click -> Commit -> Select List of Files -> Right Click -> Add it will
launch a git process for each file.

I've attached a test repo where you can compare the performance of Right
Click -> TortoiseGit -> Add vs Right Click -> Commit -> Right Click -> Add.

I also suggest you
1. Extract the .git into a folder, navigate to it in windows explorer, with
explorer still open extract the *.txt's you'll see a massive performance
lockup.

Attachments:
libgit2addtest.7z 38.0 KB

torto...@googlecode.com

unread,
Jan 21, 2012, 11:36:59 PM1/21/12
to tortois...@googlegroups.com
Updates:
Status: Discuss

Comment #13 on issue 867 by pro-lo...@optusnet.com.au: Add should not run a

(No comment was entered for this change.)

torto...@googlecode.com

unread,
Jan 23, 2012, 12:30:20 PM1/23/12
to tortois...@googlegroups.com

Comment #14 on issue 867 by sstrickr...@gmail.com: Add should not run a new
I suggest to revert to use libgit2 for adding files in 1.7.7:
- issue #1051: Commit doesn't normalize line endings
- issue #1044: Add file to index with special character (ei – "dash") or
accent fail without any message
- issue #1015: when I "Git add", it then says add failed
- issue #1022: Cannot add files to submodules

Opinions?

torto...@googlecode.com

unread,
Jan 23, 2012, 12:34:22 PM1/23/12
to tortois...@googlegroups.com

Comment #15 on issue 867 by sstrickr...@gmail.com: Add should not run a new

or at least add a registry flag to explicitely enable it, but disable it by
default.

torto...@googlecode.com

unread,
Jan 23, 2012, 2:50:05 PM1/23/12
to tortois...@googlegroups.com

Comment #16 on issue 867 by pro-lo...@optusnet.com.au: Add should not run a

You mean revert to use of git.exe to add files? Given the number of issue
caused by libgit2 it's probably a good idea. (We should probably report all
these issues to libgit2, at the end of the day we want to use it since it's
much faster and easier to work with than git.exe)

My suggestion would be to pass many file names to each git process at a
time, as opposed to just 1, this will at least speed it up a bit. A single
command can be at most 8191 characters in windows XP[1].

Eg: 'git.exe add "file1" "folder with space in it\file 2"' is 52
characters, so 8191 should fit a fair few files to the add parameter.

My guess would be, have an array of files to add, and start formatting
the "add" string, keep adding files in "" and relative posix paths until
the array of files to add is empty, or the string exceeds 8191 characters.
If there are no more files to add to the sting, execute it, if it exceeds
8191 remove the last string, execute the add, and start formatting a new
add string.

This way we save a lot of time for the process start/terminate cycle.

[1] http://support.microsoft.com/kb/830473

torto...@googlecode.com

unread,
Jan 23, 2012, 3:15:22 PM1/23/12
to tortois...@googlegroups.com

Comment #17 on issue 867 by sstrickr...@gmail.com: Add should not run a new

pro-logic: I already had this idea: See my massive-git-task branch.

torto...@googlecode.com

unread,
Jan 23, 2012, 4:35:07 PM1/23/12
to tortois...@googlegroups.com

Comment #18 on issue 867 by pro-lo...@optusnet.com.au: Add should not run a

I like the idea your massive-git-task :) I personally think it should be
the default for all operations that a list of files. Like add / revert /
remove etc etc the CPU time spent formatting the string is insignificant
compared to the CPU time of the git process starting up.

The only issue I see with it (from a very quick read) is you have
MAX_COMMANDLINE_LENGTH = 30000 where it has to be 8191

My idea - which i think you have - is esentially

string baseCommand = "git.exe add"
string command = baseCommand;

foreach fileYouWantToAdd
if(command.length + fileYouWantToAdd.length + 3 < 8191)
command += " " + "\"" + fileYouWantToAdd.path + "\"
else
command.execute
command = baseCommand
command += " " + "\"" + fileYouWantToAdd.path + "\"
end foreach
command.execute

torto...@googlecode.com

unread,
Jan 23, 2012, 5:40:51 PM1/23/12
to tortois...@googlegroups.com

Comment #19 on issue 867 by sstrickr...@gmail.com: Add should not run a new

I suppose 8191 is only for parameters of the cmd.exe command and not a
general limitation.

"CreateProcess function": "The command line to be executed. The maximum
length of this string is 32,768 characters" (for W2K and newer)

Taken from:
http://msdn.microsoft.com/en-us/library/windows/desktop/ms682425%28v=vs.85%29.aspx

torto...@googlecode.com

unread,
Feb 17, 2012, 11:34:14 PM2/17/12
to tortois...@googlegroups.com

Comment #20 on issue 867 by pro-lo...@optusnet.com.au: Add should not run a

Any plans to include this in 1.7.8 :) Happy to beta test ;)

torto...@googlecode.com

unread,
Feb 18, 2012, 6:35:52 AM2/18/12
to tortois...@googlegroups.com

Comment #21 on issue 867 by sstrickr...@gmail.com: Add should not run a new

it's already in 1.7.7

torto...@googlecode.com

unread,
Feb 18, 2012, 4:02:43 PM2/18/12
to tortois...@googlegroups.com

Comment #22 on issue 867 by pro-lo...@optusnet.com.au: Add should not run a

Do you just use it for a specific list of operations?

I just tested doing an add of 1200 files (using the repo from comment 12),
and it's FAAAST.

But selecting those same 1200 files and doing a right click "revert" seems
to start a few git processes as opposed to just 1 for the start.

torto...@googlecode.com

unread,
Feb 20, 2012, 6:36:24 AM2/20/12
to tortois...@googlegroups.com

Comment #23 on issue 867 by sstrickr...@gmail.com: Add should not run a new

Revert isn't that easy (interaction with windows recycle bin and "checkout"
vs. delete file), so it doesn't use this approach atm.

torto...@googlecode.com

unread,
Aug 4, 2012, 11:18:48 AM8/4/12
to tortois...@googlegroups.com

Comment #24 on issue 867 by ch3co...@gmail.com: Add should not run a new
For newly added files, current TortoiseGit calls this for each file
git add "file"

For hundreds / thousands of new files, it takes long time.

I suggest to add "assume-all" option, which calls this to boost performance
git add .

assume-all means the user will not change files in working copy from the
time he launch the commit dialog to the time the commit process finishes.
All non-ignored files are assumed to be committed, regardless whether the
files appeared in GUI.

torto...@googlecode.com

unread,
Aug 4, 2012, 8:06:28 PM8/4/12
to tortois...@googlegroups.com

Comment #25 on issue 867 by sstrickr...@gmail.com: Add should not run a new
The add dialog uses libgit2 in 1.7.11.5+ again.

torto...@googlecode.com

unread,
Aug 10, 2012, 12:48:56 AM8/10/12
to tortois...@googlegroups.com

Comment #27 on issue 867 by ch3co...@gmail.com: Add should not run a new
Although "add dialog" is improved, how many of you use it often instead of
directly using "commit dialog"? When should we use "add dialog"?

torto...@googlecode.com

unread,
Aug 10, 2012, 5:07:24 AM8/10/12
to tortois...@googlegroups.com

Comment #28 on issue 867 by sstrickr...@gmail.com: Add should not run a new
Please open a new issue (however, it's planned for 1.7.13).

Reply all
Reply to author
Forward
0 new messages