git clone/pull with multiple users hangs (newbie alert)

[Mark]

Hi all,

This is probably - even likely - user error. I've recently setup
Gerrit for me organization and we're just easing into using it. We're
not, afaik, doing anything out of the ordinary - gerrit manages a
central repository for us - we all pull from that and push changes to
refs/for/<foo> for reviews.

We've got a reasonable sized existing repository (~60Mb) which I've
pushed into Gerrit and we've done some changes and reviews, merges
etc. Everything seems to working perfectly. Except - we noticed today
while getting a few extra developers onto it that it seems only one
person can clone (or pull!) at a time. If there is someone else
already pulling then git-upload-pack seems to just hang.

Our traces show:

$ GIT_TRACE=1 git pull
trace: exec: 'git-pull'
trace: run_command: 'git-pull'
trace: built-in: git 'rev-parse' '--git-dir'
trace: built-in: git 'rev-parse' '--show-toplevel'
trace: built-in: git 'ls-files' '-u'
trace: built-in: git 'symbolic-ref' '-q' 'HEAD'
trace: built-in: git 'config' '--bool' 'branch.master.rebase'
trace: built-in: git 'rev-parse' '-q' '--verify' 'HEAD'
trace: built-in: git 'fetch' '--update-head-ok'
trace: run_command: 'ssh' 'gerrit' 'git-upload-pack '\''/idod/
core'\'''
Enter passphrase for key '/home/Mark/.ssh/id_rsa':


And then nothing! As some of our developers are remote and on less-
than-speedy connections this is proving to be quite a bottle neck. Is
this normal? Is there something I'm obviously doing wrong? I don't
really know where to start - googling around hasn't turned up
anything.

We're on version 2.1.2.2, postgresql backend, ubuntu server with
clients connecting over ssh. We get the same behaviour with git
1.6.3.3 on Linux clients and on 1.7.0.4 on cygwin.

Any ideas or thoughts would be very appreciated!

thanks,

-mark

--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

[Shawn Pearce]

Mark <mark.b...@gmail.com> wrote:
> Except - we noticed today
> while getting a few extra developers onto it that it seems only one
> person can clone (or pull!) at a time. If there is someone else
> already pulling then git-upload-pack seems to just hang.

...

> And then nothing! As some of our developers are remote and on less-
> than-speedy connections this is proving to be quite a bottle neck. Is
> this normal?

Yes, on a single-CPU server.

Modify your gerrit.config and set sshd.threads [1] to a higher
number. You may also need to increase the memory available to
Gerrit (this is set by the container.heapLimit [2] variable),
as each additional thread needs additional working space.

[1] http://gerrit.googlecode.com/svn/documentation/2.1.2/config-gerrit.html#sshd.threads
[2] http://gerrit.googlecode.com/svn/documentation/2.1.2/config-gerrit.html#container.heapLimit

I'm suspecting we might be a bit too conservative here and are
setting the default number of concurrent tasks too low. E.g.
my 4 CPU server at $DAY_JOB is using sshd.threads = 24.

Remote users with slower network connections tie up a thread for a
lot longer than local users, due to the lower throughput. If you get
too many of them going at once, the whole server appears to grind
to a halt and stop answering users, but the CPU is barely used,
because its spending all of its time waiting for the network.

We should be smarter about this, and automatically increase the
number of threads in this situation. But its not that easy to
detect automatically; I don't think we have easy access to all
of the stats we need to make the decision to increase the thread
count automatically.

[Mark]

Thanks Shawn, I'll have to test later (when our remote developers wake
up) but I'm sure thats exactly what I was looking for.

cheers,

-mark

On Apr 20, 7:38 am, Shawn Pearce <s...@google.com> wrote:

> Mark <mark.butc...@gmail.com> wrote:
> > Except - we noticed today
> > while getting a few extra developers onto it that it seems only one
> > person can clone (or pull!) at a time. If there is someone else
> > already pulling then git-upload-pack seems to just hang.
> ...
> > And then nothing! As some of our developers are remote and on less-
> > than-speedy connections this is proving to be quite a bottle neck. Is
> > this normal?
>
> Yes, on a single-CPU server.
>
> Modify your gerrit.config and set sshd.threads [1] to a higher
> number.  You may also need to increase the memory available to
> Gerrit (this is set by the container.heapLimit [2] variable),
> as each additional thread needs additional working space.
>

> [1]http://gerrit.googlecode.com/svn/documentation/2.1.2/config-gerrit.ht...
> [2]http://gerrit.googlecode.com/svn/documentation/2.1.2/config-gerrit.ht...