Repo sync fails when using -j to Gerrit's port 29418

[Earl Gresh]

Hello -

I am observing problems when using repo to fetch projects
simultaneously via
Gerrit. That is, "repo sync -j" to Gerrit's port 29418. The problem
manifests
inconsistently: it might happen every 4th or 5th sync, every 10th
sync, or
sometimes every other sync.

Most times, repo will end with this error message:

error: Exited sync due to fetch errors

And sometimes repo will simply hang and stop producing any output to
the
terminal. I have seen this problem in a couple of environments, each
one
syncing using Gerrit and the -j option to repo.

I can't reproduce the problem using any other syncing protocol. For
example, changing repo's manifest URLs to sync via SSH or git-daemon
doesn't
break under any -j number ( ssh://server/path/to/project, git://server/project).

I dug into the repo code, and edited this file:

repo/git_ssh

so that it would not use the ControlMaster functionality of SSH and
instead
creates a new SSH connection for each -j instance.

changed:
exec ssh -o "ControlMaster no" -o "ControlPath $REPO_SSH_SOCK" "$@"

to:
exec ssh "@"

Now when running repo, I can use any -j option and not observe the
error. It
seems the problem is isolated to the Mina SSHD server built into
Gerrit's war
file and its ControlMaster implementation.

I turned on SSH debugging, along with --trace to repo, and was able to
capture
the error happening.

I am running Gerrit version 2.1.8 on top of Ubuntu 10.04.02 LTS.

Any insight would be appreciated.

Thanks!

Initializing project platform/packages/apps/Project.git ...
: export GIT_DIR=/local/mnt/workspace/repo_user/.repo/projects/
packages/apps/Project.git
: git init 1>| 2>|
: git config --file /local/mnt/workspace/repo_user/.repo/projects/
packages/apps/Project.git/config --null --list 1>| 2>|
: git config --file /local/mnt/workspace/repo_user/.repo/projects/
packages/apps/Project.git/config --unset-all core.bare 1>| 2>|
: git config --file /local/mnt/workspace/repo_user/.repo/projects/
packages/apps/Project.git/config --replace-all remote.server-
secure.url ssh://review-server.corp.com:29418/platform/packages/apps/Project.git
1>| 2>|
: git config --file /local/mnt/workspace/repo_user/.repo/projects/
packages/apps/Project.git/config --replace-all remote.server-
secure.review review-server.corp.com 1>| 2>|
: git config --file /local/mnt/workspace/repo_user/.repo/projects/
packages/apps/Project.git/config --replace-all remote.server-
secure.projectname platform/packages/apps/Project.git 1>| 2>|
: git config --file /local/mnt/workspace/repo_user/.repo/projects/
packages/apps/Project.git/config --replace-all remote.server-
secure.fetch +refs/heads/*:refs/remotes/server-secure/* 1>| 2>|
: export GIT_DIR=/local/mnt/workspace/repo_user/.repo/projects/
packages/apps/Project.git
: git fetch --verbose server-secure
debug3: ssh_msg_send: type 2
debug3: ssh_msg_recv entering
debug3: fd 49 is not O_NONBLOCK
debug3: ssh_msg_recv entering
debug3: ssh_msg_send: type 2
debug3: ssh_msg_recv entering
debug3: ssh_msg_send: type 2
debug3: muxserver_accept_control: receiving 0 env vars
debug3: ssh_msg_recv entering
debug2: muxserver_accept_control: accepted tty 0, subsys 0, cmd git-
upload-pack '/platform/packages/apps/Project.git'
debug2: muxserver_accept_control: got fds stdin 50, stdout 51, stderr
52
debug3: ssh_msg_send: type 2
debug2: fd 50 setting O_NONBLOCK
debug2: fd 51 setting O_NONBLOCK
debug3: fd 52 is O_NONBLOCK
debug2: fd 49 setting O_NONBLOCK
debug1: channel 11: new [client-session]
debug3: muxserver_accept_control: channel_new: 11
debug2: channel 11: send open
debug3: Wrote 112 bytes for a total of 737695
debug2: callback start
debug1: Requesting authentication agent forwarding.
debug2: channel 11: request auth-ag...@openssh.com confirm 0
debug2: client_session2_setup: id 11
debug1: Sending command: git-upload-pack '/platform/packages/apps/
Project.git'
debug2: channel 11: request exec confirm 1
debug2: callback done
debug2: channel 11: open confirm rwindow 2097152 rmax 32768
debug3: Wrote 160 bytes for a total of 738399
debug2: channel_input_status_confirm: type 99 id 11
debug2: exec request accepted on channel 11
debug3: Wrote 15984 bytes for a total of 765951
debug2: channel 11: window 1969515 sent adjust 127637
debug3: Wrote 48 bytes for a total of 784735
debug2: channel 11: window 1966080 sent adjust 131072
debug3: Wrote 48 bytes for a total of 801119
debug2: channel 11: window 1966080 sent adjust 131072
debug3: Wrote 48 bytes for a total of 801647
debug2: channel 11: window 1967579 sent adjust 129573
debug3: Wrote 48 bytes for a total of 802079
debug2: channel 11: read<=0 rfd 50 len 0
debug2: channel 11: read failed
debug2: channel 11: close_read
debug2: channel 11: input open -> drain
debug2: channel 11: ibuf empty
debug2: channel 11: send eof
debug2: channel 11: input drain -> closed
debug3: Wrote 32 bytes for a total of 802303
debug2: channel 11: rcvd eof
debug2: channel 11: output open -> drain
debug2: channel 11: obuf empty
debug2: channel 11: close_write
debug2: channel 11: output drain -> closed
debug2: channel 11: send close
debug3: Wrote 80 bytes for a total of 802383
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug3: channel 11: will not send data after close
debug2: channel 11: rcvd close
debug3: channel 11: will not send data after close
debug2: channel 11: is deadProject.git
debug2: channel 11: garbage collecting
debug1: channel 11: free: client-session, nchannels 20
debug3: channel 11: status: The following connections are open:
#0 client-session (t4 r133 i0/0 o0/0 fd 6/7 cfd 5)
#1 client-session (t4 r118 i0/0 o0/0 fd 10/11 cfd 9)
#2 client-session (t4 r128 i0/0 o0/0 fd 14/15 cfd 13)
#3 client-session (t4 r140 i0/0 o0/0 fd 18/19 cfd 17)
#4 client-session (t4 r97 i0/0 o0/0 fd 22/23 cfd 21)
#5 client-session (t4 r131 i0/0 o0/0 fd 27/28 cfd 25)
#6 client-session (t4 r104 i0/0 o0/0 fd 30/31 cfd 29)
#7 client-session (t4 r127 i0/0 o0/0 fd 34/35 cfd 33)
#8 client-session (t4 r130 i0/0 o0/0 fd 38/39 cfd 37)
#9 client-session (t4 r141 i0/0 o0/0 fd 42/43 cfd 41)
#10 client-session (t4 r126 i0/0 o0/0 fd 45/46 cfd 26)
#11 client-session (t4 r142 i3/0 o3/0 fd -1/-1 cfd 49)
#12 client-session (t4 r120 i0/0 o0/0 fd 54/55 cfd 53)
#13 client-session (t4 r134 i0/0 o0/0 fd 58/59 cfd 57)
#14 client-session (t4 r123 i0/0 o0/0 fd 62/63 cfd 61)
#15 client-session (t4 r135 i0/0 o0/0 fd 66/67 cfd 65)
#17 client-session (t4 r137 i0/0 o0/0 fd 74debug3: channel 11:
close_fds r -1 w -1 e 52 c 49
debug2: Received EOF from master
debug2: Control master terminated unexpectedly
: export GIT_DIR=/local/mnt/workspace/repo_user/.repo/projects/
packages/apps/Project.git
: git pack-refs --all --prune 1>| 2>|
debug1: client_input_channel_req: channel 11 rtype exit-status reply 0
channel_by_id: 11: bad id: channel free
client_input_channel_req: channel 11: unknown channel
error: Cannot fetch platform/packages/apps/Project.git