git push hangs with git-lfs pre-push

[Лёша М]

Hello Repo and Gerrit Discussion,

I am trying to get git-lfs working with gerrit 2.14.1.  I managed to get it to the point of accepting git lfs push and fetch.  Git lfs clone works just fine:

$ GIT_TRACE=1 git lfs clone ssh://us...@gerrit.local:29418/demo

13:28:58.318698 git.c:594               trace: exec: 'git-lfs' 'clone' 'ssh://us...@gerrit.local:29418/demo'

13:28:58.319265 run-command.c:369       trace: run_command: 'git-lfs' 'clone' 'ssh://us...@gerrit.local:29418/demo'

trace git-lfs: run_command: 'git' version

Cloning into 'demo'...

remote: Counting objects: 12, done

remote: Finding sources: 100% (12/12)

remote: Total 14 (delta 1), reused 14 (delta 1)

Receiving objects: 100% (14/14), done.

Resolving deltas: 100% (1/1), done.

trace git-lfs: run_command: 'git' config -l

trace git-lfs: run_command: 'git' rev-parse HEAD --symbolic-full-name HEAD

trace git-lfs: tq: running as batched queue, batch size of 100

trace git-lfs: run_command: git ls-tree -r -l -z --full-tree c7f6edbc6d7dae793708a457f92419147bfc9c8c

trace git-lfs: run_command: git cat-file --batch

trace git-lfs: fetch first.bin [cba5243834a58801d5f3460c1d21fe28c33b1e1c1bb8ce7513e1948eed3a19e4]

trace git-lfs: tq: sending batch of size 1

Git LFS: (0 of 1 files) 0 B / 12 B                                                                                                                                                                                                         trace git-lfs: run_command: ssh -p 29418 -- us...@gerrit.local git-lfs-authenticate demo download

trace git-lfs: api: batch 1 files

trace git-lfs: HTTP: POST https://gerrit.local/g/demo/info/lfs/objects/batch

trace git-lfs: HTTP: 200

trace git-lfs: HTTP: {"objects":[{"oid":"cba5243834a58801d5f3460c1d21fe28c33b1e1c1bb8ce7513e1948eed3a19e4","size":12,"actions":{"download":{"expires_at":"2017-06-20T20:29:09.533Z","href":"https://gerrit.local/g/plugins/lfs/content/default/cba5243834a58801d5f3460c1d21fe28c33b1e1c1bb8ce7513e1948eed3a19e4","header":{"Authorization":"U5Rvt9e79itLXXREVWSkLE5qE9a+CXc4PfJr7Vbm1Gx2o1/k+RSP/MM4TvNdkCjsjGr63AcjdWDl1TrZzcD1dCNmyGZ38m8dDAcK2qea4bH3uHX5DvFDUytVZ7AyNADXqU8ZVQ6tX4Zcyr920mlk2lKDsLO0qq12+CdULam1CTk="}}}}]}

trace git-lfs: tq: starting transfer adapter "basic"

Git LFS: (0 of 1 files) 0 B / 12 B

trace git-lfs: HTTP: GET https://gerrit.local/g/plugins/lfs/content/default/cba5243834a58801d5f3460c1d21fe28c33b1e1c1bb8ce7513e1948eed3a19e4

trace git-lfs: HTTP: 200

Git LFS: (1 of 1 files) 12 B / 12 B                                      

Now I commit something and try pushing it.  Git lfs pre-push hook or some interaction thereof causes it to hang:

$ git commit --allow-empty -m hello

[master dc7f019] hello

$ GIT_TRACE=1 git push

13:33:38.818124 git.c:369               trace: built-in: git 'push'

13:33:38.822048 run-command.c:369       trace: run_command: 'ssh' '-p' '29418' 'us...@gerrit.local' 'git-receive-pack '\''/demo'\'''

13:33:38.939427 run-command.c:369       trace: run_command: '.git/hooks/pre-push' 'origin' 'ssh://us...@gerrit.local:29418/demo'

13:33:38.946999 git.c:594               trace: exec: 'git-lfs' 'pre-push' 'origin' 'ssh://us...@gerrit.local:29418/demo'

13:33:38.947687 run-command.c:369       trace: run_command: 'git-lfs' 'pre-push' 'origin' 'ssh://us...@gerrit.local:29418/demo'

trace git-lfs: run_command: 'git' config -l

trace git-lfs: run_command: 'git' version

trace git-lfs: tq: running as batched queue, batch size of 100

trace git-lfs: run_command: ssh -p 29418 -- us...@gerrit.local git-lfs-authenticate demo upload

***hangs here***

error_log complains that stream is already closed:

[2017-06-20 13:33:49,498] [SSH git-receive-pack '/demo' (user)] ERROR com.google.gerrit.sshd.BaseCommand : Internal server error (user user account 1000000) during git-receive-pack '/demo'

org.apache.sshd.common.SshException: flush(ChannelOutputStream[ChannelSession[id=0, recipient=0]-ServerSessionImpl[user@/192.168.0.1:58229]] SSH_MSG_CHANNEL_EXTENDED_DATA) length=0 - stream is already closed

        at org.apache.sshd.common.channel.ChannelOutputStream.flush(ChannelOutputStream.java:167)

        at org.eclipse.jgit.transport.BaseReceivePack.close(BaseReceivePack.java:1804)

        at org.eclipse.jgit.transport.ReceivePack.receive(ReceivePack.java:211)

        at com.google.gerrit.sshd.commands.Receive.runImpl(Receive.java:96)

        at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:97)

        at com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:30)

        at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:63)

        at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:418)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:418)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:748)

[2017-06-20 13:33:49,499] [SSH git-lfs-authenticate demo upload (user)] ERROR com.google.gerrit.sshd.BaseCommand : Internal server error (user user account 1000000) during git-lfs-authenticate demo upload

org.apache.sshd.common.SshException: flush(ChannelOutputStream[ChannelSession[id=0, recipient=0]-ServerSessionImpl[user@/192.168.0.1:58232]] SSH_MSG_CHANNEL_DATA) length=0 - stream is already closed

        at org.apache.sshd.common.channel.ChannelOutputStream.flush(ChannelOutputStream.java:167)

        at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:425)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:418)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:748)

I can work around this by using git push --no-verify and git lfs push, but that is not acceptable for production.

Any clues how to fix it or where to look for more information?

Thank you,

Alexey