LFS: Authorization error

[cdesai]

I'm trying to setup LFS with Gerrit and I'm not able to push

Pushing as the admin user so access shouldn't be an issue.

$ git push gerrit HEAD:refs/for/branch

Locking support detected on remote "gerrit". Consider enabling it with:

  $ git config lfs.https://gerrit/repo/info/lfs.locksverify true

Authentication required: Authorization error: https://gerrit/plugins/lfs/content/default/6490d7c4ffd3f8ffb54b43d845afe01cf80abd74c91375d2860e659ecdc7a145

Check that you have proper access to the repository

Uploading LFS objects:  83% (5/6), 2.7 MB | 567 KB/s, done.

error: failed to push some refs to 'ssh://gerrit:29418/repo'

$ git remote get-url gerrit

ssh://user@gerrit:29418/repo

Gerrit 3.5.1

git-lfs/3.1.4 (GitHub; linux amd64; go 1.18.1)

lfs plugin 1f4c4127e7

Tried both https://gerrit-ci.gerritforge.com/job/plugin-lfs-bazel-master-stable-3.5/ and https://gerrit-ci.gerritforge.com/view/Plugins-master/job/plugin-lfs-bazel-master-master/

Both SSH and authenticated HTTP pushes fail.

gerrit.config:

[lfs]

     plugin = lfs

All-Projects lfs.config:

[lfs "?/*"]

        enabled = true

        maxObjectSize = 500m

Any ideas?

I don't see anything in either nginx logs or gerrit logs when this failure happens.

Pushing other non-lfs commits to the same repo has worked just fine.

[cdesai]

I've also tried

etc/lfs.config

[auth]

    sshExpirationSeconds = 600

[cdesai]

I tried to play around with this a bit more:

1. bumped sshExpirationSeconds to 6000, didn't help
     Verified timeout has been bumped using
     ssh gerrit git-lfs-authenticate repo download # And then checking the expires values

     git-lfs/3.1.4 (GitHub; linux amd64; go 1.18.1)

2. Copied the repo to a normal linux user on the gerrit server manually (archive + scp), and then pushed from there (to take the network out of the picture)

     That worked, however the server has an older version of git lfs
     git-lfs/2.7.1 (GitHub; linux amd64; go 1.11.6)

3. Tried it from another server (same datacenter), that worked too.
     git-lfs/2.13.1 (GitHub; linux amd64; go 1.14)
4. Tried using the same versions locally, compiling from tags
     git-lfs/2.7.1 wouldn't run
     git-lfs/2.13.1 worked locally once but not reliably

I've also noticed that it always fail when uploading the last (or last few) LFS objects, which are the biggest size wise (~50MB or more)
Smaller objects get uploaded fine, so it might be hitting a timeout somewhere.