gerrit 3.6 LFS problem

Roman Karlstetter

unread,

Aug 31, 2022, 6:43:20 AM8/31/22

to repo-d...@googlegroups.com

Hi everyone,

we see a strange problem in our gerrit (v3.6.1) LFS (lfs plugin version 1f4c4127e7) setup, where LFS is working for alle projects except for those under “tools/*”:

The “lfs” plugin is enabled for all projects via the following All-Projects configuration (the second entry is just desperation based):

[lfs "^.*"]

enabled = true

maxObjectSize = 500m

[lfs "?/*"]

enabled = true

maxObjectSize = 500m

In the UI, LFS Info is present on every project’s “general” page, showing that LFS is enabled as expected. However, pushing commits that contain LFS tracked files for review does not work for all projects. Specifically, it fails for all projects under “tools/*”. It works for all root projects, and for all projects under other folders (concretely at least for projects under “sandbox/*” and “tools-lfs/*”).

The error me and my colleagues get for all projects under “tools/*” is an HTTP 405 Method Not Allowed.

I’ve attached the output of a failing push for the “bad” namespace. We don’t see any errors in the gerrit error_log or sshd_log.

Do you have any ideas what could cause this problem? Or maybe even just some pointers on how to better debug the issue?

Kind regards

Roman

Here’s the output:

09:28:31.470417 git.c:439 trace: built-in: git push origin HEAD:refs/for/main

09:28:31.470827 run-command.c:663 trace: run_command: unset GIT_PREFIX; GIT_PROTOCOL=version=1 ssh -o SendEnv=GIT_PROTOCOL gerrit 'git-receive-pack '\''tools/ProjectName'\'''

09:28:31.554347 run-command.c:663 trace: run_command: .git/hooks/pre-push origin gerrit:tools/ProjectName

09:28:31.556117 git.c:703 trace: exec: git-lfs pre-push origin gerrit:tools/ProjectName

09:28:31.556153 run-command.c:663 trace: run_command: git-lfs pre-push origin gerrit:tools/ProjectName

09:28:31.559784 trace git-lfs: exec: git 'version'

09:28:31.562438 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'

09:28:31.565414 trace git-lfs: exec: git 'config' '-l'

09:28:31.566949 trace git-lfs: pre-push: HEAD e1ac27fb191ae736f5a66154ad95d47f565d73f8 refs/for/main 0000000000000000000000000000000000000000

09:28:31.700939 trace git-lfs: run_command: ssh -- gerrit git-lfs-authenticate tools/ProjectName upload

09:28:31.788150 trace git-lfs: HTTP: POST https://our.gerrit.url/tools/ProjectName/info/lfs/locks/verify

> POST /tools/ProjectName/info/lfs/locks/verify HTTP/1.1

> Host: our.gerrit.url

> Accept: application/vnd.git-lfs+json; charset=utf-8

> Authorization: Ssh: t+7LmOiQXJRcDkCC0lSLX5djkK3ULZdv0sxeMRuNJSdZbukW3XuJ0S3MFLcocYomf1UhOUBp/PFTlGhNw/aSyGv7VZKf11+2/F/Avkn8arY=

> Content-Length: 32

> Content-Type: application/vnd.git-lfs+json; charset=utf-8

> User-Agent: git-lfs/2.9.2 (GitHub; linux amd64; go 1.13.5)

>

{"ref":{"name":"refs/for/main"}}09:28:31.807422 trace git-lfs: HTTP: 405

< HTTP/1.1 405 Method Not Allowed

< Content-Length: 18

< Cache-Control: no-cache, no-store, max-age=0, must-revalidate

< Content-Type: text/plain;charset=iso-8859-1

< Date: Wed, 31 Aug 2022 07:28:32 GMT

< Expires: Mon, 01 Jan 1990 00:00:00 GMT

< Pragma: no-cache

< Server: Apache/2.4.18 (Ubuntu)

< X-Frame-Options: DENY

<

Remote "origin" does not support the LFS locking API. Consider disabling it with:

$ git config lfs.https://gerrit/tools/ProjectName.git/info/lfs.locksverify false

09:28:31.807547 trace git-lfs: tq: running as batched queue, batch size of 100

09:28:31.807666 trace git-lfs: run_command: git rev-list --stdin --objects --not --remotes=origin --

09:28:31.813105 trace git-lfs: tq: sending batch of size 6

09:28:31.813201 trace git-lfs: ssh cache: gerrit git-lfs-authenticate tools/ProjectName upload

09:28:31.813313 trace git-lfs: api: batch 6 files

09:28:31.813416 trace git-lfs: HTTP: POST https://our.gerrit.url/tools/ProjectName/info/lfs/objects/batch

> POST /tools/ProjectName/info/lfs/objects/batch HTTP/1.1

> Host: our.gerrit.url

> Accept: application/vnd.git-lfs+json; charset=utf-8

> Authorization: Ssh: t+7LmOiQXJRcDkCC0lSLX5djkK3ULZdv0sxeMRuNJSdZbukW3XuJ0S3MFLcocYomf1UhOUBp/PFTlGhNw/aSyGv7VZKf11+2/F/Avkn8arY=

> Content-Length: 654

> Content-Type: application/vnd.git-lfs+json; charset=utf-8

> User-Agent: git-lfs/2.9.2 (GitHub; linux amd64; go 1.13.5)

>

{"operation":"upload","objects":[{"oid":"a346017c7c9ea59cdc4a3663f3556f32449ab415de74e2c37351ed433a13199a","size":254134704},{"oid":"8a9fdb067304d08b2cdb08a93eb06860ecbbe0a39e721877e16bd63aa73c0de7","size":76909572},{"oid":"2b3e354ced44dda3aa49b4cf8a107cefe4396b2a5fcf2935710e66d8a984bfbf","size":66867275},{"oid":"4b33ab7652d9b756027a71375a1fb858c888fe2c1449e13fd5ad9ae1d7fb9157","size":40598989},{"oid":"143924a98276629aea6906adc19f8dbc9cff6e09c658fa93eeaaf10cf197970a","size":30544786},{"oid":"2e01a3a15cf277d242e81b8cde8ed6f20278f9916634b147e705901835de2228","size":204236}],"transfers":["basic","lfs-standalone-file"],"ref":{"name":"refs/for/main"}}09:28:31.821413 trace git-lfs: HTTP: 405

< HTTP/1.1 405 Method Not Allowed

< Content-Length: 18

< Cache-Control: no-cache, no-store, max-age=0, must-revalidate

< Content-Type: text/plain;charset=iso-8859-1

< Date: Wed, 31 Aug 2022 07:28:32 GMT

< Expires: Mon, 01 Jan 1990 00:00:00 GMT

< Pragma: no-cache

< Server: Apache/2.4.18 (Ubuntu)

< X-Frame-Options: DENY

<

09:28:31.821539 trace git-lfs: api error: Client error: https://our.gerrit.url/tools/ProjectName/info/lfs/objects/batch from HTTP 405

batch response: Client error: https://our.gerrit.url/tools/ProjectName/info/lfs/objects/batch from HTTP 405

error: failed to push some refs to 'gerrit:tools/ProjectName'

Björn Pedersen

unread,

Aug 31, 2022, 6:57:13 AM8/31/22

to Repo and Gerrit Discussion

Roman Karlstetter schrieb am Mittwoch, 31. August 2022 um 12:43:20 UTC+2:

< Server: Apache/2.4.18 (Ubuntu)

This hints that there is a reverse proxy in front of gerrit. I would check that there is no special logic for tools configured there.

Roman Karlstetter

unread,

Aug 31, 2022, 9:50:44 AM8/31/22

to Repo and Gerrit Discussion

Thanks for the quick response. Yes, our instance is behind an Apache Reverse Proxy – if we disable the proxy and directly access our gerrit instance we experience the same issue. Any other ideas?

Roman

Mikko Paukkila

unread,

Sep 2, 2022, 6:58:32 AM9/2/22

to Repo and Gerrit Discussion

Hi!

I don't have a solution, but please check also https://groups.google.com/g/repo-discuss/c/MxD0jIL3Slo/m/rvMDFwKOAgAJ that is a duplicate of this.

(Apache reverse proxy also in use, but I don't believe it matters.)

Br. Mikko

Roman Karlstetter

unread,

Sep 2, 2022, 9:08:15 AM9/2/22

to Mikko Paukkila, Repo and Gerrit Discussion

Hi Mikko,

thanks for the pointer – I created a bug report for tracking this issue: https://bugs.chromium.org/p/gerrit/issues/detail?id=16237

Roman

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/0f6884da-e276-4a0f-901d-68fbc14978a6n%40googlegroups.com.

Reply all

Reply to author

Forward