[Gerrit 2.11.10] git push fails for only one git-repo
Remy Bohmer
We are facing the last couple of days a very weird issue with Gerrit,
and we are looking for pointers on how to solve this. We hope someone
has an idea.
What we see is that a git push to that git-repo ends immediately after:
$ git push origin HEAD:refs/for/master/test; echo $?
141
There is no other feedback in the terminal than this error code 141...
Same happens for a push to any other ref in that repo.
If we run the following command we see a difference:
ssh -p 29418 <server> git-receive-pack '/<repo-name>', we see a long
list of all refs, ending with:
.....
003318963e870ad9b42c672c726b361fec8bc6bda44e .have
00338973e44ac0601e9acf1c3d37083e263e11455165 .have
00336620b5eaa13e1fc5ed7fba577872b900d94611b1 .have
00330c1e59ab8e919123a60476b3da1ed5b6118e028b .have
0000
The command on the error repository immediately exits here after the
0000. Exit code ($?) 0.
Gerrit terminates this command itself immediately.
On a GOOD repo we can give a dummy command and then Gerrit waits for X
time, and if we type something stupid it reports an error:
....
00330ca52e34982ac793386367b008327fc5b2f84652 .have
00338525790653750cb8de07a77f591f1bba3127474a .have
0033dcca0eeafd6582828b29ebcaf264c67994e746fe .have
0000blah
fatal: internal server error
So, if we type command blah with <enter> it reports an error (of course!)
The problem we see is that Gerrit immediately returns after running
this command on a bad repository, we do not even have the chance to
type such a stupid command...
We looked at the following:
* Git-repo is valid, no errors in there, and even cmdline git behaves
as you would expect on the 'blah' command. Even replacing it with an
empty git repo results in the same.
* Secondary index has been rebuild using the online reindex script
(the groovy based plugin script). No effect.
* The error log does not give any error. Nothing. Even full debugging
logging does not show any sign of root cause.
Anyone any idea?
Kind regards,
Remy
Matthias Sohn
Remy Bohmer
Hi,
Thanks.
Packet tracing and the git protocol internals manual actually helped us in getting to the point where we noticed that the ssh command for receivepack is behaving weird. Where it immediately exits such that the client does not get a chance of posting the command it needs to execute. So, we can limit the problem now to only run ssh git-receive-pack and see the difference. No git client needed to reproduce the issue.
It is something in the server side for sure. And it is proven not to be a client issue. Copying the git repo on the server to a new name makes it work again, but that is off course not a valid solution.
Somehow the server got into this state... We do not understand how it got there or how to solve it.
On a side note: we also noticed that in gerrit.config we have put the receive timeout to 10 minutes, but it times out after exactly 1 minute. But that is seperate from the problem we talk about here.
Kind regards,
Remy
Remy Bohmer
We found a database corruption related to draft patch deletion that
was not fully executed.
Fixed the database by removing the change --> problem is gone...
Kind regards,
Remy
Remy Bohmer
The problem popped up again. By diving further it turned out to be the
quota plugin that actually caused these issues.
This does not report any error if a quota exceeded. We removed this
plugin... It works again.
Apparently the removing of the draft changes and run proper cleanup
with git gc just got us below the limit, but in a couple of days it
came above it again...
Kind regards,
Remy