Post Upgrade "errors"

78 views
Skip to first unread message

Nate Dreier

unread,
Jul 28, 2021, 12:47:45 PM7/28/21
to Repo and Gerrit Discussion
Hello, 

After a major upgrade from 2.11.2 -> 3.4 there is a new behavior that sometimes happens.
Is there some new functionality or changes to the way Gerrit handles commits that causes this error to happen more often than not? It did not happen almost ever in 2.11.2.

Thanks!

```
remote: Counting objects: 62004, done
error: remote unpack failed: error Missing tree
123344a567b889a900c2934785
fatal: Unpack error, check server log
```

Nasser Grainawi

unread,
Jul 28, 2021, 12:50:42 PM7/28/21
to Nate Dreier, Repo and Gerrit Discussion
Hey Nate,

Many many things changed between 2.11 and 3.4. It might be hard to narrow it down.

What’s the server-side error in error_log that’s recorded when this happens?

Thanks,
Nasser




--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/7d2e8894-cc5f-4f11-a941-88260d6bb5a1n%40googlegroups.com.

Clark Boylan

unread,
Jul 28, 2021, 1:05:09 PM7/28/21
to Nate Dreier, Repo and Gerrit Discussion
On Wed, Jul 28, 2021 at 9:47 AM Nate Dreier <ndr...@maystreet.com> wrote:
Hello, 

After a major upgrade from 2.11.2 -> 3.4 there is a new behavior that sometimes happens.
Is there some new functionality or changes to the way Gerrit handles commits that causes this error to happen more often than not? It did not happen almost ever in 2.11.2.

It seems this issue has long been a problem for us [0], but hard to say if it has gotten worse since our 3.2 upgrade. We did update git-review to add a --no-thin option for git push [1] which seems to workaround this. As mentioned in the commit message there, I suspect some sort of mismatch in expectations between C git and JGit which --no-thin bypasses by being far more expensive but also complete? This quickly gets beyond my ability to debug though.

As another note we have seen this with clients using git protocol v1 and v2.
 

Thanks!

```
remote: Counting objects: 62004, done
error: remote unpack failed: error Missing tree
123344a567b889a900c2934785
fatal: Unpack error, check server log
```

Nate Dreier

unread,
Jul 28, 2021, 1:12:47 PM7/28/21
to Repo and Gerrit Discussion
Thanks Nassar,

Nassar, unfortunately nothing in error_log, but Clark hit the nail on the head (so it seems).

Nate Dreier

unread,
Jul 28, 2021, 1:13:25 PM7/28/21
to Repo and Gerrit Discussion
Thanks for adding color to the issue, Clark

This helps a lot, gives me something to track and something to tell other devs as they come across the issue.

Nate

Clark Boylan

unread,
Jul 28, 2021, 1:23:23 PM7/28/21
to Nate Dreier, Repo and Gerrit Discussion
On Wed, Jul 28, 2021 at 10:13 AM Nate Dreier <ndr...@maystreet.com> wrote:
Thanks Nassar,

Nassar, unfortunately nothing in error_log, but Clark hit the nail on the head (so it seems).

I was able to dig up logs from a recent occurrence on our server and have attached them. I've edited them to remove any specific info, but if necessary we can probably share more. These logs were generated by Gerrit 3.2.11-7-gb1242861dd-dirty. The user reported they were running Ubuntu Focal's git version 2.25.1.
gerrit_unpack_error.txt

Nate Dreier

unread,
Jul 28, 2021, 1:32:26 PM7/28/21
to Repo and Gerrit Discussion
Ah okay, I found some error related to this. They are pretty close to identical. 

Is there much debugging that can be done given this is a bug with JGit? They are able to successfully push when using --no-thin

Though is this can be remedied without the above workaround that would be sweet!
nate

Nasser Grainawi

unread,
Jul 28, 2021, 1:46:21 PM7/28/21
to Nate Dreier, Repo and Gerrit Discussion
On Jul 28, 2021, at 11:32 AM, Nate Dreier <ndr...@maystreet.com> wrote:

Ah okay, I found some error related to this. They are pretty close to identical. 

Is there much debugging that can be done given this is a bug with JGit? They are able to successfully push when using --no-thin

Ah yes, this classic issue. You can see in https://git.eclipse.org/r/c/jgit/jgit/+/31081 that it was fixed at some point, but that fix introduced an enormous performance problem. The solution needs someone to spend some serious time in JGit to resolve it without adding a perf penalty. The other workaround that has been used is setting the Gerrit server config “receive.checkReferencedObjectsAreReachable” to false, as you see suggested in https://bugs.chromium.org/p/gerrit/issues/detail?id=1582. That provides a Gerrit-wide solution that doesn’t require clients to change their behavior, but it is a security tradeoff since JGit will do fewer checks on pushes.


Though is this can be remedied without the above workaround that would be sweet!
nate

On Wednesday, July 28, 2021 at 10:23:23 AM UTC-7 clark....@gmail.com wrote:
On Wed, Jul 28, 2021 at 10:13 AM Nate Dreier <ndr...@maystreet.com> wrote:
Thanks Nassar,

Nassar, unfortunately nothing in error_log, but Clark hit the nail on the head (so it seems).

I was able to dig up logs from a recent occurrence on our server and have attached them. I've edited them to remove any specific info, but if necessary we can probably share more. These logs were generated by Gerrit 3.2.11-7-gb1242861dd-dirty. The user reported they were running Ubuntu Focal's git version 2.25.1.
 

On Wednesday, July 28, 2021 at 9:50:42 AM UTC-7 nas...@codeaurora.org wrote:
On Jul 28, 2021, at 10:47 AM, Nate Dreier <ndr...@maystreet.com> wrote:

Hello, 

After a major upgrade from 2.11.2 -> 3.4 there is a new behavior that sometimes happens.
Is there some new functionality or changes to the way Gerrit handles commits that causes this error to happen more often than not? It did not happen almost ever in 2.11.2.

Thanks!

```
remote: Counting objects: 62004, done
error: remote unpack failed: error Missing tree
123344a567b889a900c2934785
fatal: Unpack error, check server log
```

Hey Nate,

Many many things changed between 2.11 and 3.4. It might be hard to narrow it down.

What’s the server-side error in error_log that’s recorded when this happens?

Thanks,
Nasser



--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

Nate Dreier

unread,
Jul 28, 2021, 2:02:15 PM7/28/21
to Repo and Gerrit Discussion
Oh thanks! I appreciate all that extra info. What do you mean by "security tradeoff"? What exactly is JGit checking per push?

Nasser Grainawi

unread,
Jul 28, 2021, 4:28:48 PM7/28/21
to Nate Dreier, Repo and Gerrit Discussion
On Jul 28, 2021, at 12:02 PM, Nate Dreier <ndr...@maystreet.com> wrote:

Oh thanks! I appreciate all that extra info. What do you mean by "security tradeoff"? What exactly is JGit checking per push?

Please try not to top-post, it makes it harder to follow the conversation.

Gerrit (through JGit) is confirming that all objects (commits) in the history of the branch(es) you’re pushing are reachable from a branch that’s visible to you. So it’s an access control check.

Clark Boylan

unread,
Jul 29, 2021, 5:55:48 PM7/29/21
to Nasser Grainawi, Nate Dreier, Repo and Gerrit Discussion
On Wed, Jul 28, 2021 at 10:46 AM Nasser Grainawi <nas...@codeaurora.org> wrote:

On Jul 28, 2021, at 11:32 AM, Nate Dreier <ndr...@maystreet.com> wrote:

Ah okay, I found some error related to this. They are pretty close to identical. 

Is there much debugging that can be done given this is a bug with JGit? They are able to successfully push when using --no-thin

Ah yes, this classic issue. You can see in https://git.eclipse.org/r/c/jgit/jgit/+/31081 that it was fixed at some point, but that fix introduced an enormous performance problem. The solution needs someone to spend some serious time in JGit to resolve it without adding a perf penalty. The other workaround that has been used is setting the Gerrit server config “receive.checkReferencedObjectsAreReachable” to false, as you see suggested in https://bugs.chromium.org/p/gerrit/issues/detail?id=1582. That provides a Gerrit-wide solution that doesn’t require clients to change their behavior, but it is a security tradeoff since JGit will do fewer checks on pushes.


https://bugs.chromium.org/p/gerrit/issues/detail?id=1582 was closed due to its age with the suggestion that it can be reopened if the issue can be reproduced with recent Gerrit. While I cannot artificially reproduce it, we are still observing it in the wild on two different Gerrit >=3.2 installations as indicated by this thread. Can this issue be reopened? I don't see a way to flip that state myself otherwise I would do it.

Luca Milanesio

unread,
Jul 29, 2021, 6:04:10 PM7/29/21
to Repo and Gerrit Discussion, Luca Milanesio
Can you post your steps to reproduce the issue with Gerrit >= 3.2 to the ticket?
Bear in mind that if we cannot reproduce the problem, we won’t be able to test it.

If you cannot test it or reproduce it, you cannot fix it.

Luca.

Clark Boylan

unread,
Jul 29, 2021, 6:07:39 PM7/29/21
to Luca Milanesio, Repo and Gerrit Discussion
It sounded like this was a known problem with JGit and other people do
understand it. As mentioned before we don't know what causes it to
happen to our users, but it does happen, it gets reported, and we have
them push with --no-thin to work around it. I understand that not
having reliable reproducers makes fixing this more difficult, but if
it is a known issue that is affecting real world users shouldn't it
continue to be tracked?

I'll have to defer to those that understand JGit and its interactions
with C git better for reliable reproduction.
Reply all
Reply to author
Forward
0 new messages