java.io.IOException: Unpack error after org.eclipse.jgit.errors.MissingObjectException: Missing tree (Gerrit 2.12.1)

[Matthew Webber]

Yesterday I upgraded our Gerrit instance from 2.10 to 2.12.1, and today we are getting a significant number of failures when people push to Gerrit. The key parts of the stack track (from the Gerrit server error_log) are

[2016-03-08 13:54:10,034] [SSH git-receive-pack '/gda/gda-core.git' (xxxx.xxxx)] ERROR com.google.gerrit.sshd.BaseCommand : Internal server error (user xxxx.xxxx account 1000023) during git-receive-pack '/gda/gda-core.git'

com.google.gerrit.sshd.BaseCommand$Failure: fatal: Unpack error, check server log

at com.google.gerrit.sshd.commands.Receive.runImpl(Receive.java:176)

Caused by: java.io.IOException: Unpack error on project "gda/gda-core":

  AdvertiseRefsHook: org.eclipse.jgit.transport.AdvertiseRefsHookChain@404d370eclass org.eclipse.jgit.transport.AdvertiseRefsHookChain

Caused by: org.eclipse.jgit.errors.UnpackException: Exception while parsing pack stream

Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing tree 0a6d97add35f8cfe79e5c9fc645caa97082234ae

Other occurrences have org.eclipse.jgit.errors.MissingObjectException: Missing blob (rather than Missing tree) but are otherwise the same.

A web search shows various mentions of this, but the underlying cause is not clear. Some people suggest that it occurs after a "gerrit gc" is run on the repo. This does not apply in our case: prior to the upgrade, and while Gerrit was stopped, we ran a "git gc" from the command line in our repositories. The first occurrence of the problem was after than. The overnight, Gerrit scheduled its own gc, but the problem still occurs.

I have seen a suggestion that setting receive.checkReferencedObjectsAreReachable=false will avoid the error, but I have not yet tried that.

Does anyone have any suggestions about how to proceed? We don't really want to roll back to 2.10 (indeed, I'm not even sure if that is possible).

Thanks

[Matthew Webber]

One of our developers did some further digging into this problem, and came up with some useful information:

Case 1:

Pushing using EGit had failed. Developer then tried pushing from the command line, which worked.

Case 2:

Pushing using EGit had failed. Developer then tried pushing from the command line, which failed, but gave a much more useful error message:

$git push origin HEAD:refs/for/gda-8.master
Counting objects: 43, done.
remote: Processing changes: refs: 1, done    

To ssh://gerrit.xxx:29418/gda/gda-core.git
 ! [remote rejected] HEAD -> refs/for/gda-8.master (change http://gerrit.xxx/8080/1814 closed)
error: failed to push some refs to 'ssh://gerrit.xxx:29418/gda/gda-core.git'

The error message (change 1814 closed) was enough to identify the problem.

• commit b148882 was gerrit pushed to gda-8.master as Change-Id: Ic7f3a7 (change/1814)
  
• This was reviewed and submitted successfully.
  
• It also ended up in the developers local repo as commit 26c354c (presumably cherry picked) with the same Change-Id.
  
• Developer was committing his changes on top of this commit.
  
• Pushing from Egit resulted in the unpacking error rather than the change "closed error" on the command line.
  
• After rebasing the children of 26c354c onto its parents, the gerrit push to gda-8.master succeeded from Egit.
  

The lesson: if EGit fails, try from the command line!

[Matthias Sohn]

which EGit version did he use ?

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Matthew Webber]

which EGit version did he use ?

Ah, that's a good point. Obviously the JGit in Gerrit 2.12.1 is the latest, but on the other side, the developer was using Eclipse Luna SR2, and its EGit/JGit is about a year old. We are about to try Eclipse Mars, and I'll post the result here.