Cannot read project error (jgit MissingObjectException)

[Khai Do]

We are running Gerrit 2.11.4 and have recently had a problem in that Gerrit couldn't read one of our repositories making it totally unusable.   We tried running flush-caches --all but that did not fix the problem.  The only way we were able to fix it was to do a Gerrit restart.  I was wondering if anybody else has run into this issue or might know why this would be happening? 

Stack trace of the error is below and the change that Gerrit says is missing is a change to project.config in refs/meta/config (http://git.openstack.org/cgit/openstack/ec2-api/commit/?id=72803eb8013ff25d7751b4a7c6cd6571d6631435)

[2016-03-27 05:41:19,448] WARN  org.eclipse.jgit.internal.storage.file.ObjectDirectory : Pack file /home/gerrit2/review_site/git/openstack/ec2-api.git/objects/pack/pack-3a686787179a5e70d4c53e9be6897ec5ed0821b2.pack was deleted, removing it from pack list
[2016-03-27 05:41:19,450] WARN  com.google.gerrit.server.project.ProjectCacheImpl : Cannot read project openstack/ec2-api
java.util.concurrent.ExecutionException: org.eclipse.jgit.errors.MissingObjectException: Missing unknown 72803eb8013ff25d7751b4a7c6cd6571d6631435
        at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
        at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
        at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
        at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:137)
        at com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2348)
        at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2320)
        at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
        at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
        at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
        at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
        at com.google.gerrit.server.project.ProjectCacheImpl.checkedGet(ProjectCacheImpl.java:141)
        at com.google.gerrit.server.project.ProjectCacheImpl.get(ProjectCacheImpl.java:128)
        at com.google.gerrit.server.project.ListProjects.display(ListProjects.java:268)
        at com.google.gerrit.sshd.commands.ListProjectsCommand.run(ListProjectsCommand.java:43)
        at com.google.gerrit.sshd.SshCommand$1.run(SshCommand.java:35)
        at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:445)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:379)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing unknown 72803eb8013ff25d7751b4a7c6cd6571d6631435
        at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:145)
        at org.eclipse.jgit.lib.ObjectReader.open(ObjectReader.java:226)
        at org.eclipse.jgit.revwalk.RevWalk.parseAny(RevWalk.java:859)
        at org.eclipse.jgit.revwalk.RevWalk.parseCommit(RevWalk.java:772)
        at com.google.gerrit.server.git.VersionedMetaData.load(VersionedMetaData.java:123)
        at com.google.gerrit.server.git.VersionedMetaData.load(VersionedMetaData.java:100)
        at com.google.gerrit.server.project.ProjectCacheImpl$Loader.load(ProjectCacheImpl.java:294)
        at com.google.gerrit.server.project.ProjectCacheImpl$Loader.load(ProjectCacheImpl.java:278)
        at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
        at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
        ... 19 more

[Bassem Rabil]

We are facing similar errors with 2.11.4, and sometimes we are able to overcome this by pushing this SHA-1 from an older version of the repository. Can you relate such incidents at your end with garbage collection running on this repository ? If yes, what type of gc you are using ?

Regards

Bassem

...

[Khai Do]

We do not run git gc at all, we only run `git repack -afd` once a week.  We use git ver 1.9.1, not jgit.  I'm assuming your asking about gc because you think it might be related?    

-Khai

[Bassem Rabil]

I think it might be related specially you are using an older version for git for the repacking. You can confirm this if these issues are occurring right after running that during the weekend for example. I would try to run repacking using a newer git client, or you can try using the standalone JGit for full gc on your repositories. If you choose to use JGit standalone, you should get the bitmaps generated which is improving the performance for different git operations from the JGit within Gerrit application.

Regards

Bassem

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to a topic in the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/repo-discuss/g2W-m12fwPs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.