Another "org.eclipse.jgit.errors.MissingObjectException" scenario

[Dave Castagna (Motorola Mobility)]

So we're running 2.6.1 (we'll be upgrading hopefully to 2.8.4 soon) with a MySQL backend.

We have a setup with 4 mirrors.  Call them "A", "B", "C" and "D".  Mirrors "A" and "B" have their own git storage.  Mirrors "C" and "D" NFS-mount mirror "A"s storage.  All 4 run the GerritCodeReview process in slave mode and all four lock out pushes (allow pulls/clones only).

On mirrors "C" and "D" we are seeing this exception occur seemingly at random:

[2014-05-05 09:06:25,576] ERROR com.google.gerrit.sshd.BaseCommand : Internal server error (user <USER1> account 1001120) during git-upload-pack '/<PROJECT1>'

org.eclipse.jgit.transport.UploadPackInternalServerErrorException

        at org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:1041)

        at org.eclipse.jgit.transport.UploadPack.service(UploadPack.java:612)

        at org.eclipse.jgit.transport.UploadPack.upload(UploadPack.java:521)

        at com.google.gerrit.sshd.commands.Upload.runImpl(Upload.java:57)

        at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:101)

        at com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:32)

        at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:70)

        at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:429)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

        at java.util.concurrent.FutureTask.run(FutureTask.java:262)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)

        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:337)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:744)

Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing tree 5c96af9445e1b4c7ba1814b21ef26487f90b0bca

        at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:149)

        at org.eclipse.jgit.revwalk.ObjectWalk.markTreeUninteresting(ObjectWalk.java:673)

        at org.eclipse.jgit.revwalk.ObjectWalk.next(ObjectWalk.java:268)

        at org.eclipse.jgit.internal.storage.pack.PackWriter.findObjectsToPack(PackWriter.java:1753)

        at org.eclipse.jgit.internal.storage.pack.PackWriter.preparePack(PackWriter.java:797)

        at org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:1150)

        at org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:1035)

        ... 15 more

This is followed some time later by this exception signature:

[2014-05-05 09:07:08,719] WARN  com.google.gerrit.server.git.TagSet : Error on refs/heads/<BRANCH1> of <PROJECT1>

org.eclipse.jgit.errors.MissingObjectException: Missing unknown 82519f1a3fc974908b3a87b2fe7e4beb929d2924

        at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:148)

        at org.eclipse.jgit.lib.ObjectReader.open(ObjectReader.java:229)

        at org.eclipse.jgit.revwalk.RevWalk.parseAny(RevWalk.java:809)

        at org.eclipse.jgit.revwalk.RevWalk.parseCommit(RevWalk.java:722)

        at com.google.gerrit.server.git.TagSet.addRef(TagSet.java:318)

        at com.google.gerrit.server.git.TagSet.build(TagSet.java:172)

        at com.google.gerrit.server.git.TagSetHolder.rebuild(TagSetHolder.java:88)

        at com.google.gerrit.server.git.TagSetHolder.matcher(TagSetHolder.java:54)

        at com.google.gerrit.server.git.VisibleRefFilter.filter(VisibleRefFilter.java:108)

        at com.google.gerrit.server.git.VisibleRefFilter.filter(VisibleRefFilter.java:129)

        at com.google.gerrit.server.git.VisibleRefFilter.getAdvertisedRefs(VisibleRefFilter.java:125)

        at org.eclipse.jgit.transport.AbstractAdvertiseRefsHook.advertiseRefs(AbstractAdvertiseRefsHook.java:62)

        at org.eclipse.jgit.transport.UploadPack.sendAdvertisedRefs(UploadPack.java:669)

        at org.eclipse.jgit.transport.UploadPack.service(UploadPack.java:554)

        at org.eclipse.jgit.transport.UploadPack.upload(UploadPack.java:521)

        at com.google.gerrit.sshd.commands.Upload.runImpl(Upload.java:57)

        at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:101)

        at com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:32)

        at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:70)

        at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:429)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

        at java.util.concurrent.FutureTask.run(FutureTask.java:262)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)

        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:337)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:744)

[2014-05-05 09:07:08,720] WARN  com.google.gerrit.server.git.TagSet : Error on refs/heads/<BRANCH1> of <PROJECT1>

org.eclipse.jgit.errors.MissingObjectException: Missing unknown ffb873887bcb264db66f02e7c600d405acd5cf57

        at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:148)

        at org.eclipse.jgit.lib.ObjectReader.open(ObjectReader.java:229)

        at org.eclipse.jgit.revwalk.RevWalk.parseAny(RevWalk.java:809)

        at org.eclipse.jgit.revwalk.RevWalk.parseCommit(RevWalk.java:722)

        at com.google.gerrit.server.git.TagSet.addRef(TagSet.java:318)

        at com.google.gerrit.server.git.TagSet.build(TagSet.java:172)

        at com.google.gerrit.server.git.TagSetHolder.rebuild(TagSetHolder.java:88)

        at com.google.gerrit.server.git.TagSetHolder.matcher(TagSetHolder.java:54)

        at com.google.gerrit.server.git.VisibleRefFilter.filter(VisibleRefFilter.java:108)

        at com.google.gerrit.server.git.VisibleRefFilter.filter(VisibleRefFilter.java:129)

        at com.google.gerrit.server.git.VisibleRefFilter.getAdvertisedRefs(VisibleRefFilter.java:125)

        at org.eclipse.jgit.transport.AbstractAdvertiseRefsHook.advertiseRefs(AbstractAdvertiseRefsHook.java:62)

        at org.eclipse.jgit.transport.UploadPack.sendAdvertisedRefs(UploadPack.java:669)

        at org.eclipse.jgit.transport.UploadPack.service(UploadPack.java:554)

        at org.eclipse.jgit.transport.UploadPack.upload(UploadPack.java:521)

        at com.google.gerrit.sshd.commands.Upload.runImpl(Upload.java:57)

        at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:101)

        at com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:32)

        at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:70)

        at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:429)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

        at java.util.concurrent.FutureTask.run(FutureTask.java:262)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)

        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:337)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:744)

Once this starts the same exception pattern starts occurring at a very high rate.  Yesterday we saw in excess of 200,000 of these in 16 hours of operation.

When this happens the projects start to disappear from the project listing on that mirror.  And the problem starts to spread.  When we found it yesterday we were up to 178 missing projects out of several thousand.

When we see it happen only 2 things fix it:

1) A git gc (we tried several other commands - none of which seemed to help - including doing things like "find . -exec cat {} \; > /dev/null",  "git fsck", "git repack" from inside the git).  Also a "gerrit flush-caches" followed by a "gerrit ls-projects" also did nothing.

2) A server process restart.

The git gc will only fix the one git whereas the server process restart seems to take care of everything at once.

The other two mirrors "A" and "B" that host their own storage are not affected.

There appears to be no correlation to server load / thread load on the Gerrit process as all four mirrors had a light to moderate load at the time the problem started.

I suspect some kind of NFS related stale file kind of problem.  But I am not an expert on NFS and have no way to prove the assertion.  It would seem to be corroborated by other posts related to this error.  In this post Shawn suggests that a similar error could be caused by:

Gerrit can't see the pack file anymore, because its in-memory cache of the $GIT_DIR/objects/pack directory is out of sync with the filesystem.  It doesn't know the pack that contains the commit exists, so it can't find the commit.

That would seem to be confirmed by the "git gc" which would refresh the git sufficiently for it to need to be re-parsed right?

So three questions:

1) How do I confirm that this is being caused by NFS?

2) Short of doing hourly mirror restarts - which will still not fix the issue - what can I do to keep this from happening?

3) Has this particular scenario been fixed in something post 2.6.1?

[Saša Živkov]

Check also this JGit bug:

https://bugs.eclipse.org/bugs/show_bug.cgi?id=433387

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dave Castagna (Motorola Mobility)]

That looks completely unrelated to my situation.

The gits work fine on one mirror but not on another mirror using the same physical storage.

[Saša Živkov]

On Tue, May 6, 2014 at 4:29 PM, Dave Castagna (Motorola Mobility) <cast...@motorola.com> wrote:

That looks completely unrelated to my situation.

The gits work fine on one mirror but not on another mirror using the same physical storage.

Sorry, I didn't read your post carefully.

[Bassem Rabil]

We experience the same behavior with Gerrit 2.8.4, I expect that the 2 mirrors sharing the same NFS for repositories is causing such behaviour where one mirror puts some pack file information in memory and gets stuck there. This information probably is not consistent due to changes from the other mirror using the same NFS. This outdated information in the memory of one mirror prevents many transactions from being executed on the repository until the application is restarted and the up-to-date information from NFS is retrieved and then the repository is back to normal. We tried to flush all application caches and none of them helped to get rid of this issue. 

I think this might be related to this bug in Gerrit which turned to be jgit bug which is fixed in jgit 3.4

https://code.google.com/p/gerrit/issues/detail?id=2025#c13

https://bugs.eclipse.org/bugs/show_bug.cgi?id=426044

Is there a way to expose such jgit caches so admins can flush such caches without the need to restart the instance ?

Thanks and Regards

Bassem Guendy

[clou...@qq.com]

The version of our jgit is "4.0.0.201506090130-r", and our gerrit is 2.8.5, but wo still encountered this problem.

[2015-10-22 16:28:04,573] WARN  com.google.gerrit.server.git.TagSet : Error on refs/heads/XXX of PROJECT

org.eclipse.jgit.errors.MissingObjectException: Missing unknown d65e2650ce211393395390d52fde0b955e81921e

        at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:145)

        at org.eclipse.jgit.lib.ObjectReader.open(ObjectReader.java:226)

        at org.eclipse.jgit.revwalk.RevWalk.parseAny(RevWalk.java:859)

        at org.eclipse.jgit.revwalk.RevWalk.parseCommit(RevWalk.java:772)

        at com.google.gerrit.server.git.TagSet.addRef(TagSet.java:319)

        at com.google.gerrit.server.git.TagSet.build(TagSet.java:173)

        at com.google.gerrit.server.git.TagSetHolder.rebuild(TagSetHolder.java:88)

        at com.google.gerrit.server.git.TagSetHolder.matcher(TagSetHolder.java:54)

        at com.google.gerrit.server.git.VisibleRefFilter.filter(VisibleRefFilter.java:113)

        at com.google.gerrit.server.git.VisibleRefFilter.filter(VisibleRefFilter.java:142)

        at com.google.gerrit.server.git.VisibleRefFilter.getAdvertisedRefs(VisibleRefFilter.java:131)

        at org.eclipse.jgit.transport.AbstractAdvertiseRefsHook.advertiseRefs(AbstractAdvertiseRefsHook.java:62)

        at org.eclipse.jgit.transport.UploadPack.sendAdvertisedRefs(UploadPack.java:785)

        at org.eclipse.jgit.transport.UploadPack.service(UploadPack.java:664)

        at org.eclipse.jgit.transport.UploadPack.upload(UploadPack.java:630)

        at com.google.gerrit.sshd.commands.Upload.runImpl(Upload.java:80)

        at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:101)

        at com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:32)

        at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:70)

        at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:442)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

        at java.util.concurrent.FutureTask.run(FutureTask.java:262)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)

        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:370)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:745)

Thanks

在 2014年5月7日星期三 UTC+8上午3:18:13，Bassem Rabil写道：

[David Ostrovsky]

Am Donnerstag, 29. Oktober 2015 11:41:05 UTC+1 schrieb clou...@qq.com:

The version of our jgit is "4.0.0.201506090130-r", and our gerrit is 2.8.5, but wo still encountered this problem.

Intermittent pack file corruption is a known Gerrit bug and was fixed in

Gerrit 2.10.7 and 2.11.4 releases.

[clou...@qq.com]

Can you tell me the patch which is fixed this bug. And Can I aplly this patch to the gerrit 2.10.5 (sorry for that I writed the wrong gerrit version last time).

Thanks.
在 2015年10月29日星期四 UTC+8下午6:53:47，David Ostrovsky写道：

[Bassem Rabil]

The patch for 2.10 based releases is: https://gerrit-review.googlesource.com/#/c/71381/