Lot's of MissingObjectException after upgrading to 2.1.*
Eric Tsai
MissingObjectException in error_log.
The client may see
fatal: internal server error
fatal: The remote end hung up unexpectedly
fatal: protocol error: bad pack header
OR
Fetching projects: 62% (10/16) fatal: internal server error
fatal: The remote end hung up unexpectedly
error: Cannot fetch kernel_abc
Sometimes I don't need to do anything then the problem disappears.
Sometimes I restart Gerrit then the problem disappears.
This problem happens on all Gerrit managed repositories.
The missing commit in the log for a specific repository may be
different when the symptom occurs..
Here are some different kinds of log:
[2010-02-25 12:17:08,889] ERROR com.google.gerrit.sshd.BaseCommand :
Internal server error (user andy account 1000007) during git-upload-
pack '/kernel_abc.git'
org.eclipse.jgit.errors.MissingObjectException: Missing commit
eedb89f8337413476f43b432beee6c94f8fac578
at
org.eclipse.jgit.revwalk.RevObject.loadCanonical(RevObject.java:77)
at
org.eclipse.jgit.revwalk.RevCommit.parseHeaders(RevCommit.java:83)
at
org.eclipse.jgit.revwalk.PendingGenerator.next(PendingGenerator.java:
138)
at
org.eclipse.jgit.revwalk.TopoSortGenerator.<init>(TopoSortGenerator.java:
72)
at
org.eclipse.jgit.revwalk.StartGenerator.next(StartGenerator.java:145)
at org.eclipse.jgit.revwalk.RevWalk.next(RevWalk.java:381)
at org.eclipse.jgit.revwalk.ObjectWalk.next(ObjectWalk.java:
208)
at
org.eclipse.jgit.lib.PackWriter.findObjectsToPack(PackWriter.java:856)
at org.eclipse.jgit.lib.PackWriter.preparePack(PackWriter.java:
494)
at
org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:455)
at
org.eclipse.jgit.transport.UploadPack.service(UploadPack.java:246)
at
org.eclipse.jgit.transport.UploadPack.upload(UploadPack.java:227)
at com.google.gerrit.sshd.commands.Upload.runImpl(Upload.java:
26)
at
com.google.gerrit.sshd.commands.AbstractGitCommand.service(AbstractGitCommand.java:
62)
at com.google.gerrit.sshd.commands.AbstractGitCommand.access
$100(AbstractGitCommand.java:30)
at com.google.gerrit.sshd.commands.AbstractGitCommand
$1.run(AbstractGitCommand.java:46)
at com.google.gerrit.sshd.BaseCommand
$TaskThunk.run(BaseCommand.java:405)
at java.util.concurrent.Executors
$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask
$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ScheduledThreadPoolExecutor
$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor
$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at com.google.gerrit.server.git.WorkQueue
$Task.run(WorkQueue.java:310)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
[2010-02-25 16:10:18,275] ERROR com.google.gerrit.sshd.BaseCommand :
Internal server error (user hk account 1000020) during git-upload-pack
'/kernel_abc.git'
java.io.InterruptedIOException
at
org.apache.sshd.common.channel.ChannelPipedInputStream.read(ChannelPipedInputStream.java:
83)
at org.eclipse.jgit.util.IO.readFully(IO.java:122)
at
org.eclipse.jgit.transport.PacketLineIn.readLength(PacketLineIn.java:
120)
at
org.eclipse.jgit.transport.PacketLineIn.readString(PacketLineIn.java:
92)
at
org.eclipse.jgit.transport.UploadPack.recvWants(UploadPack.java:269)
at
org.eclipse.jgit.transport.UploadPack.service(UploadPack.java:241)
at
org.eclipse.jgit.transport.UploadPack.upload(UploadPack.java:227)
at com.google.gerrit.sshd.commands.Upload.runImpl(Upload.java:
26)
at
com.google.gerrit.sshd.commands.AbstractGitCommand.service(AbstractGitCommand.java:
62)
at com.google.gerrit.sshd.commands.AbstractGitCommand.access
$100(AbstractGitCommand.java:30)
at com.google.gerrit.sshd.commands.AbstractGitCommand
$1.run(AbstractGitCommand.java:46)
at com.google.gerrit.sshd.BaseCommand
$TaskThunk.run(BaseCommand.java:405)
at java.util.concurrent.Executors
$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask
$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ScheduledThreadPoolExecutor
$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor
$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at com.google.gerrit.server.git.WorkQueue
$Task.run(WorkQueue.java:310)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.sshd.common.channel.ChannelPipedInputStream.read(ChannelPipedInputStream.java:
81)
... 20 more
[2010-02-25 20:16:04,674] ERROR
com.google.gerrit.server.cache.SelfPopulatingCache : Cannot lookup
com.google.gerrit.server.patch.PatchListKey@7a28758b in "diff"
net.sf.ehcache.CacheException: Could not fetch object for cache entry
with key "com.google.gerrit.server.patch.PatchListKey@7a28758b".
at
net.sf.ehcache.constructs.blocking.SelfPopulatingCache.get(SelfPopulatingCache.java:
88)
at
com.google.gerrit.server.cache.SelfPopulatingCache.get(SelfPopulatingCache.java:
107)
at
com.google.gerrit.server.patch.PatchListCacheImpl.get(PatchListCacheImpl.java:
85)
at
com.google.gerrit.server.patch.PatchListCacheImpl.get(PatchListCacheImpl.java:
97)
at
com.google.gerrit.server.patch.PatchListCacheImpl.get(PatchListCacheImpl.java:
89)
at
com.google.gerrit.httpd.rpc.changedetail.PatchSetDetailFactory.call(PatchSetDetailFactory.java:
83)
at
com.google.gerrit.httpd.rpc.changedetail.ChangeDetailFactory.loadCurrentPatchSet(ChangeDetailFactory.java:
188)
at
com.google.gerrit.httpd.rpc.changedetail.ChangeDetailFactory.call(ChangeDetailFactory.java:
108)
at
com.google.gerrit.httpd.rpc.changedetail.ChangeDetailFactory.call(ChangeDetailFactory.java:
53)
at com.google.gerrit.httpd.rpc.Handler.to(Handler.java:55)
at
com.google.gerrit.httpd.rpc.changedetail.ChangeDetailServiceImpl.changeDetail(ChangeDetailServiceImpl.java:
42)
at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown
Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
com.google.gwtjsonrpc.server.MethodHandle.invoke(MethodHandle.java:91)
at
com.google.gwtjsonrpc.server.JsonServlet.doService(JsonServlet.java:
376)
at
com.google.gwtjsonrpc.server.JsonServlet.service(JsonServlet.java:268)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:
820)
at
com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:
216)
at
com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:
141)
at
com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:
93)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:
63)
at
com.google.gwtexpui.server.CacheControlFilter.doFilter(CacheControlFilter.java:
76)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:
129)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:
59)
at
com.google.gerrit.httpd.RequestCleanupFilter.doFilter(RequestCleanupFilter.java:
54)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:
129)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:
59)
at
com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:
122)
at
com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:110)
at org.eclipse.jetty.servlet.ServletHandler
$CachedChain.doFilter(ServletHandler.java:1187)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:
425)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:
931)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:
362)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:
867)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:
117)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:
113)
at org.eclipse.jetty.server.Server.handle(Server.java:334)
at
org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:
559)
at org.eclipse.jetty.server.HttpConnection
$RequestHandler.content(HttpConnection.java:1007)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:
747)
at
org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:209)
at
org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:
406)
at
org.eclipse.jetty.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:
462)
at org.eclipse.jetty.util.thread.QueuedThreadPool
$2.run(QueuedThreadPool.java:436)
at java.lang.Thread.run(Thread.java:619)
Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing
unknown 53cace30b276b33a5adbcb4bdff30bf60e70d5ed
at org.eclipse.jgit.revwalk.RevWalk.parseAny(RevWalk.java:725)
at org.eclipse.jgit.revwalk.RevWalk.parseCommit(RevWalk.java:
654)
at
com.google.gerrit.server.patch.PatchListCacheImpl.readPatchList(PatchListCacheImpl.java:
112)
at
com.google.gerrit.server.patch.PatchListCacheImpl.compute(PatchListCacheImpl.java:
104)
at com.google.gerrit.server.patch.PatchListCacheImpl.access
$000(PatchListCacheImpl.java:50)
at com.google.gerrit.server.patch.PatchListCacheImpl
$2.createEntry(PatchListCacheImpl.java:79)
at com.google.gerrit.server.patch.PatchListCacheImpl
$2.createEntry(PatchListCacheImpl.java:76)
at com.google.gerrit.server.cache.SelfPopulatingCache
$1.createEntry(SelfPopulatingCache.java:60)
at
net.sf.ehcache.constructs.blocking.SelfPopulatingCache.get(SelfPopulatingCache.java:
72)
... 45 more
[2010-02-25 20:22:06,100] ERROR com.google.gerrit.sshd.BaseCommand :
Internal server error (user davy account 1000099) during git-upload-
pack '/kernel_abc.git'
org.eclipse.jgit.errors.PackProtocolException: internal revision error
at
org.eclipse.jgit.transport.UploadPack.okToGiveUpImp(UploadPack.java:
409)
at
org.eclipse.jgit.transport.UploadPack.okToGiveUp(UploadPack.java:393)
at
org.eclipse.jgit.transport.UploadPack.negotiate(UploadPack.java:348)
at
org.eclipse.jgit.transport.UploadPack.service(UploadPack.java:245)
at
org.eclipse.jgit.transport.UploadPack.upload(UploadPack.java:227)
at com.google.gerrit.sshd.commands.Upload.runImpl(Upload.java:
26)
at
com.google.gerrit.sshd.commands.AbstractGitCommand.service(AbstractGitCommand.java:
62)
at com.google.gerrit.sshd.commands.AbstractGitCommand.access
$100(AbstractGitCommand.java:30)
at com.google.gerrit.sshd.commands.AbstractGitCommand
$1.run(AbstractGitCommand.java:46)
at com.google.gerrit.sshd.BaseCommand
$TaskThunk.run(BaseCommand.java:405)
at java.util.concurrent.Executors
$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask
$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ScheduledThreadPoolExecutor
$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor
$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at com.google.gerrit.server.git.WorkQueue
$Task.run(WorkQueue.java:310)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing
commit 459831401236fbfc6c489fa3b4e603103a0a86c5
at
org.eclipse.jgit.revwalk.RevObject.loadCanonical(RevObject.java:77)
at
org.eclipse.jgit.revwalk.RevCommit.parseHeaders(RevCommit.java:83)
at
org.eclipse.jgit.revwalk.PendingGenerator.next(PendingGenerator.java:
138)
at org.eclipse.jgit.revwalk.RevWalk.next(RevWalk.java:381)
at
org.eclipse.jgit.transport.UploadPack.wantSatisfied(UploadPack.java:
418)
at
org.eclipse.jgit.transport.UploadPack.okToGiveUpImp(UploadPack.java:
405)
... 18 more
Fredrik Luthander
Here at my company we also have seen the same problems. Typically it
occurs more often on gits that see very large objects (binaries) in
their commits.
We usually don't have to resort to a Gerrit restart (even if it helps,
it's sort of overkill), usually we can open a shell on the hosting
server, cd into the git directory and just run a
$ git gc
That seems to force Gerrit to recache the contents of the repository.
Right now we don't have any good approach on how we avoid the event
that triggers this state on the server. Also looking for help on that
one!
BR,
Fredrik Luthander
Ulrik Sjölin
poking around...
It seems that the commit actually has been pushed properly (we can
access it via giweb
if we provide the SHA-1). But when the user clicks the change-id in
the Gerrit web interface
the "not found" message is shown on the client and the stacktrace that
starts with
"selfpopulatingcache" is in the server log. I could not find any other
related error messages
in the server log. Looking on the file system the change is not under
refs/changes even though
the commit is in the git.
Any hints, tips on how to extract more info to debug this problem?
/Ulrik
On Mar 1, 12:47 pm, Fredrik Luthander
> ...
>
> read more »
Shawn Pearce
> I have just been with one our users that has this problem and did some
> poking around...
> It seems that the commit actually has been pushed properly (we can
> access it via giweb
> if we provide the SHA-1). But when the user clicks the change-id in
> the Gerrit web interface
> the "not found" message is shown on the client and the stacktrace that
> starts with
> "selfpopulatingcache" is in the server log. I could not find any other
> related error messages
> in the server log. Looking on the file system the change is not under
> refs/changes even though
> the commit is in the git.
>
> Any hints, tips on how to extract more info to debug this problem?
The issue is Gerrit can't see the pack file anymore, because its
in-memory cache of the $GIT_DIR/objects/pack directory is out of
sync with the filesystem. It doesn't know the pack that contains
the commit exists, so it can't find the commit.
You *might* be able to fix it simply by touching the directory and
forcing its modification time to change. I've actually never tried
that on a running server, but it should cause JGit to rescan the
directory and update its cache.
As for why the cache is out of sync, there might be a race condition
deep within JGit, where an update gets lost as we add a new incoming
pack file to the repository. But I was pretty sure that code was
correct the last time I was in there.
Nicholas Mucci
[2010-02-25 20:16:04,674] ERROR
com.google.gerrit.server.cache.SelfPopulatingCache : Cannot lookup
com.google.gerrit.server.patch.PatchListKey@7a28758b in "diff"
net.sf.ehcache.CacheException: Could not fetch object for cache entry
with key "com.google.gerrit.server.patch.PatchListKey@7a28758b".
the user sees a popup window indicating an internal server error when
they try and view their change in Gerrit. I ran "git gc" on the
affected repository on the server and immediately after that the user
was able to see his change.
-Nick
On Mar 3, 3:01 pm, Shawn Pearce <s...@google.com> wrote:
> > I have just been with one our users that has this problem and did some
> > poking around...
> > It seems that the commit actually has been pushed properly (we can
> > access it via giweb
> > if we provide the SHA-1). But when the user clicks the change-id in
> > the Gerrit web interface
> > starts with
> > "selfpopulatingcache" is in the server log. Icouldnotfind any other
> > in the server log. Looking on the file system the change isnotunder
> > refs/changes even though
> > the commit is in the git.
>
> > Any hints, tips on how to extract more info to debug this problem?
>
> The issue is Gerrit can't see the pack file anymore, because its
> the commit exists, so it can't find the commit.
>
> You *might* be able to fix it simply by touching the directory and
> forcing its modification time to change. I've actually never tried
> that on a running server, but it should cause JGit to rescan the
> directory and update itscache.
>
> pack file to the repository. But I was pretty sure that code was
> correct the last time I was in there.
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en
Shawn Pearce
> Hate to wake an old thread, but I have seen the
>
> [2010-02-25 20:16:04,674] ERROR
> com.google.gerrit.server.cache.SelfPopulatingCache : Cannot lookup
> com.google.gerrit.server.patch.PatchListKey@7a28758b in "diff"
> net.sf.ehcache.CacheException: Could not fetch object for cache entry
> with key "com.google.gerrit.server.patch.PatchListKey@7a28758b".
>
> exception on my 2.0.23 installation. When the exception is thrown,
> the user sees a popup window indicating an internal server error when
> they try and view their change in Gerrit. I ran "git gc" on the
> affected repository on the server and immediately after that the user
> was able to see his change.
http://code.google.com/p/gerrit/issues/detail?id=390
Nicholas Mucci
possible if the server was under heavy load that this issue could
still come up? Here's what we saw from the command line:
[user@box project]$ repo upload .
Counting objects: 43, done.
Compressing objects: 100% (12/12), done.
Writing objects: 100% (15/15), 2.07 KiB, done.
Total 15 (delta 10), reused 0 (delta 0)
fatal: The remote end hung up unexpectedly
[FAILED] packages/apps/project/ project1retry (Upload failed)
-Nick
On Apr 21, 10:56 am, Shawn Pearce <s...@google.com> wrote:
> > Hate to wake an old thread, but I have seen the
>
> > [2010-02-25 20:16:04,674] ERROR
> > com.google.gerrit.server.cache.SelfPopulatingCache : Cannot lookup
> > com.google.gerrit.server.patch.PatchListKey@7a28758b in "diff"
> > net.sf.ehcache.CacheException:Couldnotfetchobjectforcacheentry
Shawn Pearce
> The particular repository that was affected was only 2MB. Is it
> possible if the server was under heavy load that this issue could
> still come up? Here's what we saw from the command line:
>
> [user@box project]$ repo upload .
> Counting objects: 43, done.
> Compressing objects: 100% (12/12), done.
> Writing objects: 100% (15/15), 2.07 KiB, done.
> Total 15 (delta 10), reused 0 (delta 0)
> fatal: internal server error
> fatal: The remote end hung up unexpectedly
of the in-memory list of known pack files, and a stat of the
repository doesn't notice it needs to rebuild that list from scratch.
This is more likely caused by the server being under load and having
two packs get inserted at the same time, with one of them being
dropped out accidentally, than it is having anything to do with the
size of the pack (or project) involved.
I really need to work on this bug. I'm hoping to wrap up this branch
level read stuff today, and then the rest of today is already shot for
me, but I really want to get to this bug today if I can. Its driving
everyone nuts. Its not the highest priority bug on the issue tracker,
but it should be.
Mihai Rusu
to do something (ie instead of getting an instant error with "git
fetch" it takes about 3 seconds and then I get the error) but does not
fix the problem. Then I tried the "git gc" workaround and looking at
"show-caches" output it does seem to have cleared a lot of the JGit
cache but it still errors on "git fetch" with MissingObjectException.
Then I restarted Gerrit which fixed it :)
--
Mihai Rusu
Eric Anderson
duft....@gmail.com
Markus Duft
this just happened to us when fetching from two related repositories from the same server (i.e. clone the first one, add a remote, fetch the second one). when fetching from the second one it says missing commit blah blah in the second repo (which i want to fetch now), where i verified that the commit is in the first repo really (which i allready have cloned). neither git gc nor a gerrit restart helps. gerrit version is 2.2.2.1. any hints?