A user did:
$ git push origin HEAD:refs/for/next
and got the following error:
Counting objects: 40, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (34/34), done.
Writing objects: 100% (38/38), 6.72 KiB, done.
Total 38 (delta 9), reused 0 (delta 0)
fatal: The remote end hung up unexpectedly
fatal: internal server error
After that the change is still listed in Gerrit but the page to view
the Change details reports "Not Found".
Gerrit stack trace of the error is:
[2010-08-06 17:00:30,679] ERROR
com.google.gerrit.server.cache.PopulatingCache : Cannot lookup
PatchListKey[prod/omnivore/main
BASE..49559f08f48f565e97270906984356a5e09df61f IGNORE_NONE] in "diff"
net.sf.ehcache.CacheException: Could not fetch object for cache entry
with key "PatchListKey[prod/omnivore/main
BASE..49559f08f48f565e97270906984356a5e09df61f IGNORE_NONE]".
at net.sf.ehcache.constructs.blocking.SelfPopulatingCache.get(SelfPopulatingCache.java:88)
at com.google.gerrit.server.cache.PopulatingCache.get(PopulatingCache.java:85)
at com.google.gerrit.server.patch.PatchListCacheImpl.get(PatchListCacheImpl.java:149)
at com.google.gerrit.server.patch.PatchListCacheImpl.get(PatchListCacheImpl.java:161)
at com.google.gerrit.server.patch.PatchListCacheImpl.get(PatchListCacheImpl.java:153)
at com.google.gerrit.httpd.rpc.changedetail.PatchSetDetailFactory.call(PatchSetDetailFactory.java:83)
at com.google.gerrit.httpd.rpc.changedetail.ChangeDetailFactory.loadCurrentPatchSet(ChangeDetailFactory.java:195)
at com.google.gerrit.httpd.rpc.changedetail.ChangeDetailFactory.call(ChangeDetailFactory.java:109)
at com.google.gerrit.httpd.rpc.changedetail.ChangeDetailFactory.call(ChangeDetailFactory.java:54)
at com.google.gerrit.httpd.rpc.Handler.to(Handler.java:65)
at com.google.gerrit.httpd.rpc.changedetail.ChangeDetailServiceImpl.changeDetail(ChangeDetailServiceImpl.java:46)
at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.google.gwtjsonrpc.server.MethodHandle.invoke(MethodHandle.java:91)
at com.google.gwtjsonrpc.server.JsonServlet.doService(JsonServlet.java:382)
at com.google.gwtjsonrpc.server.JsonServlet.service(JsonServlet.java:268)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:216)
at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:141)
at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:93)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:63)
at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:134)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:59)
at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:134)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:59)
at com.google.gwtexpui.server.CacheControlFilter.doFilter(CacheControlFilter.java:76)
at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:129)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:59)
at com.google.gerrit.httpd.RequireSslFilter.doFilter(RequireSslFilter.java:68)
at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:129)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:59)
at com.google.gerrit.httpd.RequestCleanupFilter.doFilter(RequestCleanupFilter.java:54)
at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:129)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:59)
at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:134)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:59)
at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:122)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:110)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1190)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:424)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:931)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:361)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:867)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:113)
at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:50)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:113)
at org.eclipse.jetty.server.Server.handle(Server.java:337)
at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:581)
at org.eclipse.jetty.server.HttpConnection$RequestHandler.content(HttpConnection.java:1020)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:775)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:228)
at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:417)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:474)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:437)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IndexOutOfBoundsException: Index: 10, Size: 10
at java.util.ArrayList.RangeCheck(ArrayList.java:547)
at java.util.ArrayList.get(ArrayList.java:322)
at org.eclipse.jgit.diff.RenameDetector.findExactRenames(RenameDetector.java:411)
at org.eclipse.jgit.diff.RenameDetector.compute(RenameDetector.java:277)
at org.eclipse.jgit.diff.RenameDetector.compute(RenameDetector.java:258)
at com.google.gerrit.server.patch.PatchListCacheImpl$Loader.readPatchList(PatchListCacheImpl.java:225)
at com.google.gerrit.server.patch.PatchListCacheImpl$Loader.createEntry(PatchListCacheImpl.java:178)
at com.google.gerrit.server.patch.PatchListCacheImpl$Loader.createEntry(PatchListCacheImpl.java:164)
at com.google.gerrit.server.cache.PopulatingCache$1.createEntry(PopulatingCache.java:55)
at net.sf.ehcache.constructs.blocking.SelfPopulatingCache.get(SelfPopulatingCache.java:72)
... 56 more
What could be done to either remove the change from Gerrit so that it
can be pused again
or to make data in Gerrit consistent again?
Is there a possibility that the Gir repo is now also in an inconsistent state?
Sasa Zivkov
Ouch! That is a bug in the new rename code. And there's no way to
work around it. :-(
> What could be done to either remove the change from Gerrit so that it
> can be pused again
> or to make data in Gerrit consistent again?
Well, if the user amended his/her change to keep it from renaming
files with identical SHA-1s they might be able to work around the bug.
This is horrible but... if the file was a source file, maybe just add
a "// Gerrit is dumb" comment to the renamed file to force a new SHA-1
and break out of the exact rename code into the inexact rename
detection logic? Push that as a new patch set to the change. You
won't be able to look at patch set 1, but patch set 2 might be OK.
> Is there a possibility that the Gir repo is now also in an inconsistent state?
The Git repository is fine. There's no data corruption there. Gerrit
just can't figure out what the modified file list is. If we manage to
fix JGit this change would start to display correctly.
Good news is I think I found the bug. If there are more than one
files with the same SHA-1 being added or deleted in the commit we
build a matrix and try to rank the files by similarity on path name.
If the matrix is non-square, we get the ArrayIndexOutOfBoundsException
you saw in your server log because the matrix indices were crossed
with each other by accident. :-(
I'm working up a fix in JGit and will spin a new -rc later today. But
in the mean time another work around is to just delete another file
with the same SHA-1 so that the rename matrix is square again. :-)
FYI, fixed in Gerrit 2.1.4-rc4.
This was fast!
Thanks for your support.
I'll deploy it early on Monday.
Saša Živkov
FWIW, I think the change will automatically fix itself after you
upgrade the server.
Upgrade done and, yes, the change fixed itself as you said.
This quick bugfix contributed significantly to already good reputation
of Gerrit in my company.