internal server error when pushing to Gerrit
Sasa Zivkov
A user did:
$ git push origin HEAD:refs/for/next
and got the following error:
Counting objects: 40, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (34/34), done.
Writing objects: 100% (38/38), 6.72 KiB, done.
Total 38 (delta 9), reused 0 (delta 0)
fatal: The remote end hung up unexpectedly
fatal: internal server error
After that the change is still listed in Gerrit but the page to view
the Change details reports "Not Found".
Gerrit stack trace of the error is:
[2010-08-06 17:00:30,679] ERROR
com.google.gerrit.server.cache.PopulatingCache : Cannot lookup
PatchListKey[prod/omnivore/main
BASE..49559f08f48f565e97270906984356a5e09df61f IGNORE_NONE] in "diff"
net.sf.ehcache.CacheException: Could not fetch object for cache entry
with key "PatchListKey[prod/omnivore/main
BASE..49559f08f48f565e97270906984356a5e09df61f IGNORE_NONE]".
at net.sf.ehcache.constructs.blocking.SelfPopulatingCache.get(SelfPopulatingCache.java:88)
at com.google.gerrit.server.cache.PopulatingCache.get(PopulatingCache.java:85)
at com.google.gerrit.server.patch.PatchListCacheImpl.get(PatchListCacheImpl.java:149)
at com.google.gerrit.server.patch.PatchListCacheImpl.get(PatchListCacheImpl.java:161)
at com.google.gerrit.server.patch.PatchListCacheImpl.get(PatchListCacheImpl.java:153)
at com.google.gerrit.httpd.rpc.changedetail.PatchSetDetailFactory.call(PatchSetDetailFactory.java:83)
at com.google.gerrit.httpd.rpc.changedetail.ChangeDetailFactory.loadCurrentPatchSet(ChangeDetailFactory.java:195)
at com.google.gerrit.httpd.rpc.changedetail.ChangeDetailFactory.call(ChangeDetailFactory.java:109)
at com.google.gerrit.httpd.rpc.changedetail.ChangeDetailFactory.call(ChangeDetailFactory.java:54)
at com.google.gerrit.httpd.rpc.Handler.to(Handler.java:65)
at com.google.gerrit.httpd.rpc.changedetail.ChangeDetailServiceImpl.changeDetail(ChangeDetailServiceImpl.java:46)
at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.google.gwtjsonrpc.server.MethodHandle.invoke(MethodHandle.java:91)
at com.google.gwtjsonrpc.server.JsonServlet.doService(JsonServlet.java:382)
at com.google.gwtjsonrpc.server.JsonServlet.service(JsonServlet.java:268)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:216)
at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:141)
at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:93)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:63)
at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:134)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:59)
at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:134)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:59)
at com.google.gwtexpui.server.CacheControlFilter.doFilter(CacheControlFilter.java:76)
at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:129)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:59)
at com.google.gerrit.httpd.RequireSslFilter.doFilter(RequireSslFilter.java:68)
at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:129)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:59)
at com.google.gerrit.httpd.RequestCleanupFilter.doFilter(RequestCleanupFilter.java:54)
at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:129)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:59)
at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:134)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:59)
at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:122)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:110)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1190)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:424)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:931)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:361)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:867)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:113)
at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:50)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:113)
at org.eclipse.jetty.server.Server.handle(Server.java:337)
at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:581)
at org.eclipse.jetty.server.HttpConnection$RequestHandler.content(HttpConnection.java:1020)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:775)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:228)
at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:417)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:474)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:437)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IndexOutOfBoundsException: Index: 10, Size: 10
at java.util.ArrayList.RangeCheck(ArrayList.java:547)
at java.util.ArrayList.get(ArrayList.java:322)
at org.eclipse.jgit.diff.RenameDetector.findExactRenames(RenameDetector.java:411)
at org.eclipse.jgit.diff.RenameDetector.compute(RenameDetector.java:277)
at org.eclipse.jgit.diff.RenameDetector.compute(RenameDetector.java:258)
at com.google.gerrit.server.patch.PatchListCacheImpl$Loader.readPatchList(PatchListCacheImpl.java:225)
at com.google.gerrit.server.patch.PatchListCacheImpl$Loader.createEntry(PatchListCacheImpl.java:178)
at com.google.gerrit.server.patch.PatchListCacheImpl$Loader.createEntry(PatchListCacheImpl.java:164)
at com.google.gerrit.server.cache.PopulatingCache$1.createEntry(PopulatingCache.java:55)
at net.sf.ehcache.constructs.blocking.SelfPopulatingCache.get(SelfPopulatingCache.java:72)
... 56 more
What could be done to either remove the change from Gerrit so that it
can be pused again
or to make data in Gerrit consistent again?
Is there a possibility that the Gir repo is now also in an inconsistent state?
Sasa Zivkov
Shawn Pearce
> Gerrit version: 2.1.4-rc2
...
> Gerrit stack trace of the error is:
> [2010-08-06 17:00:30,679] ERROR
> com.google.gerrit.server.cache.PopulatingCache : Cannot lookup
> PatchListKey[prod/omnivore/main
> BASE..49559f08f48f565e97270906984356a5e09df61f IGNORE_NONE] in "diff"
> net.sf.ehcache.CacheException: Could not fetch object for cache entry
> with key "PatchListKey[prod/omnivore/main
> BASE..49559f08f48f565e97270906984356a5e09df61f IGNORE_NONE]".
> Caused by: java.lang.IndexOutOfBoundsException: Index: 10, Size: 10
> at java.util.ArrayList.RangeCheck(ArrayList.java:547)
> at java.util.ArrayList.get(ArrayList.java:322)
> at org.eclipse.jgit.diff.RenameDetector.findExactRenames(RenameDetector.java:411)
Ouch! That is a bug in the new rename code. And there's no way to
work around it. :-(
> What could be done to either remove the change from Gerrit so that it
> can be pused again
> or to make data in Gerrit consistent again?
Well, if the user amended his/her change to keep it from renaming
files with identical SHA-1s they might be able to work around the bug.
This is horrible but... if the file was a source file, maybe just add
a "// Gerrit is dumb" comment to the renamed file to force a new SHA-1
and break out of the exact rename code into the inexact rename
detection logic? Push that as a new patch set to the change. You
won't be able to look at patch set 1, but patch set 2 might be OK.
> Is there a possibility that the Gir repo is now also in an inconsistent state?
The Git repository is fine. There's no data corruption there. Gerrit
just can't figure out what the modified file list is. If we manage to
fix JGit this change would start to display correctly.
Shawn Pearce
> Well, if the user amended his/her change to keep it from renaming
> files with identical SHA-1s they might be able to work around the bug.
Good news is I think I found the bug. If there are more than one
files with the same SHA-1 being added or deleted in the commit we
build a matrix and try to rank the files by similarity on path name.
If the matrix is non-square, we get the ArrayIndexOutOfBoundsException
you saw in your server log because the matrix indices were crossed
with each other by accident. :-(
I'm working up a fix in JGit and will spin a new -rc later today. But
in the mean time another work around is to just delete another file
with the same SHA-1 so that the rename matrix is square again. :-)
Shawn Pearce
>> Gerrit version: 2.1.4-rc2
> ...
>> at java.util.ArrayList.RangeCheck(ArrayList.java:547)
>> at java.util.ArrayList.get(ArrayList.java:322)
>> at org.eclipse.jgit.diff.RenameDetector.findExactRenames(RenameDetector.java:411)
>
> Ouch! That is a bug in the new rename code.
FYI, fixed in Gerrit 2.1.4-rc4.
Saša Živkov
This was fast!
Thanks for your support.
I'll deploy it early on Monday.
Saša Živkov
Shawn Pearce
>>>
>>> Ouch! That is a bug in the new rename code.
>>
>> FYI, fixed in Gerrit 2.1.4-rc4.
>
FWIW, I think the change will automatically fix itself after you
upgrade the server.
Sasa Zivkov
Upgrade done and, yes, the change fixed itself as you said.
This quick bugfix contributed significantly to already good reputation
of Gerrit in my company.