Merge attempt failed

[Nicholas Mucci]

Hello Gerrit Friends,

I have a Gerrit 2.4.2 box that has suddenly decided to stop merging new changes on one project.  The project uses the cherry-pick strategy, and yesterday I started seeing the following in the logs:

[2012-12-19 10:20:23,945] ERROR com.google.gerrit.server.git.ChangeMergeQueue : Merge attempt for foo/bar/project,refs/heads/master failed

java.lang.NullPointerException

        at com.google.gerrit.server.git.MergeOp.writeCherryPickCommit(MergeOp.java:912)

        at com.google.gerrit.server.git.MergeOp.cherryPickChanges(MergeOp.java:724)

        at com.google.gerrit.server.git.MergeOp.preMerge(MergeOp.java:301)

        at com.google.gerrit.server.git.MergeOp.merge(MergeOp.java:277)

        at com.google.gerrit.server.git.ChangeMergeQueue.mergeImpl(ChangeMergeQueue.java:181)

        at com.google.gerrit.server.git.ChangeMergeQueue.merge(ChangeMergeQueue.java:101)

        at com.google.gerrit.server.changedetail.Submit.call(Submit.java:184)

        at com.google.gerrit.httpd.rpc.changedetail.SubmitAction.call(SubmitAction.java:54)

        at com.google.gerrit.httpd.rpc.changedetail.SubmitAction.call(SubmitAction.java:30)

        at com.google.gerrit.httpd.rpc.Handler.to(Handler.java:65)

        at com.google.gerrit.httpd.rpc.changedetail.ChangeManageServiceImpl.submit(ChangeManageServiceImpl.java:52)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

        at java.lang.reflect.Method.invoke(Method.java:597)

        at com.google.gwtjsonrpc.server.MethodHandle.invoke(MethodHandle.java:91)

        at com.google.gwtjsonrpc.server.JsonServlet.doService(JsonServlet.java:379)

        at com.google.gwtjsonrpc.server.JsonServlet.service(JsonServlet.java:265)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

        at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)

        at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)

        at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)

        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)

        at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)

        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)

        at com.google.gwtexpui.server.CacheControlFilter.doFilter(CacheControlFilter.java:76)

        at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)

        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)

        at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)

        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)

        at com.google.gerrit.httpd.RequireSslFilter.doFilter(RequireSslFilter.java:68)

        at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)

        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)

        at com.google.gerrit.httpd.RequestCleanupFilter.doFilter(RequestCleanupFilter.java:54)

        at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)

        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)

        at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)

        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)

        at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)

        at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)

        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1322)

        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:473)

        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:921)

        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:403)

        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:856)

        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)

        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:114)

        at org.eclipse.jetty.server.Server.handle(Server.java:352)

        at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:596)

        at org.eclipse.jetty.server.HttpConnection$RequestHandler.content(HttpConnection.java:1069)

        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:805)

        at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:218)

        at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:426)

        at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:510)

        at org.eclipse.jetty.io.nio.SelectChannelEndPoint.access$000(SelectChannelEndPoint.java:34)

        at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:40)

        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:450)

        at java.lang.Thread.run(Thread.java:662)

Looking through the code, it looks like submitAudit is null when it shouldn't be, and we don't know why.  We're trying to throw some error messages into the code to debug this.  What could be causing this to occur?

-Nick

[Nicholas Mucci]

A ha!  We figured it out.  Somehow a change got into a funky state; patch_set_approvals was missing information on patch set 2 for the change.  The patch set was there, but it never got the necessary scores to submit, yet somehow the change ended up in the submitted state, breaking all subsequent submissions.

-Nick

[Gaurav Negi]

Hi Nick, How did you fix this?

[Doug Kelly]

I doubt a thread this old (from 2.4.2) is going to be relevant, unless you really are on that old of a version.  As of 2.12, the merge queue no longer exists, so patches will either merge or not: a failure won't get stuck like this.  Even on 2.11, if you figure out which change has the issue, abandoning the change usually is enough to unblock the queue, then you can re-submit.  I know I've seen issues like this a few times prior to the 2.12 upgrade, and rarely (but occasionally) some careful database surgery is necessary to fix the bad state, but usually abandoning to get it out of the failure is good enough.

(Also, note that current versions of Gerrit insert an extra patch set if cherry-picking the change, and they copy the approvals over from the last patch set.)

--Doug