Change depends on change that was not submitted

2,074 views
Skip to first unread message

Darragh Bailey

unread,
Nov 23, 2017, 10:38:32 AM11/23/17
to Repo and Gerrit Discussion
Hi,

We've encountered a bit of an odd issue with Gerrit 2.13.5.

A couple of reviews have appeared where changes are reporting that they cannot be submitted due to the error'Change X: depends on change that was not submitted'.

I say odd, because it only affects a few changes where their parent commit is the merge commit performed by Gerrit when an earlier change was submitted. If this affected all changes we would have ground to a halt.

This looks to be related to https://bugs.chromium.org/p/gerrit/issues/detail?id=4930 which was fixed and released for 2.13.3, but looking at the fix https://gerrit-review.googlesource.com/#/c/gerrit/+/91610/ shows that it only fixes the issue for RebaseIfNecessary, while we are using MergeIfNecessary. Why this works most of the time and not in this specific case for 2 changes baffles me.


It looks like the two changes affected are both based on the same parent commit, that is a merge commit created about 3 days prior. In turn there was maybe about 30 changes that landed since, however no merge conflict has occurred, and querying the change shows no dependencies. So it seems that for some reason that Gerrit no longer recognizes that the parent commit is a merge that was created due to a submit occurring previously.

Experimenting with a local version indicates that generally this is not an issue, so I'm not entirely sure how to reproduce, and I've noticed that there are some errors in the gerrit log for this repository:

[2017-11-23 15:17:15,011] [HTTP-233185] WARN  / : Internal error during upload-pack from /var/gerrit/review_site/git/ncs-qe/images.git
org.eclipse.jetty.io.EofException
    at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:192)
    at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:408)
    at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:302)
    at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:129)
    at org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:708)
    at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:246)
    at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:208)
    at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:480)
    at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:768)
    at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:801)
    at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:147)
    at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:140)
    at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:383)
    at net.bull.javamelody.FilterServletOutputStream.write(FilterServletOutputStream.java:88)
    at net.bull.javamelody.CounterResponseStream.write(CounterResponseStream.java:82)
    at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
    at org.eclipse.jgit.util.TemporaryBuffer.write(TemporaryBuffer.java:160)
    at org.eclipse.jgit.transport.SideBandOutputStream.writeBuffer(SideBandOutputStream.java:171)
    at org.eclipse.jgit.transport.SideBandOutputStream.flushBuffer(SideBandOutputStream.java:127)
    at org.eclipse.jgit.transport.SideBandOutputStream.flush(SideBandOutputStream.java:132)
    at org.eclipse.jgit.internal.storage.pack.PackOutputStream.flush(PackOutputStream.java:136)
    at org.eclipse.jgit.internal.storage.pack.PackWriter.writePack(PackWriter.java:1056)
    at org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:1540)
    at org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:1390)
    at org.eclipse.jgit.transport.UploadPack.service(UploadPack.java:774)
    at org.eclipse.jgit.transport.UploadPack.upload(UploadPack.java:667)
    at org.eclipse.jgit.http.server.UploadPackServlet.doPost(UploadPackServlet.java:191)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:648)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
    at org.eclipse.jgit.http.server.glue.UrlPipeline$Chain.doFilter(UrlPipeline.java:237)
    at com.google.gerrit.httpd.GitOverHttpServlet$UploadFilter.doFilter(GitOverHttpServlet.java:269)
    at org.eclipse.jgit.http.server.glue.UrlPipeline$Chain.doFilter(UrlPipeline.java:235)
    at org.eclipse.jgit.http.server.UploadPackServlet$Factory.doFilter(UploadPackServlet.java:149)
    at org.eclipse.jgit.http.server.glue.UrlPipeline$Chain.doFilter(UrlPipeline.java:235)
    at org.eclipse.jgit.http.server.RepositoryFilter.doFilter(RepositoryFilter.java:151)
    at org.eclipse.jgit.http.server.glue.UrlPipeline$Chain.doFilter(UrlPipeline.java:235)
    at org.eclipse.jgit.http.server.NoCacheFilter.doFilter(NoCacheFilter.java:80)
    at org.eclipse.jgit.http.server.glue.UrlPipeline$Chain.doFilter(UrlPipeline.java:235)
    at org.eclipse.jgit.http.server.glue.UrlPipeline.service(UrlPipeline.java:215)
    at org.eclipse.jgit.http.server.glue.SuffixPipeline.service(SuffixPipeline.java:101)
    at org.eclipse.jgit.http.server.glue.MetaFilter.doFilter(MetaFilter.java:175)
    at org.eclipse.jgit.http.server.glue.MetaServlet.service(MetaServlet.java:133)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
    at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286)
    at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:276)
    at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:181)
    at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
    at com.google.gerrit.httpd.GetUserFilter.doFilter(GetUserFilter.java:82)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at com.google.gerrit.httpd.RequireSslFilter.doFilter(RequireSslFilter.java:77)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at com.google.gwtexpui.server.CacheControlFilter.doFilter(CacheControlFilter.java:73)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:122)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at com.google.gerrit.httpd.ProjectDigestFilter.doFilter(ProjectDigestFilter.java:107)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at com.google.gerrit.httpd.RequestMetricsFilter.doFilter(RequestMetricsFilter.java:60)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:136)
    at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:201)
    at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:178)
    at com.googlesource.gerrit.plugins.javamelody.GerritMonitoringFilter.doFilter(GerritMonitoringFilter.java:65)
    at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:132)
    at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:105)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:75)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at com.google.gerrit.pgm.http.jetty.ProjectQoSFilter.doFilter(ProjectQoSFilter.java:131)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120)
    at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
    at org.eclipse.jetty.server.Server.handleAsync(Server.java:549)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:318)
    at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:262)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Broken pipe
    at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
    at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
    at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
    at sun.nio.ch.IOUtil.write(IOUtil.java:65)
    at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
    at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:170)
    ... 87 more


With a very naive look, I'm wondering if the check for isAlreadyMerged(c) in https://gerrit-review.googlesource.com/#/c/gerrit/+/91610/7/gerrit-server/src/main/java/com/google/gerrit/server/git/RebaseSorter.java something that should also be present at https://gerrit.googlesource.com/gerrit/+/stable-2.13/gerrit-server/src/main/java/com/google/gerrit/server/git/MergeSorter.java#60 ?

Or is it that Gerrit has lost track of the merge commit (removed from cache?) and is unable to update due to some issue with the repository? However 'git fsck' directly on the git repo directly on Gerrit's filesystem shows no issues.

For now, I've recommended for the user to simply rebase the change so they can move on, but not sure what I can capture that would help explain this issue.


Darragh

Sven Selberg

unread,
Nov 23, 2017, 11:28:38 AM11/23/17
to Repo and Gerrit Discussion
Hi,

Since you obviously have merged changes that had the same dependency as the changes that you couldn't merge it's difficult to say if there isn't actually something else. Furthermore if you cannot replicate the issue locally it's hard to test, but I'm wondering if this commit would have fixed your issue:

I cannot find the change on gerrit-review ( https://gerrit-review.googlesource.com/#/q/If9a3c6f03f6dab4074440472faa17d7af565f717 ), probably some issue there.

/Sven

David Pursehouse

unread,
Nov 23, 2017, 11:01:21 PM11/23/17
to Sven Selberg, Repo and Gerrit Discussion, Zhen Chen
On Fri, Nov 24, 2017 at 1:28 AM Sven Selberg <sven.s...@axis.com> wrote:
Hi,

Since you obviously have merged changes that had the same dependency as the changes that you couldn't merge it's difficult to say if there isn't actually something else. Furthermore if you cannot replicate the issue locally it's hard to test, but I'm wondering if this commit would have fixed your issue:

I cannot find the change on gerrit-review ( https://gerrit-review.googlesource.com/#/q/If9a3c6f03f6dab4074440472faa17d7af565f717 ), probably some issue there.


It might have been set private during the migration to remove the draft status.  If any of the patch sets were draft, the whole change would have been set private.  This commonly happens to changes that were created via the UI, since it always created the first empty patch set as draft.

@czhen, can you see that change?


 
--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Zhen Chen

unread,
Nov 23, 2017, 11:32:42 PM11/23/17
to David Pursehouse, Sven Selberg, Repo and Gerrit Discussion
Hi David,

I can see that change, but it marked "Private" and "Merged", I guess it is because of the migration, please see the attachment.

I've "unmarked private", so everyone else can see it. 

Zhen
9TwoJMdXanB.png

Darragh Bailey

unread,
Nov 24, 2017, 9:23:24 AM11/24/17
to Repo and Gerrit Discussion
Seems like that should be for something else, as we only have one branch in this repo, master. So it's not a case of not looking at other branches for the merge commit.

What I've suddenly noticed, when looking for the commit in gitiles, I can see it in the history, but gitiles cannot display the merge commit, but if I clone the repository it's possible to look at the merge commit, and if I select an earlier merge commit that was created by Gerrit (need to go a few 3) it displays fine. But this commit along with another 5 merge commits created by Gerrit submitting commits do not display when trying to view with gitiles.

Does this provide a further clue on what to look at?

Is there a cache/index issue here for some of these merge commits? or something wrong with the repository at that point that causes issues with jgit?

Darragh

Reply all
Reply to author
Forward
0 new messages