Can't submit to new branch, LOCK_FAILURE

387 views
Skip to first unread message

John de Largentaye

unread,
May 30, 2024, 2:37:38 PM5/30/24
to Repo and Gerrit Discussion
Hi repo-discuss,

I've gotten myself in a tangle, and in Gerrit 3.8.2, I cannot submit to a new branch I just created. When I click submit in the UI, it thinks for a bit, then pops up a dialog saying:

> An error occurred
> Could not perform action: Lock failure 
> * Updating a ref failed with LOCK_FAILURE.
> This may be a temporary issue due to concurrent updates. Please retry later.

The traceback from Gerrit's error.log is below, but first some more context...

I fumbled when creating this branch. I corrected things, but I'm worried there's something lingering that's causing the issue.

I initially created it by doing `ssh -p 29418 my.gerrit.server gerrit create-branch PROJECT NEWBRANCH master`. However they wanted it pulled from a different commit, so I changed the project's access permissions through the UI to give myself force-push permissions on the branch. *I fumbled the permissions change*, accidentally changing the reference for the main "Label/submit" permission group alongside adding the reference for NEWBRANCH, and also tripped up the reference name for NEWBRANCH, as it needed to be a regex because I was actually dealing with two branches. I fixed the main permission group reference back to "refs/for/*", fixed NEWBRANCH force push permissions, and was able to force-push to NEWBRANCH to the commit it should have been. I then deleted all the permissions change. "git diff refs/meta/config SHA-BEFORE-ALL-THE-FUMBLING" shows no difference; the permissions are what they're supposed to be. I'll paste the project's refs/meta/config:project.config below

Other things checked:
- In the bare repo, there is nothing under refs/heads/. The directory is empty.
- The permissions to the refs/heads/ directory look correct (rw for gerrit user)
- the permissions on packed-refs look correct (rw for gerrit user)

Traceback:
```
[CONTEXT SUBMISSION_ID="6103-1717092311563-ad930fd2" project="PERTOS" request="REST /gerrit/*/PERTOS~6103/*/1/*" ]
[2024-05-30T18:05:57.281Z] [HTTP POST /gerrit/changes/PERTOS~6103/revisions/1/submit (jlargentaye from 172.20.0.3)] ERROR com.google.gerrit.httpd.restapi.RestApiServlet : Error in POST /gerrit/changes/PERTOS~6103/revisions/1/submit [CONTEXT project="PERTOS" request="REST /gerrit/*/PERTOS~6103/*/1/*" ]
com.google.gerrit.server.update.UpdateException: com.google.gerrit.git.LockFailureException: Update aborted with one or more lock failures: SnapshotPackedBatchRefUpdate[
  UPDATE: 773dc1887bd489add9006807f266424986ab2ce5 d7569e85da219301838c303cf02c0770d9fa2944 refs/heads/NEWBRANCH  (LOCK_FAILURE)
  UPDATE: 1c1d319549cd4e7bb9237b1f345e1bef3012a462 0b978e3ea33d22f88e671583dbcd453a80095683 refs/changes/03/6103/meta  (REJECTED_OTHER_REASON: transaction aborted)
]
at com.google.gerrit.server.update.BatchUpdate.wrapAndThrowException(BatchUpdate.java:250)
at com.google.gerrit.server.update.BatchUpdate.execute(BatchUpdate.java:193)
at com.google.gerrit.server.update.SubmissionExecutor.execute(SubmissionExecutor.java:61)
at com.google.gerrit.server.submit.MergeOp.integrateIntoHistory(MergeOp.java:666)
at com.google.gerrit.server.submit.MergeOp.lambda$merge$1(MergeOp.java:535)
at com.google.gerrit.server.update.RetryableChangeAction.lambda$new$0(RetryableChangeAction.java:48)
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
at com.github.rholder.retry.Retryer.call(Retryer.java:160)
at com.google.gerrit.server.update.RetryHelper.executeWithTimeoutCount(RetryHelper.java:564)
at com.google.gerrit.server.update.RetryHelper.execute(RetryHelper.java:507)
at com.google.gerrit.server.update.RetryableAction.call(RetryableAction.java:173)
at com.google.gerrit.server.update.RetryableChangeAction.call(RetryableChangeAction.java:84)
at com.google.gerrit.server.submit.MergeOp.merge(MergeOp.java:544)
at com.google.gerrit.server.restapi.change.Submit.mergeChange(Submit.java:214)
at com.google.gerrit.server.restapi.change.Submit.apply(Submit.java:191)
at com.google.gerrit.server.restapi.change.Submit.apply(Submit.java:88)
at com.google.gerrit.httpd.restapi.RestApiServlet.lambda$invokeRestModifyViewWithRetry$7(RestApiServlet.java:915)
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
at com.github.rholder.retry.Retryer.call(Retryer.java:160)
at com.google.gerrit.server.update.RetryHelper.executeWithTimeoutCount(RetryHelper.java:564)
at com.google.gerrit.server.update.RetryHelper.execute(RetryHelper.java:507)
at com.google.gerrit.server.update.RetryableAction.call(RetryableAction.java:173)
at com.google.gerrit.httpd.restapi.RestApiServlet.invokeRestEndpointWithRetry(RestApiServlet.java:990)
at com.google.gerrit.httpd.restapi.RestApiServlet.invokeRestModifyViewWithRetry(RestApiServlet.java:910)
at com.google.gerrit.httpd.restapi.RestApiServlet.service(RestApiServlet.java:560)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290)
at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280)
at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)
at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:89)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
at com.google.gerrit.httpd.raw.StaticModule$PolyGerritFilter.doFilter(StaticModule.java:412)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.gerrit.httpd.GetUserFilter.doFilter(GetUserFilter.java:92)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.gerrit.httpd.RequireSslFilter.doFilter(RequireSslFilter.java:72)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:120)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.gerrit.httpd.SetThreadNameFilter.doFilter(SetThreadNameFilter.java:62)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:139)
at com.google.gerrit.httpd.AllowRenderInFrameFilter.doFilter(AllowRenderInFrameFilter.java:56)
at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:135)
at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:141)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.gerrit.httpd.RequestCleanupFilter.doFilter(RequestCleanupFilter.java:60)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.gerrit.httpd.RequestMetricsFilter.doFilter(RequestMetricsFilter.java:92)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:64)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: com.google.gerrit.git.LockFailureException: Update aborted with one or more lock failures: SnapshotPackedBatchRefUpdate[
  UPDATE: 773dc1887bd489add9006807f266424986ab2ce5 d7569e85da219301838c303cf02c0770d9fa2944 refs/heads/NEWBRANCH  (LOCK_FAILURE)
  UPDATE: 1c1d319549cd4e7bb9237b1f345e1bef3012a462 0b978e3ea33d22f88e671583dbcd453a80095683 refs/changes/03/6103/meta  (REJECTED_OTHER_REASON: transaction aborted)
]
at com.google.gerrit.git.RefUpdateUtil.checkResults(RefUpdateUtil.java:100)
at com.google.gerrit.git.RefUpdateUtil.executeChecked(RefUpdateUtil.java:57)
at com.google.gerrit.server.notedb.NoteDbUpdateManager.execute(NoteDbUpdateManager.java:399)
at com.google.gerrit.server.notedb.NoteDbUpdateManager.execute(NoteDbUpdateManager.java:340)
at com.google.gerrit.server.update.BatchUpdate$ChangesHandle.execute(BatchUpdate.java:716)
at com.google.gerrit.server.update.BatchUpdate.execute(BatchUpdate.java:163)
... 82 more
```

refs/meta/config: project.config
```
[submit]
        action = inherit
[access "refs/for/*"]
        exclusiveGroupPermissions = label-Code-Review submit
        label-Code-Review = -2..+2 group Registered Users
        submit = group Registered Users
[access "^refs/heads/DEV.*"]
        push = +force group Registered Users
[access "^refs/heads/(CM|SER|DBG)-.+"]
        create = group Registered Users
        push = +force group Registered Users
[plugin "uploadvalidator"]
        blockedContentTypeWhitelist = false
        maxPathLength = 0
        rejectDuplicatePathnamesLocale = en
        rejectSubmodule = false
        rejectSymlink = false
        rejectWindowsLineEndings = false
```

Matthias Sohn

unread,
May 30, 2024, 3:24:15 PM5/30/24
to John de Largentaye, Repo and Gerrit Discussion
I recommend to use force push only in exceptional cases and not grant it to everybody by default.
I'd rather grant this in the exceptional case and revoke it afterwards.
 
[access "^refs/heads/(CM|SER|DBG)-.+"]
        create = group Registered Users
        push = +force group Registered Users
[plugin "uploadvalidator"]
        blockedContentTypeWhitelist = false
        maxPathLength = 0
        rejectDuplicatePathnamesLocale = en
        rejectSubmodule = false
        rejectSymlink = false
        rejectWindowsLineEndings = false

Check if there are any .lock files in this repository.
I guess there's a stale packed-refs.lock, maybe caused by a server crash or kill ?

-Matthias
 
```

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/CAMxs8317mDNa9a%2B-oau1ZCiD580R8Fg6YVkfZH1XrL5GC_Pctg%40mail.gmail.com.

John de Largentaye

unread,
May 30, 2024, 8:31:18 PM5/30/24
to Matthias Sohn, Repo and Gerrit Discussion
Thanks for your suggestion, Mathias!

Unfortunately, there are no .lock files in the repository (or any others!), as evidenced by `fd '\.lock$' (and without the '$').

On the topic of Force permissions, I recognize it's not great, but it's the behavior our team has expected on these "wild-west" branches since before we used Gerrit. I'll fight on that hill later ;)

John de Largentaye

unread,
May 31, 2024, 9:44:12 PM5/31/24
to Matthias Sohn, Repo and Gerrit Discussion
Good news: I worked around the problem by deleting and recreating the branch.

Reply all
Reply to author
Forward
0 new messages