High availability active/active setup event exception

Opportunity alie

unread,

Oct 19, 2025, 7:04:43 AMOct 19

to Repo and Gerrit Discussion

Hi,all.

gerrit 3.12.2

Locally pushed code error_log reports an error, and Jenkins cannot detect the patchset create event.Do you know what's causing this? Is there some configuration error?

error_log:

[2025-10-18T23:06:10.436+08:00] [ReceiveCommits-1[receive-commits]-for-SSH git-receive-pack /Code/vendor_feature_001 (admin)] ERROR com.google.gerrit.server.events.StreamEventsApiListener : Failed to dispatch event [CONTEXT PLUGIN="gerrit" RECEIVE_ID="Code/vendor_feature_001-1760799969914-77bf7bcb" TRACE_ID="1760799969913-77bf7bcb" project="Code/vendor_feature_001" request="GIT_RECEIVE" ]
com.google.gerrit.exceptions.StorageException: com.google.gerrit.server.project.NoSuchChangeException: 61
at com.google.gerrit.server.events.StreamEventsApiListener.getNotes(StreamEventsApiListener.java:185)
at com.google.gerrit.server.events.StreamEventsApiListener.onRevisionCreated(StreamEventsApiListener.java:307)
at com.google.gerrit.server.extensions.events.RevisionCreated.lambda$fire$0(RevisionCreated.java:85)
at com.google.gerrit.server.plugincontext.PluginContext.runLogExceptions(PluginContext.java:214)
at com.google.gerrit.server.plugincontext.PluginSetContext.lambda$runEach$1(PluginSetContext.java:148)
at java.base/java.lang.Iterable.forEach(Iterable.java:75)
at com.google.gerrit.server.plugincontext.PluginSetContext.runEach(PluginSetContext.java:148)
at com.google.gerrit.server.extensions.events.RevisionCreated.fire(RevisionCreated.java:85)
at com.google.gerrit.server.change.ChangeInserter.postUpdate(ChangeInserter.java:657)
at com.google.gerrit.server.update.BatchUpdate.executePostOps(BatchUpdate.java:799)
at com.google.gerrit.server.update.BatchUpdates.execute(BatchUpdates.java:141)
at com.google.gerrit.server.update.BatchUpdate.execute(BatchUpdate.java:372)
at com.google.gerrit.server.git.receive.ReceiveCommits.insertChangesAndPatchSets(ReceiveCommits.java:1297)
at com.google.gerrit.server.git.receive.ReceiveCommits.insertChangesAndPatchSets(ReceiveCommits.java:1190)
at com.google.gerrit.server.git.receive.ReceiveCommits.processCommandsUnsafe(ReceiveCommits.java:949)
at com.google.gerrit.server.git.receive.ReceiveCommits.processCommands(ReceiveCommits.java:757)
at com.google.gerrit.server.git.receive.AsyncReceiveCommits.lambda$preReceive$2(AsyncReceiveCommits.java:402)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at com.google.gerrit.server.util.RequestScopePropagator.lambda$cleanup$1(RequestScopePropagator.java:187)
at com.google.gerrit.server.util.RequestScopePropagator.lambda$context$0(RequestScopePropagator.java:174)
at com.google.gerrit.server.util.ThreadLocalRequestScopePropagator.lambda$wrapImpl$0(ThreadLocalRequestScopePropagator.java:47)
at com.google.gerrit.server.util.RequestScopePropagator$1.call(RequestScopePropagator.java:85)
at com.google.gerrit.server.util.RequestScopePropagator$2.run(RequestScopePropagator.java:116)
at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:113)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:912)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: com.google.gerrit.server.project.NoSuchChangeException: 61
at com.google.gerrit.server.notedb.ChangeNotes.onLoad(ChangeNotes.java:657)
at com.google.gerrit.server.notedb.AbstractChangeNotes.load(AbstractChangeNotes.java:170)
at com.google.gerrit.server.notedb.AbstractChangeNotes.load(AbstractChangeNotes.java:149)
at com.google.gerrit.server.notedb.ChangeNotes$Factory.createChecked(ChangeNotes.java:155)
at com.google.gerrit.server.notedb.ChangeNotes$Factory.createChecked(ChangeNotes.java:159)
at com.google.gerrit.server.events.StreamEventsApiListener.getNotes(StreamEventsApiListener.java:182)
... 31 more

Luca Milanesio

unread,

Oct 19, 2025, 1:39:49 PMOct 19

to Repo and Gerrit Discussion, Luca Milanesio

Looks like a shared filesystem caching issue.

Can you share the $GERRIT_SITE/etc/jgit.config of all the nodes that are sharing the same filesystem?

Luca.

Opportunity alie

unread,

Oct 20, 2025, 2:15:12 AMOct 20

to Repo and Gerrit Discussion

Hi, Luca

The configuration is as follows：

[core]
supportsAtomicFileCreation = false
trustStat = after_open

Luca Milanesio

unread,

Oct 20, 2025, 3:30:16 AMOct 20

to Repo and Gerrit Discussion, Luca Milanesio

On 20 Oct 2025, at 02:35, Opportunity alie <houji...@gmail.com> wrote:

Hi, Luca
The configuration is as follows：
[core]
supportsAtomicFileCreation = false
trustStat = after_open

This looks right, have you checked if the change 61 actually exists on the repo?

The associated ref is: refs/changes/61/61/meta

Luca.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/0838bdab-8b99-49fb-8bb2-1f9a67ef5107n%40googlegroups.com.

Opportunity alie

unread,

Oct 20, 2025, 5:15:10 AMOct 20

to Repo and Gerrit Discussion

The change exists. The online environment also reports the same error. The patset create hook Jenkins cannot detect it, but other ref update, abandon, etc. are fine.

Luca Milanesio

unread,

Oct 20, 2025, 5:33:01 AMOct 20

to Repo and Gerrit Discussion, Luca Milanesio

> On 20 Oct 2025, at 10:15, Opportunity alie <houji...@gmail.com> wrote:
>
> The change exists. The online environment also reports the same error. The patset create hook Jenkins cannot detect it, but other ref update, abandon, etc. are fine.

Can you reproduce the problem if you set core.trustStat=never?

Luca.

> To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/afdbe825-1c6d-4e99-8d0c-f63d4b152976n%40googlegroups.com.

Opportunity alie

unread,

Oct 20, 2025, 6:13:37 AMOct 20

to Repo and Gerrit Discussion

You reported that there might be a problem with jgit. I tried to modify core.trustStat=after_open->core.trustStat=always -> core.trustStat=never, but the final push resulted in an internal error.

git bash client:

remote: Processing changes: done
remote: error: internal error
To ssh://xxxxxx:29418/Code/vendor_feature_004
! [remote rejected] HEAD -> refs/for/master (internal error)

error_log:

[2025-10-20T11:17:38.905+08:00] [SSH git-receive-pack /Code/vendor_feature_001 (admin)] WARN com.google.gerrit.server.git.MultiProgressMonitor : MultiProgressMonitor worker did not call end() before returning
[2025-10-20T11:17:38.905+08:00] [SSH git-receive-pack /Code/vendor_feature_001 (admin)] ERROR com.google.gerrit.server.git.receive.AsyncReceiveCommits : error while processing push
java.util.concurrent.ExecutionException: com.google.gerrit.exceptions.StorageException: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Not attempted: refs/sequences/changes
at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
at com.google.gerrit.server.git.receive.AsyncReceiveCommits.preReceive(AsyncReceiveCommits.java:423)
at com.google.gerrit.server.git.receive.AsyncReceiveCommits.lambda$asHook$0(AsyncReceiveCommits.java:367)
at org.eclipse.jgit.transport.ReceivePack.service(ReceivePack.java:2225)
at org.eclipse.jgit.transport.ReceivePack.receive(ReceivePack.java:2138)
at com.google.gerrit.sshd.commands.Receive.runImpl(Receive.java:100)
at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:110)
at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:75)
at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:506)

at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:113)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:912)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)

Caused by: com.google.gerrit.exceptions.StorageException: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Not attempted: refs/sequences/changes
at com.google.gerrit.server.notedb.RepoSequence.next(RepoSequence.java:329)
at com.google.gerrit.server.Sequences.nextChangeIds(Sequences.java:85)
at com.google.gerrit.server.git.receive.ReceiveCommits.selectNewAndReplacedChangesFromMagicBranch(ReceiveCommits.java:2828)
at com.google.gerrit.server.git.receive.ReceiveCommits.processCommandsUnsafe(ReceiveCommits.java:939)

at com.google.gerrit.server.git.receive.ReceiveCommits.processCommands(ReceiveCommits.java:757)
at com.google.gerrit.server.git.receive.AsyncReceiveCommits.lambda$preReceive$2(AsyncReceiveCommits.java:402)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at com.google.gerrit.server.util.RequestScopePropagator.lambda$cleanup$1(RequestScopePropagator.java:187)
at com.google.gerrit.server.util.RequestScopePropagator.lambda$context$0(RequestScopePropagator.java:174)
at com.google.gerrit.server.util.ThreadLocalRequestScopePropagator.lambda$wrapImpl$0(ThreadLocalRequestScopePropagator.java:47)
at com.google.gerrit.server.util.RequestScopePropagator$1.call(RequestScopePropagator.java:85)
at com.google.gerrit.server.util.RequestScopePropagator$2.run(RequestScopePropagator.java:116)

... 8 more
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Not attempted: refs/sequences/changes
at com.github.rholder.retry.Retryer$ExceptionAttempt.<init>(Retryer.java:254)
at com.github.rholder.retry.Retryer.call(Retryer.java:163)
at com.google.gerrit.server.notedb.RepoSequence.next(RepoSequence.java:293)
... 20 more
Caused by: java.lang.IllegalArgumentException: Not attempted: refs/sequences/changes
at com.google.gerrit.git.RefUpdateUtil.checkResult(RefUpdateUtil.java:153)
at com.google.gerrit.server.notedb.RepoSequence.store(RepoSequence.java:453)
at com.google.gerrit.server.notedb.RepoSequence.acquire(RepoSequence.java:362)
at com.google.gerrit.server.notedb.RepoSequence.lambda$next$1(RepoSequence.java:299)
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)

Luca Milanesio

unread,

Oct 20, 2025, 6:20:17 AMOct 20

to Repo and Gerrit Discussion, Luca Milanesio

Well, can you create *ANY* change on your setup?

Luca.

To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/b795349f-3785-4711-81e6-b8caf4feb1a0n%40googlegroups.com.

Opportunity alie

unread,

Oct 21, 2025, 2:41:26 AMOct 21

to Repo and Gerrit Discussion

Sorry, I don't understand what this means.

Luca Milanesio

unread,

Oct 21, 2025, 3:24:05 AMOct 21

to Repo and Gerrit Discussion, Luca Milanesio

Can you create a repository and then create a change on the UI?

Does the above work?

Luca.

To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/28d44b31-9695-4e10-b5ea-72f1467cf4fbn%40googlegroups.com.

Opportunity alie

unread,

Oct 21, 2025, 10:11:35 PMOct 21

to Repo and Gerrit Discussion

Repositories->Commands->create change
UI error:
An error occurred
Error 500: Internal server error
Endpoint: /changes/
Trace Id: 1761098367626-b602c11f
error_log:
[2025-10-22T09:59:27.662+08:00] [HTTP POST /gerrit/changes/ (compiler from 192.168.1.1)] ERROR com.google.gerrit.httpd.restapi.RestApiServlet : Error in POST /gerrit/changes/ (view: restapi.change.CreateChange): IllegalArgumentException [CONTEXT TRACE_ID="1761098367626-b602c11f" request="REST /gerrit/*/" ]

com.google.gerrit.exceptions.StorageException: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Not attempted: refs/sequences/changes
at com.google.gerrit.server.notedb.RepoSequence.next(RepoSequence.java:329)

at com.google.gerrit.server.notedb.RepoSequence.next(RepoSequence.java:274)
at com.google.gerrit.server.Sequences.nextChangeId(Sequences.java:78)
at com.google.gerrit.server.restapi.change.CreateChange.createNewChange(CreateChange.java:533)
at com.google.gerrit.server.restapi.change.CreateChange.execute(CreateChange.java:281)
at com.google.gerrit.server.restapi.change.CreateChange.execute(CreateChange.java:252)
at com.google.gerrit.server.restapi.change.CreateChange.apply(CreateChange.java:207)
at com.google.gerrit.server.restapi.change.CreateChange.apply(CreateChange.java:123)
at com.google.gerrit.httpd.restapi.RestApiServlet.lambda$invokeRestCollectionModifyViewWithRetry$9(RestApiServlet.java:902)
at com.google.gerrit.server.update.RetryableAction.lambda$new$0(RetryableAction.java:94)
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
at com.github.rholder.retry.Retryer.call(Retryer.java:160)
at com.google.gerrit.server.update.RetryHelper.executeWithTimeoutCount(RetryHelper.java:616)
at com.google.gerrit.server.update.RetryHelper.execute(RetryHelper.java:557)
at com.google.gerrit.server.update.RetryableAction.call(RetryableAction.java:192)
at com.google.gerrit.httpd.restapi.RestApiServlet.invokeRestEndpointWithRetry(RestApiServlet.java:927)
at com.google.gerrit.httpd.restapi.RestApiServlet.invokeRestCollectionModifyViewWithRetry(RestApiServlet.java:897)
at com.google.gerrit.httpd.restapi.RestApiServlet.service(RestApiServlet.java:583)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:293)
at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:283)
at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)
at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:89)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
at com.google.gerrit.httpd.raw.StaticModule$PolyGerritFilter.doFilter(StaticModule.java:403)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.gerrit.httpd.GetUserFilter.doFilter(GetUserFilter.java:92)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.gerrit.httpd.RequireSslFilter.doFilter(RequireSslFilter.java:72)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:120)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.gerrit.httpd.EnableTracingFilter.doFilter(EnableTracingFilter.java:57)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.gerrit.httpd.SetThreadNameFilter.doFilter(SetThreadNameFilter.java:62)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:139)
at com.ericsson.gerrit.plugins.highavailability.XGerritInstanceFilter.doFilter(XGerritInstanceFilter.java:51)
at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:135)
at com.google.gerrit.httpd.AllowRenderInFrameFilter.doFilter(AllowRenderInFrameFilter.java:56)
at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:135)
at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:141)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.gerrit.httpd.RequestCleanupFilter.doFilter(RequestCleanupFilter.java:60)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.gerrit.httpd.RequestMetricsFilter.doFilter(RequestMetricsFilter.java:92)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:64)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:54)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.base/java.lang.Thread.run(Thread.java:1583)

Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Not attempted: refs/sequences/changes
at com.github.rholder.retry.Retryer$ExceptionAttempt.<init>(Retryer.java:254)
at com.github.rholder.retry.Retryer.call(Retryer.java:163)
at com.google.gerrit.server.notedb.RepoSequence.next(RepoSequence.java:293)

... 81 more

Caused by: java.lang.IllegalArgumentException: Not attempted: refs/sequences/changes
at com.google.gerrit.git.RefUpdateUtil.checkResult(RefUpdateUtil.java:153)
at com.google.gerrit.server.notedb.RepoSequence.store(RepoSequence.java:453)
at com.google.gerrit.server.notedb.RepoSequence.acquire(RepoSequence.java:362)
at com.google.gerrit.server.notedb.RepoSequence.lambda$next$1(RepoSequence.java:299)
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)

at com.github.rholder.retry.Retryer.call(Retryer.java:160)
... 82 more

Luca Milanesio

unread,

Oct 22, 2025, 3:32:23 AMOct 22

to Repo and Gerrit Discussion, Luca Milanesio

You receive a NOT_ATTEMPTED response from the Git ref updates: there is no way your Gerrit can work if JGit isn’t able to perform ref updates.

Have you tried with a fresh setup?

Luca.

at com.google.gerrit.server.notedb.RepoSequence.acquire(RepoSequence.java:362)
at com.google.gerrit.server.notedb.RepoSequence.lambda$next$1(RepoSequence.java:299)
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
at com.github.rholder.retry.Retryer.call(Retryer.java:160)
... 82 more

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/c18dfe3f-5d3d-47fa-9abc-619b84000f81n%40googlegroups.com.

Opportunity alie

unread,

Oct 22, 2025, 4:28:42 AMOct 22

to Repo and Gerrit Discussion

Today I switched back to the Active/Passive setup, and with or without jgit configured, I can push normally, and Jenkins events can be triggered normally. We upgraded from 2.16 to 3.12. Does the fresh setup refer to all the init data?

Opportunity alie

unread,

Oct 22, 2025, 4:34:17 AMOct 22

to Repo and Gerrit Discussion

Active/Passive setup，The normal logs look like this：

[2025-10-22T14:23:25.706+08:00] [ReceiveCommits-1[receive-commits]-for-SSH git-receive-pack /Code/vendor_feature_001 (admin)] INFO com.googlesource.gerrit.plugins.hooks.HookFactory : hooks.path: /data01/review_site/hooks [CONTEXT RECEIVE_ID="Code/vendor_feature_001-1761114205666-facb06ce" TRACE_ID="1761114205665-facb06ce" project="Code/vendor_feature_001" request="GIT_RECEIVE" ]
[2025-10-22T14:23:25.707+08:00] [ReceiveCommits-1[receive-commits]-for-SSH git-receive-pack /Code/vendor_feature_001 (admin)] INFO com.googlesource.gerrit.plugins.hooks.HookFactory : hooks.commitReceivedHook resolved to /data01/review_site/hooks/commit-received [CONTEXT RECEIVE_ID="Code/vendor_feature_001-1761114205666-facb06ce" TRACE_ID="1761114205665-facb06ce" project="Code/vendor_feature_001" request="GIT_RECEIVE" ]
[2025-10-22T14:23:25.770+08:00] [ReceiveCommits-1[receive-commits]-for-SSH git-receive-pack /Code/vendor_feature_001 (admin)] INFO com.googlesource.gerrit.plugins.hooks.HookFactory : hooks.refUpdatedHook resolved to /data01/review_site/hooks/ref-updated [CONTEXT RECEIVE_ID="Code/vendor_feature_001-1761114205666-facb06ce" TRACE_ID="1761114205665-facb06ce" project="Code/vendor_feature_001" request="GIT_RECEIVE" ]
[2025-10-22T14:23:26.174+08:00] [ReceiveCommits-1[receive-commits]-for-SSH git-receive-pack /Code/vendor_feature_001 (admin)] INFO com.googlesource.gerrit.plugins.hooks.HookFactory : hooks.patchsetCreatedHook resolved to /data01/review_site/hooks/patchset-created [CONTEXT RECEIVE_ID="Code/vendor_feature_001-1761114205666-facb06ce" TRACE_ID="1761114205665-facb06ce" project="Code/vendor_feature_001" request="GIT_RECEIVE" ]

Opportunity alie

unread,

Oct 22, 2025, 4:36:42 AMOct 22

to Repo and Gerrit Discussion

active/active setup，There are currently four abnormal logs, which I sent in four batches.
[2025-10-22T14:57:00.477+08:00] [ReceiveCommits-1[receive-commits]-for-SSH git-receive-pack /Code/vendor_feature_001 (admin)] WARN com.gerritforge.gerrit.globalrefdb.validation.RefUpdateValidator : Failure while running with policy enforcement REQUIRED. Error message: Local ref refs/sequences/changes (ObjectId=c390b4cff15f60e3c7c12561c444c6f029cb8ef2) on project All-Projects is out of sync with the shared ref-db [CONTEXT RECEIVE_ID="Code/vendor_feature_001-1761116220339-a43ae62d" TRACE_ID="1761116220339-a43ae62d" project="Code/vendor_feature_001" request="GIT_RECEIVE" ]
com.gerritforge.gerrit.globalrefdb.validation.dfsrefdb.OutOfSyncException: Local ref refs/sequences/changes (ObjectId=c390b4cff15f60e3c7c12561c444c6f029cb8ef2) on project All-Projects is out of sync with the shared ref-db
at com.gerritforge.gerrit.globalrefdb.validation.RefUpdateValidator.compareAndGetLatestLocalRef(RefUpdateValidator.java:289)
at com.gerritforge.gerrit.globalrefdb.validation.RefUpdateValidator.doExecuteRefUpdate(RefUpdateValidator.java:188)
at com.gerritforge.gerrit.globalrefdb.validation.RefUpdateValidator.executeRefUpdate(RefUpdateValidator.java:155)
at com.gerritforge.gerrit.globalrefdb.validation.SharedRefDbRefUpdate.update(SharedRefDbRefUpdate.java:146)
at com.google.gerrit.server.notedb.IntBlob.tryStore(IntBlob.java:106)
at com.google.gerrit.server.notedb.RepoSequence.store(RepoSequence.java:452)

at com.google.gerrit.server.notedb.RepoSequence.acquire(RepoSequence.java:362)
at com.google.gerrit.server.notedb.RepoSequence.lambda$next$1(RepoSequence.java:299)
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
at com.github.rholder.retry.Retryer.call(Retryer.java:160)

at com.google.gerrit.server.notedb.RepoSequence.next(RepoSequence.java:293)

at com.google.gerrit.server.Sequences.nextChangeIds(Sequences.java:85)
at com.google.gerrit.server.git.receive.ReceiveCommits.selectNewAndReplacedChangesFromMagicBranch(ReceiveCommits.java:2828)
at com.google.gerrit.server.git.receive.ReceiveCommits.processCommandsUnsafe(ReceiveCommits.java:939)
at com.google.gerrit.server.git.receive.ReceiveCommits.processCommands(ReceiveCommits.java:757)
at com.google.gerrit.server.git.receive.AsyncReceiveCommits.lambda$preReceive$2(AsyncReceiveCommits.java:402)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at com.google.gerrit.server.util.RequestScopePropagator.lambda$cleanup$1(RequestScopePropagator.java:187)
at com.google.gerrit.server.util.RequestScopePropagator.lambda$context$0(RequestScopePropagator.java:174)
at com.google.gerrit.server.util.ThreadLocalRequestScopePropagator.lambda$wrapImpl$0(ThreadLocalRequestScopePropagator.java:47)
at com.google.gerrit.server.util.RequestScopePropagator$1.call(RequestScopePropagator.java:85)
at com.google.gerrit.server.util.RequestScopePropagator$2.run(RequestScopePropagator.java:116)

Opportunity alie

unread,

Oct 22, 2025, 4:37:45 AMOct 22

to Repo and Gerrit Discussion

[2025-10-22T14:57:00.485+08:00] [ReceiveCommits-1[receive-commits]-for-SSH git-receive-pack /Code/vendor_feature_001 (admin)] WARN com.gerritforge.gerrit.globalrefdb.validation.RefUpdateValidator : Local node is out of sync with ref-db: Local ref refs/sequences/changes (ObjectId=c390b4cff15f60e3c7c12561c444c6f029cb8ef2) on project All-Projects is out of sync with the shared ref-db [CONTEXT RECEIVE_ID="Code/vendor_feature_001-1761116220339-a43ae62d" TRACE_ID="1761116220339-a43ae62d" project="Code/vendor_feature_001" request="GIT_RECEIVE" ]

Opportunity alie

unread,

Oct 22, 2025, 4:39:09 AMOct 22

to Repo and Gerrit Discussion

[2025-10-22T14:57:00.492+08:00] [ReceiveCommits-1[receive-commits]-for-SSH git-receive-pack /Code/vendor_feature_001 (admin)] ERROR com.google.gerrit.server.git.receive.ReceiveCommits : ReceiveCommits failed due to StorageException

com.google.gerrit.exceptions.StorageException: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Not attempted: refs/sequences/changes
at com.google.gerrit.server.notedb.RepoSequence.next(RepoSequence.java:329)

at com.google.gerrit.server.Sequences.nextChangeIds(Sequences.java:85)
at com.google.gerrit.server.git.receive.ReceiveCommits.selectNewAndReplacedChangesFromMagicBranch(ReceiveCommits.java:2828)
at com.google.gerrit.server.git.receive.ReceiveCommits.processCommandsUnsafe(ReceiveCommits.java:939)
at com.google.gerrit.server.git.receive.ReceiveCommits.processCommands(ReceiveCommits.java:757)
at com.google.gerrit.server.git.receive.AsyncReceiveCommits.lambda$preReceive$2(AsyncReceiveCommits.java:402)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at com.google.gerrit.server.util.RequestScopePropagator.lambda$cleanup$1(RequestScopePropagator.java:187)
at com.google.gerrit.server.util.RequestScopePropagator.lambda$context$0(RequestScopePropagator.java:174)
at com.google.gerrit.server.util.ThreadLocalRequestScopePropagator.lambda$wrapImpl$0(ThreadLocalRequestScopePropagator.java:47)
at com.google.gerrit.server.util.RequestScopePropagator$1.call(RequestScopePropagator.java:85)
at com.google.gerrit.server.util.RequestScopePropagator$2.run(RequestScopePropagator.java:116)
at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:113)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:912)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)

Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Not attempted: refs/sequences/changes
at com.github.rholder.retry.Retryer$ExceptionAttempt.<init>(Retryer.java:254)
at com.github.rholder.retry.Retryer.call(Retryer.java:163)
at com.google.gerrit.server.notedb.RepoSequence.next(RepoSequence.java:293)

... 20 more

Caused by: java.lang.IllegalArgumentException: Not attempted: refs/sequences/changes
at com.google.gerrit.git.RefUpdateUtil.checkResult(RefUpdateUtil.java:153)
at com.google.gerrit.server.notedb.RepoSequence.store(RepoSequence.java:453)

at com.google.gerrit.server.notedb.RepoSequence.acquire(RepoSequence.java:362)
at com.google.gerrit.server.notedb.RepoSequence.lambda$next$1(RepoSequence.java:299)
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
at com.github.rholder.retry.Retryer.call(Retryer.java:160)

... 21 more

Opportunity alie

unread,

Oct 22, 2025, 4:39:49 AMOct 22

to Repo and Gerrit Discussion

[2025-10-22T14:57:00.838+08:00] [SSH git-receive-pack /Code/vendor_feature_001 (admin)] WARN com.google.gerrit.server.git.MultiProgressMonitor : MultiProgressMonitor worker did not call end() before returning
[2025-10-22T14:57:00.839+08:00] [SSH git-receive-pack /Code/vendor_feature_001 (admin)] ERROR com.google.gerrit.server.git.receive.AsyncReceiveCommits : error while processing push

java.util.concurrent.ExecutionException: com.google.gerrit.exceptions.StorageException: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Not attempted: refs/sequences/changes
at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
at com.google.gerrit.server.git.receive.AsyncReceiveCommits.preReceive(AsyncReceiveCommits.java:423)
at com.google.gerrit.server.git.receive.AsyncReceiveCommits.lambda$asHook$0(AsyncReceiveCommits.java:367)
at org.eclipse.jgit.transport.ReceivePack.service(ReceivePack.java:2225)
at org.eclipse.jgit.transport.ReceivePack.receive(ReceivePack.java:2138)
at com.google.gerrit.sshd.commands.Receive.runImpl(Receive.java:100)
at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:110)
at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:75)
at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:506)

at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:113)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:912)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)

Caused by: com.google.gerrit.exceptions.StorageException: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Not attempted: refs/sequences/changes

at com.google.gerrit.server.notedb.RepoSequence.next(RepoSequence.java:329)
at com.google.gerrit.server.Sequences.nextChangeIds(Sequences.java:85)
at com.google.gerrit.server.git.receive.ReceiveCommits.selectNewAndReplacedChangesFromMagicBranch(ReceiveCommits.java:2828)
at com.google.gerrit.server.git.receive.ReceiveCommits.processCommandsUnsafe(ReceiveCommits.java:939)
at com.google.gerrit.server.git.receive.ReceiveCommits.processCommands(ReceiveCommits.java:757)
at com.google.gerrit.server.git.receive.AsyncReceiveCommits.lambda$preReceive$2(AsyncReceiveCommits.java:402)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at com.google.gerrit.server.util.RequestScopePropagator.lambda$cleanup$1(RequestScopePropagator.java:187)
at com.google.gerrit.server.util.RequestScopePropagator.lambda$context$0(RequestScopePropagator.java:174)
at com.google.gerrit.server.util.ThreadLocalRequestScopePropagator.lambda$wrapImpl$0(ThreadLocalRequestScopePropagator.java:47)
at com.google.gerrit.server.util.RequestScopePropagator$1.call(RequestScopePropagator.java:85)
at com.google.gerrit.server.util.RequestScopePropagator$2.run(RequestScopePropagator.java:116)

... 8 more

Luca Milanesio

unread,

Oct 22, 2025, 4:45:12 AMOct 22

to Repo and Gerrit Discussion, Luca Milanesio

On 22 Oct 2025, at 09:28, Opportunity alie <houji...@gmail.com> wrote:

Today I switched back to the Active/Passive setup, and with or without jgit configured, I can push normally, and Jenkins events can be triggered normally. We upgraded from 2.16 to 3.12. Does the fresh setup refer to all the init data?

Well, a Gerrit v2.16 to v3.12 is a quite big upgrade.

What was your cutover plan like?

Did it work on staging?

Did you run an HA setup also in v2.16?

Was it on the same NFS?

Is it on bare metal of cloud setup?

Luca.

To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/813e3e22-82d5-4e68-bb14-936b916d5589n%40googlegroups.com.

Matthias Sohn

unread,

Oct 22, 2025, 5:22:37 AMOct 22

to Opportunity alie, Repo and Gerrit Discussion

check the value of refs/sequences/changes in the All-Projects repository.

This ref doesn't refer to a commit but to a blob. It should be a larger number than the highest change

number across all repositories.

$ cd ${gerrit_site}/git/All-Projects.git

$ git show refs/sequences/changes

to guard ref updates in high-availability setup it's a good idea to install the global-refdb module and one of

its implementation plugins https://gerrit-review.googlesource.com/admin/repos/q/filter:refdb

at com.github.rholder.retry.Retryer$ExceptionAttempt.<init>(Retryer.java:254)
at com.github.rholder.retry.Retryer.call(Retryer.java:163)
at com.google.gerrit.server.notedb.RepoSequence.next(RepoSequence.java:293)
... 81 more
Caused by: java.lang.IllegalArgumentException: Not attempted: refs/sequences/changes
at com.google.gerrit.git.RefUpdateUtil.checkResult(RefUpdateUtil.java:153)
at com.google.gerrit.server.notedb.RepoSequence.store(RepoSequence.java:453)
at com.google.gerrit.server.notedb.RepoSequence.acquire(RepoSequence.java:362)
at com.google.gerrit.server.notedb.RepoSequence.lambda$next$1(RepoSequence.java:299)
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
at com.github.rholder.retry.Retryer.call(Retryer.java:160)
... 82 more

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/c18dfe3f-5d3d-47fa-9abc-619b84000f81n%40googlegroups.com.

Opportunity alie

unread,

Oct 22, 2025, 5:30:54 AMOct 22

to Repo and Gerrit Discussion

What was your cutover plan like? -- Just init version 3.12.2 based on version 2.16.28, and then do offline index

Did it work on staging? -- Data comparison before and after the upgrade, UI operation, API and other basic tests are all ok

Did you run an HA setup also in v2.16? - 2.16 Using Active/passive

Was it on the same NFS? - same NFS

Is it on bare metal of cloud setup? -- NFS v3 mounted on two physical machines
The error "ERROR com.google.gerrit.server.events.StreamEventsApiListener : Failed to dispatch event" occurred in a completely new environment built with 3.12, without any historical data involved. This means the initial error message was identical between the test and production environments. Subsequent internal service errors also occurred in this newly built environment with 3.12.

Opportunity alie

unread,

Oct 22, 2025, 5:38:06 AMOct 22

to Repo and Gerrit Discussion

Hi， Matthias.

I have installed the global-refdb module in the lib directory。

Both gerrit.conf nodes have added this configuration installModule = com.gerritforge.gerrit.globalrefdb.validation.LibModule

Luca Milanesio

unread,

Oct 22, 2025, 6:18:39 AMOct 22

to Repo and Gerrit Discussion, Luca Milanesio

On 22 Oct 2025, at 10:30, Opportunity alie <houji...@gmail.com> wrote:

What was your cutover plan like? -- Just init version 3.12.2 based on version 2.16.28, and then do offline index

I don’t believe that’s enough.

Did you read all the relevant release notes and watched the presentations I made on migrations? [1]

Did it work on staging? -- Data comparison before and after the upgrade, UI operation, API and other basic tests are all ok

What about the JGit settings in staging?

Did you run an HA setup also in v2.16? - 2.16 Using Active/passive

Interesting, so it happens only if you enable the HAProxy traffic on both?

Was it on the same NFS? - same NFS
Is it on bare metal of cloud setup? -- NFS v3 mounted on two physical machines

Can you share the mount options?

Why aren’t you using NFS v4?

The error "ERROR com.google.gerrit.server.events.StreamEventsApiListener : Failed to dispatch event" occurred in a completely new environment built with 3.12, without any historical data involved.

I am confused: how come then that you said that after downgrading it it worked?

This means the initial error message was identical between the test and production environments. Subsequent internal service errors also occurred in this newly built environment with 3.12.

I am confused again, you mentioned that it was working on staging, or did you see the problems on staging and you rolled out to prod anyway?

HTH

Luca.

[1] https://www.youtube.com/playlist?list=PLySCWiWz9cNum6wmySRlIRLkmXdh_jBoT

To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/a48082f2-042d-4875-981f-34711cb4673an%40googlegroups.com.

Opportunity alie

unread,

Oct 22, 2025, 7:17:21 AMOct 22

to Repo and Gerrit Discussion

1.Upgrade my reference here。https://www.gerritcodereview.com/3.12.html#offline-upgrade。I looked at it and it seems that this operation is missing java -jar gerrit.war copy-approvals -d site_path
2.On 2.16 we run active/passive without configuring jgit
3.192.168.2.1:/gerrit_repositories on /gerrit_repositories type nfs (rw,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.0.1,mountvers=3,mountport=635,mountproto=udp,local_lock=none,addr=192.168.0.1)
Our operation and maintenance team all use the modified v3
4.We upgraded to an active/passive setup and then switched directly to an active/active setup before discovering the problem.

After upgrading our production environment to 3.12, we continued to use the default active/passive setup without any issues.
However, we subsequently switched directly to an active/active setup, and discovered an error in the errorlog (ERROR com.google.gerrit.server.events.StreamEventsApiListener : Failed to dispatch event).

Prior to the upgrade, we conducted preliminary research on the new active/active setup directly on 3.12, without any external or historical data. Perhaps we didn't pay much attention to log information at the time, but we found no issues with commits, the UI, or events. After a brief test, we moved directly to production after the upgrade.

However, we discovered issues in the production environment, and subsequent preliminary research on this environment also revealed the same issue, so this is likely unrelated to the upgrade process.
Initial jgit configuration for the pre-research environment：
[core]
supportsAtomicFileCreation = false
trustFolderStat = false
trustLooseRefStat = after_open
trustPackedRefsStat = after_open
Found something wrong，Change to：

[core]
supportsAtomicFileCreation = false
trustStat = after_open

Want to solve the Failed to dispatch event problem ，tried to modify core.trustStat=after_open->core.trustStat=always -> core.trustStat=never, but the final push resulted in an internal error.

Let me summarize：
1.The pre-research environment was newly built based on 3.12. After the production environment was upgraded to 3.12, the problem of ERROR com.google.gerrit.server.events.StreamEventsApiListener : Failed to dispatch event occurred when switching between active/active.
2. I want to solve the problem in the pre-research environment and try to modify the jgit configuration. The error "Failed to dispatch" becomes an internal service error. But there is no problem switching back to active/passive.

Luca Milanesio

unread,

Oct 22, 2025, 12:29:15 PMOct 22

to Repo and Gerrit Discussion, Luca Milanesio

Can you please respect the rules of engagement of the community support? [1]

Specifically, you should use the interleaved posting [2].

On 22 Oct 2025, at 12:17, Opportunity alie <houji...@gmail.com> wrote:

1.Upgrade my reference here。https://www.gerritcodereview.com/3.12.html#offline-upgrade。I looked at it and it seems that this operation is missing java -jar gerrit.war copy-approvals -d site_path

You haven’t upgraded from v3.11, but from 2.16.

You should read *all* the release notes from v2.16 up to v3.12, including all the intermediate releases.

2.On 2.16 we run active/passive without configuring jgit

The JGit configuration was in a different place at that time.

3.192.168.2.1:/gerrit_repositories on /gerrit_repositories type nfs (rw,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.0.1,mountvers=3,mountport=635,mountproto=udp,local_lock=none,addr=192.168.0.1)

You do not have any attribute caching configured, see [3] for a quick reference.

Our operation and maintenance team all use the modified v3
4.We upgraded to an active/passive setup and then switched directly to an active/active setup before discovering the problem.

Did you setup a global-refdb? Which backend have you used?

After upgrading our production environment to 3.12, we continued to use the default active/passive setup without any issues.

If you don’t use active/active, then you’re not exposed to any of the quirks of NFS.

However, we subsequently switched directly to an active/active setup, and discovered an error in the errorlog (ERROR com.google.gerrit.server.events.StreamEventsApiListener : Failed to dispatch event).

What’s your cutover plan for switching from active/passive to active/active?

What configuration changes have you performed?

Which additional modules/plugins have you installed?

Which documentation did you follow?

Prior to the upgrade, we conducted preliminary research on the new active/active setup directly on 3.12, without any external or historical data.

There’s a lot of resources available on this topic: where did you perform your research?

There are also experienced companies providing enterprise support and consulting on this topic. Have you consulted them?

Perhaps we didn't pay much attention to log information at the time, but we found no issues with commits, the UI, or events. After a brief test, we moved directly to production after the upgrade.

It looks like you have some serious gaps in your testing phase, if your tests did not include the creation of a change.

However, we discovered issues in the production environment, and subsequent preliminary research on this environment also revealed the same issue, so this is likely unrelated to the upgrade process.
Initial jgit configuration for the pre-research environment：
[core]
supportsAtomicFileCreation = false
trustFolderStat = false
trustLooseRefStat = after_open
trustPackedRefsStat = after_open
Found something wrong，Change to：
[core]
supportsAtomicFileCreation = false
trustStat = after_open

Want to solve the Failed to dispatch event problem ，tried to modify core.trustStat=after_open->core.trustStat=always -> core.trustStat=never, but the final push resulted in an internal error.

I believe you have to first review the components you have and make sure that a basic test would work:

- you are able to create a project

- you are able to create a change

- you are able to review and merge a change

Use as a reference the high-availability test environment on [4].

If you need more assistance, you can access the Enterprise Support providers at [5].

Let me summarize：
1.The pre-research environment was newly built based on 3.12. After the production environment was upgraded to 3.12, the problem of ERROR com.google.gerrit.server.events.StreamEventsApiListener : Failed to dispatch event occurred when switching between active/active.
2. I want to solve the problem in the pre-research environment and try to modify the jgit configuration. The error "Failed to dispatch" becomes an internal service error. But there is no problem switching back to active/passive.

I’d recommend to rollback to an active/passive until you have all the proper components in place.

Then review your upgrade strategy and go through the points you’ve missed.

Use [4] as your reference.

HTH

Luca.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/a6dfa1ae-2506-4ace-a7b1-ede021ed2592n%40googlegroups.com.

[1] https://www.gerritcodereview.com/support.html

[2] https://www.gerritcodereview.com/support.html#general-support

[3] https://www.ibm.com/docs/en/aix/7.2.0?topic=client-nfs-file-attribute-cache-tuning

[4] https://gerrit.googlesource.com/plugins/high-availability/+/refs/heads/stable-3.12/src/test/docker/

[5] https://www.gerritcodereview.com/support.html#enterprise-support

Reply all

Reply to author

Forward