ERROR com.google.gerrit.server.events.StreamEventsApiListener : Failed to dispatch event

[Opportunity alie]

Hi,all：

gerrit 3.12.3 active/active setup，ZooKeeper is built using a cluster.I configured it like this.

[ref-database "zookeeper"]
  connectString = 192.168.2.101:2181；192.168.2.102:2181；192.168.2.103:2181；

Local Git Bash push，git push origin HEAD:refs/for/master, the error log:

[ReceiveCommits-1[receive-commits]-for-SSH git-receive-pack /code/test (admin)] ERROR com.google.gerrit.server.events.StreamEventsApiListener : Failed to dispatch event [CONTEXT PLUGIN="gerrit" RECEIVE_ID="code/test-1766133196669-07d664bf" TRACE_ID="1766133196668-07d664bf" project="code/test" request="GIT_RECEIVE" ]
com.google.gerrit.exceptions.StorageException: com.google.gerrit.server.project.NoSuchChangeException: 21

......

Caused by: com.google.gerrit.server.project.NoSuchChangeException: 21

.....

the backup http_log:

"POST /gerrit/plugins/high-availability/index/change/code%2Ftest~21 HTTP/1.1" 204
"POST /gerrit/plugins/high-availability/event/ref-updated HTTP/1.1" 204 - 2
"POST /gerrit/plugins/high-availability/event/ref-updated HTTP/1.1" 204 - 2

missing "POST /gerrit/plugins/high-availability/event/patchset-created HTTP/1.1" 204

All newly built，

[ref-database "zookeeper"]

    connectString = 192.168.2.101:2181 

Everything is working normally, and the online environment is currently running according to the previous configuration. How can we revert to the previous setup?

I tried the following solutions：

   stop gerrit，ref-database Configured as a single

   Clear cache，clear zookeeper version-2 ，Reinit 

   reindex projects changes 

None of them can solve it.

[Luca Milanesio]

On 19 Dec 2025, at 09:02, Opportunity alie <houji...@gmail.com> wrote:

Hi,all：

gerrit 3.12.3 active/active setup，ZooKeeper is built using a cluster.I configured it like this.

[ref-database "zookeeper"]
  connectString = 192.168.2.101:2181；192.168.2.102:2181；192.168.2.103:2181；

Local Git Bash push，git push origin HEAD:refs/for/master, the error log:

[ReceiveCommits-1[receive-commits]-for-SSH git-receive-pack /code/test (admin)] ERROR com.google.gerrit.server.events.StreamEventsApiListener : Failed to dispatch event [CONTEXT PLUGIN="gerrit" RECEIVE_ID="code/test-1766133196669-07d664bf" TRACE_ID="1766133196668-07d664bf" project="code/test" request="GIT_RECEIVE" ]
com.google.gerrit.exceptions.StorageException: com.google.gerrit.server.project.NoSuchChangeException: 21

......

Caused by: com.google.gerrit.server.project.NoSuchChangeException: 21

.....

There could have been some errors preventing the change from being stored on the repo.

Can you share the error_log?

Luca.

the backup http_log:

"POST /gerrit/plugins/high-availability/index/change/code%2Ftest~21 HTTP/1.1" 204
"POST /gerrit/plugins/high-availability/event/ref-updated HTTP/1.1" 204 - 2
"POST /gerrit/plugins/high-availability/event/ref-updated HTTP/1.1" 204 - 2

missing "POST /gerrit/plugins/high-availability/event/patchset-created HTTP/1.1" 204

All newly built，

[ref-database "zookeeper"]

    connectString = 192.168.2.101:2181 

Everything is working normally, and the online environment is currently running according to the previous configuration. How can we revert to the previous setup?

I tried the following solutions：

   stop gerrit，ref-database Configured as a single

   Clear cache，clear zookeeper version-2 ，Reinit 

   reindex projects changes 

None of them can solve it.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/86a885e9-0864-410a-8661-45fe040f839dn%40googlegroups.com.

[Opportunity alie]

Pushing from local Git Bash and writing to the node's error_log：

[2025-12-22T10:22:36.628+08:00] [ReceiveCommits-1[receive-commits]-for-SSH git-receive-pack /code/test (admin)] ERROR com.google.gerrit.server.events.StreamEventsApiListener : Failed to dispatch event [CONTEXT PLUGIN="gerrit" RECEIVE_ID="code/test-1766370156401-07d664bf" TRACE_ID="1766370156400-07d664bf" project="code/test" request="GIT_RECEIVE" ]
com.google.gerrit.exceptions.StorageException: com.google.gerrit.server.project.NoSuchChangeException: 23
at com.google.gerrit.server.events.StreamEventsApiListener.getNotes(StreamEventsApiListener.java:185)
at com.google.gerrit.server.events.StreamEventsApiListener.onRevisionCreated(StreamEventsApiListener.java:307)
at com.google.gerrit.server.extensions.events.RevisionCreated.lambda$fire$0(RevisionCreated.java:85)
at com.google.gerrit.server.plugincontext.PluginContext.runLogExceptions(PluginContext.java:214)
at com.google.gerrit.server.plugincontext.PluginSetContext.lambda$runEach$1(PluginSetContext.java:148)
at java.base/java.lang.Iterable.forEach(Iterable.java:75)
at com.google.gerrit.server.plugincontext.PluginSetContext.runEach(PluginSetContext.java:148)
at com.google.gerrit.server.extensions.events.RevisionCreated.fire(RevisionCreated.java:85)
at com.google.gerrit.server.change.ChangeInserter.postUpdate(ChangeInserter.java:657)
at com.google.gerrit.server.update.BatchUpdate.executePostOps(BatchUpdate.java:799)
at com.google.gerrit.server.update.BatchUpdates.execute(BatchUpdates.java:141)
at com.google.gerrit.server.update.BatchUpdate.execute(BatchUpdate.java:372)
at com.google.gerrit.server.git.receive.ReceiveCommits.insertChangesAndPatchSets(ReceiveCommits.java:1297)
at com.google.gerrit.server.git.receive.ReceiveCommits.insertChangesAndPatchSets(ReceiveCommits.java:1190)
at com.google.gerrit.server.git.receive.ReceiveCommits.processCommandsUnsafe(ReceiveCommits.java:949)
at com.google.gerrit.server.git.receive.ReceiveCommits.processCommands(ReceiveCommits.java:757)
at com.google.gerrit.server.git.receive.AsyncReceiveCommits.lambda$preReceive$2(AsyncReceiveCommits.java:402)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at com.google.gerrit.server.util.RequestScopePropagator.lambda$cleanup$1(RequestScopePropagator.java:187)
at com.google.gerrit.server.util.RequestScopePropagator.lambda$context$0(RequestScopePropagator.java:174)
at com.google.gerrit.server.util.ThreadLocalRequestScopePropagator.lambda$wrapImpl$0(ThreadLocalRequestScopePropagator.java:47)
at com.google.gerrit.server.util.RequestScopePropagator$1.call(RequestScopePropagator.java:85)
at com.google.gerrit.server.util.RequestScopePropagator$2.run(RequestScopePropagator.java:116)
at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:113)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:912)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: com.google.gerrit.server.project.NoSuchChangeException: 23
at com.google.gerrit.server.notedb.ChangeNotes.onLoad(ChangeNotes.java:657)
at com.google.gerrit.server.notedb.AbstractChangeNotes.load(AbstractChangeNotes.java:170)
at com.google.gerrit.server.notedb.AbstractChangeNotes.load(AbstractChangeNotes.java:149)
at com.google.gerrit.server.notedb.ChangeNotes$Factory.createChecked(ChangeNotes.java:155)
at com.google.gerrit.server.notedb.ChangeNotes$Factory.createChecked(ChangeNotes.java:159)
at com.google.gerrit.server.events.StreamEventsApiListener.getNotes(StreamEventsApiListener.java:182)
... 31 more

The read node's HTTP log only contains three entries, lacking the patch set create event.

[2025-12-22T10:22:36.607+08:00] "POST /gerrit/plugins/high-availability/index/change/code%2Ftest~23 HTTP/1.1" 204
[2025-12-22T10:22:36.610+08:00] "POST /gerrit/plugins/high-availability/event/ref-updated HTTP/1.1" 204
[2025-12-22T10:22:36.612+08:00] "POST /gerrit/plugins/high-availability/event/ref-updated HTTP/1.1" 204

The repository contains an empty directory like repositories/code/test.git/refs/changes/23/23.

In the Gerrit page, under commands -> create change, the error log will not show any errors; the logs of the read node contain the synchronization of the pattern create event.

[Luca Milanesio]

Thanks for sharing this fragment.

It looks like this is a high-availability issue, can you explain in details where the operation was initiated and where the error appears?

The read node's HTTP log only contains three entries, lacking the patch set create event.

[2025-12-22T10:22:36.607+08:00] "POST /gerrit/plugins/high-availability/index/change/code%2Ftest~23 HTTP/1.1" 204
[2025-12-22T10:22:36.610+08:00] "POST /gerrit/plugins/high-availability/event/ref-updated HTTP/1.1" 204
[2025-12-22T10:22:36.612+08:00] "POST /gerrit/plugins/high-availability/event/ref-updated HTTP/1.1" 204

What do you mean by “read” node?

Can you share the logs of both nodes (error_log, sshd_log, httpd_log) for the failing operation?

The repository contains an empty directory like repositories/code/test.git/refs/changes/23/23.

You should check if the ref exists, using a ‘git show-ref’ command [1].

HTH

Luca.

[1] https://git-scm.com/docs/git-show-ref

[Opportunity alie]

gerrit01 192.168.2.101 HAProxy traffic is defined as writes, or simply write nodes； gerrit02 192.168.2.102 HAProxy traffic is defined as read, or simply read nodes. HAProxy has temporarily commented out gerrit01.

Configure ZooKeeper in cluster mode, connectString = 192.168.2.101:2181；192.168.2.102:2181；192.168.2.103:2181；

gerrit01 clone repository，git bash，add file， commit，git push origin HEAD：refs/for/branchname

gerrit01 error_log：

[2025-12-23T15:47:07.871+08:00] [ReceiveCommits-1[receive-commits]-for-SSH git-receive-pack /code/test (admin)] ERROR com.google.gerrit.server.events.StreamEventsApiListener : Failed to dispatch event [CONTEXT PLUGIN="gerrit" RECEIVE_ID="code/test-1766476027185-26124522" TRACE_ID="1766476027184-26124522" project="code/test" request="GIT_RECEIVE" ]
com.google.gerrit.exceptions.StorageException: com.google.gerrit.server.project.NoSuchChangeException: 61

Caused by: com.google.gerrit.server.project.NoSuchChangeException: 61

at com.google.gerrit.server.notedb.ChangeNotes.onLoad(ChangeNotes.java:657)
at com.google.gerrit.server.notedb.AbstractChangeNotes.load(AbstractChangeNotes.java:170)
at com.google.gerrit.server.notedb.AbstractChangeNotes.load(AbstractChangeNotes.java:149)
at com.google.gerrit.server.notedb.ChangeNotes$Factory.createChecked(ChangeNotes.java:155)
at com.google.gerrit.server.notedb.ChangeNotes$Factory.createChecked(ChangeNotes.java:159)
at com.google.gerrit.server.events.StreamEventsApiListener.getNotes(StreamEventsApiListener.java:182)
... 31 more

gerrit01 ssh_log:

[2025-12-23T15:47:07.893+08:00] 5448fb61 [SSH git-receive-pack /code/test (admin)] admin a/1000000 git-receive-pack./code/test 46ms 1091ms - 0 git/2.31.1.windows.1 148ms 120ms 5564384 1766476027184-26124522
[2025-12-23T15:47:07.902+08:00] 5448fb61 [sshd-SshDaemon[5deae5ef](port=22)-nio2-thread-2] admin a/1000000 LOGOUT -

gerrit01 http_log:  There are no logs for this commit.

gerrit02 error_log：

[2025-12-23T15:47:07.809+08:00] [Forwarded-Index-Event-1[java.util.concurrent.FutureTask@2239ef46[Not completed, task = com.google.gerrit.server.logging.LoggingContextAwareCallable@6c8f4c23]]] WARN  org.eclipse.jgit.lib.CoreConfig : Both core.trustFolderStat and core.trustStat are set, ignoring trustFolderStat since trustStat takes precedence. Remove core.trustFolderStat from your configuration.
[2025-12-23T15:47:07.829+08:00] [Forwarded-Index-Event-1[java.util.concurrent.FutureTask@2239ef46[Not completed, task = com.google.gerrit.server.logging.LoggingContextAwareCallable@6c8f4c23]]] INFO  com.google.gerrit.server.index.change.ChangeIndexer : Reindexing stale document StalenessCheckResult{isStale=true, reason=Optional[Document 61 missing from index]} [CONTEXT TRACE_ID="1766476027802-735d601e" ]

gerrit02 http_log:

"POST /gerrit/plugins/high-availability/event/ref-updated HTTP/1.1" 204 - 19 - "Apache-HttpClient/4.5.14 (Java/21.0.7)" 18 20 753248 - 1766476027441-ea7be896
"POST /gerrit/plugins/high-availability/index/change/code%2Ftest~61 HTTP/1.1" 204 - 3 - "Apache-HttpClient/4.5.14 (Java/21.0.7)" 1 - 118168 - 1766476027802-735d601e
"POST /gerrit/plugins/high-availability/event/ref-updated HTTP/1.1" 204 - 1 - "Apache-HttpClient/4.5.14 (Java/21.0.7)" 1 - 115832 - 1766476027816-ea7be896
"POST /gerrit/plugins/high-availability/event/ref-updated HTTP/1.1" 204 - 1 - "Apache-HttpClient/4.5.14 (Java/21.0.7)" - - 103400 - 1766476027823-735d601e

----- This is what I meant by "missing 'POST /gerrit/plugins/high-availability/event/patchset-created HTTP/1.1' 204".

gerrit02  sshd_log： There are no logs for this commit.

but browse -> repositories -> “name” -> commands -> create change

gerrit01 error_log: no error

gerrit01  http_log :

admin [2025-12-23T16:04:59.775+08:00] "POST /gerrit/changes/ HTTP/1.1" 201 488 248 - "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36" 92 90 5120736 - 1766477099551-ad7e300e

gerrit01 ssh_log:  no log

gerrit02 http_log:

[HTTP-403] - - [2025-12-23T16:04:57.095+08:00] "GET /gerrit/config/server/version HTTP/1.0" 200 14 1 - - - - 118664 - 1766477097095-2ab0100f
[HTTP-289] - - [2025-12-23T16:04:59.721+08:00] "POST /gerrit/plugins/high-availability/index/change/code%2Ftest~62 HTTP/1.1" 204 - 2 - "Apache-HttpClient/4.5.14 (Java/21.0.7)" 1 - 108640 - 1766477099720-ea7be896
[HTTP-403] - - [2025-12-23T16:04:59.726+08:00] "POST /gerrit/plugins/high-availability/event/ref-updated HTTP/1.1" 204 - 1 - "Apache-HttpClient/4.5.14 (Java/21.0.7)" 1 10 128560 - 1766477099725-2ab0100f
[HTTP-289] - - [2025-12-23T16:04:59.737+08:00] "POST /gerrit/plugins/high-availability/event/ref-updated HTTP/1.1" 204 - 1 - "Apache-HttpClient/4.5.14 (Java/21.0.7)" 1 - 103504 - 1766477099736-ea7be896
[HTTP-403] - - [2025-12-23T16:04:59.805+08:00] "POST /gerrit/plugins/high-availability/event/patchset-created HTTP/1.1" 204 - 6 - "Apache-HttpClient/4.5.14 (Java/21.0.7)" 6 - 380568 - 1766477099799-2ab0100f

This appears to be working normally.

All newly built，

[ref-database "zookeeper"]

    connectString = 192.168.2.101:2181 

Everything is working normally.

 ZooKeeper was configured as a single node. Considering the risk of application downtime, it was configured as a cluster. After configuration, Gerrit started without any errors.In the online environment, ZooKeeper is configured in a cluster. Switching ZooKeeper to a single-node configuration did not resolve the `git push gerrit01 error_log` issue, and Only patchset create events were not synced to gerrit02. The Gerrit triggers configured in jenkins also failed to fire. I apologize for not following the official single-node configuration. I urgently need a solution and would appreciate any assistance. Thank you.

I tried the following solutions：

   Clear cache，clear zookeeper version-2 ，Reinit 

   reindex projects changes 

None of them can solve it.

After multiple verifications, Configure ZooKeeper as a cluster，it was found to be reproducible.

[Opportunity alie]

Hi everyone, could you help me reproduce and verify this, and provide a solution?