Zookeeper-refdb expected behavior for project deletion

Serena He

unread,

May 22, 2025, 12:07:40 PMMay 22

to Repo and Gerrit Discussion

Hi team,

I have a K8sgerrit with Gerrit v3.8.1, installing delete-project v3.8.1 and zookeeper-refdb v3.3.0-25-ge46327b96a.

When I force delete a project, I notice that zookeeper ref-db entry for deleted project is not removed. I have to delete it manually if I want to recreate a project with the same name. Otherwise, there will be error showing on UI "the name is already occupied by another project." and from log "gerritforge.gerrit.globalrefdb.validation.dfsrefdb.OutOfSyncException: Local ref refs/meta/config (ObjectId=0000000000000000000000000000000000000000) on project test2 is out of sync with the shared ref-db".

Is this expected behavior? Or am I missing some configuration to make zookeeper refdb automatically clean up?

Best Regards,

Serena

Luca Milanesio

unread,

May 22, 2025, 12:50:59 PMMay 22

to Repo and Gerrit Discussion, Luca Milanesio

On 22 May 2025, at 17:07, Serena He <Seren...@arm.com> wrote:

Hi team,

I have a K8sgerrit with Gerrit v3.8.1, installing delete-project v3.8.1 and zookeeper-refdb v3.3.0-25-ge46327b96a.

Gerrit v3.8.x is unsupported, as you may know (see [1]).

When I force delete a project,

What do you mean by “force delete”?

I notice that zookeeper ref-db entry for deleted project is not removed. I have to delete it manually if I want to recreate a project with the same name. Otherwise, there will be error showing on UI "the name is already occupied by another project." and from log "gerritforge.gerrit.globalrefdb.validation.dfsrefdb.OutOfSyncException: Local ref refs/meta/config (ObjectId=0000000000000000000000000000000000000000) on project test2 is out of sync with the shared ref-db".

Deleting a project also suffers from a known bug that is fixed with [2] in Gerrit v3.10.x.

Please upgrade and the delete-project will work as expected.

Is this expected behavior? Or am I missing some configuration to make zookeeper refdb automatically clean up?

No, it’s a bug and it’s fixed from v3.10.x onwards.

HTH

Luca.

[1] https://www.gerritcodereview.com/support.html#supported-versions

[2] https://gerrit-review.googlesource.com/c/gerrit/+/455781

Best Regards,
Serena

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/3f547383-26fb-47a3-9572-3b16a1037944n%40googlegroups.com.

Serena He

unread,

May 30, 2025, 11:23:41 AMMay 30

to Repo and Gerrit Discussion

On Thursday, May 22, 2025 at 5:50:59 PM UTC+1 Luca Milanesio wrote:

On 22 May 2025, at 17:07, Serena He <Seren...@arm.com> wrote:

Hi team,

I have a K8sgerrit with Gerrit v3.8.1, installing delete-project v3.8.1 and zookeeper-refdb v3.3.0-25-ge46327b96a.

Gerrit v3.8.x is unsupported, as you may know (see [1]).

When I force delete a project,

What do you mean by “force delete”?

I mean "Delete repo even if open changes exist".

I notice that zookeeper ref-db entry for deleted project is not removed. I have to delete it manually if I want to recreate a project with the same name. Otherwise, there will be error showing on UI "the name is already occupied by another project." and from log "gerritforge.gerrit.globalrefdb.validation.dfsrefdb.OutOfSyncException: Local ref refs/meta/config (ObjectId=0000000000000000000000000000000000000000) on project test2 is out of sync with the shared ref-db".

Deleting a project also suffers from a known bug that is fixed with [2] in Gerrit v3.10.x.
Please upgrade and the delete-project will work as expected.

Is this expected behavior? Or am I missing some configuration to make zookeeper refdb automatically clean up?

No, it’s a bug and it’s fixed from v3.10.x onwards.

Thank you for the information. I've upgraded to the latest 3.12, but still see that problem.

When I delete from UI, I get "Error: Response payload is not prefixed json. Payload: ", and from logs I can see [1][2][3], no corresponding log in sharedref_log. The project is deleted from NoteDB and cache but not deleted from zookeeper. The version if zookeeper is 3.9.3.

Would you please suggest what am I missing?

Thank you,

Serena

[1] httpd_log
[HTTP-200] - - [2025-05-30T13:00:58.652Z] "POST /plugins/high-availability/cache/projects HTTP/1.1" 204 - 12 - "Apache-HttpClient/4.5.14 (Java/21.0.7)" 1 - 64304 - 1748610058640-445666cb
[HTTP-308] - [2025-05-30T13:00:58.653Z] "POST /projects/test/delete-project~delete HTTP/1.1" 204 - 267 - "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:137.0) Gecko/20100101 Firefox/137.0" 39 40 1154296 - 1748610058401-84d46d45
[HTTP-255] - - [2025-05-30T13:00:58.667Z] "POST /plugins/high-availability/cache/projects HTTP/1.1" 204 - 7 - "Apache-HttpClient/4.5.14 (Java/21.0.7)" - - 63992 - 1748610058661-544469a9

[2]error_log

[2025-05-30T13:00:58.634Z] [pool-45-thread-3] DEBUG com.ericsson.gerrit.plugins.highavailability.forwarder.rest.RestForwarder : Executing Update project_list, remove test towards http://gerrit-1.gerrit.gerrit.svc.cluster.local:8080
[2025-05-30T13:00:58.636Z] [pool-45-thread-10] DEBUG com.ericsson.gerrit.plugins.highavailability.forwarder.rest.RestForwarder : Executing invalidate cache projects projects towards http://gerrit-1.gerrit.gerrit.svc.cluster.local:8080
[2025-05-30T13:00:58.641Z] [pool-45-thread-4] DEBUG com.ericsson.gerrit.plugins.highavailability.forwarder.rest.RestForwarder : Executing invalidate cache projects projects towards http://gerrit-1.gerrit.gerrit.svc.cluster.local:8080
[2025-05-30T13:00:58.641Z] [HTTP POST /plugins/high-availability/cache/projects (N/A from 127.0.0.6)] DEBUG com.ericsson.gerrit.plugins.highavailability.forwarder.ForwardedCacheEvictionHandler : Invalidated cache projects[test] [CONTEXT TRACE_ID="1748610058640-445666cb" ]
[2025-05-30T13:00:58.640Z] [HTTP POST /projects/test/delete-project~delete ] WARN com.googlesource.gerrit.plugins.webhooks.EventHandler : Ignoring event for a non-existing project test, com.googlesource.gerrit.plugins.deleteproject.ProjectDeletedEvent [CONTEXT PLUGIN="webhooks" TRACE_ID="1748610058401-84d46d45" project="test" request="REST /projects/*/delete-project~delete" ]
[2025-05-30T13:00:58.640Z] [pool-45-thread-6] DEBUG com.ericsson.gerrit.plugins.highavailability.forwarder.rest.RestForwarder : Executing send event project-deleted towards http://gerrit-1.gerrit.gerrit.svc.cluster.local:8080
[2025-05-30T13:00:58.650Z] [pool-45-thread-10] DEBUG com.ericsson.gerrit.plugins.highavailability.forwarder.rest.RestForwarder : invalidate cache projects projects towards http://gerrit-1.gerrit.gerrit.svc.cluster.local:8080 OK
[2025-05-30T13:00:58.661Z] [HTTP POST /plugins/high-availability/cache/projects] DEBUG com.ericsson.gerrit.plugins.highavailability.forwarder.ForwardedCacheEvictionHandler : Invalidated cache projects[test] [CONTEXT TRACE_ID="1748610058661-544469a9" ]
[2025-05-30T13:00:58.671Z] [pool-45-thread-4] DEBUG com.ericsson.gerrit.plugins.highavailability.forwarder.rest.RestForwarder : invalidate cache projects projects towards http://gerrit-1.gerrit.gerrit.svc.cluster.local:8080 OK
[2025-05-30T13:00:58.671Z] [pool-45-thread-6] DEBUG com.ericsson.gerrit.plugins.highavailability.forwarder.rest.RestForwarder : send event project-deleted towards http://gerrit-1.gerrit.gerrit.svc.cluster.local:8080 OK
[2025-05-30T13:00:58.899Z] [pool-45-thread-3] DEBUG com.ericsson.gerrit.plugins.highavailability.forwarder.rest.RestForwarder : Update project_list, remove test towards http://gerrit-1.gerrit.gerrit.svc.cluster.local:8080 OK

[3]delete_log

[2025-05-30 13:00:58,652 +0000] INFO userid username OK test {"preserve":false,"force":true}

Luca Milanesio

unread,

May 30, 2025, 3:23:43 PMMay 30

to Repo and Gerrit Discussion, Luca Milanesio

On 30 May 2025, at 17:23, Serena He <Seren...@arm.com> wrote:

On Thursday, May 22, 2025 at 5:50:59 PM UTC+1 Luca Milanesio wrote:

On 22 May 2025, at 17:07, Serena He <Seren...@arm.com> wrote:

Hi team,

I have a K8sgerrit with Gerrit v3.8.1, installing delete-project v3.8.1 and zookeeper-refdb v3.3.0-25-ge46327b96a.

Gerrit v3.8.x is unsupported, as you may know (see [1]).

When I force delete a project,

What do you mean by “force delete”?

I mean "Delete repo even if open changes exist".
I notice that zookeeper ref-db entry for deleted project is not removed. I have to delete it manually if I want to recreate a project with the same name. Otherwise, there will be error showing on UI "the name is already occupied by another project." and from log "gerritforge.gerrit.globalrefdb.validation.dfsrefdb.OutOfSyncException: Local ref refs/meta/config (ObjectId=0000000000000000000000000000000000000000) on project test2 is out of sync with the shared ref-db".

Deleting a project also suffers from a known bug that is fixed with [2] in Gerrit v3.10.x.
Please upgrade and the delete-project will work as expected.

Is this expected behavior? Or am I missing some configuration to make zookeeper refdb automatically clean up?

No, it’s a bug and it’s fixed from v3.10.x onwards.

Thank you for the information. I've upgraded to the latest 3.12, but still see that problem.
When I delete from UI, I get "Error: Response payload is not prefixed json. Payload: ", and from logs I can see [1][2][3], no corresponding log in sharedref_log. The project is deleted from NoteDB and cache but not deleted from zookeeper. The version if zookeeper is 3.9.3.
Would you please suggest what am I missing?

You are missing the fix for the bug [4].

If you have upgraded to Gerrit v3.12 and you still see the issue, make sure you’ve cleared out the browser cache completely.

It was a client issue and it has been fixed with [5].

HTH

Luca.

[4] https://issues.gerritcodereview.com/issues/379762004

[5] https://gerrit-review.googlesource.com/c/gerrit/+/443482

To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/6880b9b4-61f2-4aec-bb29-2fcd5bf45e93n%40googlegroups.com.

lucamilanesio

unread,

Jun 8, 2025, 6:09:36 AMJun 8

to Repo and Gerrit Discussion

I've discovered a new bug [6] and fixed it with [7].

I believe this is the problem you've experiencing and it should not happen anymore after applying the fix.

HTH

Luca.

[6] https://issues.gerritcodereview.com/issues/422947553

[7] https://gerrit-review.googlesource.com/c/plugins/high-availability/+/481781

Serena He

unread,

Jun 10, 2025, 10:20:41 AMJun 10

to Repo and Gerrit Discussion

Thank you very much! That explains our issue.

Many Thanks,

Serena

Serena He

unread,

Sep 21, 2025, 6:46:40 PM (2 days ago) Sep 21

to Repo and Gerrit Discussion

On Sunday, June 8, 2025 at 11:09:36 AM UTC+1 lucamilanesio wrote:

Hi Luca,

I'm upgrading to Gerrit 3.12.2-27-gcf7d13a276 with high-availability v3.7.8-135-g5af509297c and 2 primary nodes. I can now delete and recreate a project successfully, but I meet another problem when testing with a project with changes. An example log for project deletion is shown in error_log[1]. After I recreate that project, I can see changes that should be deleted with the previous project still in the changes list. When the frontend reaches gerrit-1, it's all working fine, but when it reaches gerrit-0, the changes should be deleted appears and I can find [2]. I guess delete event doesn't update changes so not captured by indexSync. I then extend initialSyncAge to cover those changes and restart gerrit-0. From [3], the stale changes are fixed.

I hope I explain this clearly. My question is whether this is expected for current delete-project behavior or am I missing something? Is there a way to avoid this manual fix?

Many Thanks,
Serena

error_log
[1]
[[gerrit-0]]
[2025-09-21T17:07:59.376Z] [HTTP DELETE /plugins/high-availability/cache/project_list/serena-test (N/A from 127.0.0.6)] DEBUG com.ericsson.gerrit.plugins.highavailability.forwarder.ForwardedProjectListUpdateHandler : Removed serena-test from project list [CONTEXT TRACE_ID="1758474479363-859032e9" ]
[2025-09-21T17:07:59.502Z] [HTTP POST /plugins/high-availability/cache/projects (N/A from 127.0.0.6)] DEBUG com.ericsson.gerrit.plugins.highavailability.forwarder.ForwardedCacheEvictionHandler : Invalidated cache projects[serena-test] [CONTEXT TRACE_ID="1758474479502-c5bcb1aa" ]
[2025-09-21T17:07:59.503Z] [pool-45-thread-3] DEBUG com.ericsson.gerrit.plugins.highavailability.forwarder.rest.RestForwarder : Executing invalidate cache projects projects towards http://gerrit-1.gerrit.gerrit.svc.cluster.local:8080
[2025-09-21T17:07:59.512Z] [HTTP POST /plugins/high-availability/event/project-deleted (N/A from 127.0.0.6)] DEBUG com.ericsson.gerrit.plugins.highavailability.forwarder.ForwardedEventHandler : dispatching event project-deleted [CONTEXT TRACE_ID="1758474479511-859032e9" ]
[2025-09-21T17:07:59.518Z] [pool-45-thread-3] DEBUG com.ericsson.gerrit.plugins.highavailability.forwarder.rest.RestForwarder : invalidate cache projects projects towards http://gerrit-1.gerrit.gerrit.svc.cluster.local:8080 OK
[2025-09-21T17:07:59.521Z] [HTTP POST /plugins/high-availability/cache/projects (N/A from 127.0.0.6)] DEBUG com.ericsson.gerrit.plugins.highavailability.forwarder.ForwardedCacheEvictionHandler : Invalidated cache projects[serena-test] [CONTEXT TRACE_ID="1758474479520-c5bcb1aa" ]
[[gerrit-1]]
[2025-09-21T17:07:59.360Z] [pool-45-thread-5] DEBUG com.ericsson.gerrit.plugins.highavailability.forwarder.rest.RestForwarder : Executing Update project_list, remove serena-test towards http://gerrit-0.gerrit.gerrit.svc.cluster.local:8080
[2025-09-21T17:07:59.360Z] [HTTP POST /projects/serena-test/delete-project~delete (user from 10.0.0.104)] INFO com.gerritforge.gerrit.globalrefdb.validation.ProjectDeletedSharedDbCleanup : Deleting project 'serena-test'. Will perform a cleanup in Shared-Ref database. [CONTEXT TRACE_ID="1758474479071-bd26400a" project="serena-test" request="REST /projects/*/delete-project~delete" ]
[2025-09-21T17:07:59.377Z] [pool-45-thread-5] DEBUG com.ericsson.gerrit.plugins.highavailability.forwarder.rest.RestForwarder : Update project_list, remove serena-test towards http://gerrit-0.gerrit.gerrit.svc.cluster.local:8080 OK
[2025-09-21T17:07:59.500Z] [pool-45-thread-8] DEBUG com.ericsson.gerrit.plugins.highavailability.forwarder.rest.RestForwarder : Executing invalidate cache projects projects towards http://gerrit-0.gerrit.gerrit.svc.cluster.local:8080
[2025-09-21T17:07:59.503Z] [pool-45-thread-8] DEBUG com.ericsson.gerrit.plugins.highavailability.forwarder.rest.RestForwarder : invalidate cache projects projects towards http://gerrit-0.gerrit.gerrit.svc.cluster.local:8080 OK
[2025-09-21T17:07:59.506Z] [HTTP POST /projects/serena-test/delete-project~delete (user from 10.0.0.104)] WARN com.googlesource.gerrit.plugins.webhooks.EventHandler : Ignoring event for a non-existing project serena-test, com.googlesource.gerrit.plugins.deleteproject.ProjectDeletedEvent@180a04bc [CONTEXT PLUGIN="webhooks" TRACE_ID="1758474479071-bd26400a" project="serena-test" request="REST /projects/*/delete-project~delete" ]
[2025-09-21T17:07:59.505Z] [pool-45-thread-7] DEBUG com.ericsson.gerrit.plugins.highavailability.forwarder.rest.RestForwarder : Executing send event project-deleted towards http://gerrit-0.gerrit.gerrit.svc.cluster.local:8080
[2025-09-21T17:07:59.514Z] [pool-45-thread-1] DEBUG com.ericsson.gerrit.plugins.highavailability.forwarder.rest.RestForwarder : Executing invalidate cache projects projects towards http://gerrit-0.gerrit.gerrit.svc.cluster.local:8080
[2025-09-21T17:07:59.514Z] [HTTP POST /plugins/high-availability/cache/projects (N/A from 127.0.0.6)] DEBUG com.ericsson.gerrit.plugins.highavailability.forwarder.ForwardedCacheEvictionHandler : Invalidated cache projects[serena-test] [CONTEXT TRACE_ID="1758474479514-16a5c323" ]
[2025-09-21T17:07:59.520Z] [pool-45-thread-7] DEBUG com.ericsson.gerrit.plugins.highavailability.forwarder.rest.RestForwarder : send event project-deleted towards http://gerrit-0.gerrit.gerrit.svc.cluster.local:8080 OK
[2025-09-21T17:07:59.521Z] [pool-45-thread-1] DEBUG com.ericsson.gerrit.plugins.highavailability.forwarder.rest.RestForwarder : invalidate cache projects projects towards http://gerrit-0.gerrit.gerrit.svc.cluster.local:8080 OK
[2025-09-21T17:08:00.247Z] [pool-45-thread-3] DEBUG com.ericsson.gerrit.plugins.highavailability.forwarder.rest.RestForwarder : Executing invalidate cache projects projects towards http://gerrit-0.gerrit.gerrit.svc.cluster.local:8080
[2025-09-21T17:08:00.251Z] [pool-45-thread-3] DEBUG com.ericsson.gerrit.plugins.highavailability.forwarder.rest.RestForwarder : invalidate cache projects projects towards http://gerrit-0.gerrit.gerrit.svc.cluster.local:8080 OK

[2]
[2025-09-21T17:16:29.869Z] [HTTP GET /changes/?O=10102&S=0&q=change%3A39197%20OR%20change%3A37708%20OR%20change%3A37791%20OR%20change%3A35494%20OR%2 (user from 10.0.0.244)] WARN com.google.gerrit.server.change.ChangeJson : Omitting corrupt change 39197 from results [CONTEXT TRACE_ID="1758474989829-e9a498f0" request="REST /changes/" ]
[2025-09-21T17:16:32.351Z] [HTTP GET /changes/?O=10102&S=0&q=change%3A39178%20OR%20change%3A39177%20OR%20change%3A37712%20OR%20change%3A37711%20OR%2 (user from 10.0.0.244)] WARN com.google.gerrit.server.change.ChangeJson : Omitting corrupt change 39178 from results [CONTEXT TRACE_ID="1758474992318-f2bbcde8" request="REST /changes/" ]
[2025-09-21T17:16:32.372Z] [HTTP GET /changes/?O=10102&S=0&q=change%3A39178%20OR%20change%3A39177%20OR%20change%3A37712%20OR%20change%3A37711%20OR%2 (user from 10.0.0.244)] WARN com.google.gerrit.server.change.ChangeJson : Omitting corrupt change 39177 from results [CONTEXT TRACE_ID="1758474992318-f2bbcde8" request="REST /changes/" ]
[2025-09-21T17:16:47.183Z] [HTTP GET /changes/?O=10102&S=0&q=change%3A39180%20OR%20change%3A39179%20OR%20change%3A35814%20OR%20change%3A37111%20OR%2 (user from 10.0.0.244)] WARN com.google.gerrit.server.change.ChangeJson : Omitting corrupt change 39180 from results [CONTEXT TRACE_ID="1758475007127-0ead5ab7" request="REST /changes/" ]
[2025-09-21T17:16:47.189Z] [HTTP GET /changes/?O=10102&S=0&q=change%3A39180%20OR%20change%3A39179%20OR%20change%3A35814%20OR%20change%3A37111%20OR%2 (user from 10.0.0.244)] WARN com.google.gerrit.server.change.ChangeJson : Omitting corrupt change 39179 from results [CONTEXT TRACE_ID="1758475007127-0ead5ab7" request="REST /changes/" ]

[3]
[2025-09-21T17:22:46.728Z] [IndexSyncRunner-1[java.util.concurrent.FutureTask@60ce75d0[Not completed, task = com.google.gerrit.server.logging.LoggingContextAwareCallable@507a39b6]]] INFO com.ericsson.gerrit.plugins.highavailability.indexsync.IndexSyncRunner : Change test/serena-test~39180 present in index but not in noteDb. Deleting from index
[2025-09-21T17:22:46.732Z] [IndexSyncRunner-1[java.util.concurrent.FutureTask@60ce75d0[Not completed, task = com.google.gerrit.server.logging.LoggingContextAwareCallable@507a39b6]]] INFO com.ericsson.gerrit.plugins.highavailability.indexsync.IndexSyncRunner : Change test/serena-test~39179 present in index but not in noteDb. Deleting from index
[2025-09-21T17:22:46.765Z] [IndexSyncRunner-1[java.util.concurrent.FutureTask@60ce75d0[Not completed, task = com.google.gerrit.server.logging.LoggingContextAwareCallable@507a39b6]]] INFO com.ericsson.gerrit.plugins.highavailability.indexsync.IndexSyncRunner : Change serena-test~39177 present in index but not in noteDb. Deleting from index
[2025-09-21T17:22:46.767Z] [IndexSyncRunner-1[java.util.concurrent.FutureTask@60ce75d0[Not completed, task = com.google.gerrit.server.logging.LoggingContextAwareCallable@507a39b6]]] INFO com.ericsson.gerrit.plugins.highavailability.indexsync.IndexSyncRunner : Change serena-test~39178 present in index but not in noteDb. Deleting from index
[2025-09-21T17:22:46.777Z] [IndexSyncRunner-1[java.util.concurrent.FutureTask@60ce75d0[Not completed, task = com.google.gerrit.server.logging.LoggingContextAwareCallable@507a39b6]]] INFO com.ericsson.gerrit.plugins.highavailability.indexsync.IndexSyncRunner : Change serena-test~39197 present in index but not in noteDb. Deleting from index

Luca Milanesio

unread,

Sep 22, 2025, 3:21:05 AM (yesterday) Sep 22

to Repo and Gerrit Discussion, Luca Milanesio

Please do not use this topic if the issue is different, but open a separate one.

Thanks for respecting the general support guidelines at [3].

Luca.

[3] https://www.gerritcodereview.com/support.html#general-support

To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/1627b975-304d-41db-9d6d-e411630791d4n%40googlegroups.com.

Reply all

Reply to author

Forward