Gerrit index very very very slow when nfs mount with noac

bonnie chen

unread,

Aug 8, 2024, 11:33:08 PM8/8/24

to Repo and Gerrit Discussion

Hi team,

We have two Gerrit masters in high-availability via high available plugin mount nfs server

when we nfs mount with ac，we find sometimes the changes will be lock failure，and find this topic： https://gerrit.googlesource.com/jgit/+/refs/tags/v5.13.0.202108250949-m3/Documentation/config-options.md ， it say it is because of nfs cache

however we find when nfs mount with noac，Gerrit index is very very very slow.

If it is an issue

Thanks,

Bonnie

Luca Milanesio

unread,

Aug 8, 2024, 11:38:36 PM8/8/24

to Repo and Gerrit Discussion, Luca Milanesio

On 8 Aug 2024, at 20:33, bonnie chen <bonnie....@gmail.com> wrote:

Hi team,

We have two Gerrit masters in high-availability via high available plugin mount nfs server

when we nfs mount with ac，we find sometimes the changes will be lock failure

How can Gerrit indexing cause a lock failure?

You should share your error_log and the exact user operation performed.

，and find this topic： https://gerrit.googlesource.com/jgit/+/refs/tags/v5.13.0.202108250949-m3/Documentation/config-options.md ，

That’s not a topic but the JGit documentation.

it say it is because of nfs cache

however we find when nfs mount with noac，Gerrit index is very very very slow.

That’s not Gerrit, that NFS slow without caching, which is expected.

HTH

Luca.

If it is an issue

Thanks,
Bonnie

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/fe6105a1-01f0-442d-863c-5ce112a387e2n%40googlegroups.com.

bonnie chen

unread,

Aug 8, 2024, 11:58:30 PM8/8/24

to Repo and Gerrit Discussion

Thanks for reply

Find the following log in error_log

[2024-08-02T18:35:34.266+08:00] [HTTP POST /changes/topsop~163286/revisions/9/rebase(kartik from 10.253.6.1)] WARN com.google.gerrit.server.update.RetryHelper : REST_WRITE_REQUEST was attempted 15 times [CONTEXT project="topsop" ]
[2024-08-02T18:35:34.266+08:00] [HTTP POST /changes/topsop~163286/revisions/9/rebase(kartik from 10.253.6.1)] ERROR com.google.gerrit.httpd.restapi.RestApiServlet : Error in POST /changes/topsop~163286/revisions/9/rebase [CONTEXT project="topsop" ]
CREATE: 0000000000000000000000000000000000000000 180b8d5f6a1f24e6d9cd915a3a22a595ad9ce24a refs/changes/86/163286/10 (LOCK_FAILURE)
UPDATE: e6d330cd9b1f95f82ef2a13965afd417013c0d50 b1aa1f5559dc2ccd7bc2b728f1d8b1640b1b9e8f refs/changes/86/163286/meta (REJECTED_OTHER_REASON: transaction aborted)
CREATE: 0000000000000000000000000000000000000000 180b8d5f6a1f24e6d9cd915a3a22a595ad9ce24a refs/changes/86/163286/10 (LOCK_FAILURE)
UPDATE: e6d330cd9b1f95f82ef2a13965afd417013c0d50 b1aa1f5559dc2ccd7bc2b728f1d8b1640b1b9e8f refs/changes/86/163286/meta (REJECTED_OTHER_REASON: transaction aborted)
[2024-08-02T18:36:05.458+08:00] [HTTP POST /changes/topsop~163286/revisions/9/rebase(kartik from 10.253.6.1)] WARN com.google.gerrit.server.update.RetryHelper : REST_WRITE_REQUEST was attempted 15 times [CONTEXT project="topsop" ]
[2024-08-02T18:36:05.459+08:00] [HTTP POST /changes/topsop~163286/revisions/9/rebase(kartik from 10.253.6.1)] ERROR com.google.gerrit.httpd.restapi.RestApiServlet : Error in POST /changes/topsop~163286/revisions/9/rebase [CONTEXT project="topsop" ]
CREATE: 0000000000000000000000000000000000000000 faaf06b65e411241e4d0baccf695e8d90ec46128 refs/changes/86/163286/10 (LOCK_FAILURE)
UPDATE: e6d330cd9b1f95f82ef2a13965afd417013c0d50 1eff27d183f086b69e9310355696fc96be36b3ad refs/changes/86/163286/meta (REJECTED_OTHER_REASON: transaction aborted)
CREATE: 0000000000000000000000000000000000000000 faaf06b65e411241e4d0baccf695e8d90ec46128 refs/changes/86/163286/10 (LOCK_FAILURE)
UPDATE: e6d330cd9b1f95f82ef2a13965afd417013c0d50 1eff27d183f086b69e9310355696fc96be36b3ad refs/changes/86/163286/meta (REJECTED_OTHER_REASON: transaction aborted)
[2024-08-02T20:29:40.588+08:00] [HTTP POST /changes/topsop~163286/revisions/9/rebase(kartik from 10.253.6.1)] WARN com.google.gerrit.server.update.RetryHelper : REST_WRITE_REQUEST was attempted 15 times [CONTEXT project="topsop" ]
[2024-08-02T20:29:40.588+08:00] [HTTP POST /changes/topsop~163286/revisions/9/rebase(kartik from 10.253.6.1)] ERROR com.google.gerrit.httpd.restapi.RestApiServlet : Error in POST /changes/topsop~163286/revisions/9/rebase [CONTEXT project="topsop" ]
CREATE: 0000000000000000000000000000000000000000 36d1d4eab03753938af20182a1816f31ed11dd5f refs/changes/86/163286/10 (LOCK_FAILURE)
UPDATE: 391e2cc3c0c10168f03fc8e9199b045c436180af d8962082a0ed9f51c160fcd527bd4c70b1af2c17 refs/changes/86/163286/meta (REJECTED_OTHER_REASON: transaction aborted)
CREATE: 0000000000000000000000000000000000000000 36d1d4eab03753938af20182a1816f31ed11dd5f refs/changes/86/163286/10 (LOCK_FAILURE)
UPDATE: 391e2cc3c0c10168f03fc8e9199b045c436180af d8962082a0ed9f51c160fcd527bd4c70b1af2c17 refs/changes/86/163286/meta (REJECTED_OTHER_REASON: transaction aborted)

bonnie chen

unread,

Aug 9, 2024, 12:01:32 AM8/9/24

to Repo and Gerrit Discussion

And also a lot " could not be indexed after 3 retries. Change index could be stale " error

[2024-08-02T11:48:07.871+08:00] [Forwarded-Index-Event-7[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@6d281849[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@70991f34[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@1f127c86]]]] ERROR com.ericsson.gerrit.plugins.highavailability.forwarder.ForwardedIndexingHandler : Change tops~163675 could not be indexed after 3 retries. Change index could be stale.
[2024-08-02T11:48:26.199+08:00] [Forwarded-Index-Event-6[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@215c23d0[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@55ac1c91[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@2eef80a1]]]] ERROR com.ericsson.gerrit.plugins.highavailability.forwarder.ForwardedIndexingHandler : Change tops~144961 could not be indexed after 3 retries. Change index could be stale.
[2024-08-02T11:48:42.555+08:00] [Forwarded-Index-Event-5[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@358a7714[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@45db9737[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@34e02061]]]] ERROR com.ericsson.gerrit.plugins.highavailability.forwarder.ForwardedIndexingHandler : Change tops~163675 could not be indexed after 3 retries. Change index could be stale.
[2024-08-02T11:57:53.131+08:00] [Forwarded-Index-Event-3[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@67051b8[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@5c75cd66[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@1f71e72b]]]] ERROR com.ericsson.gerrit.plugins.highavailability.forwarder.ForwardedIndexingHandler : Change tops~144961 could not be indexed after 3 retries. Change index could be stale.
[2024-08-02T12:13:56.640+08:00] [Forwarded-Index-Event-1[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@54f9e6f9[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@2d9d70c5[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@26e6093a]]]] ERROR com.ericsson.gerrit.plugins.highavailability.forwarder.ForwardedIndexingHandler : Change topsop~163338 could not be indexed after 3 retries. Change index could be stale.
[2024-08-02T15:55:28.302+08:00] [Forwarded-Index-Event-4[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@6545db95[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@5eb59e68[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@3af64f0d]]]] ERROR com.ericsson.gerrit.plugins.highavailability.forwarder.ForwardedIndexingHandler : Change topsop~163719 could not be indexed after 3 retries. Change index could be stale.
[2024-08-02T17:41:49.665+08:00] [Forwarded-Index-Event-2[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@3fedc49a[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@767c1c92[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@1f4319c2]]]] ERROR com.ericsson.gerrit.plugins.highavailability.forwarder.ForwardedIndexingHandler : Change tops~163787 could not be indexed after 3 retries. Change index could be stale.
[2024-08-02T19:00:36.676+08:00] [Forwarded-Index-Event-3[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@2c9f30a3[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@63ba58eb[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@2253f7d8]]]] ERROR com.ericsson.gerrit.plugins.highavailability.forwarder.ForwardedIndexingHandler : Change topsop~163558 could not be indexed after 3 retries. Change index could be stale.

Luca Milanesio

unread,

Aug 9, 2024, 7:39:50 AM8/9/24

to Repo and Gerrit Discussion, Luca Milanesio

On 8 Aug 2024, at 20:58, bonnie chen <bonnie....@gmail.com> wrote:

Thanks for reply

Find the following log in error_log

[2024-08-02T18:35:34.266+08:00] [HTTP POST /changes/topsop~163286/revisions/9/rebase(kartik from 10.253.6.1)] WARN com.google.gerrit.server.update.RetryHelper : REST_WRITE_REQUEST was attempted 15 times [CONTEXT project="topsop" ]

From the REST-API URL, you can clearly see that this is a rebase: nothing to do with a Gerrit indexing operation.

If you have concurrency on the repository, that’s absolutely expected.

Luca.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/d8a60a80-76e7-450c-8f13-209c2820cde1n%40googlegroups.com.

Luca Milanesio

unread,

Aug 9, 2024, 7:45:32 AM8/9/24

to Repo and Gerrit Discussion, Luca Milanesio

On 8 Aug 2024, at 21:01, bonnie chen <bonnie....@gmail.com> wrote:

And also a lot " could not be indexed after 3 retries. Change index could be stale " error

[2024-08-02T11:48:07.871+08:00] [Forwarded-Index-Event-7[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@6d281849[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@70991f34[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@1f127c86]]]] ERROR com.ericsson.gerrit.plugins.highavailability.forwarder.ForwardedIndexingHandler : Change tops~163675 could not be indexed after 3 retries. Change index could be stale.
[2024-08-02T11:48:26.199+08:00] [Forwarded-Index-Event-6[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@215c23d0[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@55ac1c91[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@2eef80a1]]]] ERROR com.ericsson.gerrit.plugins.highavailability.forwarder.ForwardedIndexingHandler : Change tops~144961 could not be indexed after 3 retries. Change index could be stale.
[2024-08-02T11:48:42.555+08:00] [Forwarded-Index-Event-5[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@358a7714[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@45db9737[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@34e02061]]]] ERROR com.ericsson.gerrit.plugins.highavailability.forwarder.ForwardedIndexingHandler : Change tops~163675 could not be indexed after 3 retries. Change index could be stale.
[2024-08-02T11:57:53.131+08:00] [Forwarded-Index-Event-3[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@67051b8[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@5c75cd66[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@1f71e72b]]]] ERROR com.ericsson.gerrit.plugins.highavailability.forwarder.ForwardedIndexingHandler : Change tops~144961 could not be indexed after 3 retries. Change index could be stale.
[2024-08-02T12:13:56.640+08:00] [Forwarded-Index-Event-1[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@54f9e6f9[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@2d9d70c5[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@26e6093a]]]] ERROR com.ericsson.gerrit.plugins.highavailability.forwarder.ForwardedIndexingHandler : Change topsop~163338 could not be indexed after 3 retries. Change index could be stale.
[2024-08-02T15:55:28.302+08:00] [Forwarded-Index-Event-4[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@6545db95[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@5eb59e68[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@3af64f0d]]]] ERROR com.ericsson.gerrit.plugins.highavailability.forwarder.ForwardedIndexingHandler : Change topsop~163719 could not be indexed after 3 retries. Change index could be stale.
[2024-08-02T17:41:49.665+08:00] [Forwarded-Index-Event-2[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@3fedc49a[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@767c1c92[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@1f4319c2]]]] ERROR com.ericsson.gerrit.plugins.highavailability.forwarder.ForwardedIndexingHandler : Change tops~163787 could not be indexed after 3 retries. Change index could be stale.
[2024-08-02T19:00:36.676+08:00] [Forwarded-Index-Event-3[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@2c9f30a3[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@63ba58eb[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@2253f7d8]]]] ERROR com.ericsson.gerrit.plugins.highavailability.forwarder.ForwardedIndexingHandler : Change topsop~163558 could not be indexed after 3 retries. Change index could be stale.

The above issue has been fixed with [1].

Maybe you’re running al older version?

Your NFS caching assumption is putting you off track.

HTH

Luca.

[1] https://gerrit-review.googlesource.com/c/plugins/high-availability/+/431677

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/8d38e5ee-c2d1-422c-b213-48c8d652aa08n%40googlegroups.com.

bonnie chen

unread,

Aug 14, 2024, 12:01:50 AM8/14/24

to Repo and Gerrit Discussion

Hi Luca,

Thanks for explain.

If so, what would cause a lock failure, and when would it prevent for submit?

And the failure has become intermittent and we are unable to identify the root cause

Regards，

Bonnie

bonnie chen

unread,

Aug 14, 2024, 2:44:21 AM8/14/24

to Repo and Gerrit Discussion

yup，we use 3.4.3

"existence of draft comments doesn't have any effect on changes staleness check" what mean this error "Change topsop~163558 could not be indexed after 3 retries. Change index could be stale." will not affect the following action for changes

am I right?

Reply all

Reply to author

Forward