Hi,
Recently, while running an offline reindex, I encountered numerous warnings. Although the process printed the completion message at the end, it never returned control to the shell.
The output stopped at:
Index projects in version 10 is ready
[2026-02-26 22:40:30,264] [main] INFO com.google.gerrit.server.cache.h2.H2CacheFactory : Finishing 502 disk cache updates I had to manually terminate the process using Ctrl+C. Interestingly, despite these warnings and the abnormal behavior during reindexing, the Gerrit server starts successfully afterwards.
Now I can still reproduce these issuses with another server.
My server have 32 cores and 128G mem; the command:
java -jar -Xmx48g gerrit-home/bin/gerrit.war reindex -d gerrit-home --threads=4
(here I use 4 threads for trying to reduce some warnings, but seems no
any effect)
Before reindex, I removed all caches files.
put some logs here:
org.h2.jdbc.JdbcSQLTimeoutException: Timeout trying to lock table "DATA"; SQL statement:
ALTER TABLE data ADD COLUMN IF NOT EXISTS space BIGINT AS OCTET_LENGTH(k) + OCTET_LENGTH(v) [50200-232]
org.h2.jdbc.JdbcSQLTimeoutException: Timeout trying to lock table "DATA"; SQL statement:
ALTER TABLE data ADD COLUMN IF NOT EXISTS version INT DEFAULT 0 NOT NULL [50200-232]
org.h2.jdbc.JdbcSQLSyntaxErrorException: Column "VERSION" not found; SQL statement:
CREATE INDEX IF NOT EXISTS version_key ON data(version, k) [42122-232]
[2026-02-26 22:06:55,481] [Index-Batch-12[Index all changes of project bes-aos/sport-algo][bes-aos/sport-algo-0]] WARN com.google.gerrit.server.cache.h2.H2CacheImpl : Cannot read cache jdbc:h2:file:///data/gerrit-data/cache/modified_files-v2 for ModifiedFilesCacheKey{project=bes-aos/sport-algo, aCommit=commit bacb6d9ee2c90343c470d5f7afe5ded1c18d1760 0 -------, bCommit=AnyObjectId[068e4d4f716cadbb32173b69d8c29e63cdff69dc], renameScore=60}
org.h2.jdbc.JdbcBatchUpdateException: Timeout trying to lock table "SYS"; SQL statement:
CREATE TABLE IF NOT EXISTS data(k OTHER NOT NULL PRIMARY KEY HASH,v OTHER NOT NULL,created TIMESTAMP NOT NULL,accessed TIMESTAMP NOT NULL) [50200-232]
at org.h2.jdbc.JdbcStatement.executeBatch(JdbcStatement.java:828)
at com.google.gerrit.server.cache.h2.H2CacheImpl$SqlHandle.<init>(H2CacheImpl.java:761)
at com.google.gerrit.server.cache.h2.H2CacheImpl$SqlStore.acquire(H2CacheImpl.java:716)
at com.google.gerrit.server.cache.h2.H2CacheImpl$SqlStore.getIfPresent(H2CacheImpl.java:468)
at com.google.gerrit.server.cache.h2.H2CacheImpl$Loader.load(H2CacheImpl.java:260)
at com.google.gerrit.server.cache.h2.H2CacheImpl$Loader.load(H2CacheImpl.java:244)
at com.github.benmanes.caffeine.guava.CaffeinatedGuavaLoadingCache$SingleLoader.load(CaffeinatedGuavaLoadingCache.java:136)
at com.github.benmanes.caffeine.cache.LocalLoadingCache.lambda$newMappingFunction$2(LocalLoadingCache.java:141)
at com.github.benmanes.caffeine.cache.LocalCache.lambda$statsAware$0(LocalCache.java:139)
at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2413)
at java.base/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1916)
at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2411)
at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2394)
at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:108)
at com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:54)
at com.github.benmanes.caffeine.guava.CaffeinatedGuavaLoadingCache.get(CaffeinatedGuavaLoadingCache.java:59)
at com.google.gerrit.server.cache.h2.H2CacheImpl.get(H2CacheImpl.java:130)
at com.google.gerrit.server.patch.diff.ModifiedFilesCacheImpl.get(ModifiedFilesCacheImpl.java:90)
at com.google.gerrit.server.patch.DiffOperationsImpl.getModifiedFiles(DiffOperationsImpl.java:256)
at com.google.gerrit.server.patch.DiffOperationsImpl.listModifiedFilesAgainstParent(DiffOperationsImpl.java:130)
at com.google.gerrit.server.patch.DiffSummaryLoader.call(DiffSummaryLoader.java:51)
at com.google.gerrit.server.patch.DiffSummaryLoader.call(DiffSummaryLoader.java:28)
at com.google.gerrit.server.cache.h2.H2CacheImpl.lambda$get$1(H2CacheImpl.java:163)
at com.github.benmanes.caffeine.guava.CaffeinatedGuavaCache.lambda$get$0(CaffeinatedGuavaCache.java:69)
at com.github.benmanes.caffeine.cache.LocalCache.lambda$statsAware$0(LocalCache.java:139)
at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2413)
at java.base/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1955)
at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2411)
at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2394)
at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:108)
at com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:62)
at com.github.benmanes.caffeine.guava.CaffeinatedGuavaCache.get(CaffeinatedGuavaCache.java:67)
at com.google.gerrit.server.cache.h2.H2CacheImpl.get(H2CacheImpl.java:158)
at com.google.gerrit.server.patch.PatchListCacheImpl.getDiffSummary(PatchListCacheImpl.java:101)
at com.google.gerrit.server.query.change.ChangeData.getDiffSummary(ChangeData.java:601)
at com.google.gerrit.server.query.change.ChangeData.computeChangedLines(ChangeData.java:610)
at com.google.gerrit.server.query.change.ChangeData.changedLines(ChangeData.java:622)
at com.google.gerrit.server.index.change.ChangeField.lambda$static$56(ChangeField.java:1207)
at com.google.gerrit.index.IndexedField.get(IndexedField.java:437)
at com.google.gerrit.index.IndexedField$SearchSpec.get(IndexedField.java:151)
at com.google.gerrit.index.Schema.fieldValues(Schema.java:273)
at com.google.gerrit.index.Schema.lambda$buildFields$0(Schema.java:306)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at com.google.common.collect.CollectSpliterators$1WithCharacteristics.lambda$forEachRemaining$1(CollectSpliterators.java:72)
at java.base/java.util.stream.Streams$RangeIntSpliterator.forEachRemaining(Streams.java:104)
at com.google.common.collect.CollectSpliterators$1WithCharacteristics.forEachRemaining(CollectSpliterators.java:72)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
at com.google.gerrit.index.Schema.buildFields(Schema.java:308)
at com.google.gerrit.lucene.AbstractLuceneIndex.toDocument(AbstractLuceneIndex.java:359)
at com.google.gerrit.lucene.LuceneChangeIndex.insert(LuceneChangeIndex.java:237)
at com.google.gerrit.lucene.LuceneChangeIndex.insert(LuceneChangeIndex.java:105)
at com.google.gerrit.server.index.change.ChangeIndexer.indexImpl(ChangeIndexer.java:338)
at com.google.gerrit.server.index.change.ChangeIndexer.doIndex(ChangeIndexer.java:298)
at com.google.gerrit.server.index.change.ChangeIndexer.index(ChangeIndexer.java:294)
at com.google.gerrit.server.index.change.AllChangesIndexer$ProjectSliceIndexer.lambda$new$1(AllChangesIndexer.java:301)
at com.google.gerrit.server.index.change.AllChangesIndexer$ProjectSliceIndexer.index(AllChangesIndexer.java:347)
at com.google.gerrit.server.index.change.AllChangesIndexer$ProjectSliceIndexer.lambda$call$3(AllChangesIndexer.java:331)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:1024)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
at com.google.gerrit.server.index.change.AllChangesIndexer$ProjectSliceIndexer.call(AllChangesIndexer.java:329)
at com.google.gerrit.server.index.change.AllChangesIndexer$ProjectSliceIndexer.call(AllChangesIndexer.java:281)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:75)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:113)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:940)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
...
Reindexing changes: changes: 72% (216500/299257), failed: 4, project-slices: 87% (606/691), Slicing projects: 100% (1054/1054) (-)[2026-02-26 22:36:31,739] [main] INFO com.google.gerrit.server.git.MultiProgressMonitor : Reindexing changes: changes: 72% (216760/299257), failed: 4, project-slices: 87% (606/691), Slicing projects: 100% (1054/1054) (\) [CONTEXT ratelimit_period="1 MINUTES" skipped=119 ]
Reindexing changes: changes: 78% (235622/299257), failed: 4, project-slices: 90% (625/691), Slicing projects: 100% (1054/1054) (-)
Reindexing changes: changes: 79% (237851/299257), failed: 4, project-slices: 90% (627/691), Slicing projects: 100% (1054/1054) (-)[2026-02-26 22:37:31,964] [main] INFO com.google.gerrit.server.git.MultiProgressMonitor : Reindexing changes: changes: 79% (238111/299257), failed: 4, project-slices: 90% (627/691), Slicing projects: 100% (1054/1054) (\) [CONTEXT ratelimit_period="1 MINUTES" skipped=119 ]
Reindexing changes: changes: 86% (258370/299257), failed: 4, project-slices: 93% (648/691), Slicing projects: 100% (1054/1054) (-)[2026-02-26 22:38:32,084] [main] INFO com.google.gerrit.server.git.MultiProgressMonitor : Reindexing changes: changes: 86% (258606/299257), failed: 4, project-slices: 93% (648/691), Slicing projects: 100% (1054/1054) (\) [CONTEXT ratelimit_period="1 MINUTES" skipped=119 ]
Reindexing changes: changes: 92% (278131/299257), failed: 4, project-slices: 96% (668/691), Slicing projects: 100% (1054/1054) (-)[2026-02-26 22:39:32,572] [main] INFO com.google.gerrit.server.git.MultiProgressMonitor : Reindexing changes: changes: 92% (278267/299257), failed: 4, project-slices: 96% (668/691), Slicing projects: 100% (1054/1054) (\) [CONTEXT ratelimit_period="1 MINUTES" skipped=119 ]
Reindexing changes: changes: 97% (293244/299257), failed: 4, project-slices: 98% (683/691), Slicing projects: 100% (1054/1054) (\)
Reindexing changes: changes: 99% (299253/299257), failed: 4, project-slices: 100% (691/691), Slicing projects: 100% (1054/1054), done
[2026-02-26 22:40:30,022] [main] WARN com.google.gerrit.server.index.change.AllChangesIndexer : Failed 4/299257 changes
Reindexed 299257 documents in changes index in 1261.8s (237.2/s)
Index changes in version 87 is ready
Reindexing projects: 100% (1054/1054)
Reindexed 1054 documents in projects index in 0.2s (4466.1/s)
Index projects in version 10 is ready
[2026-02-26 22:40:30,264] [main] INFO com.google.gerrit.server.cache.h2.H2CacheFactory : Finishing 502 disk cache updates
From here I have to ctrl-c to terminate the proccess. but after that,
it still printed some warnings:
^C[2026-02-26 22:43:30,371] [main] WARN com.google.gerrit.server.cache.h2.H2CacheImpl : Cannot put into cache jdbc:h2:file:///data/gerrit-data/cache/gerrit_file_diff-v2
org.h2.jdbc.JdbcSQLNonTransientConnectionException: Database is already closed (to disable automatic closing at VM shutdown, add ";DB_CLOSE_ON_EXIT=FALSE" to the db URL) [90121-232]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:690)
at org.h2.message.DbException.getJdbcSQLException(DbException.java:489)
at org.h2.message.DbException.get(DbException.java:223)
at org.h2.message.DbException.get(DbException.java:199)
at org.h2.message.DbException.get(DbException.java:188)
at org.h2.jdbc.JdbcConnection.checkClosed(JdbcConnection.java:1423)
at org.h2.jdbc.JdbcStatement.checkClosed(JdbcStatement.java:1302)
at org.h2.jdbc.JdbcPreparedStatement.clearParameters(JdbcPreparedStatement.java:292)
at com.google.gerrit.server.cache.h2.H2CacheImpl$SqlStore.put(H2CacheImpl.java:576)
at com.google.gerrit.server.cache.h2.H2CacheImpl.stop(H2CacheImpl.java:217)
at com.google.gerrit.server.cache.h2.H2CacheFactory.stop(H2CacheFactory.java:161)
at com.google.gerrit.lifecycle.LifecycleManager.stop(LifecycleManager.java:109)
at com.google.gerrit.pgm.Reindex.run(Reindex.java:152)
at com.google.gerrit.pgm.util.AbstractProgram.main(AbstractProgram.java:62)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at com.google.gerrit.launcher.GerritLauncher.invokeProgram(GerritLauncher.java:251)
at com.google.gerrit.launcher.GerritLauncher.mainImpl(GerritLauncher.java:147)
at com.google.gerrit.launcher.GerritLauncher.main(GerritLauncher.java:92)
at Main.main(Main.java:30)
[2026-02-26 22:43:30,372] [main] WARN com.google.gerrit.server.cache.h2.H2CacheImpl : Cannot put into cache jdbc:h2:file:///data/gerrit-data/cache/gerrit_file_diff-v2
org.h2.jdbc.JdbcSQLNonTransientConnectionException: Database is already closed (to disable automatic closing at VM shutdown, add ";DB_CLOSE_ON_EXIT=FALSE" to the db URL) [90121-232]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:690)
at org.h2.message.DbException.getJdbcSQLException(DbException.java:489)
at org.h2.message.DbException.get(DbException.java:223)
at org.h2.message.DbException.get(DbException.java:199)
at org.h2.message.DbException.get(DbException.java:188)
at org.h2.jdbc.JdbcConnection.checkClosed(JdbcConnection.java:1423)
at org.h2.jdbc.JdbcStatement.checkClosed(JdbcStatement.java:1302)
at org.h2.jdbc.JdbcPreparedStatement.clearParameters(JdbcPreparedStatement.java:292)
at com.google.gerrit.server.cache.h2.H2CacheImpl$SqlStore.put(H2CacheImpl.java:576)
at com.google.gerrit.server.cache.h2.H2CacheImpl.stop(H2CacheImpl.java:217)
at com.google.gerrit.server.cache.h2.H2CacheFactory.stop(H2CacheFactory.java:161)
at com.google.gerrit.lifecycle.LifecycleManager.stop(LifecycleManager.java:109)
at com.google.gerrit.pgm.Reindex.run(Reindex.java:152)
at com.google.gerrit.pgm.util.AbstractProgram.main(AbstractProgram.java:62)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at com.google.gerrit.launcher.GerritLauncher.invokeProgram(GerritLauncher.java:251)
at com.google.gerrit.launcher.GerritLauncher.mainImpl(GerritLauncher.java:147)
at com.google.gerrit.launcher.GerritLauncher.main(GerritLauncher.java:92)
at Main.main(Main.java:30)
[2026-02-26 22:43:30,373] [main] WARN com.google.gerrit.server.cache.h2.H2CacheImpl : Cannot put into cache jdbc:h2:file:///data/gerrit-data/cache/gerrit_file_diff-v2
org.h2.jdbc.JdbcSQLNonTransientConnectionException: Database is already closed (to disable automatic closing at VM shutdown, add ";DB_CLOSE_ON_EXIT=FALSE" to the db URL) [90121-232]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:690)
at org.h2.message.DbException.getJdbcSQLException(DbException.java:489)
at org.h2.message.DbException.get(DbException.java:223)
at org.h2.message.DbException.get(DbException.java:199)
at org.h2.message.DbException.get(DbException.java:188)
at org.h2.jdbc.JdbcConnection.checkClosed(JdbcConnection.java:1423)
at org.h2.jdbc.JdbcStatement.checkClosed(JdbcStatement.java:1302)
at org.h2.jdbc.JdbcPreparedStatement.clearParameters(JdbcPreparedStatement.java:292)
at com.google.gerrit.server.cache.h2.H2CacheImpl$SqlStore.put(H2CacheImpl.java:576)
at com.google.gerrit.server.cache.h2.H2CacheImpl.stop(H2CacheImpl.java:217)
at com.google.gerrit.server.cache.h2.H2CacheFactory.stop(H2CacheFactory.java:161)
at com.google.gerrit.lifecycle.LifecycleManager.stop(LifecycleManager.java:109)
at com.google.gerrit.pgm.Reindex.run(Reindex.java:152)
at com.google.gerrit.pgm.util.AbstractProgram.main(AbstractProgram.java:62)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at com.google.gerrit.launcher.GerritLauncher.invokeProgram(GerritLauncher.java:251)
at com.google.gerrit.launcher.GerritLauncher.mainImpl(GerritLauncher.java:147)
at com.google.gerrit.launcher.GerritLauncher.main(GerritLauncher.java:92)
at Main.main(Main.java:30)
[2026-02-26 22:43:30,374] [main] WARN com.google.gerrit.server.cache.h2.H2CacheImpl : Cannot put into cache jdbc:h2:file:///data/gerrit-data/cache/gerrit_file_diff-v2
org.h2.jdbc.JdbcSQLNonTransientConnectionException: Database is already closed (to disable automatic closing at VM shutdown, add ";DB_CLOSE_ON_EXIT=FALSE" to the db URL) [90121-232]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:690)
at org.h2.message.DbException.getJdbcSQLException(DbException.java:489)
at org.h2.message.DbException.get(DbException.java:223)
at org.h2.message.DbException.get(DbException.java:199)
at org.h2.message.DbException.get(DbException.java:188)
at org.h2.jdbc.JdbcConnection.checkClosed(JdbcConnection.java:1423)
at org.h2.jdbc.JdbcStatement.checkClosed(JdbcStatement.java:1302)
at org.h2.jdbc.JdbcPreparedStatement.clearParameters(JdbcPreparedStatement.java:292)
at com.google.gerrit.server.cache.h2.H2CacheImpl$SqlStore.put(H2CacheImpl.java:576)
at com.google.gerrit.server.cache.h2.H2CacheImpl.stop(H2CacheImpl.java:217)
at com.google.gerrit.server.cache.h2.H2CacheFactory.stop(H2CacheFactory.java:161)
at com.google.gerrit.lifecycle.LifecycleManager.stop(LifecycleManager.java:109)
at com.google.gerrit.pgm.Reindex.run(Reindex.java:152)
at com.google.gerrit.pgm.util.AbstractProgram.main(AbstractProgram.java:62)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at com.google.gerrit.launcher.GerritLauncher.invokeProgram(GerritLauncher.java:251)
at com.google.gerrit.launcher.GerritLauncher.mainImpl(GerritLauncher.java:147)
at com.google.gerrit.launcher.GerritLauncher.main(GerritLauncher.java:92)
at Main.main(Main.java:30)
If needed, I will attach the whole log file, it is more larger.
Br
Yingchun