cache ArrayIndexOutOfBoundsException

Åsmund Østvold

unread,

Jul 16, 2024, 8:23:11 AM (6 days ago) Jul 16

to repo-discuss

Hi,

I did updates 3.8.2 -> 3.9.4 -> 3.10.0. Functionally all works as expected. I am getting below exceptions several times in error_log. Guessing it had to do with some inconsistencies in caches did a clearing of all caches. But the stactraces keep on coming. Am I missing configuration or have I done something wrong?

[2024-07-16T11:42:35.089Z] [DiskCache-Store-0] WARN com.google.gerrit.server.cache.h2.H2CacheImpl : Cannot put into cache jdbc:h2:file:///var/cache/gerrit/git_modified_files [CONTEXT project="resoptima/irma" request="REST /changes/*/revisions/*/files" ]
org.h2.jdbc.JdbcSQLException: General error: "java.lang.ArrayIndexOutOfBoundsException" [50000-176]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:344)
at org.h2.message.DbException.get(DbException.java:167)
at org.h2.message.DbException.convert(DbException.java:294)
at org.h2.engine.Database.openDatabase(Database.java:291)
at org.h2.engine.Database.<init>(Database.java:254)
at org.h2.engine.Engine.openSession(Engine.java:57)
at org.h2.engine.Engine.openSession(Engine.java:164)
at org.h2.engine.Engine.createSessionAndValidate(Engine.java:142)
at org.h2.engine.Engine.createSession(Engine.java:125)
at org.h2.engine.Engine.createSession(Engine.java:27)
at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:331)
at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:107)
at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:91)
at org.h2.Driver.connect(Driver.java:74)
at com.google.gerrit.server.cache.h2.H2CacheImpl$SqlHandle.<init>(H2CacheImpl.java:753)
at com.google.gerrit.server.cache.h2.H2CacheImpl$SqlStore.acquire(H2CacheImpl.java:720)
at com.google.gerrit.server.cache.h2.H2CacheImpl$SqlStore.put(H2CacheImpl.java:572)
at com.google.gerrit.server.cache.h2.H2CacheImpl$Loader.lambda$load$0(H2CacheImpl.java:277)
at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:113)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1570)
Caused by: java.lang.ArrayIndexOutOfBoundsException

I am also getting:

[2024-07-16T07:46:24.196Z] [Index-Interactive-1[index-change-22210]] WARN com.google.gerrit.server.cache.h2.H2CacheImpl : Cannot build BloomFilter for jdbc:h2:file:///var/cache/gerrit/git_modified_files: General error: "java.lang.ArrayIndexOutOfBoundsException" [50000-176] [CONTEXT SUBMISSION_ID="22210-1721115981666-74d7dd9a" project="resoptima/irma" request="REST /changes/*/revisions/*/submit" ]
[2024-07-16T07:46:24.741Z] [Index-Interactive-1[index-change-22210]] WARN com.google.gerrit.server.cache.h2.H2CacheImpl : Cannot read cache jdbc:h2:file:///var/cache/gerrit/git_modified_files for GitModifiedFilesCacheKey{project=resoptima/irma, aTree=tree a449752e0d0ebb91ca48033d8c6006a09fa1ee53 -------, bTree=tree b2929339a3d49d1dc40e58460c1ca273a112c658 -------, renameScore=60} [CONTEXT SUBMISSION_ID="22210-1721115981666-74d7dd9a" project="resoptima/irma" request="REST /changes/*/revisions/*/submit" ]
org.h2.jdbc.JdbcSQLException: General error: "java.lang.ArrayIndexOutOfBoundsException" [50000-176]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:344)
at org.h2.message.DbException.get(DbException.java:167)
at org.h2.message.DbException.convert(DbException.java:294)
at org.h2.engine.Database.openDatabase(Database.java:291)
at org.h2.engine.Database.<init>(Database.java:254)
at org.h2.engine.Engine.openSession(Engine.java:57)
at org.h2.engine.Engine.openSession(Engine.java:164)
at org.h2.engine.Engine.createSessionAndValidate(Engine.java:142)
at org.h2.engine.Engine.createSession(Engine.java:125)
at org.h2.engine.Engine.createSession(Engine.java:27)
at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:331)
at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:107)
at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:91)
at org.h2.Driver.connect(Driver.java:74)
at com.google.gerrit.server.cache.h2.H2CacheImpl$SqlHandle.<init>(H2CacheImpl.java:753)
at com.google.gerrit.server.cache.h2.H2CacheImpl$SqlStore.acquire(H2CacheImpl.java:720)
at com.google.gerrit.server.cache.h2.H2CacheImpl$SqlStore.getIfPresent(H2CacheImpl.java:477)
at com.google.gerrit.server.cache.h2.H2CacheImpl$Loader.load(H2CacheImpl.java:269)
at com.google.gerrit.server.cache.h2.H2CacheImpl$Loader.load(H2CacheImpl.java:252)
at com.github.benmanes.caffeine.guava.CaffeinatedGuavaLoadingCache$SingleLoader.load(CaffeinatedGuavaLoadingCache.java:136)
at com.github.benmanes.caffeine.cache.LocalLoadingCache.lambda$newMappingFunction$2(LocalLoadingCache.java:141)
at com.github.benmanes.caffeine.cache.LocalCache.lambda$statsAware$0(LocalCache.java:139)
at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2413)
at java.base/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1957)
at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2411)
at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2394)
at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:108)
at com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:54)
at com.github.benmanes.caffeine.guava.CaffeinatedGuavaLoadingCache.get(CaffeinatedGuavaLoadingCache.java:59)
at com.google.gerrit.server.cache.h2.H2CacheImpl.get(H2CacheImpl.java:134)
at com.google.gerrit.server.patch.gitdiff.GitModifiedFilesCacheImpl.get(GitModifiedFilesCacheImpl.java:79)
at com.google.gerrit.server.patch.diff.ModifiedFilesLoader.getModifiedFiles(ModifiedFilesLoader.java:245)
at com.google.gerrit.server.patch.diff.ModifiedFilesLoader.load(ModifiedFilesLoader.java:153)
at com.google.gerrit.server.patch.diff.ModifiedFilesCacheImpl$Loader.load(ModifiedFilesCacheImpl.java:126)
at com.google.gerrit.server.patch.diff.ModifiedFilesCacheImpl$Loader.load(ModifiedFilesCacheImpl.java:104)
at com.google.gerrit.server.cache.h2.H2CacheImpl$Loader.load(H2CacheImpl.java:276)
at com.google.gerrit.server.cache.h2.H2CacheImpl$Loader.load(H2CacheImpl.java:252)
at com.github.benmanes.caffeine.guava.CaffeinatedGuavaLoadingCache$SingleLoader.load(CaffeinatedGuavaLoadingCache.java:136)
at com.github.benmanes.caffeine.cache.LocalLoadingCache.lambda$newMappingFunction$2(LocalLoadingCache.java:141)
at com.github.benmanes.caffeine.cache.LocalCache.lambda$statsAware$0(LocalCache.java:139)
at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2413)
at java.base/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1918)
at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2411)
at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2394)
at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:108)
at com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:54)
at com.github.benmanes.caffeine.guava.CaffeinatedGuavaLoadingCache.get(CaffeinatedGuavaLoadingCache.java:59)
at com.google.gerrit.server.cache.h2.H2CacheImpl.get(H2CacheImpl.java:134)
at com.google.gerrit.server.patch.diff.ModifiedFilesCacheImpl.get(ModifiedFilesCacheImpl.java:90)
at com.google.gerrit.server.patch.DiffOperationsImpl.getModifiedFiles(DiffOperationsImpl.java:255)
at com.google.gerrit.server.patch.DiffOperationsImpl.listModifiedFilesAgainstParent(DiffOperationsImpl.java:129)
at com.google.gerrit.server.patch.DiffSummaryLoader.call(DiffSummaryLoader.java:51)
at com.google.gerrit.server.patch.DiffSummaryLoader.call(DiffSummaryLoader.java:28)
at com.google.gerrit.server.cache.h2.H2CacheImpl.lambda$get$1(H2CacheImpl.java:173)
at com.github.benmanes.caffeine.guava.CaffeinatedGuavaCache.lambda$get$0(CaffeinatedGuavaCache.java:69)
at com.github.benmanes.caffeine.cache.LocalCache.lambda$statsAware$0(LocalCache.java:139)
at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2413)
at java.base/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1957)
at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2411)
at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2394)
at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:108)
at com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:62)
at com.github.benmanes.caffeine.guava.CaffeinatedGuavaCache.get(CaffeinatedGuavaCache.java:67)
at com.google.gerrit.server.cache.h2.H2CacheImpl.get(H2CacheImpl.java:162)
at com.google.gerrit.server.patch.PatchListCacheImpl.getDiffSummary(PatchListCacheImpl.java:101)
at com.google.gerrit.server.query.change.ChangeData.getDiffSummary(ChangeData.java:600)
at com.google.gerrit.server.query.change.ChangeData.computeChangedLines(ChangeData.java:609)
at com.google.gerrit.server.query.change.ChangeData.changedLines(ChangeData.java:621)
at com.google.gerrit.server.index.change.ChangeField.lambda$static$56(ChangeField.java:1204)
at com.google.gerrit.index.IndexedField.get(IndexedField.java:437)
at com.google.gerrit.index.IndexedField$SearchSpec.get(IndexedField.java:151)
at com.google.gerrit.index.Schema.fieldValues(Schema.java:273)
at com.google.gerrit.index.Schema.lambda$buildFields$0(Schema.java:306)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:212)
at com.google.common.collect.CollectSpliterators$1WithCharacteristics.lambda$forEachRemaining$1(CollectSpliterators.java:72)
at java.base/java.util.stream.Streams$RangeIntSpliterator.forEachRemaining(Streams.java:104)
at com.google.common.collect.CollectSpliterators$1WithCharacteristics.forEachRemaining(CollectSpliterators.java:72)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:556)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:546)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:265)
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:702)
at com.google.gerrit.index.Schema.buildFields(Schema.java:308)
at com.google.gerrit.lucene.AbstractLuceneIndex.toDocument(AbstractLuceneIndex.java:343)
at com.google.gerrit.lucene.LuceneChangeIndex.replace(LuceneChangeIndex.java:218)
at com.google.gerrit.lucene.LuceneChangeIndex.replace(LuceneChangeIndex.java:104)
at com.google.gerrit.server.index.change.ChangeIndexer.indexImpl(ChangeIndexer.java:339)
at com.google.gerrit.server.index.change.ChangeIndexer.doIndex(ChangeIndexer.java:297)
at com.google.gerrit.server.index.change.ChangeIndexer$IndexTask.callImpl(ChangeIndexer.java:602)
at com.google.gerrit.server.index.change.ChangeIndexer$IndexTask.callImpl(ChangeIndexer.java:563)
at com.google.gerrit.server.index.change.ChangeIndexer$AbstractIndexTask.call(ChangeIndexer.java:551)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:76)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:113)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:703)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1570)
Caused by: java.lang.ArrayIndexOutOfBoundsException

Files I assume are of interest:

ll -la /var/cache/gerrit/git_modified_files.*
-rw-r--r-- 1 git git 378214400 Jul 16 12:10 /var/cache/gerrit/git_modified_files.h2.db
-rw-r--r-- 1 git git 22333453 Jul 10 07:27 /var/cache/gerrit/git_modified_files.trace.db

Any hint on what I could do?

Regards,

Åsmund

Nasser Grainawi

unread,

Jul 16, 2024, 2:45:57 PM (6 days ago) Jul 16

to Åsmund Østvold, repo-discuss

On Tue, Jul 16, 2024 at 6:23 AM Åsmund Østvold <asm...@gmail.com> wrote:

Hi,

I did updates 3.8.2 -> 3.9.4 -> 3.10.0. Functionally all works as expected. I am getting below exceptions several times in error_log. Guessing it had to do with some inconsistencies in caches did a clearing of all caches. But the stactraces keep on coming. Am I missing configuration or have I done something wrong?

Looks like this is a known issue that can happen with our version of h2: https://groups.google.com/g/h2-database/c/7IcaiL8D_Ac/m/dffLrmshoHsJ

Unfortunately, H2 changed their license when it moved from 1.3.* to 1.4.* and it is no longer compatible with Gerrit's, so we can't upgrade H2 to a version with the fix. :(

The only "fix" I can think of right now would be to stop Gerrit, remove those h2 files from the disk, and then start Gerrit again.

Longer term, I started looking into alternate solutions that we could use instead of H2, but I haven't invested significant time into it yet. There is a Gerrit module that allows you to use ChronicleMaps instead of H2, but it has some limitations that you may not like.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/CA%2BzfM3AeayxvO8h5jnvrZsML4_hiMymHtq0qZ8fhNR1UW4%3DSgw%40mail.gmail.com.

Åsmund Østvold

unread,

Jul 19, 2024, 6:06:08 AM (3 days ago) Jul 19

to Nasser Grainawi, repo-discuss

On Tue, Jul 16, 2024 at 8:45 PM Nasser Grainawi <nasser....@linaro.org> wrote:

On Tue, Jul 16, 2024 at 6:23 AM Åsmund Østvold <asm...@gmail.com> wrote:
Hi,

I did updates 3.8.2 -> 3.9.4 -> 3.10.0. Functionally all works as expected. I am getting below exceptions several times in error_log. Guessing it had to do with some inconsistencies in caches did a clearing of all caches. But the stactraces keep on coming. Am I missing configuration or have I done something wrong?

Looks like this is a known issue that can happen with our version of h2: https://groups.google.com/g/h2-database/c/7IcaiL8D_Ac/m/dffLrmshoHsJ

Unfortunately, H2 changed their license when it moved from 1.3.* to 1.4.* and it is no longer compatible with Gerrit's, so we can't upgrade H2 to a version with the fix. :(
The only "fix" I can think of right now would be to stop Gerrit, remove those h2 files from the disk, and then start Gerrit again.

Thank you Nasser!
This solved it. Our gerrit instances do not have high load so it was no problem for us.

Åsmund

Nasser Grainawi

unread,

Jul 19, 2024, 11:18:44 AM (3 days ago) Jul 19

to Åsmund Østvold, repo-discuss

Glad it worked, thanks for confirming!

Åsmund

Reply all

Reply to author

Forward