change_notes H2 cache file grows unbounded (1.2 GiB of live data in a 16 GiB file) after 3.12 → 3.13 upgrade — MVStore corruption in background housekeeping

[Yingchun Li]

Hi all,

After upgrading from Gerrit 3.12 to 3.13.8 we are seeing the change_notes persistent cache file grow without bound while the amount of live data stays flat.

## Environment
- Gerrit 3.13.8 (upgraded from 3.12)
- H2 2.4.240 (bundled)
- NoteDb, single primary, no replicas
- Java 21
- ~1200 projects
- Relevant config:

[cache "change_notes"]
        memoryLimit = 1g
        diskLimit = 2g

## Symptom
The file grows by roughly 4 GiB per hour, while SUM(space) reported by show-caches does not move at all:

13:50    12G    change_notes-v2.mv.db     Space = 1.20g
14:45    16G    change_notes-v2.mv.db     Space = 1.21g

ll data/cache -alh
total 16G
drwxrwxr-x 2 gerrit gerrit  4.0K Aug  3 12:39 ./
drwxrwxr-x 8 gerrit gerrit  4.0K Aug  1 21:04 ../
-rw-rw-r-- 1 gerrit gerrit 1016K Aug  3 15:17 accounts-v2.mv.db
-rw-rw-r-- 1 gerrit gerrit  8.2M Aug  3 15:18 change_kind-v2.mv.db
-rw-rw-r-- 1 gerrit gerrit   16G Aug  3 15:18 change_notes-v2.mv.db
-rw-rw-r-- 1 gerrit gerrit   46M Aug  3 15:01 change_notes-v2.trace.db
-rw-rw-r-- 1 gerrit gerrit   65M Aug  3 12:39 change_notes-v2.trace.db.old

...

show-caches output for the cache:

  Name                          |Entries              |  AvgGet |Hit Ratio|
                                |   Mem   Disk   Space|         |Mem  Disk|
--------------------------------+---------------------+---------+---------+
D change_notes                  | 26030 278399   1.21g| 425.8us | 19%  99%|

So Gerrit's own accounting says 1.21 GiB of live data, well below diskLimit = 2g , and the on-disk file is 16 GiB. Disk space itself is not a problem for us, but the growth is monotonic and shows no sign of levelling off.

A change_notes-v2.trace.db also appeared and grew to 29 MiB. No other cache in data/cache/ has a trace.db , so this seems specific to change_notes .

## What the trace file shows
The same exception repeats throughout:

Caused by: org.h2.message.DbException: General error:
  "org.h2.mvstore.MVStoreException: Reading from file
   sun.nio.ch.FileChannelImpl@633640a failed at 4221461579 (length -1),
   read 0, remaining 20712 [2.4.240/1]" [50000-240]
at org.h2.message.DbException.get(DbException.java:212)
at org.h2.message.DbException.convert(DbException.java:407)
at org.h2.mvstore.db.Store.lambda$new$0(Store.java:122)
at org.h2.mvstore.MVStore.handleException(MVStore.java:1675)
at org.h2.mvstore.MVStore.panic(MVStore.java:502)
at org.h2.mvstore.MVStore.tryExecuteUnderStoreLock(MVStore.java:1061)
at org.h2.mvstore.RandomAccessStore.doHousekeeping(RandomAccessStore.java:729)
at org.h2.mvstore.FileStore.writeInBackground(FileStore.java:1842)
at org.h2.mvstore.FileStore$BackgroundWriterThread.run(FileStore.java:2257)

Two things stand out:

1. read 0, remaining 20712 with length -1 — MVStore follows chunk metadata to offset 4221461579 (~3.93 GiB) and hits EOF. The chunk index and the actual file contents have diverged.
2. The failing thread is BackgroundWriterThread → writeInBackground → doHousekeeping , i.e. MVStore's free-space reclamation, and it ends in MVStore.panic() .
That second point appears to be the direct cause of the unbounded growth: the thread responsible for reusing freed chunks panics on every attempt, so freed space is never reused and every write is appended at the end of the file. This matches the observation that SUM(space) stays at 1.21 GiB while the file keeps growing.

Thanks

Br,

Yingchun