H2CacheImpl warnings: "Cannot read/put/invalidate/prune cache" (Gerrit 2.11.3)

[Leandro Fonseca]

Hi Guys,

My error_log file is being flooded with below warning messages:

[2018-05-17 13:57:43,782] WARN  com.google.gerrit.server.cache.h2.H2CacheImpl : Cannot read cache jdbc:h2:file:/<path>/cache/gerrit/git_tags for <gerrit project>

org.h2.jdbc.JdbcSQLException: General error: "java.lang.RuntimeException: old!=record pos:52019 old:page[52019] data leaf overflow parent:52018 next:52020 new:page[52019] data leaf table:14 T14 entries:0 parent:7 keys:null offsets:null" [50000-174]

at org.h2.message.DbException.getJdbcSQLException(DbException.java:332)

at org.h2.message.DbException.get(DbException.java:161)

at org.h2.message.DbException.convert(DbException.java:284)

at org.h2.table.RegularTable.addRow(RegularTable.java:137)

at org.h2.store.PageStore.redo(PageStore.java:1535)

at org.h2.store.PageLog.recover(PageLog.java:318)

at org.h2.store.PageStore.recover(PageStore.java:1371)

at org.h2.store.PageStore.openExisting(PageStore.java:361)

at org.h2.store.PageStore.open(PageStore.java:285)

at org.h2.engine.Database.getPageStore(Database.java:2277)

at org.h2.engine.Database.open(Database.java:610)

at org.h2.engine.Database.openDatabase(Database.java:236)

at org.h2.engine.Database.<init>(Database.java:231)

at org.h2.engine.Engine.openSession(Engine.java:56)

at org.h2.engine.Engine.openSession(Engine.java:160)

at org.h2.engine.Engine.createSessionAndValidate(Engine.java:139)

at org.h2.engine.Engine.createSession(Engine.java:122)

at org.h2.engine.Engine.createSession(Engine.java:28)

at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:323)

at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:105)

at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:90)

at org.h2.Driver.connect(Driver.java:73)

at com.google.gerrit.server.cache.h2.H2CacheImpl$SqlHandle.<init>(H2CacheImpl.java:636)

at com.google.gerrit.server.cache.h2.H2CacheImpl$SqlStore.acquire(H2CacheImpl.java:604)

at com.google.gerrit.server.cache.h2.H2CacheImpl$SqlStore.getIfPresent(H2CacheImpl.java:402)

at com.google.gerrit.server.cache.h2.H2CacheImpl.getIfPresent(H2CacheImpl.java:101)

at com.google.gerrit.server.git.TagCache.get(TagCache.java:92)

at com.google.gerrit.server.git.VisibleRefFilter.filter(VisibleRefFilter.java:140)

at com.google.gerrit.server.git.VisibleRefFilter.filter(VisibleRefFilter.java:169)

at com.google.gerrit.server.git.VisibleRefFilter.getAdvertisedRefs(VisibleRefFilter.java:158)

at org.eclipse.jgit.transport.AbstractAdvertiseRefsHook.advertiseRefs(AbstractAdvertiseRefsHook.java:68)

at org.eclipse.jgit.transport.AdvertiseRefsHookChain.advertiseRefs(AdvertiseRefsHookChain.java:85)

at org.eclipse.jgit.transport.BaseReceivePack.sendAdvertisedRefs(BaseReceivePack.java:1002)

at org.eclipse.jgit.transport.ReceivePack.service(ReceivePack.java:179)

at org.eclipse.jgit.transport.ReceivePack.receive(ReceivePack.java:161)

at com.google.gerrit.sshd.commands.Receive.runImpl(Receive.java:115)

at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:101)

at com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:32)

at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:70)

at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:445)

at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)

at java.util.concurrent.FutureTask.run(Unknown Source)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)

at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:379)

at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

Caused by: java.lang.RuntimeException: old!=record pos:52019 old:page[52019] data leaf overflow parent:52018 next:52020 new:page[52019] data leaf table:14 T14 entries:0 parent:7 keys:null offsets:null

at org.h2.message.DbException.throwInternalError(DbException.java:231)

at org.h2.util.CacheLRU.update(CacheLRU.java:124)

at org.h2.store.PageStore.update(PageStore.java:1065)

at org.h2.index.PageDataNode.addRowTry(PageDataNode.java:140)

at org.h2.index.PageDataIndex.addTry(PageDataIndex.java:173)

at org.h2.index.PageDataIndex.add(PageDataIndex.java:136)

at org.h2.table.RegularTable.addRow(RegularTable.java:119)

... 44 more

[2018-05-17 13:57:43,817] WARN  com.google.gerrit.server.cache.h2.H2CacheImpl : Cannot put into cache jdbc:h2:file:/<path>/cache/gerrit/git_tags

org.h2.jdbc.JdbcSQLException: General error: "java.lang.RuntimeException: old!=record pos:52019 old:page[52019] data leaf overflow parent:52018 next:52020 new:page[52019] data leaf table:14 T14 entries:0 parent:7 keys:null offsets:null" [50000-174]

at org.h2.message.DbException.getJdbcSQLException(DbException.java:332)

at org.h2.message.DbException.get(DbException.java:161)

at org.h2.message.DbException.convert(DbException.java:284)

at org.h2.table.RegularTable.addRow(RegularTable.java:137)

at org.h2.store.PageStore.redo(PageStore.java:1535)

at org.h2.store.PageLog.recover(PageLog.java:318)

at org.h2.store.PageStore.recover(PageStore.java:1371)

at org.h2.store.PageStore.openExisting(PageStore.java:361)

at org.h2.store.PageStore.open(PageStore.java:285)

at org.h2.engine.Database.getPageStore(Database.java:2277)

at org.h2.engine.Database.open(Database.java:610)

at org.h2.engine.Database.openDatabase(Database.java:236)

at org.h2.engine.Database.<init>(Database.java:231)

at org.h2.engine.Engine.openSession(Engine.java:56)

at org.h2.engine.Engine.openSession(Engine.java:160)

at org.h2.engine.Engine.createSessionAndValidate(Engine.java:139)

at org.h2.engine.Engine.createSession(Engine.java:122)

at org.h2.engine.Engine.createSession(Engine.java:28)

at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:323)

at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:105)

at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:90)

at org.h2.Driver.connect(Driver.java:73)

at com.google.gerrit.server.cache.h2.H2CacheImpl$SqlHandle.<init>(H2CacheImpl.java:636)

at com.google.gerrit.server.cache.h2.H2CacheImpl$SqlStore.acquire(H2CacheImpl.java:604)

at com.google.gerrit.server.cache.h2.H2CacheImpl$SqlStore.put(H2CacheImpl.java:473)

at com.google.gerrit.server.cache.h2.H2CacheImpl$1.run(H2CacheImpl.java:126)

at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

Caused by: java.lang.RuntimeException: old!=record pos:52019 old:page[52019] data leaf overflow parent:52018 next:52020 new:page[52019] data leaf table:14 T14 entries:0 parent:7 keys:null offsets:null

at org.h2.message.DbException.throwInternalError(DbException.java:231)

at org.h2.util.CacheLRU.update(CacheLRU.java:124)

at org.h2.store.PageStore.update(PageStore.java:1065)

at org.h2.index.PageDataNode.addRowTry(PageDataNode.java:140)

at org.h2.index.PageDataIndex.addTry(PageDataIndex.java:173)

at org.h2.index.PageDataIndex.add(PageDataIndex.java:136)

at org.h2.table.RegularTable.addRow(RegularTable.java:119)

... 25 more

I have already confirmed that the Gerrit application running user (gerrit) has read and write permissions to the configured cache folder, everything looks correct:

$ pwd

/<path>/cache/gerrit

$ ls -ld git_tags.h2.db 

-rw-r--r-- 1 gerrit <group> 231888896 May 17 14:01 git_tags.h2.db

$ du -sh git_tags.h2.db 

222M git_tags.h2.db

$ ls -ld .

drwxr-xr-x 2 gerrit <group> 4096 May 17 14:01 .

$

I have found below old topic, with a similar issue, where Shawn suggests to remove the git_tags.h2.db file, however, it is not the exact same problem (this one specifically says that the file is corrupted, mine doesn't).

https://groups.google.com/forum/#!topic/repo-discuss/ll35PqjsC94

Any ideas?

Thanks a lot!

Leandro.

[Leandro Fonseca]

Guys,

Can someone please confirm if that it is ok to re-create Gerrit's cache using below procedure?

  shutdown gerrit 
  rm -rf cache/* 
  start gerrit 

Thanks,

Leandro.

[Matthew Webber]

On Sunday, 8 July 2018 23:08:40 UTC+1, Leandro Fonseca wrote:

Guys,

Can someone please confirm if that it is ok to re-create Gerrit's cache using below procedure?

  shutdown gerrit 
  rm -rf cache/* 
  start gerrit 

Thanks,

Leandro.

I do that whenever I take a copy of my production system for testing, and it seems to work fine.

I think your should be ok, but I would prefer someone else to confirm this.

Matthew

[RJ]

I do it all the time before bring up after scheduled maintenance as well clean tmp directory too. No issues observed till now