Protection against H2 file corruption

[h2new]

Hello all,

we are using H2 in our java application.

For a while I am deploying it on kubernetes and testing app resilience with chaoskube which randomly kills some pods from time to time.

It turned out that one component of the app leaves h2 file corrupted:

[ERROR] 2020-03-10T16:08:15,009Z [main] pool.HikariPool.throwPoolInitializationException - HikariPool-1 - Exception during pool initialization.
  org.h2.jdbc.JdbcSQLException: File corrupted while reading record: null. Possible solution: use the recovery tool [90030-197]
   at org.h2.message.DbException.getJdbcSQLException(DbException.java:357) ~[myjar.jar:?]
   at org.h2.message.DbException.get(DbException.java:168) ~[myjar.jar:?]

Is there any option or parameter which should be used to prevent the corruption ? Any hint how to cope with it ?

[h2new]

Any hint please ?

[Evgenij Ryazanov]

Hello.

You need to post a complete stack trace, first few lines don't contain enough information about your problem.

You can try to use a more recent version of H2 or even try to build H2 from its current sources, current H2 is more reliable than 1.4.197. You can also try to use the legacy PageStore backend instead of default one.

If your application interrupts threads that work with the database, you need to use the async: file system supported by recent versions of H2 (jdbc:h2:async:/path/to/database). This file system is not supported by 1.4.197.

If the whole processes are killed, it's better not to use the embedded database. H2 can be used as a normal database server.

[h2new]

@Evgenij

[ERROR] 2020-03-10T16:08:15,009Z [main] pool.HikariPool.throwPoolInitializationException - HikariPool-1 - Exception during pool initialization.
org.h2.jdbc.JdbcSQLException: File corrupted while reading record: null. Possible solution: use the recovery tool [90030-197]

    at org.h2.message.DbException.getJdbcSQLException(DbException.java:357) ~[.jar:?]
    at org.h2.message.DbException.get(DbException.java:168) ~[app.jar:?]
    at org.h2.mvstore.db.MVTableEngine$Store.convertIllegalStateException(MVTableEngine.java:196) ~[app.jar:?]
    at org.h2.mvstore.db.MVTableEngine$Store.open(MVTableEngine.java:168) ~[app.jar:?]
    at org.h2.mvstore.db.MVTableEngine.init(MVTableEngine.java:100) ~[app.jar:?]
    at org.h2.engine.Database.getPageStore(Database.java:2538) ~[app.jar:?]
    at org.h2.engine.Database.open(Database.java:709) ~[app.jar:?]
    at org.h2.engine.Database.openDatabase(Database.java:286) ~[app.jar:?]
    at org.h2.engine.Database.<init>(Database.java:280) ~[app.jar:?]
    at org.h2.engine.Engine.openSession(Engine.java:66) ~[app.jar:?]
    at org.h2.engine.Engine.openSession(Engine.java:179) ~[app.jar:?]
    at org.h2.engine.Engine.createSessionAndValidate(Engine.java:157) ~[app.jar:?]
    at org.h2.engine.Engine.createSession(Engine.java:140) ~[app.jar:?]
    at org.h2.engine.Engine.createSession(Engine.java:28) ~[app.jar:?]
    at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:351) ~[app.jar:?]
    at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:124) ~[app.jar:?]
    at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:103) ~[app.jar:?]
    at org.h2.Driver.connect(Driver.java:69) ~[app.jar:?]
    at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:121) ~[app.jar:?]
    at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:353) ~[app.jar:?]
    at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:201) ~[app.jar:?]
    at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:473) [app.jar:?]
    at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:562) [app.jar:?]
    at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:115) [app.jar:?]
    at com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:81) [app.jar:?]
    at com.r3.enm.nspersistence.PersistenceUtilsKt$configureDatabase$1.invoke(PersistenceUtils.kt:234) [app.jar:?]
    at com.r3.enm.nspersistence.PersistenceUtilsKt$configureDatabase$1.invoke(PersistenceUtils.kt) [app.jar:?]
    at com.r3.enm.nspersistence.PersistenceUtilsKt.withJDBCDriverLoaded(PersistenceUtils.kt:197) [app.jar:?]
    at com.r3.enm.nspersistence.PersistenceUtilsKt.configureDatabase(PersistenceUtils.kt:217) [app.jar:?]
    at com.r3.enm.services.myservice.NetworkManagementServer.configureDatabasePersistence(NetworkManagementServer.kt:62) [app.jar:?]
    at com.r3.enm.services.myservice.NetworkManagementServer.access$configureDatabasePersistence(NetworkManagementServer.kt:26) [app.jar:?]
    at com.r3.enm.services.myservice.NetworkManagementServer$database$2.invoke(NetworkManagementServer.kt:37) [app.jar:?]
    at com.r3.enm.services.myservice.NetworkManagementServer$database$2.invoke(NetworkManagementServer.kt:26) [app.jar:?]
    at kotlin.SynchronizedLazyImpl.getValue(LazyJVM.kt:74) [app.jar:?]
    at com.r3.enm.services.myservice.NetworkManagementServer.getDatabase(NetworkManagementServer.kt) [app.jar:?]
    at com.r3.enm.services.myservice.IMService.getmyserviceService(IMService.kt:178) [app.jar:?]
    at com.r3.enm.services.myservice.IMService.getServices(IMService.kt:150) [app.jar:?]
    at com.r3.enm.services.myservice.NetworkManagementServer.start(NetworkManagementServer.kt:45) [app.jar:?]
    at com.r3.enm.services.myservice.RunmyserviceCommand.startNetworkServicesContainer(RunmyserviceCommand.kt:79) [app.jar:?]
    at com.r3.enm.services.myservice.RunmyserviceCommand.access$startNetworkServicesContainer(RunmyserviceCommand.kt:25) [app.jar:?]
    at com.r3.enm.services.myservice.RunmyserviceCommand$executeCommand$2.invoke(RunmyserviceCommand.kt:70) [app.jar:?]
    at com.r3.enm.services.myservice.RunmyserviceCommand$executeCommand$2.invoke(RunmyserviceCommand.kt:25) [app.jar:?]
    at com.r3.myapp.networkmanage.common.utils.ServiceCommand.suppressSqlException(CLIUtils.kt:185) [app.jar:?]
    at com.r3.enm.services.myservice.RunmyserviceCommand.executeCommand(RunmyserviceCommand.kt:70) [app.jar:?]
    at com.r3.myapp.networkmanage.common.utils.BaseCommand.run(CLIUtils.kt:117) [app.jar:?]
    at com.r3.myapp.networkmanage.common.utils.BaseServiceCommand.run(CLIUtils.kt:145) [app.jar:?]
    at picocli.CommandLine.execute(CommandLine.java:1043) [app.jar:?]
    at picocli.CommandLine.access$900(CommandLine.java:142) [app.jar:?]
    at picocli.CommandLine$RunLast.handle(CommandLine.java:1246) [app.jar:?]
    at picocli.CommandLine$RunLast.handle(CommandLine.java:1214) [app.jar:?]
    at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:1122) [app.jar:?]
    at picocli.CommandLine.parseWithHandlers(CommandLine.java:1405) [app.jar:?]
    at com.r3.myapp.networkmanage.common.utils.CLIUtilsKt.commandLineRunExitOnError(CLIUtils.kt:224) [app.jar:?]
    at com.r3.enm.services.myservice.MainKt.main(Main.kt:17) [app.jar:?]
Caused by: java.lang.IllegalStateException: File corrupted in chunk 920, expected page length 4..384, got 1852521011 [1.4.197/6]
    at org.h2.mvstore.DataUtils.newIllegalStateException(DataUtils.java:870) ~[app.jar:?]
    at org.h2.mvstore.Page.read(Page.java:652) ~[app.jar:?]
    at org.h2.mvstore.Page.read(Page.java:186) ~[app.jar:?]
    at org.h2.mvstore.MVStore.readPage(MVStore.java:1936) ~[app.jar:?]
    at org.h2.mvstore.MVMap.readPage(MVMap.java:698) ~[app.jar:?]
    at org.h2.mvstore.Page.getChildPage(Page.java:208) ~[app.jar:?]
    at org.h2.mvstore.MVMap.binarySearch(MVMap.java:454) ~[app.jar:?]
    at org.h2.mvstore.MVMap.binarySearch(MVMap.java:455) ~[app.jar:?]
    at org.h2.mvstore.MVMap.get(MVMap.java:436) ~[app.jar:?]
    at org.h2.mvstore.MVStore.getChunkIfFound(MVStore.java:924) ~[app.jar:?]
    at org.h2.mvstore.MVStore.getChunk(MVStore.java:901) ~[app.jar:?]
    at org.h2.mvstore.MVStore.readPage(MVStore.java:1927) ~[app.jar:?]
    at org.h2.mvstore.MVMap.readPage(MVMap.java:698) ~[app.jar:?]
    at org.h2.mvstore.Page.getChildPage(Page.java:208) ~[app.jar:?]
    at org.h2.mvstore.Cursor.fetchNext(Cursor.java:150) ~[app.jar:?]
    at org.h2.mvstore.Cursor.next(Cursor.java:50) ~[app.jar:?]
    at org.h2.mvstore.MVStore.loadChunkMeta(MVStore.java:684) ~[app.jar:?]
    at org.h2.mvstore.MVStore.readStoreHeader(MVStore.java:664) ~[app.jar:?]
    at org.h2.mvstore.MVStore.<init>(MVStore.java:361) ~[app.jar:?]
    at org.h2.mvstore.MVStore$Builder.open(MVStore.java:2934) ~[app.jar:?]
    at org.h2.mvstore.db.MVTableEngine$Store.open(MVTableEngine.java:155) ~[app.jar:?]
    ... 50 more