How to Recover a Corrupted Database

[Michael Franz]

Hello,

I have been using H2 as the backend for a gerrit project for months.  We had a restart of the instance over the weekend for some unrelated OS maintenance (Linux).  When the server came back up our gerrit instance would not start.  The database is corrupt.

After investigating, it seems that our database has been corrupt for months but since we did not restart it was working (well mostly working, there is only one feature that I found that stopped working) fine.  We are taking daily backups of the database directory, which seemed like a good idea, but now all we have is backups of a corrupted database for months.

I have use the Recover feature as was suggested in the error_log.  This has allowed me to bring the database and gerrit back up, however we are missing the history of our changes since the corruption started (April 1).  Is there anyway to recover more of the history?

commands run so far:

java -cp h2.jar org.h2.tools.Recover

java -cp h2.jar org.h2.tools.RunScript -url jdbc:h2:./ReviewDB -script ReviewDB.h2.sql

I have tried to use the

java -cp h2.jar org.h2.tools.Script -url jdbc:h2:./ReviewDB  -script ReviewDB.zip -options compress zip

I had to modify the Script to use null/null for userid/password as that is apparently how gerrit connects to h2

We are currently using h2 1.3.174

Original exception:

ERROR com.google.gerrit.pgm.Daemon : Unable to start daemon

com.google.gerrit.common.Die: Unable to determine SqlDialect

  caused by org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (File corrupted while reading record: "[1705] stream data key:213 pos:11 remaining:0". Possible solution: use the recovery tool [90030-174])

  caused by org.h2.jdbc.JdbcSQLException: File corrupted while reading record: "[1705] stream data key:213 pos:11 remaining:0". Possible solution: use the recovery tool [90030-174]

        at com.google.gerrit.pgm.util.AbstractProgram.die(AbstractProgram.java:88)

        at com.google.gerrit.pgm.util.SiteProgram.createDbInjector(SiteProgram.java:181)

        at com.google.gerrit.pgm.Daemon.start(Daemon.java:276)

        at com.google.gerrit.pgm.Daemon.run(Daemon.java:205)

        at com.google.gerrit.pgm.util.AbstractProgram.main(AbstractProgram.java:64)

Any help is appreciated!

Michael

[Michael Franz]

I gave up on recovering the missing data at this time.  I still have the corrupt DB files and might eventually try to extract the data, but for now we are ok with the missing data.  All of the changes are documented in emails that were sent by Gerrit.

In the end it was 1 table that was corrupted, the CHANGE_MESSAGES table.  In another thread I mentioned that the recover was corrupt as well.  To work around that, i used Squirrel QL to copy the data out of this table - export to CSV, dropped the table and recreated and then reloaded (using a custom script).

To prevent problems going forward we are using an export backup instead of the DB file copy.  We will also migrate to Postgres or MySQL when time permits.