Database corruption after file system full

144 views
Skip to first unread message

Fulvio

unread,
Dec 2, 2011, 3:48:31 AM12/2/11
to H2 Database, marco.b...@primeur.com
We are using h2 1.2.147.
At a certain point in time, a VM machine running h2 as exausted the
disk space. When the problem was discovered, we sopped our
applications, enlaged the disk space, and than we restarted the
database.
We suddently got some errors, so we sopped our applications, we run
the Recover tool, and the the RunScript tool.
Running the script we got:


RunScript -script ./a3data.h2.sql -url jdbc:h2:./a3data -password
primeur -user primeur
Exception in thread "main" org.h2.jdbc.JdbcSQLException: Unique index
or primary key violation: "A3_ACCOUNT_IX1 ON
PUBLIC.A3_ACCOUNT(USERNAME, ACCOUNT_STATE)"; SQL statement:


Having a look at the sql script, there were some errors: too many
inserts for a table with a unique constraint!

Is H2 'safe' (ACID) agaist file system full problems ?


Thanks in advance,

Fulvio

Fulvio

unread,
Dec 2, 2011, 4:01:22 AM12/2/11
to H2 Database
The error we got in the trace.db file during the file systenm full
was:


11-29 17:02:46 database: log 0
12-01 17:04:48 database: flush
org.h2.message.DbException: IO Exception: "java.io.IOException: No
space left on device"; "/opt/spazio/spazio/spdata/rt/a3sp/
a3data.h2.db" [90031-147]
at org.h2.message.DbException.get(DbException.java:156)
at org.h2.message.DbException.convertIOException(DbException.java:
313)
at org.h2.store.FileStore.write(FileStore.java:342)
at org.h2.store.PageStore.writePage(PageStore.java:1309)
at org.h2.store.PageStreamData.write(PageStreamData.java:105)
at org.h2.store.PageOutputStream.storePage(PageOutputStream.java:148)
at org.h2.store.PageOutputStream.flush(PageOutputStream.java:156)
at org.h2.store.PageLog.flushOut(PageLog.java:842)
at org.h2.store.PageLog.flush(PageLog.java:659)
at org.h2.store.PageStore.flushLog(PageStore.java:956)
at org.h2.engine.Database.flush(Database.java:1699)
at org.h2.store.WriterThread.run(WriterThread.java:85)
at java.lang.Thread.run(Unknown Source)
Caused by: org.h2.jdbc.JdbcSQLException: IO Exception:
"java.io.IOException: No space left on device"; "/opt/spazio/spazio/
spdata/rt/a3sp/a3data.h2.db" [90031-147]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:
327)
... 13 more
Caused by: java.io.IOException: No space left on device
at java.io.RandomAccessFile.writeBytes(Native Method)
at java.io.RandomAccessFile.write(Unknown Source)
at org.h2.store.FileStore.write(FileStore.java:340)
... 10 more
12-01 17:04:49 database: flush
org.h2.message.DbException: IO Exception: "java.io.IOException: No
space left on device"; "/opt/spazio/spazio/spdata/rt/a3sp/
a3data.h2.db" [90031-147]
at org.h2.message.DbException.get(DbException.java:156)
at org.h2.message.DbException.convertIOException(DbException.java:
313)
at org.h2.store.FileStore.write(FileStore.java:342)
at org.h2.store.PageStore.writePage(PageStore.java:1309)
at org.h2.store.PageStreamData.write(PageStreamData.java:105)
at org.h2.store.PageOutputStream.storePage(PageOutputStream.java:148)
at org.h2.store.PageOutputStream.flush(PageOutputStream.java:156)
at org.h2.store.PageLog.flushOut(PageLog.java:842)
at org.h2.store.PageLog.flush(PageLog.java:659)
at org.h2.store.PageStore.flushLog(PageStore.java:956)
at org.h2.engine.Database.flush(Database.java:1699)
at org.h2.store.WriterThread.run(WriterThread.java:85)
at java.lang.Thread.run(Unknown Source)
Caused by: org.h2.jdbc.JdbcSQLException: IO Exception:
"java.io.IOException: No space left on device"; "/opt/spazio/spazio/
spdata/rt/a3sp/a3data.h2.db" [90031-147]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:
327)
... 13 more
Caused by: java.io.IOException: No space left on device
at java.io.RandomAccessFile.writeBytes(Native Method)
at java.io.RandomAccessFile.write(Unknown Source)
at org.h2.store.FileStore.write(FileStore.java:340)
... 10 more
12-01 17:04:49 database: flush
org.h2.message.DbException: IO Exception: "java.io.IOException: No
space left on device"; "/opt/spazio/spazio/spdata/rt/a3sp/
a3data.h2.db" [90031-147]
at org.h2.message.DbException.get(DbException.java:156)
at org.h2.message.DbException.convertIOException(DbException.java:
313)
at org.h2.store.FileStore.write(FileStore.java:342)
at org.h2.store.PageStore.writePage(PageStore.java:1309)
at org.h2.store.PageStreamData.write(PageStreamData.java:105)
at org.h2.store.PageOutputStream.storePage(PageOutputStream.java:148)
at org.h2.store.PageOutputStream.flush(PageOutputStream.java:156)
at org.h2.store.PageLog.flushOut(PageLog.java:842)
at org.h2.store.PageLog.flush(PageLog.java:659)
at org.h2.store.PageStore.flushLog(PageStore.java:956)
at org.h2.engine.Database.flush(Database.java:1699)
at org.h2.store.WriterThread.run(WriterThread.java:85)
at java.lang.Thread.run(Unknown Source)
Caused by: org.h2.jdbc.JdbcSQLException: IO Exception:
"java.io.IOException: No space left on device"; "/opt/spazio/spazio/
spdata/rt/a3sp/a3data.h2.db" [90031-147]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:
327)
... 13 more
Caused by: java.io.IOException: No space left on device
at java.io.Rorg.h2.message.DbException: IO Exception:
"java.io.IOException: No space left on device"; "/opt/spazio/spazio/
spdata/rt/a3sp/a3data.h2.db" [90031-147]
at org.h2.message.DbException.get(DbException.java:156)
at org.h2.message.DbException.convertIOException(DbException.java:
313)
at org.h2.store.FileStore.write(FileStore.java:342)
at org.h2.store.PageStore.writePage(PageStore.java:1309)
at org.h2.store.PageStreamData.write(PageStreamData.java:105)
at org.h2.store.PageOutputStream.storePage(PageOutputStream.java:148)
at org.h2.store.PageOutputStream.flush(PageOutputStream.java:156)
at org.h2.store.PageLog.flushOut(PageLog.java:842)
at org.h2.store.PageLog.flush(PageLog.java:659)
at org.h2.store.PageStore.flushLog(PageStore.java:956)
at org.h2.engine.Database.flush(Database.java:1699)
at org.h2.store.WriterThread.run(WriterThread.java:85)
at java.lang.Thread.run(Unknown Source)
Caused by: org.h2.jdbc.JdbcSQLException: IO Exception:
"java.io.IOException: No space left on device"; "/opt/spazio/spazio/
spdata/rt/a3sp/a3data.h2.db" [90031-147]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:
327)
... 13 more
Caused by: java.io.IOException: No space left on device
at java.io.RandomAccessFile.writeBytes(Native Method)
at java.io.RandomAccessFile.write(Unknown Source)
at org.h2.store.FileStore.write(FileStore.java:340)
... 10 more
12-01 17:08:07 database: flush
org.h2.message.DbException: IO Exception: "java.io.IOException: No
space left on device"; "/opt/spazio/spazio/spdata/rt/a3sp/
a3data.h2.db" [90031-147]
at org.h2.message.DbException.get(DbException.java:156)
at org.h2.message.DbException.convertIOException(DbException.java:
313)
at org.h2.store.FileStore.write(FileStore.java:342)
at org.h2.store.PageStore.writePage(PageStore.java:1309)
at org.h2.store.PageStreamData.write(PageStreamData.java:105)
at org.h2.store.PageOutputStream.storePage(PageOutputStream.java:148)
at org.h2.store.PageOutputStream.flush(PageOutputStream.java:156)
at org.h2.store.PageLog.flushOut(PageLog.java:842)
at org.h2.store.PageLog.flush(PageLog.java:659)
at org.h2.store.PageStore.flushLog(PageStore.java:956)
at org.h2.engine.Database.flush(Database.java:1699)
at org.h2.store.WriterThread.run(WriterThread.java:85)
at java.lang.Thread.run(Unknown Source)
Caused by: org.h2.jdbc.JdbcSQLException: IO Exception:
"java.io.IOException: No space left on device"; "/opt/spazio/spazio/
spdata/rt/a3sp/a3data.h2.db" [90031-147]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:
327)
... 13 more
Caused by: java.io.IOException: No space left on device
at java.io.RandomAccessFile.writeBytes(Native Method)
at java.io.RandomAccessFile.write(Unknown Source)
at org.h2.store.FileStore.write(FileStore.java:340)
... 10 more
12-01 17:08:07 database: flush
org.h2.message.DbException: IO Exception: "java.io.IOException: No
space left on device"; "/opt/spazio/spazio/spdata/rt/a3sp/
a3data.h2.db" [90031-147]
at org.h2.message.DbException.get(DbException.java:156)
at org.h2.message.DbException.convertIOException(DbException.java:
313)
at org.h2.store.FileStore.write(FileStore.java:342)
at org.h2.store.PageStore.writePage(PageStore.java:1309)
at org.h2.store.PageStreamData.write(PageStreamData.java:105)
at org.h2.store.PageOutputStream.storePage(PageOutputStream.java:148)
at org.h2.store.PageOutputStream.flush(PageOutputStream.java:156)
at org.h2.store.PageLog.flushOut(PageLog.java:842)
at org.h2.store.PageLog.flush(PageLog.java:659)
at org.h2.store.PageStore.flushLog(PageStore.java:956)
at org.h2.engine.Database.flush(Database.java:1699)
at org.h2.store.WriterThread.run(WriterThread.java:85)
at java.lang.Thread.run(Unknown Source)
Caused by: org.h2.jdbc.JdbcSQLException: IO Exception:
"java.io.IOException: No space left on device"; "/opt/spazio/spazio/
spdata/rt/a3sp/a3data.h2.db" [90031-147]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:
327)
... 13 more
Caused by: java.io.IOException: No space left on device
at java.io.RandomAccessFile.writeBytes(Native Method)
at java.io.RandomAccessFile.write(Unknown Source)
at org.h2.store.FileStore.write(FileStore.java:340)
... 10 more
12-01 17:08:08 database: flush
org.h2.message.DbException: IO Exception: "java.io.IOException: No
space left on device"; "/opt/spazio/spazio/spdata/rt/a3sp/
a3data.h2.db" [90031-147]
at org.h2.message.DbException.get(DbException.java:156)
at org.h2.message.DbException.convertIOException(DbException.java:
313)
at org.h2.store.FileStore.write(FileStore.java:342)
at org.h2.store.PageStore.writePage(PageStore.java:1309)
at org.h2.store.PageStreamData.write(PageStreamData.java:105)
at org.h2.store.PageOutputStream.storePage(PageOutputStream.java:148)
at org.h2.store.PageOutputStream.flush(PageOutputStream.java:156)
at org.h2.store.PageLog.flushOut(PageLog.java:842)
at org.h2.store.PageLog.flush(PageLog.java:659)
at org.h2.store.PageStore.flushLog(PageStore.java:956)
at org.h2.engine.Database.flush(Database.java:1699)
at org.h2.store.WriterThread.run(WriterThread.java:85)
at java.lang.Thread.run(Unknown Source)
Caused by: org.h2.jdbc.JdbcSQLException: IO Exception:
"java.io.IOException: No space left on device"; "/opt/spazio/spazio/
spdata/rt/a3sp/a3data.h2.db" [90031-147]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:
327)
... 13 more
Caused by: java.io.IOException: No space left on device
at java.io.RandomAccessFile.writeBytes(Native Method)
at java.io.RandomAccessFile.write(Unknown Source)
at org.h2.store.FileStore.write(FileStore.java:340)
... 10 more
12-01 17:08:08 database: flush
org.h2.message.DbException: IO Exception: "java.io.IOException: No
space left on device"; "/opt/spazio/spazio/spdata/rt/a3sp/
a3data.h2.db" [90031-147]
at org.h2.message.DbException.get(DbException.java:156)
at org.h2.message.DbException.convertIOException(DbException.java:
313)
at org.h2.store.FileStore.write(FileStore.java:342)
at org.h2.store.PageStore.writePage(PageStore.java:1309)
at org.h2.store.PageStreamData.write(PageStreamData.java:105)
at org.h2.store.PageOutputStream.storePage(PageOutputStream.java:148)
at org.h2.store.PageOutputStream.flush(PageOutputStream.java:156)
at org.h2.store.PageLog.flushOut(PageLog.java:842)
at org.h2.store.PageLog.flush(PageLog.java:659)
at org.h2.store.PageStore.flushLog(PageStore.java:956)
at org.h2.engine.Database.flush(Database.java:1699)
at org.h2.store.WriterThread.run(WriterThread.java:85)
at java.lang.Thread.run(Unknown Source)
Caused by: org.h2.jdbc.JdbcSQLException: IO Exception:
"java.io.IOException: No space left on device"; "/opt/spazio/spazio/
spdata/rt/a3sp/a3data.h2.db" [90031-147]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:
327)
... 13 more
Caused by: java.io.IOException: No space left on device
at java.io.RandomAccessFile.writeBytes(Native Method)
at java.io.RandomAccessFile.write(Unknown Source)
at org.h2.store.FileStore.write(FileStore.java:340)
... 10 more
12-01 17:08:09 database: flush
org.h2.message.DbException: IO Exception: "java.io.IOException: No
space left on device"; "/opt/spazio/spazio/spdata/rt/a3sp/
a3data.h2.db" [90031-147]
at org.h2.message.DbException.get(DbException.java:156)
at org.h2.message.DbException.convertIOException(DbException.java:
313)
at org.h2.store.FileStore.write(FileStore.java:342)
at org.h2.store.PageStore.writePage(PageStore.java:1309)
at org.h2.store.PageStreamData.write(PageStreamData.java:105)
at org.h2.store.PageOutputStream.storePage(PageOutputStream.java:148)
at org.h2.store.PageOutputStream.flush(PageOutputStream.java:156)
at org.h2.store.PageLog.flushOut(PageLog.java:842)
at org.h2.store.PageLog.flush(PageLog.java:659)
at org.h2.store.PageStore.flushLog(PageStore.java:956)
at org.h2.engine.Database.flush(Database.java:1699)
at org.h2.store.WriterThread.run(WriterThread.java:85)
at java.lang.Thread.run(Unknown Source)
Caused by: org.h2.jdbc.JdbcSQLException: IO Exception:
"java.io.IOException: No space left on device"; "/opt/spazio/spazio/
spdata/rt/a3sp/a3data.h2.db" [90031-147]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:
327)
... 13 more
Cau


This lead to a database index corruption ...

Thanks in andvance,

Fulvio

Thomas Mueller

unread,
Dec 5, 2011, 2:30:23 PM12/5/11
to h2-da...@googlegroups.com
Hi,

I'm not sure what the problem could be, but possibly it is still solved in a more recent version of H2. What file system and operating system do you use?

More recent version of H2 also support the recover-test feature: http://h2database.com/html/advanced.html#using_recover_tool

"To verify the database can recover at any time, append ;RECOVER_TEST=64 to the database URL in your test environment. This will simulate an application crash after each 64 writes to the database file. A log file named databaseName.h2.db.log is created that lists the operations. The recovery is tested using an in-memory file system, that means it may require a larger heap setting."

Regards,
Thomas


--
You received this message because you are subscribed to the Google Groups "H2 Database" group.
To post to this group, send email to h2-da...@googlegroups.com.
To unsubscribe from this group, send email to h2-database...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/h2-database?hl=en.

Marco Biancini

unread,
Dec 7, 2011, 12:01:45 PM12/7/11
to H2 Database
Hi,

I am a colleague of Fulvio.
The operating system is Linux eugsl-l-mft005 2.6.18-164.11.1.el5 #1
SMP Wed Jan 6 13:26:31 EST 2010 i686 athlon i386 GNU/Linux
The filesystem is ext3.

Thanks in advance.

Marco

On Dec 5, 8:30 pm, Thomas Mueller <thomas.tom.muel...@gmail.com>
wrote:

> ...
>
> read more »

Thomas Mueller

unread,
Dec 22, 2011, 2:47:25 AM12/22/11
to h2-da...@googlegroups.com
Hi,

I think I found the problem. On out of disk space, the database can get corrupt sometimes, if later write operations succeed. The same problem happens on other kinds of I/O exceptions (where one or some of the writes fail, but subsequent writes succeed). 

In the next version of H2, the file is closed on the first unsuccessful write operation, so that later requests fail consistently.

Thanks a lot for your help!

Regards,
Thomas
Reply all
Reply to author
Forward
0 new messages