MVCC + Multithreaded

[Rami Ojares]

Hi,

I put MVCC + Multithreaded into production with the latest version.
Out of around 11.000 images (type BINARY) about 7.500 got corrupted
giving me exceptions like

org.h2.jdbc.JdbcSQLException: IO Exception: "java.io.IOException:
org.h2.message.DbException: IO Exception: ""java.io.IOException: Block
not found in id [1, -29, -17, 7, -90, 81] [1.4.187/50]"" [90028-186]";
"lob: null table: -3 id: 119466" [90031-186]

So I used the recover tool to recover the corrupted database.
Recover tool gave me error line for every problematic image like this:

Error: java.io.IOException: Block not found in id [1, -80, -118, 7,
-124, 91] [1.4.187/50]
Error: java.io.IOException: Block not found in id [1, -50, -71, 7, -123,
91] [1.4.187/50]
Error: java.io.IOException: Block not found in id [1, -72, -118, 6,
-122, 91] [1.4.187/50]

So from an older database version I manually updated those images that
had become empty.
All went fine except one to which I was not able to write anything.
It gave me the error

org.h2.jdbc.JdbcSQLException: IO Exception: "java.io.IOException:
org.h2.message.DbException: IO Exception: ""java.io.IOException:
java.nio.BufferUnderflowException"" [90028-186]"; "lob: null table: -3
id: 21742" [90031-186]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)
at org.h2.message.DbException.get(DbException.java:168)
at org.h2.message.DbException.convertIOException(DbException.java:330)
at org.h2.value.ValueLobDb.getBytesNoCopy(ValueLobDb.java:328)
at org.h2.value.ValueLobDb.getBytes(ValueLobDb.java:312)
at org.h2.jdbc.JdbcResultSet.getBytes(JdbcResultSet.java:1054)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at bsh.Reflect.invokeMethod(Unknown Source)
at bsh.Reflect.invokeObjectMethod(Unknown Source)
at bsh.Name.invokeMethod(Unknown Source)
at bsh.BSHMethodInvocation.eval(Unknown Source)
at bsh.BSHPrimaryExpression.eval(Unknown Source)
at bsh.BSHPrimaryExpression.eval(Unknown Source)
at bsh.BSHArguments.getArguments(Unknown Source)
at bsh.BSHMethodInvocation.eval(Unknown Source)
at bsh.BSHPrimaryExpression.eval(Unknown Source)
at bsh.BSHPrimaryExpression.eval(Unknown Source)
at bsh.BSHBlock.evalBlock(Unknown Source)
at bsh.BSHBlock.eval(Unknown Source)
at bsh.BSHBlock.eval(Unknown Source)
at bsh.BSHIfStatement.eval(Unknown Source)
at bsh.BSHBlock.evalBlock(Unknown Source)
at bsh.BSHBlock.eval(Unknown Source)
at bsh.BSHBlock.eval(Unknown Source)
at bsh.BSHWhileStatement.eval(Unknown Source)
at bsh.BSHBlock.evalBlock(Unknown Source)
at bsh.BSHBlock.eval(Unknown Source)
at bsh.BSHBlock.eval(Unknown Source)
at bsh.BSHTryStatement.eval(Unknown Source)
at bsh.Interpreter.eval(Unknown Source)
at bsh.Interpreter.source(Unknown Source)
at bsh.Interpreter.main(Unknown Source)
Caused by: java.io.IOException: org.h2.message.DbException: IO
Exception: "java.io.IOException: java.nio.BufferUnderflowException"
[90028-186]
at
org.h2.message.DbException.convertToIOException(DbException.java:364)
at
org.h2.store.LobStorageRemoteInputStream.read(LobStorageRemoteInputStream.java:73)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at org.h2.util.IOUtils.copy(IOUtils.java:167)
at org.h2.util.IOUtils.readBytesAndClose(IOUtils.java:282)
at org.h2.value.ValueLobDb.getBytesNoCopy(ValueLobDb.java:326)
... 30 more
Caused by: org.h2.message.DbException: IO Exception:
"java.io.IOException: java.nio.BufferUnderflowException" [90028-186]
at org.h2.message.DbException.convert(DbException.java:283)
at org.h2.engine.SessionRemote.done(SessionRemote.java:629)
at org.h2.engine.SessionRemote.readLob(SessionRemote.java:779)
at
org.h2.store.LobStorageRemoteInputStream.read(LobStorageRemoteInputStream.java:71)
... 36 more
Caused by: org.h2.jdbc.JdbcSQLException: IO Exception:
"java.io.IOException: java.nio.BufferUnderflowException" [90028-187]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)
at org.h2.message.DbException.get(DbException.java:168)
at org.h2.message.DbException.convert(DbException.java:287)
at org.h2.server.TcpServerThread.sendError(TcpServerThread.java:221)
at org.h2.server.TcpServerThread.run(TcpServerThread.java:161)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: java.nio.BufferUnderflowException
at
org.h2.message.DbException.convertToIOException(DbException.java:364)
at org.h2.util.IOUtils.readFully(IOUtils.java:338)
at org.h2.server.TcpServerThread.process(TcpServerThread.java:462)
at org.h2.server.TcpServerThread.run(TcpServerThread.java:159)
... 1 more
Caused by: java.nio.BufferUnderflowException
at java.nio.Buffer.nextGetIndex(Buffer.java:498)
at java.nio.HeapByteBuffer.getShort(HeapByteBuffer.java:304)
at org.h2.mvstore.Page.read(Page.java:648)
at org.h2.mvstore.Page.read(Page.java:196)
at org.h2.mvstore.MVStore.readPage(MVStore.java:1843)
at org.h2.mvstore.MVMap.readPage(MVMap.java:736)
at org.h2.mvstore.Page.getChildPage(Page.java:218)
at org.h2.mvstore.MVMap.binarySearch(MVMap.java:468)
at org.h2.mvstore.MVMap.binarySearch(MVMap.java:469)
at org.h2.mvstore.MVMap.binarySearch(MVMap.java:469)
at org.h2.mvstore.MVMap.get(MVMap.java:450)
at org.h2.mvstore.StreamStore.getBlock(StreamStore.java:386)
at org.h2.mvstore.StreamStore$Stream.nextBuffer(StreamStore.java:503)
at org.h2.mvstore.StreamStore$Stream.read(StreamStore.java:460)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at
org.h2.server.TcpServerThread$CachedInputStream.read(TcpServerThread.java:551)
at org.h2.util.IOUtils.readFully(IOUtils.java:329)
... 3 more
at org.h2.engine.SessionRemote.done(SessionRemote.java:622)
... 38 more

I hope this could be helpful in ironing out the bugs.

- Rami

[Noel Grandin]

You seem to be running 1.4.186. Do you know that 1.4.187 is out?

[Rami Ojares]

It seems that the database was running 187 but my webserver (and dev
client) was running 186.
Could that explain the problem?

- Rami

[Thomas Mueller]

Hi,

Yes this could explain the problem. See also the change log.

Regards,

Thomas

--
You received this message because you are subscribed to the Google Groups "H2 Database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to h2-database...@googlegroups.com.
To post to this group, send email to h2-da...@googlegroups.com.
Visit this group at http://groups.google.com/group/h2-database.
For more options, visit https://groups.google.com/d/optout.

[Rami Ojares]

Ok, I will put the multithreaded back on making sure all clients are also using version 187.
I will report if the corruption turns up again after that.

- Rami

[Rami Ojares]

I found 44 newly corrupted images that I was not able to read anymore.
There were 2 different errors associated:
java.io.IOException: java.lang.IllegalArgumentException" [90028-187]
java.io.IOException: Block not found in id [1, -36, -27, 5, -105, 6] [1.4.187/50]

I keep following this issue a bit more.

- rami

[Thomas Mueller]

Hi,

I'm afraid if the database was opened with version 1.4.186 (when using the MVStore), then the data is gone, even if you open it with a newer version afterwards.

Regards,

Thomas

[Rami Ojares]

But my import process as follows:
- Create a new db from scratch
- Create new tables
- import data from the old version using jdbc+sql
- create materilized views (calculated tables), set rights etc.
- turn h2 off in production, copy there the newly created file and start again.

So I don't think that my current db has ever touched version 186. Is there a way to see from the db file if it has at some point been touched by version 186?

- Rami

[Rami Ojares]

Hold your horses!
I found one more client that accesses the database with version 186.
I had forgotten all about it.
It is a very difficult situation that just one client reading a little bit from the database will corrupt it.
So beware of version 186 (and maybe others?).

I will regenerate the database this evening make sure all clients are 187 and report if I see any more corruptions.

- Rami

[Ryan How]

Isn't a bit unnerving if any client can actually corrupt the database?. I thought it is the servers job to keep the database safe?

[Rami Ojares]

Just wanted to let people know that I have been able to run 1.4.187 with
mvcc and multithreaded without any problems for some time now.
So far so good.

- rami