Unusual error message : java.io.SyncFailedException: sync failed

[Lizard Lizard]

I've been experiencing issues with my application, and finally managed
to trap the error at its source. It doesn't manifest until the program
has processed something on the order of 80,000 records, and I do not
believe it always manifests at the exact same point.

Here's the error message:
SQL State 90031

org.h2.jdbc.JdbcSQLException: IO Exception:
"java.io.SyncFailedException: sync failed";
"//ALIENLIZARD/Users/Public/VARisk/VARiskDB.h2.db" [90031-160]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:329)
at org.h2.message.DbException.get(DbException.java:158)
at org.h2.message.DbException.convertIOException(DbException.java:315)
at org.h2.store.FileStore.sync(FileStore.java:437)
at org.h2.store.PageStore.writeVariableHeader(PageStore.java:924)
at org.h2.store.PageStore.setLogFirstPage(PageStore.java:918)
at org.h2.store.PageLog.removeUntil(PageLog.java:698)
at org.h2.store.PageStore.checkpoint(PageStore.java:415)
at org.h2.store.PageStore.commit(PageStore.java:1406)
at org.h2.engine.Database.commit(Database.java:1733)
at org.h2.engine.Session.commit(Session.java:450)
at org.h2.command.Command.stop(Command.java:144)
at org.h2.command.Command.executeUpdate(Command.java:248)
at org.h2.server.TcpServerThread.process(TcpServerThread.java:300)
at org.h2.server.TcpServerThread.run(TcpServerThread.java:137)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.SyncFailedException: sync failed
at java.io.FileDescriptor.sync(Native Method)
at org.h2.store.fs.FileDisk.force(FilePathDisk.java:387)
at org.h2.store.FileStore.sync(FileStore.java:435)
... 12 more

IO Exception: "java.io.SyncFailedException: sync failed";
"//ALIENLIZARD/Users/Public/VARisk/VARiskDB.h2.db" [90031-147]

Here's the command that caused the crash, though I suspect it is not
really the problem (I could be wrong)
INSERT INTO Matrix_Outputs_12_14_2011_13_53_SCENARIOTEST_Flat_Rates_Flat_HPI
VALUES('RFMSII2005HI19690815','2008','12','1305','2004-10','2004','10','4','C','31','MF_31-33','44','AGE_13_to_15','NA','NA','FIX30','2','UN','UNK','UN','PRI','LA',4.8,3.4,'-0.0059128257413356','0.312','60000','Orig_Bal_50_75','44.6555','134362','681','FICO_680_699','676.551055100121','659.54','0','6','0','299','58094.63','56972.33','42.97','LTV_lt_60','132595.142799273','Prepay_Spread_030_020','0','681','268','0.719488828872811','0.673014420957512','0.0304458803961435','0.00497854173400635','0.00199014659161782','0.00201047373329569','0.00130017850434709','0.000876576653500326','0.00686275689400644','0.00448524245935612','0.00417317856878597','0.000401418165389632','0.00566545395727093','0.000468083516404374','0.258838718692376','0.0216724524348127','TRUE','40990.9549898553','251.419558512016','38343.1996855503','1734.57274506962','283.639122588582','113.383288366026','114.541372989654','74.0741988085687','49.9406143735163','390.98725047511','255.534713524449','237.755706569802','22.8697281865727','322.774112453445','26.6678085641504','957.731189640103','14902.4889408821','1243.46848168103','40990.9549898553','TRUE','13.1859363321817','586.244451001093','7.0750164425373','0.777402091387036','49.4451439474769','47.1459035462289','2.14041897104952','1.00911986357619','81','2006-02',NULL)

As noted, about 80,000 records were already inserted using this code
before it crashed, and the error doesn't seem to be due to a
misformatted field, etc.

I am running a multithreaded app, with each thread reading from and
writing to the database; I've used this code extensively in the past
without these kinds of errors, for very large data sets, though this
one has some more complex internal work than most, the actual DB I/O
should not be at all affected, all the other code is related to
internal calculations with no DB access at all.

I'm starting the server with this:
java -cp h2-l*.jar org.h2.tools.Server -tcp -tcpPort 9091 -tcpAllowOthers

I am running within Eclipse, so that I could debug/trap the error when
it happened, but I don't think that's relevant, either.

--
=======================
Personal Blog: http://www.xanga.com/lizard_sf
Facebook: http://www.facebook.com/lizard_sf
MrLizard: Gaming and Geekery: http://www.mrlizard.com

[Noel Grandin]

looks like you are running on a network drive, and I'd guess the network is becoming temporarily unavailable.

[Lizard Lizard]

It's mapped as a network drive, but it's a physical location on my c:\
drive, which is where the server and my code are also physically
located.

I'll trying changing the mapping.

My long-term intent is to have network access; is there a good
workaround for this for when it happens?

--

[Lizard Lizard]

OK, I did the following:
a)Set Eclipse to launch in c:\users\public\varisk. This is the same
physical location as the UNC path given earlier.
b)Shutdown and then restarted the server.
c)When I launch my java app, it tries to connect to the DB, and I get:

jdbc:h2:tcp://192.168.1.101:9091/file://c:\Users\Public\VARisk\VARiskDB
Could not create database, error received was: IO Exception:
"java.io.SyncFailedException: sync failed";
"C:/Users/Public/VARisk/VARiskDB.h2.db" [90031-147]. Closing program
now.
DB Connect Time=0

org.h2.jdbc.JdbcSQLException: IO Exception:
"java.io.SyncFailedException: sync failed";

"C:/Users/Public/VARisk/VARiskDB.h2.db" [90031-160]

at org.h2.message.DbException.getJdbcSQLException(DbException.java:329)
at org.h2.message.DbException.get(DbException.java:158)
at org.h2.message.DbException.convertIOException(DbException.java:315)
at org.h2.store.FileStore.sync(FileStore.java:437)
at org.h2.store.PageStore.writeVariableHeader(PageStore.java:924)
at org.h2.store.PageStore.setLogFirstPage(PageStore.java:918)

at org.h2.store.PageLog.openForWriting(PageLog.java:188)
at org.h2.store.PageStore.openExisting(PageStore.java:360)
at org.h2.store.PageStore.open(PageStore.java:278)
at org.h2.engine.Database.getPageStore(Database.java:2090)
at org.h2.engine.Database.open(Database.java:551)
at org.h2.engine.Database.openDatabase(Database.java:219)
at org.h2.engine.Database.<init>(Database.java:214)
at org.h2.engine.Engine.openSession(Engine.java:56)
at org.h2.engine.Engine.openSession(Engine.java:159)
at org.h2.engine.Engine.createSessionAndValidate(Engine.java:138)
at org.h2.engine.Engine.createSession(Engine.java:121)
at org.h2.server.TcpServerThread.run(TcpServerThread.java:124)

at java.lang.Thread.run(Unknown Source)
Caused by: java.io.SyncFailedException: sync failed
at java.io.FileDescriptor.sync(Native Method)
at org.h2.store.fs.FileDisk.force(FilePathDisk.java:387)
at org.h2.store.FileStore.sync(FileStore.java:435)

... 15 more

at org.h2.engine.SessionRemote.done(Unknown Source)
at org.h2.engine.SessionRemote.a(Unknown Source)
at org.h2.engine.SessionRemote.a(Unknown Source)
at org.h2.engine.SessionRemote.int(Unknown Source)
at org.h2.engine.SessionRemote.createSession(Unknown Source)
at org.h2.jdbc.JdbcConnection.<init>(Unknown Source)
at org.h2.jdbc.JdbcConnection.<init>(Unknown Source)
at org.h2.Driver.connect(Unknown Source)
at java.sql.DriverManager.getConnection(Unknown Source)
at java.sql.DriverManager.getConnection(Unknown Source)
at dataSource.H2LoanDataSource.<init>(H2LoanDataSource.java:31)
at dataSource.LoanSourceFactory.getLoanDataSource(LoanSourceFactory.java:60)
at dataSource.LoanSourceFactory.getLoanDataSource(LoanSourceFactory.java:27)
at tables.ScenarioTable.refreshScenarioData(ScenarioTable.java:287)
at forms.UserInterfaceForm.createContents(UserInterfaceForm.java:1091)
at forms.UserInterfaceForm$1.run(UserInterfaceForm.java:815)
at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332)
at forms.UserInterfaceForm.open(UserInterfaceForm.java:791)
at exec.FrontEndLauncher.run(FrontEndLauncher.java:54)
at exec.FrontEndLauncher.main(FrontEndLauncher.java:38)

org.h2.jdbc.JdbcSQLException: IO Exception:
"java.io.SyncFailedException: sync failed";

"C:/Users/Public/VARisk/VARiskDB.h2.db" [90031-160]

at org.h2.message.DbException.getJdbcSQLException(DbException.java:329)
at org.h2.message.DbException.get(DbException.java:158)
at org.h2.message.DbException.convertIOException(DbException.java:315)
at org.h2.store.FileStore.sync(FileStore.java:437)
at org.h2.store.PageStore.writeVariableHeader(PageStore.java:924)
at org.h2.store.PageStore.setLogFirstPage(PageStore.java:918)

at org.h2.store.PageLog.openForWriting(PageLog.java:188)
at org.h2.store.PageStore.openExisting(PageStore.java:360)
at org.h2.store.PageStore.open(PageStore.java:278)
at org.h2.engine.Database.getPageStore(Database.java:2090)
at org.h2.engine.Database.open(Database.java:551)
at org.h2.engine.Database.openDatabase(Database.java:219)
at org.h2.engine.Database.<init>(Database.java:214)
at org.h2.engine.Engine.openSession(Engine.java:56)
at org.h2.engine.Engine.openSession(Engine.java:159)
at org.h2.engine.Engine.createSessionAndValidate(Engine.java:138)
at org.h2.engine.Engine.createSession(Engine.java:121)
at org.h2.server.TcpServerThread.run(TcpServerThread.java:124)

at java.lang.Thread.run(Unknown Source)
Caused by: java.io.SyncFailedException: sync failed
at java.io.FileDescriptor.sync(Native Method)
at org.h2.store.fs.FileDisk.force(FilePathDisk.java:387)
at org.h2.store.FileStore.sync(FileStore.java:435)

... 15 more

at org.h2.engine.SessionRemote.done(Unknown Source)
at org.h2.engine.SessionRemote.a(Unknown Source)
at org.h2.engine.SessionRemote.a(Unknown Source)
at org.h2.engine.SessionRemote.int(Unknown Source)
at org.h2.engine.SessionRemote.createSession(Unknown Source)
at org.h2.jdbc.JdbcConnection.<init>(Unknown Source)
at org.h2.jdbc.JdbcConnection.<init>(Unknown Source)
at org.h2.Driver.connect(Unknown Source)
at java.sql.DriverManager.getConnection(Unknown Source)
at java.sql.DriverManager.getConnection(Unknown Source)
at dataSource.H2LoanDataSource.openDataSource(H2LoanDataSource.java:378)
at tables.ScenarioTable.refreshScenarioData(ScenarioTable.java:288)
at forms.UserInterfaceForm.createContents(UserInterfaceForm.java:1091)
at forms.UserInterfaceForm$1.run(UserInterfaceForm.java:815)
at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332)
at forms.UserInterfaceForm.open(UserInterfaceForm.java:791)
at exec.FrontEndLauncher.run(FrontEndLauncher.java:54)
at exec.FrontEndLauncher.main(FrontEndLauncher.java:38)
java.lang.NullPointerException
java.lang.NullPointerException
at dataSource.LoanDataSource.getFieldNames(LoanDataSource.java:357)
at forms.UserInterfaceForm.createContents(UserInterfaceForm.java:1094)
at forms.UserInterfaceForm$1.run(UserInterfaceForm.java:815)
at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332)
at forms.UserInterfaceForm.open(UserInterfaceForm.java:791)
at exec.FrontEndLauncher.run(FrontEndLauncher.java:54)
at exec.FrontEndLauncher.main(FrontEndLauncher.java:38)

Any idea on how to "resync" the DB would be great, because I'd hate to
lose the ~40 gig of data in it.

[Thomas Mueller]

Hi,

> looks like you are running on a network drive, and I'd guess the network is becoming temporarily unavailable.

It sounds like such a problem, yes.

H2 currently doesn't 'retry' if there is a temporary problem in the
file system (for example if it's temporarily unavailable). If this is
a problem for you, you could write a special file system adapter that
does retry.

Regards,
Thomas

[Lizard Lizard]

Is there a way to reconnect to the DB at all after the sync error?
(Shutdown & restart server, some command line option, etc?)

Would something like this work, in general?
try
{
domySQLcommand();
}
catch(IOError e)
{
//do nothing, just set some state flag that forces it to retry the
same insert command next pass through the loop
}

Basically, it's failing in a loop that inserts rows; there shouldn't
be an issue with retrying an insert with the same data if one fails,
up to some point.

Also, while this is off-topic, the "network drive" isn't; the location
of the database file is on the c:\ drive, same as the server and the
running app. In other words, it can't become "unavailable" if the app
is running in the first place, so I find the error very odd.

[Thomas Mueller]

Hi,

Is there a way to reconnect to the DB at all after the sync error?
(Shutdown & restart server, some command line option, etc?)

Currently not. As I wrote, one solution would be to implement a file system abstraction that re-tries.

Basically, it's failing in a loop that inserts rows; there shouldn't
be an issue with retrying an insert with the same data if one fails,
up to some point.

Patches are welcome. 

Also, while this is off-topic, the "network drive" isn't; the location
of the database file is on the c:\ drive, same as the server and the
running app. In other words, it can't become "unavailable" if the app
is running in the first place, so I find the error very odd.

Maybe sync simply isn't supported by this file system?

Regards,

Thomas

[Lizard Lizard]

On Tue, Dec 20, 2011 at 3:18 PM, Thomas Mueller
<thomas.to...@gmail.com> wrote:
> Hi,

>> Also, while this is off-topic, the "network drive" isn't; the location
>> of the database file is on the c:\ drive, same as the server and the
>> running app. In other words, it can't become "unavailable" if the app
>> is running in the first place, so I find the error very odd.
>
>
> Maybe sync simply isn't supported by this file system?
>

Well, the file system is Windows 7 Home Premium, if that helps.

What triggers the *need* to sync? I'm getting the impression that it's
not something happening at the H2 level.

I am, still, somewhat confused about what the file system is trying to
sync *with*. When I've restarted the computer and then try to connect
to the DB using a local path, and get the io sync error, I am not sure
what's trying to sync up with what; I can understand if the database
was corrupt and unreadable due to write errors, but my understanding
here is that this error is generated outside of H2. Various google
searches have not revealed a lot of detail on this error; it's
apparently not a common problem.

I ultimately deleted the disk file and re-imported the data, and at
the moment, it's looking good.

I will also take a look at my general threading code; it's possible I
may be sharing file resources improperly between threads, and the
tests I've been running have been extreme, straining my system in ways
likely to cause disk contention and other issues that might not occur
with smaller test runs.

[Noel Grandin]

a lot of filesystems use a write-behind cache.
calling sync() is a way of saying "make sure this data is actually on the disk"

[Thomas Mueller]

Hi,

> I am, still, somewhat confused about what the file system is trying to
> sync *with*.

See the java.io.FileDescriptor.sync documentation: "Force all system buffers to synchronize with the underlying device."

> this error is generated outside of H2.

Yes, I guess it's a file system problem.

> Various google
> searches have not revealed a lot of detail on this error; it's
> apparently not a common problem.

When searching for "java.io.SyncFailedException: sync failed" I got 30300 results on Google. But I also don't know what could be the reason in your case, but you are not the only one who got this problem.

> I will also take a look at my general threading code; it's possible I
> may be sharing file resources improperly between threads, and the
> tests I've been running have been extreme, straining my system in ways
> likely to cause disk contention and other issues that might not occur
> with smaller test runs.

Still you shouldn't get this exception.

Regards,
Thomas