H2 Durability: fsync(), F_FULLFSYNC and FlushFileBuffers()

[Gili Tzabari]

Hi Thomas,

I wanted to follow up your post about Durability problems:
http://www.h2database.com/html/advanced.html?highlight=fsync&search=fsync#durability_problems

If you follow the Mac OS X link you refer to you will find out the
following:

1. fsync() flushes the OS-level disk cache to the disk, but does not
force the disk to flush its on-board write cache.
2. Under Mac OS X you can use F_FULLFSYNC to force the disk to flush
its on-board write cache. Source: http://lists.apple.com/archives/darwin-dev/2005/Feb/msg00072.html
3. Under Windows you can use FlushFileBuffers() to force the disk to
flush its on-board write cache. Source:
http://www.mail-archive.com/sqlite...@sqlite.org/msg08132.html and
http://old.nabble.com/Re%3A-Performance-problem-with-3.2.7-p1591614.html

So first of all, we need to establish whether these sources are
correct. If you rewrite your test in terms of FlushFileBuffers() does
it flush to disk correctly? Next, assuming this works, we need to
figure out whether this API is accessible through Java. If not, we can
file a RFE with Oracle as well as looking into using JNI under H2
(ugly, but doable).

What do you think?

Gili

[Gili Tzabari]

Good news! It looks like JDK7 invokes FlushFileBuffers() in the
following cases:

1. MappedByteBuffer.force(). See http://bugs.sun.com/view_bug.do?bug_id=6816049
and http://hg.openjdk.java.net/jdk7/build/jdk/rev/3740c2da7cc5

2. AsynchronousFileChannel.force()

I found other matches but it wasn't immediately obvious which public
APIs they get invoked by. Anyway, I think we can take away two
important points from this:

1. Sun/Oracle has contacted Microsoft about the use of
FlushFileBuffers() and they implied it would do the job.

2. Java exposes this functionality through publicly accessible APIs.

This should make it easier for you to retrofit your test program.

Gili

Hopefully this means they also invoke it from other places...

Gili

On Nov 24, 12:01 am, Gili Tzabari <cow...@bbs.darktech.org> wrote:
> Hi Thomas,
>

> I wanted to follow up your post about Durability problems:http://www.h2database.com/html/advanced.html?highlight=fsync&search=f...

>
> If you follow the Mac OS X link you refer to you will find out the
> following:
>
> 1. fsync() flushes the OS-level disk cache to the disk, but does not
> force the disk to flush its on-board write cache.
> 2. Under Mac OS X you can use F_FULLFSYNC to force the disk to flush
> its on-board write cache. Source:http://lists.apple.com/archives/darwin-dev/2005/Feb/msg00072.html
> 3. Under Windows you can use FlushFileBuffers() to force the disk to
> flush its on-board write cache. Source:

> http://www.mail-archive.com/sqlite-us...@sqlite.org/msg08132.html andhttp://old.nabble.com/Re%3A-Performance-problem-with-3.2.7-p1591614.html

[Gili Tzabari]

Another interesting read: http://www.humboldt.co.uk/2009/03/fsync-across-platforms.html

Gili

On Nov 24, 12:18 am, Gili Tzabari <cow...@bbs.darktech.org> wrote:
> Good news! It looks like JDK7 invokes FlushFileBuffers() in the
> following cases:
>
> 1. MappedByteBuffer.force(). Seehttp://bugs.sun.com/view_bug.do?bug_id=6816049

> andhttp://hg.openjdk.java.net/jdk7/build/jdk/rev/3740c2da7cc5

[Thomas Mueller]

Hi,

If there is a way to call FlushFileBuffers() / F_FULLFSYNC then it
would be great (too bad no option for Linux), but I'm afraid if it's
not already supported then filing an RFE at Oracle will not help much.
The chance that they actually implement it is *very* low. I guess they
will say that fsync() should simply work.

I didn't run my 'christmas light timer' tests recently. If you want to
do that please go ahead.

Regards,
Thomas

[Gili Tzabari]

Hi Thomas,

As mentioned in my original post, Java7 already implements
FlushFileBuffers() in the Windows implementation. The first release of
OpenJDK for MacOS will be Java8 so we'll have to check for F_FULLFSYNC
when it comes out.

The real question is when do you plan on making use of these APIs
in H2?

Thanks,
Gili

On Dec 15, 5:34 am, Thomas Mueller <thomas.tom.muel...@gmail.com>
wrote:
> Hi,
>
> If there is a way to callFlushFileBuffers() / F_FULLFSYNC then it

[Thomas Mueller]

Hi,

> 2. Java exposes this functionality through publicly accessible APIs.

H2 uses FileDescriptor.sync() (RandomAccessFile.getFD().sync()) by default, but can also use RandomAccessFile.getChannel().force(true/false) if configured (System property h2.syncMethod, see the docs).

Regards,
Thomas

--
You received this message because you are subscribed to the Google Groups "H2 Database" group.
To post to this group, send email to h2-da...@googlegroups.com.
To unsubscribe from this group, send email to h2-database...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/h2-database?hl=en.

[cowwoc]

    You should be good then. FileDescriptor.sync() maps to FlushFileBuffers in the Windows implementation. RandomAccessFile.getChannel().force() does the same.

    In light of this, I recommend re-running org.h2.test.poweroff.Test under Java7. It should work fine now.

Gili