Behavior of Linux fsync on dax mount

[Mohit Verma]

Hi Everyone,

I have a system where I am using pmem in fsdax mode (by creating namespace with ndctl & then mounting with -o dax flag). I have ext4 running on this.

I know that in current Linux implementation, ext4 bypasses the page cache. My question is what happens with fsync, fdatasync like file-sytem operations in this mode?

I am running pg_test_fsync, a benchmarking tool used in postgres . I see that on pmem ext4-dax device, unsynced writes are slower than writes+fsync and I have no idea what is going on. 

Any ideas?

Thanks!

Mohit

[Andy Rudoff]

Hi Mohit,

The POSIX semantics still apply when using DAX*.  msync, fsync, fdatasync, etc. do what they promise, which means they flush what needs to be flushed.   You're right that there's no page cache to flush for DAX files, but if the CPU caches are not considered persistent (which is true for all machines shipping at the moment), then the CPU cache lines associated with the range being sync'd will be flushing.  This usually means a loop of CLWB instructions over that range, although the OS is free to optimize that by skipping clean pages, etc.

Does that account for the performance you're observing?

*Note that providing the MAP_SYNC flag to mmap asks the OS for permission to flush directly from user space, which is outside of what POSIX normally allows.

Thanks,

-andy

[Mohit Verma]

Hi Andy,

Thanks for the explanation. The postgres benchmark I am running: pg_test_fsync, uses file system calls open(), write(), fsync() and close(). It doesn't do any mmap. 

So if I understand correctly internally linux-ext4-dax issues CLWBs for all cachelines in the virtual address range being touched. Linux is free to optimize away some of these CLWBs if the pages are clean.

This still does not explain the benchmark numbers. I notice that only on pmem running ext4 in dax mode, unsynced writes are slower than writes+fsync. Particularly,

A loop of { open(), write(), close() } IS SLOWER than open() + loop of { write(), fsync() } + close().

This only happens for a file on pmem running ext4 on dax mode. On pmem running ext4 in block mode or on SSDs running ext4 filesystem, unsync'ed writes are faster than sync'ed writes.

I am not sure if I am missing something.

Thanks,

Mohit