posix direct IO and synchronisation...

[billywhizz]

am wondering if anyone can answer this for me?

is there any way of being notified asynchronously when data written to
a file (using the fs.write posix system call) is actually flushed from
the kernel buffers and persisted to the disk? the only way i can see
of doing this is by using writeSync or opening the file with the
O_SYNC flag and using write. if i was writing a database system on
node.js i would need to know when the data has actually hit the disk
but don't want to use the sync methods as they block the whole process
until they return...

Thanks

[Kapil Reddy]

I used watchFile for a small application but i see where you are
going. I considered very small number of users (around 15) so
watchFile worked for me but what you are suggesting is something like
race condition between two users trying to access the same file. But
you can start with watchFile and later and add code to handle the
condition.

Regards,
Kapil

> --
> You received this message because you are subscribed to the Google Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com.
> To unsubscribe from this group, send email to nodejs+un...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/nodejs?hl=en.
>
>

[billywhizz]

have been doing some research and it seems the only way to do this
reliably is to use O_DIRECT flag when opening the file which
completely bypasses the kernel buffers and writes directly from the
user space memory buffers to the disk. is hard to find reliable info,
but it seems that this is not a recommended way of doing things, but
from what i can discover some databases use this and manage their own
buffers. The nasty thing with O_DIRECT is it forces you to align your
buffers and file offsets with the block size of the file system you
are writing to. in theory (i can't get it to work yet in practice),
direct IO should be faster than going through the kernel, but it could
lead to all sorts of instability if not used correctly.

my scenario is basically to have multiple files being written
concurrently at very high transaction rates with as little latency as
possible and am trying to figure out best way to do so...

[Kapil Reddy]

ok, I new thing learnt i guess for me!!! Let me know when you finish it.

Regards,
Kapil

[Joran Dirk Greef]

I know this is an old thread, but if anyone is looking for more help with doing direct IO in Node.js, we released a native add on yesterday which provides helpers for things like O_DIRECT, O_SYNC, O_DSYNC, aligned buffers, F_NOCACHE on macOS, FSCTL_LOCK_VOLUME on Windows etc.

https://github.com/ronomon/direct-io