ioutil.WriteFile and Sync

[Manlio Perillo]

What is the reason why ioutil.WriteFile does not call File.Sync?

Thanks

Manlio

[Jan Mercl]

On Thu, Jul 21, 2016 at 8:17 PM Manlio Perillo <manlio....@gmail.com> wrote:

> What is the reason why ioutil.WriteFile does not call File.Sync?

That would be harmful.

--

-j

[Jakob Borg]

I would say that syncing file contents is a more advanced operation
that stems from some knowledge about the persistence requirements of
your application. If you have those requirements you need to be in
control and should not use a convenience operations like WriteFile.
Also, performance.

(I've personally only used WriteFile in tests...)

//jb

> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Konstantin Khomoutov]

On Thu, 21 Jul 2016 11:17:18 -0700 (PDT)
Manlio Perillo <manlio....@gmail.com> wrote:

> What is the reason why ioutil.WriteFile does not call File.Sync?

I'd say that's because to inhibit this behaviour when needed you'd need
to implement ioutil.WriteFileNoSync() or have an option flag as an
argument (remember CreateFile() of Win32 API?).
The way it's implemented, it does exactly one task.
A version calling Sync would be like 10 lines of code at most,
I imagine.

[Manlio Perillo]

On Thu, Jul 21, 2016 at 8:39 PM, Konstantin Khomoutov
<flat...@users.sourceforge.net> wrote:
> On Thu, 21 Jul 2016 11:17:18 -0700 (PDT)
> Manlio Perillo <manlio....@gmail.com> wrote:
>
>> What is the reason why ioutil.WriteFile does not call File.Sync?
>
> I'd say that's because to inhibit this behaviour when needed you'd need
> to implement ioutil.WriteFileNoSync() or have an option flag as an
> argument (remember CreateFile() of Win32 API?).

I'm curious to know why one wants to inhibit the durability behaviour.
AFAIK, several design choices of Go and its standard library are
designed to make program safe and robust.
WriteFile does not follow these principles.

> [...]

Manlio

[Axel Wagner]

There are almost no use cases that require syncing the file. The operating system is designed with a good buffering layer for a reason. If you do have such need, it is trivial to write your own version of WriteFile that does this.

[Manlio Perillo]

Il giorno giovedì 21 luglio 2016 21:28:09 UTC+2, Axel Wagner ha scritto:

There are almost no use cases that require syncing the file. The operating system is designed with a good buffering layer for a reason. If you do have such need, it is trivial to write your own version of WriteFile that does this.

 

Indeed, I remember having reviewed a backup tool written in Go, and there were no calls to File.Sync...

Manlio 

[Sam Whited]

On Thu, Jul 21, 2016 at 2:17 PM, Manlio Perillo
<manlio....@gmail.com> wrote:
> On Thu, Jul 21, 2016 at 8:39 PM, Konstantin Khomoutov

> I'm curious to know why one wants to inhibit the durability behaviour.
> AFAIK, several design choices of Go and its standard library are
> designed to make program safe and robust.
> WriteFile does not follow these principles.

File access in Unix and Unix-like systems is a lot like Go: If you
conform to its design decisions and do things the way it wants you to
do them, you'll end up having a pretty good time; however, if you try
to make it conform to the way you imagine things should work, you're
just going to end up creating problems for yourself down the road.

If you don't absolutely need to sync (or don't know if you need to
sync or not but aren't seeing problems), don't. Let the kernel handle
how and when your files get written (or your network gets flushed, or
whatever the case may be), and it will be able to optimize under the
hood, make the best utilization of its file cache, and generally do
all the good things it wants to do for you.

—Sam

--
Sam Whited
pub 4096R/54083AE104EA7AD3

[Konstantin Khomoutov]

On Thu, 21 Jul 2016 21:17:38 +0200

Manlio Perillo <manlio....@gmail.com> wrote:

> >> What is the reason why ioutil.WriteFile does not call File.Sync?
> >
> > I'd say that's because to inhibit this behaviour when needed you'd
> > need to implement ioutil.WriteFileNoSync() or have an option flag
> > as an argument (remember CreateFile() of Win32 API?).
>
> I'm curious to know why one wants to inhibit the durability behaviour.
> AFAIK, several design choices of Go and its standard library are
> designed to make program safe and robust.
> WriteFile does not follow these principles.

Well, I definitely sympathize your line of reasoning, but it's not that
simple. For instance, a WriteFile() truncates the file when it exists.
A well-written application which wants to _replace_ a file must go this
route:

I. Simple approach:

1) Write a _new_ file (that is, opened with O_CREAT|O_EXCL) aside the
old one.
2) rename() the new file over the old one.

II. "Robust" approach:

0) Open the directory entry containing the file.
1) Same as (1) above.
2) fsync() the new file's contents.
3) rename() the new file over the old one.
4) fsync() the directory entry containing the file.

Here, (4) is an oft-forgotten task, and it requires the openat()
/ renameat() and other whateverat() functions, and the step (0).

Now here comes another consideration: certain file operations are
atomic on a scale larger than a single file. Consider a package
management program: when it unpacks a package, it makes little sense
making sure each file it unpacks is fdsync()-ed right after it was
written -- simple because it unpacking is terminated, it doesn't matter
whether the files extracted so far actually hit the underlying medium.
So we can unpack and then issue syncfs(2) all them all at once.

What I'm leading to, is that WriteFile() is just a narrowly-scoped
utility function. For advances cases you will almost inevitably
a) think your strategy through; b) write complex custom code.

You might start exploring this interesting realm from, say [1]. ;-)

1. https://blog.valerieaurora.org/2009/04/16/dont-panic-fsync-ext34-and-your-data/

[Manlio Perillo]

Il giorno giovedì 21 luglio 2016 22:01:52 UTC+2, Konstantin Khomoutov ha scritto:

On Thu, 21 Jul 2016 21:17:38 +0200
Manlio Perillo <manlio....@gmail.com> wrote:

> >> What is the reason why ioutil.WriteFile does not call File.Sync?
> >
> > I'd say that's because to inhibit this behaviour when needed you'd
> > need to implement ioutil.WriteFileNoSync() or have an option flag
> > as an argument (remember CreateFile() of Win32 API?).
>
> I'm curious to know why one wants to inhibit the durability behaviour.
> AFAIK, several design choices of Go and its standard library are
> designed to make program safe and robust.
> WriteFile does not follow these principles.

Well, I definitely sympathize your line of reasoning, but it's not that
simple.  For instance, a WriteFile() truncates the file when it exists.
A well-written application which wants to _replace_ a file must go this
route:

> [...]

 

II. "Robust" approach:

0) Open the directory entry containing the file.
1) Same as (1) above.
2) fsync() the new file's contents.
3) rename() the new file over the old one.
4) fsync() the directory entry containing the file.

Here, (4) is an oft-forgotten task, and it requires the openat()
/ renameat() and other whateverat() functions, and the step (0).

Thanks, but I'm well aware of this.

I also wrote a package that implements an atomic storage atop a filesystem.

I have not yet released it since the golang.org/x/sys is missing some syscalls I need.

 

Now here comes another consideration: certain file operations are
atomic on a scale larger than a single file.  Consider a package
management program: when it unpacks a package, it makes little sense
making sure each file it unpacks is fdsync()-ed right after it was
written -- simple because it unpacking is terminated, it doesn't matter
whether the files extracted so far actually hit the underlying medium.
So we can unpack and then issue syncfs(2) all them all at once.  

What I'm leading to, is that WriteFile() is just a narrowly-scoped
utility function.  For advances cases you will almost inevitably
a) think your strategy through; b) write complex custom code.

I think your reasoning is wrong.

WriteFile is a "self contained" function; it does not return the handle to the file, so it can not be used inside a transaction.

I really don't see valid reasons to not call Sync.

Manlio 

[Jan Mercl]

On Fri, Jul 22, 2016 at 12:40 PM Manlio Perillo <manlio....@gmail.com> wrote:

> I really don't see valid reasons to not call Sync.

There are valid use cases where WriteFile produces a transitive file with no requirements whatsoever to reach any storage above the kernel caches. Performance in this cases would be [badly] harmed.

--

-j

[Ian Lance Taylor]

On Fri, Jul 22, 2016 at 3:40 AM, Manlio Perillo
<manlio....@gmail.com> wrote:
>
> I think your reasoning is wrong.
> WriteFile is a "self contained" function; it does not return the handle to
> the file, so it can not be used inside a transaction.
> I really don't see valid reasons to not call Sync.

WriteFile is a convenience function that can be broken down into a
series of steps. Sync is a special purpose operation. Normal file
I/O does not call Sync at any point. It would be very surprising if
WriteFile called Sync. If you want to do an operation like WriteFile
and call Sync, then break it down to the component steps yourself,
rather than using WriteFile, and call Sync yourself.

Ian

[Richard Boyer]

I made a simple library for personal use awhile ago (https://github.com/rboyer/safeio) that tries to sequence posix atomic operations and fsync operations on file and directories for those times when you just need:

1. create temp file
2. write to temp file
3. fsync and close temp file
4. rename temp file to final destination name
5. fsync parent directory inode

This library is to be used sparingly, because as other folks have pointed out, sometimes the kernel knows best (but sometimes it doesn't).