Re: [go-nuts] Temporary files in go

[Ian Lance Taylor]

On Thu, Oct 11, 2018 at 4:48 PM, Greg Saylor <greg.sa...@gmail.com> wrote:
>
> In other programming languages (this is specific to Linux/Unix systems), in
> the past to ensure security in the even of a program crash, we would do
> something like:
>
> 1. Create a temporary file and squirrel away the file handle
> 2. Unlink the temporary file by name
> 3. Various functions would write stuff to the file
> 4. If the programs completes to some successful state, create a hardlink to
> the file handle with the final filename
>
> I'm finding this very difficult to do in Go, as there does not seem to be a
> way to do #4. And this is a very important consideration for this piece of
> the system.
>
> For example, os.Rename takes filenames as the old/new filename.
>
> I figured looking in that code might reveal something lower level that could
> be used, which lead me to syscal_linuxl.Rename()
>
> That lead me to syscall_linux.RenameAt()
>
> Which led me to zsyscall_linux_amd64.go.
>
> .. at this point I got pretty lost on how to do any of this. _AT_FDCWD and
> fishing around in what appears to be some pretty low-level internals of
> Go...
>
> Is there some way to achieve this or a way that can ensure these files are
> always removed if the program is kill -9'd, terminates from a panic, etc.

Can you show us how you do this in C?

I expect that will point toward how to do it in Go.

Ian

[Justin Israel]

Probably linkat(2) ? 

http://man7.org/linux/man-pages/man2/linkat.2.html

I expect that will point toward how to do it in Go.

Ian

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sam Whited]

On Thu, Oct 11, 2018, at 18:48, Greg Saylor wrote:
> 1. Create a temporary file and squirrel away the file handle
> 2. Unlink the temporary file by name
> 3. Various functions would write stuff to the file
> 4. If the programs completes to some successful state, create a hardlink to
> the file handle with the final filename

The golang.org/x/sys/unix [1] package gives you a lower level interface that might be familiar to you if you're used to using C. Something like this is probably similar to what you were doing (though you'll need /proc mounted, I'm not sure if there's a better way to do that):

oldfd, err := unix.Open(".", unix.O_TMPFILE|unix.O_RDWR, unix.S_IRUSR|unix.S_IWUSR)
if err != nil {
panic(err)
}

f := os.NewFile(uintptr(oldfd), "myoldfile")
_, err = f.Write([]byte("Hello, World"))
if err != nil {
panic(err)
}

dir, err := os.Open(".")
if err != nil {
panic(err)
}
procdir, err := os.Open("/proc/self/fd/")
if err != nil {
panic(err)
}

err = unix.Linkat(int(procdir.Fd()), strconv.Itoa(oldfd), int(dir.Fd()), "mynewfile", unix.AT_SYMLINK_FOLLOW)
if err != nil {
panic(err)
}

—Sam

[1]: https://godoc.org/golang.org/x/sys/unix

[robert engels]

I think a more “standard” and certainly portable way to do this is with a tmp file, and a rename/move at the end - along with a clean-up of tmp at application start (if worried about crashes).

Using the proper file and process permissions on the tmp file - since you need these to protect the final file anyway.

[Greg Saylor]

If I remember correctly, it would be something like this:

fd = open("/tmp", O_TMPFILE | O_RDWR, 0600);

linkat(fd, "", AT_FDCWD, "/tmp/test", AT_EMPTY_PATH);

This is pretty specific to OS/kernel version and quite possibly the filesystem too.  This could be entirely too much of an edge case to be reasonably done.

- Greg

[Beoran]

linkat() and openat() are posix 2008 standard functions so I wouldn't call this uncommon. As such they are useful to have in Go in one way or the other. The x/unix package is fine for that, though.

[Robert Engels]

Which is not Windows, which is giving up a lot of options, and is important to some people, where as just using tmp files is ok. I am not even sure if windows supports deleting an open file yet.

> On Oct 12, 2018, at 3:09 AM, Beoran <beo...@gmail.com> wrote:
>
> linkat() and openat() are posix 2008 standard functions so I wouldn't call this uncommon. As such they are useful to have in Go in one way or the other. The x/unix package is fine for that, though.
>

[Greg Saylor]

[Ian Lance Taylor]

Seems like you can do this today in Go using os.OpenFile,
(*os.File).Fd, and unix.Linkat from the golang.org/x/sys/unix package.

Ian

[Manuel Amador (Rudd-O)]

On 12/10/2018 12.35, Robert Engels wrote:
> Which is not Windows, which is giving up a lot of options, and is important to some people, where as just using tmp files is ok. I am not even sure if windows supports deleting an open file yet.

I believe the default behavior is no it does not.

--
Rudd-O
http://rudd-o.com/