Improving the accuracy of os.Chtimes

[Nick Craig-Wood]

I am making a patch to improve the accuracy of os.Chtimes.

I've been writing some code which involves syncing files (like rsync)
and it became apparent that under Linux I could read modification
times (os.Lstat) with nanosecond precision but only write them with
microsecond precision. This difference in precision is rather annoying
when trying to discover whether files need syncing or not!

I made a patch for syscall and os which increases the accuracy of of
os.Chtimes for Linux and Windows. This involved exposing the
utimensat system call under Linux and a bit of extra code under
Windows. I decided not to expose the "at" bit of the system call as
it is impossible to replicate under Windows, so the patch adds
syscall.Utimens() to all architectures.

utimensat was added to Linux in 2.6.22 (Released, 8 July 2007) according
to the man page. Is there a policy for what kernel version go supports?
Or does the implementation need to fall back to the microsecond accurate
one?

Unfortunately Darwin doesn't seem to have a utimensat system call that
I could find so I couldn't implement it there. The BSDs do, but since
they share their syscall implementation with Darwin I couldn't figure
out how to define a syscall for *BSD and not Darwin, so any hints there
would be appreciated!

In the process I implemented the missing methods for Timespec under
windows which I needed which just happened to round out the Timespec API
for all platforms.

Let me know if you think I'm approaching this the right way and
whether I should submit a patch for review!

Thanks

Nick

------------------------------------------------------------

Test code: http://play.golang.org/p/1xnGuYOi4b

Linux Before (1000 ns precision)

$ ./utimetest.linux.before z
Setting mtime 1344937903123456789: 2012-08-14 10:51:43.123456789 +0100 BST
Reading mtime 1344937903123457000: 2012-08-14 10:51:43.123457 +0100 BST

Linux After (1 ns precision)

$ ./utimetest.linux.after z
Setting mtime 1344937903123456789: 2012-08-14 10:51:43.123456789 +0100 BST
Reading mtime 1344937903123456789: 2012-08-14 10:51:43.123456789 +0100 BST

Windows Before (1000 ns precision)

X:\>utimetest.windows.before.exe c:\Test.txt
Setting mtime 1344937903123456789: 2012-08-14 10:51:43.123456789 +0100 GMTDT
Reading mtime 1344937903123456000: 2012-08-14 10:51:43.123456 +0100 GMTDT

Windows After (100 ns precision)

X:\>utimetest.windows.after.exe c:\Test.txt
Setting mtime 1344937903123456789: 2012-08-14 10:51:43.123456789 +0100 GMTDT
Reading mtime 1344937903123456700: 2012-08-14 10:51:43.1234567 +0100 GMTDT

------------------------------------------------------------

# Checking API compatibility.
Go version is "go1.0.3", ignoring -next /home/ncw/Code/go/api/next.txt
-pkg syscall (darwin-386), func NsecToTimespec(int64) Timespec
-pkg syscall (darwin-386), func TimespecToNsec(Timespec) int64
-pkg syscall (darwin-386-cgo), func NsecToTimespec(int64) Timespec
-pkg syscall (darwin-386-cgo), func TimespecToNsec(Timespec) int64
-pkg syscall (darwin-amd64), func NsecToTimespec(int64) Timespec
-pkg syscall (darwin-amd64), func TimespecToNsec(Timespec) int64
-pkg syscall (darwin-amd64-cgo), func NsecToTimespec(int64) Timespec
-pkg syscall (darwin-amd64-cgo), func TimespecToNsec(Timespec) int64
-pkg syscall (freebsd-386), func NsecToTimespec(int64) Timespec
-pkg syscall (freebsd-386), func TimespecToNsec(Timespec) int64
-pkg syscall (freebsd-amd64), func NsecToTimespec(int64) Timespec
-pkg syscall (freebsd-amd64), func TimespecToNsec(Timespec) int64
-pkg syscall (linux-386), func NsecToTimespec(int64) Timespec
-pkg syscall (linux-386), func TimespecToNsec(Timespec) int64
-pkg syscall (linux-386-cgo), func NsecToTimespec(int64) Timespec
-pkg syscall (linux-386-cgo), func TimespecToNsec(Timespec) int64
-pkg syscall (linux-amd64), func NsecToTimespec(int64) Timespec
-pkg syscall (linux-amd64), func TimespecToNsec(Timespec) int64
-pkg syscall (linux-amd64-cgo), func NsecToTimespec(int64) Timespec
-pkg syscall (linux-amd64-cgo), func TimespecToNsec(Timespec) int64
-pkg syscall (linux-arm), func NsecToTimespec(int64) Timespec
-pkg syscall (linux-arm), func TimespecToNsec(Timespec) int64
+pkg syscall, const ImplementsUtimens bool
+pkg syscall, func NsecToTimespec(int64) Timespec
+pkg syscall, func TimespecToNsec(Timespec) int64
+pkg syscall, func Utimens(string, []Timespec) error


--
Nick Craig-Wood <ni...@craig-wood.com> -- http://www.craig-wood.com/nick

[Ian Lance Taylor]

On Mon, Dec 3, 2012 at 1:36 PM, Nick Craig-Wood <ni...@craig-wood.com> wrote:
>
> utimensat was added to Linux in 2.6.22 (Released, 8 July 2007) according
> to the man page. Is there a policy for what kernel version go supports?
> Or does the implementation need to fall back to the microsecond accurate
> one?

We require 2.6 but I don't know if we require a specific version
within 2.6. We have a fallback for epoll_create1, which was added in
2.6.27.

Since this is specialized functionality and most people should have a
kernel that supports it, I think it's OK to start out assuming that it
exists and add the fallback later if it seems necessary.
(epoll_create1 is not specialized--it's used by almost every Go
program.)

Ian

[Rémy Oudompheng]

On 2012/12/4 Ian Lance Taylor <ia...@google.com> wrote:
> On Mon, Dec 3, 2012 at 1:36 PM, Nick Craig-Wood <ni...@craig-wood.com> wrote:
>>
>> utimensat was added to Linux in 2.6.22 (Released, 8 July 2007) according
>> to the man page. Is there a policy for what kernel version go supports?
>> Or does the implementation need to fall back to the microsecond accurate
>> one?
>
> We require 2.6 but I don't know if we require a specific version
> within 2.6. We have a fallback for epoll_create1, which was added in
> 2.6.27.

The documentation at http://golang.org/doc/install says we require
2.6.23. Go is known not to work on RHEL5 due to O_CLOEXEC issues among
other things.

Rémy.

>
> Since this is specialized functionality and most people should have a
> kernel that supports it, I think it's OK to start out assuming that it
> exists and add the fallback later if it seems necessary.
> (epoll_create1 is not specialized--it's used by almost every Go
> program.)
>
> Ian
>

> --
>
>

[Andrew Gallant]

> The documentation athttp://golang.org/doc/installsays we require

> 2.6.23. Go is known not to work on RHEL5 due to O_CLOEXEC issues among
> other things.

I don't understand O_CLOEXEC completely, but I've been running Go on
RHEL5 (kernel 2.6.18) for quite some time. Right now I have
+86c7b6d67466 installed (lagging tip by a bit). Although, I've noticed
sub-par performance in terms of CPU utilization.

- Andrew

[Andy Balholm]

I'm running Go on ClearOS 5, which is essentially RHEL 5. I've noticed that it doesn't pass all the tests when I build it, but it seems to work OK. As I recall, the tests that don't pass have something to do with os.Exec.

[Dave Cheney]

> I'm running Go on ClearOS 5, which is essentially RHEL 5. I've noticed that
> it doesn't pass all the tests when I build it, but it seems to work OK. As I
> recall, the tests that don't pass have something to do with os.Exec.

Yup that is the cause, see earlier in the thread