Use monotonic clock for timers

[jays...@gmail.com]

I would like Go to use a monotonic clock for its timers. This will make applications more robust when the system's calendar time changes. https://code.google.com/p/go/issues/detail?id=6007

I have mocked up a potential implementation (linux_amd64 for now) and would like feedback.

https://github.com/jayschwa/golang/compare/master...monotonic

We can get most of the way there by making time.nano() wrap runtime.nanotime(), and change the latter to use a monotonic clock instead. I'm less sure how to handle net's SetDeadline(), which takes a calendar time instead of a duration. In the mock, I expose runtime.nanotime() via time.SysNano() so that the net package can convert the calendar time to a value that will make sense to the runtime. This seemed like the simplest way to go about it, but I'm not sure if it's the best way.

- Jay

[Dmitry Vyukov]

Hi!

I think that generally this is a move in the right direction. But
before we take any actual steps, I would like to understand larger
picture and see final destination.
Where exactly do we use mono time? Where exactly do we use real time?
How semantics of existing APIs are affected? What new APIs do we need
to add? Do we provide an ability to sleep till some real abs time?

[Jay Weisskopf]

On Mon, Jan 13, 2014 at 3:08 AM, Dmitry Vyukov <dvy...@google.com> wrote:

Where exactly do we use mono time?

Instances of func(time.Duration) should be driven by mono time underneath. Examples:

time.After(), Sleep(), Tick(), Ticker, Timer

 

Where exactly do we use real time?

Only places that deal with time.Time. For scheduling purposes, I think net.Conn.SetDeadline() functions are the only place where calendar time is used. More on that below.

 

How semantics of existing APIs are affected?

func(time.Duration) will behave with less surprises during calendar time changes.

net.Conn.SetDeadline(time.Time) is the main sore point of this. I think it should use mono time underneath, which changes semantics. I think a case for this can be made though. What do we want to happen here:

SetDeadline(time.Now().Add(30*time.Seconds))

Let's say 15 seconds after this call, system calendar time jumps backwards by an hour. If we keep calendar time under the hood for scheduling, deadline suddenly becomes ~1 hour instead of 30 seconds. This is almost certainly not what the user wants. User really wants deadline to be in 30 seconds, regardless of calendar time. I think time.Time was not the best choice for this API and that time.Duration would have been better, but it's part of the contract now.

 

What new APIs do we need to add?

Preferably none. I think the existing API is pretty good. I expose mono time via time.SysNano() in my prototype. That was mostly for benefit of changes in net package. Some users might appreciate it for obscure cases, but it won't bother me to leave it out.

 

Do we provide an ability to sleep till some real abs time?

No, I think that will muddle the API. User can do this easily enough:

for time.Now().Before(midnight) {

    time.Sleep(10*time.Minutes)

}

[Maxim Khitrov]

On Mon, Jan 13, 2014 at 3:08 AM, Dmitry Vyukov <dvy...@google.com> wrote:

Monotonic time (CLOCK_MONOTONIC, QueryPerformanceCounter, etc.) should
have been used in any place that didn't care about the meaning of the
absolute time value. Does SetDeadline care about what year it is?
Maybe for some applications, but usually it just needs a point in time
and a concept of "before" and "after". It also needs to know that
time.Now().Add(5 * time.Second) will be 5 seconds from now instead of
1 year and 5 seconds if the admin changes the system clock (yes, some
APIs handle this case, but the general point remains). I'm not sure if
there is anything in the standard library that needs a "real" time
instead of a monotonic clock with an arbitrary reference point.

One idea I'd like to throw out is using Time.loc to identify Time
values that are "located" in another frame of reference. Define a
time.Monotonic location, which would identify time values that were
obtained from a monotonic clock instead of the default system clock. A
function like time.SysNano() would return Time values with loc set to
time.Monotonic without converting Time.{sec,nsec} to UTC. SetDeadline
and friends could then decide which clock to use based on this
information (t.Location() == time.Monotonic ? CLOCK_MONOTONIC :
CLOCK_REALTIME).

Methods like time.Sub() work as before, provided that both Time values
came from the same clock. That is just common sense. time.In() could
be used to convert time values obtained from different clocks,
provided that it can figure out the difference between those clocks
with some reasonable accuracy.

Thoughts?

[Kyle Lemons]

I've thought about how to do the APIs a number of times (running a time service is my day job, so I have some familiarity with the topic and thought I might be able to come up with a solution) but haven't come up with anything suitable.  One wrinkle with your proposal is that it complicates or makes impossible conversions between that monotonic time location and other locations.  In other words, the following comparison might not be true: t.In(time.UTC).Equal(t.In(time.UTC)) if t.Location() == time.Monotonic and the system clock changes before the first call to In and the second call to In.  That may not be a problem, but it's an invariant that may be assumed present with the current library by some code.

 

Methods like time.Sub() work as before, provided that both Time values
came from the same clock. That is just common sense. time.In() could
be used to convert time values obtained from different clocks,
provided that it can figure out the difference between those clocks
with some reasonable accuracy.

Thoughts?

--

---
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Maxim Khitrov]

This is like performing mathematically identical operations on two
floats and then getting a != b. When you perform just one
t.In(time.UTC) operation, for t obtained from a different time source,
you're mapping one timeline onto another. It is never possible to do
this exactly. At some point, you'll have to do something like:

t1 = now(clock1)
t2 = now(clock2)
diff = t2 - t1

But t1 and t2 already do not refer to the same moment on the universal
timeline, even if it's just by a few nanoseconds. There is an inherent
error in diff, so both of your t.In(time.UTC) operations are inexact.
You can't take the result of two inexact operations and do a strict
comparison on them. It's a problem that will affect any solution that
introduces a second clock into the picture.

Furthermore, I'm not sure that it makes sense for time.In() to
recompute the difference between the clocks on each invocation. Doing
this once and caching the value may be sufficient. This at least
guarantees 1:1 mapping between the timelines. When you recompute the
difference you end up with times from one clock having zero or
multiple mappings to times from the other clock. Neither solution is
perfect, but caching is more efficient and will keep the
t.In(time.UTC).Equal(t.In(time.UTC)) invariant.

[Kyle Lemons]

To be clear, I'm talking about the same operation on the same value getting a different result, not on two separate values.  Currently, timezone conversion is idempotent and deterministic.

 

When you perform just one
t.In(time.UTC) operation, for t obtained from a different time source,
you're mapping one timeline onto another. It is never possible to do
this exactly. At some point, you'll have to do something like:

t1 = now(clock1)
t2 = now(clock2)
diff = t2 - t1

I'm talking about

t := mononow()

if t.In(tz) != t.In(tz) { panic("WTF") }

But t1 and t2 already do not refer to the same moment on the universal
timeline, even if it's just by a few nanoseconds. There is an inherent
error in diff, so both of your t.In(time.UTC) operations are inexact.
You can't take the result of two inexact operations and do a strict
comparison on them. It's a problem that will affect any solution that
introduces a second clock into the picture.

Furthermore, I'm not sure that it makes sense for time.In() to
recompute the difference between the clocks on each invocation.

After thinking some more, I think we were both incorrect in subtle ways.  Consider:

m := mono.Now()

u := time.Now()

println(m.In(time.UTC))

println(u.In(time.UTC))

mono.Sleep(10*time.Minute) // system clock stepped back by 1h during this

 

println(m.In(time.UTC))

println(u.In(time.UTC))

I would expect these to print different times.  How you do that is tricky... you can't really cache the difference between the two timezones globally because the offset is the thing that would change.  You would almost have to store the reference and monotonic timestamps both when calling mono.Now so you can translate the deltas over. 

[Jay Weisskopf]

So far, I don't care for the idea of storing monotonic values in time.Time. It's not clear to me what the big win for doing that is. Most of Time's methods won't make sense with monotonic values. The few that do, Add and Sub (are there more?), can be done with plain old integers. For now, I think the goal should be leaving things the way they are as much as possible, and just getting time.Foo(time.Duration) to work through calendar time changes.

[Dmitry Vyukov]

On Mon, Jan 13, 2014 at 7:35 PM, Jay Weisskopf <jays...@gmail.com> wrote:
> On Mon, Jan 13, 2014 at 3:08 AM, Dmitry Vyukov <dvy...@google.com> wrote:
>>
>> Where exactly do we use mono time?
>
>
> Instances of func(time.Duration) should be driven by mono time underneath.
> Examples:
>
> time.After(), Sleep(), Tick(), Ticker, Timer
>
>>
>> Where exactly do we use real time?
>
>
> Only places that deal with time.Time. For scheduling purposes, I think
> net.Conn.SetDeadline() functions are the only place where calendar time is
> used. More on that below.
>
>>
>> How semantics of existing APIs are affected?
>
>
> func(time.Duration) will behave with less surprises during calendar time
> changes.
>
> net.Conn.SetDeadline(time.Time) is the main sore point of this. I think it
> should use mono time underneath, which changes semantics. I think a case for
> this can be made though. What do we want to happen here:
>
> SetDeadline(time.Now().Add(30*time.Seconds))
>
> Let's say 15 seconds after this call, system calendar time jumps backwards
> by an hour. If we keep calendar time under the hood for scheduling, deadline
> suddenly becomes ~1 hour instead of 30 seconds. This is almost certainly not
> what the user wants. User really wants deadline to be in 30 seconds,
> regardless of calendar time. I think time.Time was not the best choice for
> this API and that time.Duration would have been better, but it's part of the
> contract now.

I agree that this is a bad API. Probably we can add
SetTimeout(time.Duration) and deprecate SetDeadline() since it gives
you timeout with random duration.

>> What new APIs do we need to add?
>
>
> Preferably none. I think the existing API is pretty good. I expose mono time
> via time.SysNano() in my prototype. That was mostly for benefit of changes
> in net package. Some users might appreciate it for obscure cases, but it
> won't bother me to leave it out.
>
>>
>> Do we provide an ability to sleep till some real abs time?
>
>
> No, I think that will muddle the API. User can do this easily enough:
>
> for time.Now().Before(midnight) {
> time.Sleep(10*time.Minutes)
> }

Do you know how different OSes handle sleeps and timed blocking? Don't
they convert durations back to abstime?
At least in netbsd runtime code I see mention of abstime relative to
some system clock with id 0.

[Dmitry Vyukov]

On Mon, Jan 13, 2014 at 7:35 PM, Jay Weisskopf <jays...@gmail.com> wrote:

My concern here is that this is inherently tricky, so users will do it
incorrectly, or probably even don't realize the problem at all (all
time.Sleep function take Duration -- so just calculate Duration
between Now and Then and Sleep).

[Jay Weisskopf]

On Tue, Jan 14, 2014 at 2:41 AM, Dmitry Vyukov <dvy...@google.com> wrote:

My concern here is that this is inherently tricky, so users will do it
incorrectly, or probably even don't realize the problem at all (all
time.Sleep function take Duration -- so just calculate Duration
between Now and Then and Sleep).

time.SleepUntil(time.Time) could conceivably be added, but I'm not really in favor of it.

[Dmitry Vyukov]

We can always add APIs later if needed.

In general, I support what you want to do. What we do now is
completely unreasonable.

Kyle, what do you think about just solving the smaller problem with
time.Sleep and friends, w/o trying to solve the bigger problem of
general APIs for various types of clocks for now?

[Brad Fitzpatrick]

No. We're not going back to that.

That's how the old pre-Go 1 API was, but it was too ambiguous what time.Duration meant: from when to when? Receiving a single byte? Finishing the Read call? Reading len(original Read's bytes)?  A time.Time deadline (ignoring clock jitter) is very explicit.

[Dmitry Vyukov]

That was only a problem of naming/documentation. But now we have a
broken beyond repair API.

[Maxim Khitrov]

I understood what you meant, but see below.

>> When you perform just one
>> t.In(time.UTC) operation, for t obtained from a different time source,
>> you're mapping one timeline onto another. It is never possible to do
>> this exactly. At some point, you'll have to do something like:
>>
>> t1 = now(clock1)
>> t2 = now(clock2)
>> diff = t2 - t1
>
>
> I'm talking about
>
> t := mononow()
> if t.In(tz) != t.In(tz) { panic("WTF") }

I think that even with one clock, a program that relies on such
comparisons is technically broken. Timezone information comes from a
database. Even for UTC you have leap seconds. Is there anything
stopping a Go program from updating its timezone database while it is
running? How about asking the OS to perform the conversion? How about
a program saving the output of t.In(tz) to a file, restarting, doing
the same calculation, and then comparing the result with the file
contents?

Even with one clock, without any admin intervention, the output of any
t.In(tz) operation should be considered a "best guess given the
information that we have at the moment." Remember that t could be 10
years in the future and the answer that you get is virtually
guaranteed to be wrong.

With a second clock, there is no fix for the fundamental problem that
converting from one timeline to another is an inexact operation. It is
impossible to know what values two different clocks had at the exact
same moment, so any time you compute that difference you'll get a
small error. If you compute that difference only once, then you can
keep the current t.In(tz) == t.In(tz) behavior. If you compute it each
time you call t.In(tz), even for the same t and tz, you should expect
a different result.

>> But t1 and t2 already do not refer to the same moment on the universal
>> timeline, even if it's just by a few nanoseconds. There is an inherent
>> error in diff, so both of your t.In(time.UTC) operations are inexact.
>> You can't take the result of two inexact operations and do a strict
>> comparison on them. It's a problem that will affect any solution that
>> introduces a second clock into the picture.
>>
>> Furthermore, I'm not sure that it makes sense for time.In() to
>> recompute the difference between the clocks on each invocation.
>
>
> After thinking some more, I think we were both incorrect in subtle ways.
> Consider:
>
> m := mono.Now()
> u := time.Now()
>
> println(m.In(time.UTC))
> println(u.In(time.UTC))
>
> mono.Sleep(10*time.Minute) // system clock stepped back by 1h during this
>
> println(m.In(time.UTC))
> println(u.In(time.UTC))
>
> I would expect these to print different times. How you do that is tricky...
> you can't really cache the difference between the two timezones globally
> because the offset is the thing that would change. You would almost have to
> store the reference and monotonic timestamps both when calling mono.Now so
> you can translate the deltas over.

That still won't be enough. Consider what happens when the system sleeps:

m1 := mono.Now()
u1 := time.Now()

// OS goes to sleep, CLOCK_MONOTONIC stops running

m3 := mono.Now()
u3 := time.Now()

m2 := m1.Add(m3.Sub(m1) / 2)
u2 := m2.In(time.UTC) // When is this?

In this example, u2 could be before the system went to sleep, while it
was sleeping, or after it woke up. There is no way to know and there
is no right answer to m2.In(time.UTC). As far as the monotonic
timeline is concerned, the sleep period never happened. We can either
agree on some reasonable mapping from one clock to another or forbid
the operation altogether.

This is why I think calculating the difference once and caching the
result globally is an acceptable solution. You'll have errors either
way, but this is the only method that will give you a 1:1 mapping and
it has minimal overhead.

The user is always welcome to perform their own conversion by getting
the current mono.Now() and time.Now() values when needed.

[Maxim Khitrov]

On Mon, Jan 13, 2014 at 11:09 PM, Jay Weisskopf <jays...@gmail.com> wrote:
> So far, I don't care for the idea of storing monotonic values in time.Time.
> It's not clear to me what the big win for doing that is.

As our discussion demonstrates, converting values from one clock to
another is problematic and should be avoided unless absolutely
necessary. The conversion that your code is doing may create even more
unexpected behavior.

Ideally, functions like SetDeadline should have been based on
monotonic time to begin with. My proposal attempts to solve this by
allowing time.Time to represent values obtained from other clocks,
which lets existing functions select the appropriate clock to use
without any conversions.

> Most of Time's
> methods won't make sense with monotonic values. The few that do, Add and Sub
> (are there more?), can be done with plain old integers.

I counted at least 24 / 38 methods that could be useful, but this
isn't the main point. If we want to introduce other clocks, we'll need
to define new types and methods for those clocks, convert values from
one clock to another, or allow time.Time to represent values from
other clocks using the existing Location framework. I think the third
option is the cleanest and I can't come up with a backward
compatibility argument against it.

[Ian Lance Taylor]

On Tue, Jan 14, 2014 at 3:10 AM, Dmitry Vyukov <dvy...@google.com> wrote:
>
> That was only a problem of naming/documentation. But now we have a
> broken beyond repair API.

I think you are exaggerating. SetDeadline does exactly what it says,
and it works well 99.999% of the time.

As Brad said, we used to have a timeout instead of a deadline, and it
did not work. It's not a matter of documentation. It's inherently
unclear. That is the wrong direction.

What I think you are after is a different sort of deadline: a deadline
that is N nanoseconds from now, independent of the current time. We
shouldn't confuse such a thing with the existing time.Time or
time.Duration types, both of which remain meaningful.

As far as I know the places where we need monotonic time are
net.Conn.Set{,Read,Write}Deadline, time.NewTimer, time.After,
time.AfterFunc, time.Sleep. Are there any others? I think we should
leave the existing functions and definition of time alone, and add
monotonic variants of those functions.

People who want monotonic time are never going to care about the
actual time in the time.Time sense. They are only going to care about
having something happen N nanoseconds from now. That can always be
expressed as a time.Duration. So it seems to me that we want
net.Conn.SetMonotonicDeadlineAfter(d time.Duration), which sets the
deadline to d nanoseconds from now (still a deadline, not a timeout).
And we want time.NewMonotonicTimer, time.MonotonicAfter, etc.

I think the API has to be something along those lines, perhaps with
better names. I don't know how to implement it.

Ian

[Jay Weisskopf]

It seems like there's consensus that at least time.Ticker and company should use a monotonic clock underneath. As for net.Conn.SetDeadline(), API disagreement aside (it's part of the contract - there's little point in debating it until Go 2), is there agreement that it would be good to translate values to a monotonic clock under the hood to preserve implied duration through clock jitter? Brad?

On Tue, Jan 14, 2014 at 8:38 AM, Maxim Khitrov <m...@mxcrypt.com> wrote:

As our discussion demonstrates, converting values from one clock to
another is problematic and should be avoided unless absolutely
necessary. The conversion that your code is doing may create even more
unexpected behavior. 

Ideally, functions like SetDeadline should have been based on
monotonic time to begin with. My proposal attempts to solve this by
allowing time.Time to represent values obtained from other clocks,
which lets existing functions select the appropriate clock to use
without any conversions.

The reason my code does the translation is so that everything being put into runtimeTimer has the same point of reference, which is runtime.nanotime(). Making it work with different clocks will require adding additional state to the struct and retooling more runtime code than I'd prefer to do.

[Russ Cox]

On Tue, Jan 14, 2014 at 10:15 AM, Jay Weisskopf <jays...@gmail.com> wrote:

It seems like there's consensus that at least time.Ticker and company should use a monotonic clock underneath.

I disagree. I think you should use a computer that can keep time.

Russ

[Dmitry Vyukov]

On Tue, Jan 14, 2014 at 6:50 PM, Ian Lance Taylor <ia...@golang.org> wrote:
> On Tue, Jan 14, 2014 at 3:10 AM, Dmitry Vyukov <dvy...@google.com> wrote:
>>
>> That was only a problem of naming/documentation. But now we have a
>> broken beyond repair API.
>
> I think you are exaggerating. SetDeadline does exactly what it says,
> and it works well 99.999% of the time.
>
> As Brad said, we used to have a timeout instead of a deadline, and it
> did not work. It's not a matter of documentation. It's inherently
> unclear. That is the wrong direction.

If it would be named along the lines of
MakeSoThatAllOperationsAbortAfterMonotonicTimeAdvancesByTheDuration()
it would be clear.
Current SetDeadline works 100% of time in some setups, and works from
time to time in other setups. And it does not do what ~100% of users
want -- I do not mean time.Time vs time.Duration, I mean specifying
point in time when operations must abort against system time.

> What I think you are after is a different sort of deadline: a deadline
> that is N nanoseconds from now, independent of the current time.

Yes, and that's what 99.9% of users want.

> We
> shouldn't confuse such a thing with the existing time.Time or
> time.Duration types, both of which remain meaningful.
>
> As far as I know the places where we need monotonic time are
> net.Conn.Set{,Read,Write}Deadline, time.NewTimer, time.After,
> time.AfterFunc, time.Sleep. Are there any others? I think we should
> leave the existing functions and definition of time alone, and add
> monotonic variants of those functions.
>
> People who want monotonic time are never going to care about the
> actual time in the time.Time sense. They are only going to care about
> having something happen N nanoseconds from now. That can always be
> expressed as a time.Duration. So it seems to me that we want
> net.Conn.SetMonotonicDeadlineAfter(d time.Duration), which sets the
> deadline to d nanoseconds from now (still a deadline, not a timeout).
> And we want time.NewMonotonicTimer, time.MonotonicAfter, etc.
>
> I think the API has to be something along those lines, perhaps with
> better names. I don't know how to implement it.

There are 3 problems with this proposal:
1. Current APIs does not work and never worked. And it's unclear to me
how to fix them.
2. Current APIs are needed in very-very few very-very specific cases.
It may be below threshold for inclusion into std lib.
3. 99% of Go users think that time.Timer(time.Second) works against
monotonic time, and gives you ticks "every second" regardless of
system clock changes.
All that suggests to me that current APIs (I mean time.Sleep/Ticker
and friends) needs to changed to use monotonic time.

[Dmitry Vyukov]

Why don't you want Go support to huge amount of computers that change time?
And why do you want to keep the meaningless function that sleeps for a
random amount of time?

[Josh Bleecher Snyder]

>> It seems like there's consensus that at least time.Ticker and company
>> should use a monotonic clock underneath.
>
>
> I disagree. I think you should use a computer that can keep time.

Unfortunately, this is not always a viable option, particularly with
embedded systems.

-josh

[Russ Cox]

I'll put it differently then.

A proposal to improve the implementation that does not modify any existing APIs and that does not significantly complicate the implementation would be fine.

Trying to model weird new clocks with API modifications is a waste of time.

Russ

[Kyle Lemons]

Also from what I have seen it looks like both GCE and EC2 instances have (different) problems keeping time.

I'm also not entirely sure what the current code will do in the perfectly ordinary case of leap seconds.  While our servers at Google have the luxury of having this discontinuity hidden from them, its not clear to me that every computer without a leap smear should be considered to be improperly keeping time.

[Kyle Lemons]

I got tripped up here when I went looking for ways to implement the correct behavior on our various platforms.  I wasn't able to find anything on Linux or Darwin, though I think I recall having found the right call on windows.

If we aren't going to fix it for the net deadlines, I don't think it can be called a fix.

[Ian Lance Taylor]

On Tue, Jan 14, 2014 at 7:46 AM, Dmitry Vyukov <dvy...@google.com> wrote:
>
> There are 3 problems with this proposal:
> 1. Current APIs does not work and never worked. And it's unclear to me
> how to fix them.

I can't agree that there is anything to fix for SetDeadline. And
experience shows that SetTimeout doesn't work. SetDeadlineAfter would
work for me.

But on further thought you're probably right about time.Sleep,
time.NewTimer, time.After, and time.AfterFunc. When it makes a
difference, those should operate on monotonic time. I guess that
time.NewTimer and time.After should continue to send system time on
the channel.

Ian

[Jay Weisskopf]

On Tue, Jan 14, 2014 at 12:02 PM, Kyle Lemons <kev...@google.com> wrote:

I got tripped up here when I went looking for ways to implement the correct behavior on our various platforms.  I wasn't able to find anything on Linux or Darwin, though I think I recall having found the right call on windows.

If we aren't going to fix it for the net deadlines, I don't think it can be called a fix.

SetDeadline and time.Duration's functions all use runtimeTimer underneath. The first step is to ensure that all inputs to runtimeTimer are relative to the same clock - monotonic or not. In my prototype, SetDeadline's time.Time is translated to be in terms of runtime.nanotime() before being passed in to the runtimeTimer.

Once all input to runtimeTimer is relative to one reference clock - runtime.nanotime() - the clock can be updated to monotonic on a platform-by-platform basis.

[Brad Fitzpatrick]

On Tue, Jan 14, 2014 at 7:15 AM, Jay Weisskopf <jays...@gmail.com> wrote:

It seems like there's consensus that at least time.Ticker and company should use a monotonic clock underneath. As for net.Conn.SetDeadline(), API disagreement aside (it's part of the contract - there's little point in debating it until Go 2), is there agreement that it would be good to translate values to a monotonic clock under the hood to preserve implied duration through clock jitter? Brad?

SetDeadline being converted to a duration + monotonic clock behind the scenes SGTM.  Nobody wants to stop waiting for activity on a TCP connection when the sun is at a particular place in the sky.

[Maxim Khitrov]

I think you're giving some special meaning to the absolute value of
the system clock. Let's think about the following example. Some
country decides to restart their year count from 1. Instead of using
UTC, they set their system clocks to 0001-01-01. Once the timezone
database gets updated, is this going to be a problem for time.Time?
Will Go be able to figure out that 2014-01-14 for us is 0001-01-14 for
them?

If so, how is this situation different from obtaining one time.Time
instance from the system clock and another from the monotonic clock?
The monotonic clock may have started counting up from 0 when the
system booted. That's the same as the local time for the country
that's now counting years from 1. All you need is a Location that
specifies how to map from one to the other.

[Oleku Konko]

I agree , I think everything should be left they way it it and introduce time.Monitonic()  with Ticker, After etc. instead 

[Kyle Lemons]

On Tue, Jan 14, 2014 at 9:42 AM, Maxim Khitrov <m...@mxcrypt.com> wrote:

On Tue, Jan 14, 2014 at 11:51 AM, Russ Cox <r...@golang.org> wrote:
> I'll put it differently then.
>
> A proposal to improve the implementation that does not modify any existing
> APIs and that does not significantly complicate the implementation would be
> fine.
>
> Trying to model weird new clocks with API modifications is a waste of time.
>
> Russ

I think you're giving some special meaning to the absolute value of
the system clock. Let's think about the following example. Some
country decides to restart their year count from 1. Instead of using
UTC, they set their system clocks to 0001-01-01. Once the timezone
database gets updated, is this going to be a problem for time.Time?
Will Go be able to figure out that 2014-01-14 for us is 0001-01-14 for
them?

The timezone database format either can or would need to be updated to be able to express that.  As a more real-world example, there are islands on the IDL that changed from one side to the other and skipped a day.  None of this affects unix time, which is still the number of seconds since the start of 1970 UTC.  The rest is a matter of display.

 

If so, how is this situation different from obtaining one time.Time
instance from the system clock and another from the monotonic clock?
The monotonic clock may have started counting up from 0 when the
system booted. That's the same as the local time for the country
that's now counting years from 1. All you need is a Location that
specifies how to map from one to the other.

I think that even with one clock, a program that relies on such
comparisons is technically broken. Timezone information comes from a
database. Even for UTC you have leap seconds. Is there anything
stopping a Go program from updating its timezone database while it is
running? How about asking the OS to perform the conversion? How about
a program saving the output of t.In(tz) to a file, restarting, doing
the same calculation, and then comparing the result with the file
contents?

I think you're misunderstanding what .In is doing.  All it does is change the *time.Location in the time, it doesn't change the unix time stored in the time.Time, and thus it is deterministic and idempotent.  Updating the zoneinfo database wouldn't have any effect on the representation (it may change the displayed format, of course).  Making .In alter the time itself would be surprising.

[Russ Cox]

On Tue, Jan 14, 2014 at 12:42 PM, Maxim Khitrov <m...@mxcrypt.com> wrote:

I think you're giving some special meaning to the absolute value of

the system clock. Let's think about the following example. Some
country decides to restart their year count from 1. Instead of using
UTC, they set their system clocks to 0001-01-01. Once the timezone
database gets updated, is this going to be a problem for time.Time?
Will Go be able to figure out that 2014-01-14 for us is 0001-01-14 for
them?

The actual bits stored in a Go time.Time are always UTC, in a Gregorian calendar. The bits describe a specific instant in time. The Location field controls how those bits are turned into year, month, day, hour, minute, second for formatting but do not change the meaning of the actual instant. That is, Location affects methods like String or Hour but it does not affect methods like After, Before, Equal, and Sub. The "monotonic location" proposal fails precisely because it is trying to change what the bits actually mean, not just how they are printed. By breaking that orthogonality it introduces significant (and I would argue unwarranted) confusion.

Russ

[Dmitry Vyukov]

On Tue, Jan 14, 2014 at 9:02 PM, Kyle Lemons <kev...@google.com> wrote:
>
> On Jan 14, 2014 12:33 AM, "Dmitry Vyukov" <dvy...@google.com> wrote:
>>
>> On Tue, Jan 14, 2014 at 11:48 AM, Jay Weisskopf <jays...@gmail.com>
>> wrote:
>> > On Tue, Jan 14, 2014 at 2:41 AM, Dmitry Vyukov <dvy...@google.com>
>> > wrote:
>> >>
>> >> My concern here is that this is inherently tricky, so users will do it
>> >> incorrectly, or probably even don't realize the problem at all (all
>> >> time.Sleep function take Duration -- so just calculate Duration
>> >> between Now and Then and Sleep).
>> >
>> > time.SleepUntil(time.Time) could conceivably be added, but I'm not
>> > really in
>> > favor of it.
>>
>> We can always add APIs later if needed.
>>
>> In general, I support what you want to do. What we do now is
>> completely unreasonable.
>>
>> Kyle, what do you think about just solving the smaller problem with
>> time.Sleep and friends, w/o trying to solve the bigger problem of
>> general APIs for various types of clocks for now?
>
> I got tripped up here when I went looking for ways to implement the correct
> behavior on our various platforms. I wasn't able to find anything on Linux
> or Darwin, though I think I recall having found the right call on windows.

What about select() with relative timeout? Does it convert it to
system abstime internally on linux?

> If we aren't going to fix it for the net deadlines, I don't think it can be
> called a fix.

Ian already almost agreed to c.SetDeadlineAfter(d) :)

[roger peppe]

On 14 January 2014 13:09, Maxim Khitrov <m...@mxcrypt.com> wrote:
> In this example, u2 could be before the system went to sleep, while it
> was sleeping, or after it woke up.

This raises an interesting question:

If I do:

time.Sleep(60 * time.Second)

and close my laptop lid after it's been running for 30s and open it
again a day later, should the Sleep then wake up immediately or after another
30s have passed?

I can think of arguments both ways, depending on whether the Sleep
is awaiting something internal or external to the computer.

[Dmitry Vyukov]

We need to leave it as an implementation detail.
However for QoI point of view I think we need to trigger the timer
immediately, and I think that's what OSes do with sleeps.

[Kyle Lemons]

I asked a friendly kernel hacker to take a quick look, and it looks like epoll uses monotonic time.  They all seem to use ktime_get_ts(), which is monotonic (albeit in a slightly roundabout way).

[Jay Weisskopf]

First CL for interested parties:

https://codereview.appspot.com/53010043/

To be clear, this does not switch to a monotonic clock on all platforms. It's just for linux/amd64 right now. The important thing is that it makes some other stuff internally consistent. Bringing on other platforms is now just a matter of updating their respective runtime·nanotime() implementations. No APIs were harmed in the making of this patch.

- Jay

[jays...@gmail.com]

I added a monotonic implementation for Windows a couple weeks back. One reviewer is having problems with it on 32-bit hardware, which I haven't been able to reproduce using VMs. If anyone has a Windows 32-bit machine, I'd appreciate it if you took the patch for a spin and reported results.

https://codereview.appspot.com/53010043/

- Jay

[Albert Strasheim]

Hello all

On Tuesday, February 4, 2014 8:27:06 PM UTC-8, Jay Weisskopf wrote:

On Thursday, January 16, 2014 1:47:57 AM UTC-5, Jay Weisskopf wrote:

First CL for interested parties:

https://codereview.appspot.com/53010043/

I'd be interested to know why if CLOCK_MONOTONIC is more correct/better than CLOCK_MONOTONIC_RAW here?

Some discussion on Stack Overflow:

https://stackoverflow.com/questions/14270300/what-is-the-difference-between-clock-monotonic-clock-monotonic-raw

Regards

Albert

[Jay Weisskopf]

I'd be interested to know why if CLOCK_MONOTONIC is more correct/better than CLOCK_MONOTONIC_RAW here?

As mentioned in your Stack Overflow link, CLOCK_MONOTONIC is adjusted by NTP while RAW is not. NTP compares your local hardware's ticker to highly-accurate, fancy science clocks powered by unicorn dust. If your local hardware's ticker is running faster or slower than the "official" time, CLOCK_MONOTONIC should insulate you from that.

In other news, the CL was submitted at the beginning of the week. BSD updates followed shortly thereafter by others (thanks!) I think darwin and windows are the only platforms left that need their runtime.nanotime() implementations updated. I had a Windows implementation in the initial CL, but pulled it out because it wasn't working on some 32-bit hardware. If anyone wants to take a crack at it, see the CL history or e-mail me off-list and I'll give you a copy of the patch.

Thanks,

Jay

 

[davy...@gmail.com]

I see the issue here changed to Release-None

https://code.google.com/p/go/issues/detail?id=6007

is that means won't fix it in 1.3.x ?

thanks

On Monday, January 13, 2014 3:42:30 AM UTC+8, Jay Weisskopf wrote:

I would like Go to use a monotonic clock for its timers. This will make applications more robust when the system's calendar time changes. https://code.google.com/p/go/issues/detail?id=6007

I have mocked up a potential implementation (linux_amd64 for now) and would like feedback.

https://github.com/jayschwa/golang/compare/master...monotonic

We can get most of the way there by making time.nano() wrap runtime.nanotime(), and change the latter to use a monotonic clock instead. I'm less sure how to handle net's SetDeadline(), which takes a calendar time instead of a duration. In the mock, I expose runtime.nanotime() via time.SysNano() so that the net package can convert the calendar time to a value that will make sense to the runtime. This seemed like the simplest way to go about it, but I'm not sure if it's the best way.

- Jay

[Russ Cox]

On Tue, May 20, 2014 at 11:09 AM, <davy...@gmail.com> wrote:

I see the issue here changed to Release-None

https://code.google.com/p/go/issues/detail?id=6007

is that means won't fix it in 1.3.x ?

1.3 has monotonic timers on Linux, FreeBSD, OpenBSD, and DragonflyBSD. It will not have monotonic timers on OS X, Windows, and NetBSD. The 1.3.x updates will not change this (they are only for critical bug fixes, not new features).

The marking as Release-None means that we are not planning to do the work for OS X, Windows, or NetBSD ourselves in any particular release.

More generally, it is fine if someone wants to send in CLs working on a Release-None issue; the tag just means that we won't plan to. It is also fine to comment on a Release-None issue and explain why you think the issue is important enough to be worth targeting to a specific release.

Russ

[davy...@gmail.com]

Thanks for the clarification, Linux monotonic clock is good enough for me. Looking forward to use it on my production server. Thanks again for the good work!

[nicolash...@gmail.com]

(I've not seen OSX mentioned here or in the other issues, so I'll chime in because I've recently been looking at monotonic high-resolution high-performance timers a bit)

As for OSX, one could use mach_absolute_time() to get a monotonic clock: http://stackoverflow.com/a/21352348/558819 (and many other questions/answers)

Here is libuv's implementation for a high-resolution monotonic timer: https://github.com/joyent/libuv/blob/master/src/unix/darwin.c#L55-L62 (may not be accurate if master gets updated)

Note that libuv's implementation calls mach_timebase_info every call. This is not really a good idea as it is a very slow (~180ns) that returns the same struct on every call (it is static since boot). It was done this way because two threads might request uv_hrtime at the same time for the first time. I've currently logged an issue about getting rid of that by setting the info struct just once with atomics or pthread_once (atomics is fastest of course), which gets it down to ~32ns, of which ~11ns are the integer division. A pure mach_absolute_time syscall is about ~21ns, which I believe is on the order of linux's clock_gettime(CLOCK_MONOTONIC). Please correct me if I'm wrong. One could cut down the runtime by either performing the division only after one does the subtract of the two relative values one wants to measure, or by finding another way to do the division (like __iter_div_u64_rem on linux http://lxr.free-electrons.com/source/include/linux/math64.h#L118).

I see that the Go team is not going to do this, maybe I'll take a stab at it once I get some free time.

At any rate, thanks for another great release (1.3)!