RFC: os/clock package

[Kyle Lemons]

Based on some of the conversations here on the mailing list, on IRC, and with some coworkers I've been musing about taking a shot at designing an os/clock package.

I don't have any serious designs put together, but I think it would at a minimum be good to provide e.g. clock.System(), .Monotonic(), and .HighPrecision().  They would construct Clock objects (or something of that nature).  I don't think there would be any way to interact with or extract a time.Time for any of these, though I would expect that the "time" package would have a FromTicks(clock.Ticks) time.Duration or something similar.  I would just try my hand at this as a third party package, but I think that there are strong arguments for being able to tell the "time" package that you want to use a new Clock for e.g. Sleep, After, etc (probably in a similar way that you can create your own rand.Source or log.Logger).  It's also possible that we want to use the monotonic clock when we're computing timeouts and such.

I definitely wouldn't try to get this into go 1.2, and it would probably require taking some time to soak before going out in a 1.3 or 1.4.

My motivation for this is the fact that time is difficult, and I think it's something that if done correctly in a standard package, there is actually a reasonable chance that (in a similar way that the "time" package makes it easier to keep Times and Durations separate) it could make it easier to do correctly across the board.  I also think that it might be interesting to swap out the clock with a fake clock so that you could make instant tests that nevertheless think time is passing in a similar way that is done on the playground.  One potentially huge benefit could be basing a time.Ticker or a time.After off of the HPET -- the hardware has the ability to fire very precisely timed interrupts both periodically and as a one-shot.

After having gone back and forth for awhile, I think the most straightforward way to do it is to interact with the TSC, HPET, etc directly, though I'm likely to change my mind again tomorrow and think using the OS mechanisms is easier.  The system clock probably should just use the os mechanism.  Different operating systems provide varying support for especially HPET, but I believe that Windows after XP SP3, linux 2.6+, and mac os 10+ all provide all of the facilities we'd need (and most also providing access to 64-bit HPET if available).

I'm wondering whether this is something that would be worth drawing up more formally as a design for further thought or whether it is something that would be best left as a third party package or not at all.

Comments welcome.

[Dave Cheney]

I can't speak for it's inclusion in the std lib but I would like to see some sort of clock.Clock interface. 

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Uli Kunitz]

A few thoughts:

• It would make sense to develop as a non-standard-library package like dev/os/clock, because some work on the API will be required. It will be difficult to get it right at the first cut.
  
• The clock type should be an interface (clock.Interface?), which would allow users to invent their own clocks.
  
• A standard clock would then simply clock.Monotonic as implementation of the clock interface.
• The only reason to introduce a Ticks type would be to support higher than nanosecond resolution. But this would then clearly depend on the clock. I suggest to stay with time.Duration and nanosecond support,  which is what clock_gettime() supports.
  
• While the clock package could implement their own Timer and Ticker types, it would look nicer if the time types would be extended to support clock.Interface. (One wonders whether they should have been interfaces from the very beginning at least I would make them interfaces in the clock package.)

The last point would be an argument to extend the time package to support clocks.

On Thursday, August 22, 2013 4:53:34 AM UTC+2, Kyle Lemons wrote:

Based on some of the conversations here on the mailing list, on IRC, and with some coworkers I've been musing about taking a shot at designing an os/clock package.

I don't have any serious designs put together, but I think it would at a minimum be good to provide e.g. clock.System(), .Monotonic(), and .HighPrecision().  They woul d construct Clock objects (or something of that nature).  I don't think there would be any way to interact with or extract a time.Time for any of these, though I would expect that the "time" package would have a FromTicks(clock.Ticks) time.Duration or something similar.  I would just try my hand at this as a third party package, but I think that there are strong arguments for being able to tell the "time" package that you want to use a new Clock for e.g. Sleep, After, etc (probably in a similar way that you can create your own rand.Source or log.Logger).  It's also possible that we want to use the monotonic clock when we're computing timeouts and such.

I definitely wouldn't try to get this into go 1.2, and it would probably require taking some time to soak before going out in a 1.3 or 1.4.

[Kyle Lemons]

On Wed, Aug 21, 2013 at 11:45 PM, Uli Kunitz <uli.k...@gmail.com> wrote:

A few thoughts:

• It would make sense to develop as a non-standard-library package like dev/os/clock, because some work on the API will be required. It will be difficult to get it right at the first cut.
  
• The clock type should be an interface (clock.Interface?), which would allow users to invent their own clocks.
  
• A standard clock would then simply clock.Monotonic as implementation of the clock interface.
• The only reason to introduce a Ticks type would be to support higher than nanosecond resolution. But this would then clearly depend on the clock. I suggest to stay with time.Duration and nanosecond support,  which is what clock_gettime() supports.
  

time can't depend on clocks if clocks already depends on time :). 

• While the clock package could implement their own Timer and Ticker types, it would look nicer if the time types would be extended to support clock.Interface. (One wonders whether they should have been interfaces from the very beginning at least I would make them interfaces in the clock package.)

The last point would be an argument to extend the time package to support clocks.

Your other points are well taken.

 

On Thursday, August 22, 2013 4:53:34 AM UTC+2, Kyle Lemons wrote:

Based on some of the conversations here on the mailing list, on IRC, and with some coworkers I've been musing about taking a shot at designing an os/clock package.

I don't have any serious designs put together, but I think it would at a minimum be good to provide e.g. clock.System(), .Monotonic(), and .HighPrecision().  They woul d construct Clock objects (or something of that nature).  I don't think there would be any way to interact with or extract a time.Time for any of these, though I would expect that the "time" package would have a FromTicks(clock.Ticks) time.Duration or something similar.  I would just try my hand at this as a third party package, but I think that there are strong arguments for being able to tell the "time" package that you want to use a new Clock for e.g. Sleep, After, etc (probably in a similar way that you can create your own rand.Source or log.Logger).  It's also possible that we want to use the monotonic clock when we're computing timeouts and such.

I definitely wouldn't try to get this into go 1.2, and it would probably require taking some time to soak before going out in a 1.3 or 1.4.

My motivation for this is the fact that time is difficult, and I think it's something that if done correctly in a standard package, there is actually a reasonable chance that (in a similar way that the "time" package makes it easier to keep Times and Durations separate) it could make it easier to do correctly across the board.  I also think that it might be interesting to swap out the clock with a fake clock so that you could make instant tests that nevertheless think time is passing in a similar way that is done on the playground.  One potentially huge benefit could be basing a time.Ticker or a time.After off of the HPET -- the hardware has the ability to fire very precisely timed interrupts both periodically and as a one-shot.

After having gone back and forth for awhile, I think the most straightforward way to do it is to interact with the TSC, HPET, etc directly, though I'm likely to change my mind again tomorrow and think using the OS mechanisms is easier.  The system clock probably should just use the os mechanism.  Different operating systems provide varying support for especially HPET, but I believe that Windows after XP SP3, linux 2.6+, and mac os 10+ all provide all of the facilities we'd need (and most also providing access to 64-bit HPET if available).

I'm wondering whether this is something that would be worth drawing up more formally as a design for further thought or whether it is something that would be best left as a third party package or not at all.

Comments welcome.

--

[Ian Lance Taylor]

On Wed, Aug 21, 2013 at 7:53 PM, Kyle Lemons <kev...@google.com> wrote:
>
> I don't have any serious designs put together, but I think it would at a
> minimum be good to provide e.g. clock.System(), .Monotonic(), and
> .HighPrecision(). They would construct Clock objects (or something of that
> nature). I don't think there would be any way to interact with or extract a
> time.Time for any of these, though I would expect that the "time" package
> would have a FromTicks(clock.Ticks) time.Duration or something similar. I
> would just try my hand at this as a third party package, but I think that
> there are strong arguments for being able to tell the "time" package that
> you want to use a new Clock for e.g. Sleep, After, etc (probably in a
> similar way that you can create your own rand.Source or log.Logger). It's
> also possible that we want to use the monotonic clock when we're computing
> timeouts and such.

Looking solely at implementation, the time package's implementation of
timers relies on the runtime implementation of sleeping on a note,
which in turns depends on the implementation of futexes or semaphores,
depending on the system. E.g., when using the Linux kernel, it boils
down to a call to the futex system call passing in a timespec. It
would not be easy to slide in a different Clock interface.

Ian

[Dave Cheney]

I'm interested in clocks for getting the time, Realtime, Monotonic,
Highprecision, Mock, etc.

When Kyle mentioned this package, I had in my mind, something like
this, http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2008/n2661.htm#Clocks.

Sleeping for N user microseconds vs the realtime clock that the
runtime currently uses could be problematic, but I don't think that is
a reason not to continue to sketch out this package.

[Dmitry Vyukov]

I think time.Sleep, time.Ticker, etc should simply always use monotonic clock.

[Rob Pike]

On Fri, Aug 23, 2013 at 7:06 PM, Dmitry Vyukov <dvy...@google.com> wrote:

> I think time.Sleep, time.Ticker, etc should simply always use monotonic clock.

I've been waiting for someone to say that. I figured I was missing something.

-rob

[Kyle Lemons]

I agree that monotonic sleeps seem like a reasonable default, but I don't think we can do that without potentially breaking attempting to sleep until a particular date+time, because you could (possibly wildly) overshoot in the presence of clock steps.

[Rob Pike]

Then you would wake up. I don't see the issue.

I'm not doubting the difficulty of implementing monotonic time, I just
see no reason at all that it would require an API different from what
we have.

-rob

[Kyle Lemons]

If you wish to sleep until midnight and it's 11pm, you might try to sleep for an hour.  If during this intervening time, ntpd wakes up and steps your clock by +30m, you will oversleep by 30m if time.Sleep is using a monotonic clock.  It's much easier to write a loop that keeps sleeping target-time.Now() to get this behavior in the presence of a system clock-based sleep, but with only a monotonic clock it seems impossible to get this behavior at all in the presence of clock steps without busy waiting.

[Dmitry Vyukov]

On Sat, Aug 24, 2013 at 5:16 AM, Kyle Lemons <kev...@google.com> wrote:
> I agree that monotonic sleeps seem like a reasonable default, but I don't
> think we can do that without potentially breaking attempting to sleep until
> a particular date+time, because you could (possibly wildly) overshoot in the
> presence of clock steps.

Without time steps both sleeps behave the same way, right?

In the presence of time steps, it's somewhat unclear to me what a user
can potentially want, and what a reasonable behavior of sleeps can be
at all.

I think that any function that accepts relative time (time.Sleep,
time.Ticker) must use monotonic time. It is the most reasonable thing
to do. Do you agree?

Net deadlines use absolute time. I think we need to convert the
deadline to relative time ASAP and then use relative sleep on
monotonic clock as well. Here is an issue with API that we can not fix
-- if a user adds time.Minute to time.Now(), then time step happens,
then we subtract the result from time.Now(), we get garbage.

As for your sleep-till-midnight use case. This looks like a very
special case. (1) it is not supported by current implementation in any
way, so we can not break anything existing. (2) it must necessary
accept absolute time (because if you say sleep for 1 hour, but mean
sleep till midnight, there is no way we can figure your intention
out).

So based on all that, I think the plan should be:
1. Convert most existing things to monotonic time (time.Sleep,
time.Ticker). This essentially fixes bugs.
2. Introduce a function to sleep till specified real time, e.g.
time.SleepUntil(time.Time), time.NewTimerUntil(time.Time).
3. Convert socket deadlines to monotonic time, and probably introduce
SetTimeout(time.Duration) and say that if you use SetDeadline you will
be skrewed up in presence of time steps.
4. Expose monotonic time to user, e.g. time.MonoNow().

Does it sound good to you?

[Shivakumar GN]

On Sat, Aug 24, 2013 at 3:35 PM, Dmitry Vyukov <dvy...@google.com> wrote:

I think that any function that accepts relative time (time.Sleep,

time.Ticker) must use monotonic time. It is the most reasonable thing
to do. Do you agree?

Support for monotonic time is a often occurring need in server infrastructure (not just desktop).

Ex: A server infrastructure exchanging heart-beats and making decisions based on relative time need to be independent of system time.

Having been bitten once by this issue in production environment, would like to see this behavior in std lib where relative time is not influenced by system time changes.

Not every one may realize the current surprising behavior in std lib. Even when they do, creating a solution is non-trivial. This is hard one to solve.

[Johan Bolmsjö]

Den lördagen den 24:e augusti 2013 kl. 12:05:26 UTC+2 skrev Dmitry Vyukov:

So based on all that, I think the plan should be:
1. Convert most existing things to monotonic time (time.Sleep,
time.Ticker). This essentially fixes bugs.
2. Introduce a function to sleep till specified real time, e.g.
time.SleepUntil(time.Time), time.NewTimerUntil(time.Time).
3. Convert socket deadlines to monotonic time, and probably introduce
SetTimeout(time.Duration) and say that if you use SetDeadline you will
be skrewed up in presence of time steps.
4. Expose monotonic time to user, e.g. time.MonoNow().

Sounds like a solid plan to me.

P.S.

I have also been bitten more than once by bugs in production systems caused by timers driven by real time that suddenly jumps (not in Go though, but custom timer frameworks in C).

[Kyle Lemons]

On Sat, Aug 24, 2013 at 3:05 AM, Dmitry Vyukov <dvy...@google.com> wrote:

On Sat, Aug 24, 2013 at 5:16 AM, Kyle Lemons <kev...@google.com> wrote:
> I agree that monotonic sleeps seem like a reasonable default, but I don't
> think we can do that without potentially breaking attempting to sleep until
> a particular date+time, because you could (possibly wildly) overshoot in the
> presence of clock steps.

Without time steps both sleeps behave the same way, right?

Right.

 

In the presence of time steps, it's somewhat unclear to me what a user
can potentially want, and what a reasonable behavior of sleeps can be
at all.

If we have to choose between the two, I think non-monotonic is marginally less surprising.  Many "sleep" APIs are documented as sleeping for "at least" the duration specified, though, so strictly speaking we could save the start time, and keep sleeping for monotonic durations until at least the specified duration on the system clock has passed.  In this case Sleep would never undersleep.

 

I think that any function that accepts relative time (time.Sleep,
time.Ticker) must use monotonic time. It is the most reasonable thing
to do. Do you agree?

I do agree that those APIs seem like the most straightforward users of monotonic time.

 

Net deadlines use absolute time. I think we need to convert the
deadline to relative time ASAP and then use relative sleep on
monotonic clock as well.

I'm not sure whether I agree here or not.  From an I/O timeout standpoint, I think you'd want monotonic time.  I'm not actually sure whether epoll and friends support this, though.  From an API standpoint, though, there are very profound benefits of having an absolute time deadline in terms of code simplicity.  If you only have SetTimeout (which we used to have), it's unclear what to do when the Write call will result in multiple underlying write calls.  Does each one get the specified timeout?  Does every write have to complete within that amount of time since SetTimeout was called?  The latter seems like what you propose, but would be really hard for users to understand.

 

Here is an issue with API that we can not fix
-- if a user adds time.Minute to time.Now(), then time step happens,
then we subtract the result from time.Now(), we get garbage.

Really?  I don't think our time arithmetic will ever give you garbage.  http://play.golang.org/p/FEK7AA1omh  We just need to define what happens when you try to sleep a negative amount, which should be easy: you don't sleep.

 

As for your sleep-till-midnight use case. This looks like a very
special case.

It's a specific example of a reasonably common case.  If you're scheduling things to run (system upgrades, a software rollout, cron, email notifications), you'll often be doing it in terms of wall clock time, so that a human checking his watch will know when it's going to happen.  At present you're unlikely to get too far off, because leap seconds are the only thing that should cause a time step in a steady state NTP system, but you may well write the code in the way I describe (sleep for target-time.Now() in a loop) because it is the closest analogy to what you'd do as a human (keep checking your watch until it's time to do your action).

 

(1) it is not supported by current implementation in any
way, so we can not break anything existing.

Actually, in the current implementation, I believe the API allows you to sleep until the clock says a specified time pretty easily; what it does not currently provide is a way to sleep until a certain amount of time has actually passed.

 

(2) it must necessary
accept absolute time (because if you say sleep for 1 hour, but mean
sleep till midnight, there is no way we can figure your intention
out).

It's true the Sleep(1*time.Hour) carries no intent; however, it is easier to recover if Sleep(1*time.Hour) sleeps for less than an hour -- the calling application can always sleep again.  You can't go back in time if you oversleep.

 

So based on all that, I think the plan should be:
1. Convert most existing things to monotonic time (time.Sleep,
time.Ticker). This essentially fixes bugs.

As you said above, time.Sleep carries no intent.  Therefore, you can't say whether this will introduce or fix bugs.  I have written at least one application that uses sleep in the way I describe.  Worse, making the change will not visibly break anybody that was using them in that way, so they may not notice until it's too late.

 

2. Introduce a function to sleep till specified real time, e.g.
time.SleepUntil(time.Time), time.NewTimerUntil(time.Time).

I think these introductions would be a good start.  Perhaps introducing them now and giving users until go 2 to start using them would allow you to make Sleep and Ticker use monotonic time without breaking anybody.  Of course, then the suggestion would be to make SleepMono and NewMonoTimer... but now we have a major API addition.  Thus my suggestion to introduce a Clock primitive, so you could call time.SetClock(clk) and make all calls to time.Sleep and time.NewTimer use it (or call time.New(clk).Sleep and such to not change out the clock used by the entire package).  In this way, you can maintain the current behavior in perpetuity, while still allowing users who need one behavior or the other to specify it.

 

3. Convert socket deadlines to monotonic time, and probably introduce
SetTimeout(time.Duration) and say that if you use SetDeadline you will
be skrewed up in presence of time steps.

I'm not convinced about this part quite yet; see above.

 

4. Expose monotonic time to user, e.g. time.MonoNow().

With a clock primitive, you can provide these directly from the clocks: "os/clock".Monotonic.Now() etc.  I actually don't think time.MonoNow would be a good API, because it would seem to correlate to Now() which returns a Time, and MonoNow would actually need to be a counter, e.g. uint64.

[Dmitry Vyukov]

On Sat, Aug 24, 2013 at 9:01 PM, Kyle Lemons <kev...@google.com> wrote:
> On Sat, Aug 24, 2013 at 3:05 AM, Dmitry Vyukov <dvy...@google.com> wrote:
>>
>> On Sat, Aug 24, 2013 at 5:16 AM, Kyle Lemons <kev...@google.com> wrote:
>> > I agree that monotonic sleeps seem like a reasonable default, but I
>> > don't
>> > think we can do that without potentially breaking attempting to sleep
>> > until
>> > a particular date+time, because you could (possibly wildly) overshoot in
>> > the
>> > presence of clock steps.
>>
>>
>> Without time steps both sleeps behave the same way, right?
>
>
> Right.
>
>>
>> In the presence of time steps, it's somewhat unclear to me what a user
>> can potentially want, and what a reasonable behavior of sleeps can be
>> at all.
>
>
> If we have to choose between the two, I think non-monotonic is marginally
> less surprising. Many "sleep" APIs are documented as sleeping for "at
> least" the duration specified, though, so strictly speaking we could save
> the start time, and keep sleeping for monotonic durations until at least the
> specified duration on the system clock has passed. In this case Sleep would
> never undersleep.

I believe that 99% uses of sleep/ticker actually want monotonic time.
And what happens now is either a bug in Go or in user's programs.
If you run a shall command and set 10min timeout in separate
goroutine. Do you actually mean that it's related to real time in any
way, shape or form? Do you mean that it must sleep for 1 day and 10
min in some cases, and in other abort almost instantly. That's what
such Go code currently says.

>> I think that any function that accepts relative time (time.Sleep,
>> time.Ticker) must use monotonic time. It is the most reasonable thing
>> to do. Do you agree?
>
>
> I do agree that those APIs seem like the most straightforward users of
> monotonic time.
>
>>
>> Net deadlines use absolute time. I think we need to convert the
>> deadline to relative time ASAP and then use relative sleep on
>> monotonic clock as well.
>
>
> I'm not sure whether I agree here or not. From an I/O timeout standpoint, I
> think you'd want monotonic time. I'm not actually sure whether epoll and
> friends support this, though. From an API standpoint, though, there are
> very profound benefits of having an absolute time deadline in terms of code
> simplicity. If you only have SetTimeout (which we used to have), it's
> unclear what to do when the Write call will result in multiple underlying
> write calls. Does each one get the specified timeout? Does every write
> have to complete within that amount of time since SetTimeout was called?
> The latter seems like what you propose, but would be really hard for users
> to understand.

I do not mean to change semantics of SetDeadline/Timeout calls. I mean
that relative timeout is another (better) way to specify what
SetDeadline does. But other than that it will behave exactly as
SetDeadline now.
Call as SetDeadline(time.Now() + time.Minute) is also clearly broken.
I can not imagine that user wants it to be affected by real time clock
changes.
So the SetTimeout(d) will be equivalent to
"SetMonoDeadline(time.MonoNow() + d)".
We can not just say that SetDeadline now accepts monotonic time,
because it will break existing programs. We can introduce something
like SetMonoDeadline().

>> Here is an issue with API that we can not fix
>> -- if a user adds time.Minute to time.Now(), then time step happens,
>> then we subtract the result from time.Now(), we get garbage.
>
>
> Really? I don't think our time arithmetic will ever give you garbage.
> http://play.golang.org/p/FEK7AA1omh We just need to define what happens
> when you try to sleep a negative amount, which should be easy: you don't
> sleep.

I meant that it is garbage in the sense that from users point of view
it's essentially random value.
Any call to SetDeadline(time.Now().Add(time.Minute)) is equivalent to
SetDeadline(randomTime()).

>> As for your sleep-till-midnight use case. This looks like a very
>> special case.
>
>
> It's a specific example of a reasonably common case. If you're scheduling
> things to run (system upgrades, a software rollout, cron, email
> notifications), you'll often be doing it in terms of wall clock time, so
> that a human checking his watch will know when it's going to happen. At
> present you're unlikely to get too far off, because leap seconds are the
> only thing that should cause a time step in a steady state NTP system, but
> you may well write the code in the way I describe (sleep for
> target-time.Now() in a loop) because it is the closest analogy to what you'd
> do as a human (keep checking your watch until it's time to do your action).

Ah, I see.
Talking about differences between real and monotonic time, if real
time does not change, it not interesting. Use any, they are the same.
There are also software bugs, administrators and desktop systems with users.
Ordinary user does not understand what is UTC time and timezone. So
when he travels to -12 timezone, he changes his *time* by -12hrs. Then
he connects to internet and NTP changes his time back by +12hrs. Then
it repeats for the whole travel.
During this half of his Go programs and another half crash with timeouts.

Let's assume for this discussion that time periodically randomly
changes by +/- year.
Now what does it mean to do something every midnight?
If the same midnight happens twice, do you want to do something two times?
If from one midnight to the next passes a week of monotonic time, how
the system must behave?

>> (1) it is not supported by current implementation in any
>> way, so we can not break anything existing.
>
>
> Actually, in the current implementation, I believe the API allows you to
> sleep until the clock says a specified time pretty easily; what it does not
> currently provide is a way to sleep until a certain amount of time has
> actually passed.

... until a certain amount of time has actually passed in what time
(monotonic/real)?
I believe in presence of time changes it behaves essentially randomly now.

>> (2) it must necessary
>> accept absolute time (because if you say sleep for 1 hour, but mean
>> sleep till midnight, there is no way we can figure your intention
>> out).
>
>
> It's true the Sleep(1*time.Hour) carries no intent; however, it is easier to
> recover if Sleep(1*time.Hour) sleeps for less than an hour -- the calling
> application can always sleep again. You can't go back in time if you
> oversleep.

Yes, but on the other hand, the next midnight can suddenly happen next
nanosecond.
That's why I say that I can not imagine any sane semantics for such
sleep in presence of chaotic time changes (and if they do not happen,
then all this become trivial and nothing to talk about).

>> So based on all that, I think the plan should be:
>> 1. Convert most existing things to monotonic time (time.Sleep,
>> time.Ticker). This essentially fixes bugs.
>
>
> As you said above, time.Sleep carries no intent. Therefore, you can't say
> whether this will introduce or fix bugs. I have written at least one
> application that uses sleep in the way I describe. Worse, making the change
> will not visibly break anybody that was using them in that way, so they may
> not notice until it's too late.

It does not work now. So it's not a problem :)

>> 2. Introduce a function to sleep till specified real time, e.g.
>> time.SleepUntil(time.Time), time.NewTimerUntil(time.Time).
>
>
> I think these introductions would be a good start. Perhaps introducing them
> now and giving users until go 2 to start using them would allow you to make
> Sleep and Ticker use monotonic time without breaking anybody. Of course,
> then the suggestion would be to make SleepMono and NewMonoTimer... but now
> we have a major API addition. Thus my suggestion to introduce a Clock
> primitive, so you could call time.SetClock(clk) and make all calls to
> time.Sleep and time.NewTimer use it (or call time.New(clk).Sleep and such to
> not change out the clock used by the entire package). In this way, you can
> maintain the current behavior in perpetuity, while still allowing users who
> need one behavior or the other to specify it.

You can use the clock in your program, but you can't fix all the
packages you use.

>> 3. Convert socket deadlines to monotonic time, and probably introduce
>> SetTimeout(time.Duration) and say that if you use SetDeadline you will
>> be skrewed up in presence of time steps.
>
>
> I'm not convinced about this part quite yet; see above.

I am not proposing new behavior (like timeout for each separate net
call). SetTimeout is just a non-broken way to communicate when you
want the deadline to happen.

>> 4. Expose monotonic time to user, e.g. time.MonoNow().
>
>
> With a clock primitive, you can provide these directly from the clocks:
> "os/clock".Monotonic.Now() etc. I actually don't think time.MonoNow would
> be a good API, because it would seem to correlate to Now() which returns a
> Time, and MonoNow would actually need to be a counter, e.g. uint64.

Well, I don't have a strong position right now about how exactly the
API must look like. But FWIW MonoTime() can return Time from some
unspecified point in time.

[Kyle Lemons]

That's also what "sleep 10 | command" and various other shell idioms say.  It's also what happens when I tell my android phone "Remind me to 'mail the electric bill' in 20 minutes".

Actually, though, we can't.  Not until Go 2.  We can, however, allow the user to tell (via something like time.SetClock(clock.Monotonic)) us what they want.  In addition to being backward-compatible, I think it'd be the right thing to do.

Adding even more calls to the net.Conn api is also precisely the opposite of the simplicity I'd like to see for dealing with clocks.  If you'll allow me to make up statistics for a moment, 99% of programs don't care, and of the 1% that do, 75% won't break catastrophically.  Who cares if a network connection takes a second longer to time out at midnight once a year?  Unfortunately, something more like 10% of programs *should* care, and we should make it really easy for them to be correct.  The cognitive overhead and maintenance burden of adding SetMonoDeadline and SetTimeout would seem to be both too expensive for the people who don't and/or shouldn't care, and harder to understand what the "right" thing is to do for the people who do/should care.

There's another important aspect of time that we haven't even gotten to yet, and that's all of the poor assumptions that distributed systems make about time.  If I send you a timestamp, what does that mean?  These are the places where time really starts to matter, and where the consequences of getting it wrong can be much higher.

>> Here is an issue with API that we can not fix
>> -- if a user adds time.Minute to time.Now(), then time step happens,
>> then we subtract the result from time.Now(), we get garbage.
>
>
> Really?  I don't think our time arithmetic will ever give you garbage.
> http://play.golang.org/p/FEK7AA1omh  We just need to define what happens
> when you try to sleep a negative amount, which should be easy: you don't
> sleep.

I meant that it is garbage in the sense that from users point of view
it's essentially random value.
Any call to SetDeadline(time.Now().Add(time.Minute)) is equivalent to
SetDeadline(randomTime()).

We're not talking about race conditions here.  It's not a random time, it's just a time that us subject to outside influence.

>> As for your sleep-till-midnight use case. This looks like a very
>> special case.
>
>
> It's a specific example of a reasonably common case.  If you're scheduling
> things to run (system upgrades, a software rollout, cron, email
> notifications), you'll often be doing it in terms of wall clock time, so
> that a human checking his watch will know when it's going to happen.  At
> present you're unlikely to get too far off, because leap seconds are the
> only thing that should cause a time step in a steady state NTP system, but
> you may well write the code in the way I describe (sleep for
> target-time.Now() in a loop) because it is the closest analogy to what you'd
> do as a human (keep checking your watch until it's time to do your action).

Ah, I see.
Talking about differences between real and monotonic time, if real
time does not change, it not interesting. Use any, they are the same.
There are also software bugs, administrators and desktop systems with users.
Ordinary user does not understand what is UTC time and timezone. So
when he travels to -12 timezone, he changes his *time* by -12hrs. Then
he connects to internet and NTP changes his time back by +12hrs. Then
it repeats for the whole travel.
During this half of his Go programs and another half crash with timeouts.

Let's assume for this discussion that time periodically randomly
changes by +/- year.

We didn't design the standard library to assume that someone might be standing over your DIMMs with a high energy radiation source flipping the bits in your memory either.  Assuming that the underlying system is terribly broken seems like an irresponsible stance when designing an API.  I want to design an API that makes it easy for software that cares about the behavior of its time source to behave correctly under normal circumstances.  It is unfortunate that normal circumstances include clock steps for leap seconds and the occasional clock step or slew when ntpd regains connectivity, but that's the way things are.

 

Now what does it mean to do something every midnight?
If the same midnight happens twice, do you want to do something two times?
If from one midnight to the next passes a week of monotonic time, how
the system must behave?

>> (1) it is not supported by current implementation in any
>> way, so we can not break anything existing.
>
>
> Actually, in the current implementation, I believe the API allows you to
> sleep until the clock says a specified time pretty easily; what it does not
> currently provide is a way to sleep until a certain amount of time has
> actually passed.

... until a certain amount of time has actually passed in what time
(monotonic/real)?

By actual, I meant real.

 

I believe in presence of time changes it behaves essentially randomly now.

That would seem to be an overreaction to the facts.  If that were the case, the existing mechanisms for time (in all languages) would have long since been abandoned.

 

>> (2) it must necessary
>> accept absolute time (because if you say sleep for 1 hour, but mean
>> sleep till midnight, there is no way we can figure your intention
>> out).
>
>
> It's true the Sleep(1*time.Hour) carries no intent; however, it is easier to
> recover if Sleep(1*time.Hour) sleeps for less than an hour -- the calling
> application can always sleep again.  You can't go back in time if you
> oversleep.

Yes, but on the other hand, the next midnight can suddenly happen next
nanosecond.
That's why I say that I can not imagine any sane semantics for such
sleep in presence of chaotic time changes (and if they do not happen,
then all this become trivial and nothing to talk about).

Chaotic time changes?  They may happen, but I'm not sure how you could ever trust anything under such circumstances, including whether your binary was built with up-to-date objects.  I'm interested in the run-of-the-mill time steps and slews that happen on pretty much every system, and which some programs need to understand.

 

>> So based on all that, I think the plan should be:
>> 1. Convert most existing things to monotonic time (time.Sleep,
>> time.Ticker). This essentially fixes bugs.
>
>
> As you said above, time.Sleep carries no intent.  Therefore, you can't say
> whether this will introduce or fix bugs.  I have written at least one
> application that uses sleep in the way I describe.  Worse, making the change
> will not visibly break anybody that was using them in that way, so they may
> not notice until it's too late.

It does not work now. So it's not a problem :)

My application most certainly does work now.  Even in the presence of clock steps.  It has the precise behavior that I wish of it.  It would not be improved by switching to monotonic time; in fact, if I implemented my API, I would probably explicitly tell the time API to use the system clock just for clarity of purpose.

 

>> 2. Introduce a function to sleep till specified real time, e.g.
>> time.SleepUntil(time.Time), time.NewTimerUntil(time.Time).
>
>
> I think these introductions would be a good start.  Perhaps introducing them
> now and giving users until go 2 to start using them would allow you to make
> Sleep and Ticker use monotonic time without breaking anybody.  Of course,
> then the suggestion would be to make SleepMono and NewMonoTimer... but now
> we have a major API addition.  Thus my suggestion to introduce a Clock
> primitive, so you could call time.SetClock(clk) and make all calls to
> time.Sleep and time.NewTimer use it (or call time.New(clk).Sleep and such to
> not change out the clock used by the entire package).  In this way, you can
> maintain the current behavior in perpetuity, while still allowing users who
> need one behavior or the other to specify it.

You can use the clock in your program, but you can't fix all the
packages you use.

if I could time.SetClock(time.Monotonic), it would make them all use the monotonic clock.  I'm not sure what you mean by "fix," as this implies that they're broken.  What evidence do you have that libraries in Go are widely broken with respect to time?

 

>> 3. Convert socket deadlines to monotonic time, and probably introduce
>> SetTimeout(time.Duration) and say that if you use SetDeadline you will
>> be skrewed up in presence of time steps.
>
>
> I'm not convinced about this part quite yet; see above.

I am not proposing new behavior (like timeout for each separate net
call). SetTimeout is just a non-broken way to communicate when you
want the deadline to happen.

If we're talking about clarity of intent here, let's consider the following: 

conn := net.Dial(tcp, "google.com:80")

conn.SetTimeout(1*time.Minute)

conn.Write("GET / HTTP/1.1\r\n\r\n")

buf := new(bytes.Buffer)

io.Copy(buf, conn)

...

How long could this take after the connection is established?

1 minute: if you assume that a deadline for the connection is set.

2 minutes: if you assume that the Write call and the entirety of the Copy each have their own 1 minute timeout

Forever: if you assume that each read syscall and each write syscall individually have a 1 minute timeout.

If you change that to conn.SetDeadline(time.Now().Add(1*time.Minute)), it is clear that it is intended to take at most 1 minute to complete after the connection is established.  If the time library has been switched to using a monotonic clock, this works correctly in the presence of clock steps (at the cost of making time.Now() diverge from the system clock).

 

>> 4. Expose monotonic time to user, e.g. time.MonoNow().
>
>
> With a clock primitive, you can provide these directly from the clocks:
> "os/clock".Monotonic.Now() etc.  I actually don't think time.MonoNow would
> be a good API, because it would seem to correlate to Now() which returns a
> Time, and MonoNow would actually need to be a counter, e.g. uint64.

Well, I don't have a strong position right now about how exactly the
API must look like. But FWIW MonoTime() can return Time from some
unspecified point in time.

I think the shape of the API is critical to making it easy to use, and I think maintaining the current API surfaces is possible without sacrificing correctness for the clients who require it.

[Dave Cheney]

I sketched out a few ideas, based on my favorite parts of
boost::chrono here http://godoc.org/github.com/davecheney/junk/clock

[Nick Craig-Wood]

On 12/09/13 06:48, Dave Cheney wrote:
> I sketched out a few ideas, based on my favorite parts of
> boost::chrono here http://godoc.org/github.com/davecheney/junk/clock

Looks useful!

Just wondering about the use of time.Time though. That kind of implies
that the result represents an actual time, whereas the clocks except
Realtime are returning a time reference. Perhaps make an alias for
time.Time called time.TimeReference?

--
Nick Craig-Wood <ni...@craig-wood.com> -- http://www.craig-wood.com/nick

[Dave Cheney]

Sure. These are just a few ideas that were knocking about. Hopefully something more concrete will come from them.

What specifically don't you like about returning a time.Time? If you were to define your own clock.Instant, what would make it sufficiently different to justify the cost of translating it to and from a time.Time

[Nick Craig-Wood]

On 12/09/13 12:05, Dave Cheney wrote:

> Sure. These are just a few ideas that were knocking about. Hopefully
something more concrete will come from them.

Excellent ideas IMHO

> What specifically don't you like about returning a time.Time?

I don't like the idea that you've got something called time.Time which
doesn't represent real time as we know it. It makes no sense to print
YYYYMMDDD etc from the result of clock.Monotonic.

> If you were to define your own clock.Instant, what would make it
sufficiently different to justify the cost of translating it to and from
a time.Time

I'd keep the underlying time.Time but define an alias like this

type Instant struct {
time.Time
}

You'd then be able to cast easily and cheaply to time.Time to use all
the goodness in the stdlib, but the act of casting would remind you that
you haven't actualy got a time.Time but something different so care is
needed.

Another alternative would be to use time.Duration instead and have all
the Clocks return Duration relative to the start of the runtime.

[Dave Cheney]

On 12/09/2013, at 22:28, Nick Craig-Wood <ni...@craig-wood.com> wrote:

> On 12/09/13 12:05, Dave Cheney wrote:
>
>> Sure. These are just a few ideas that were knocking about. Hopefully
> something more concrete will come from them.
>
> Excellent ideas IMHO

All credit to Kevlar, I'm just noodling with some ideas.

>
>> What specifically don't you like about returning a time.Time?
>
> I don't like the idea that you've got something called time.Time which
> doesn't represent real time as we know it. It makes no sense to print
> YYYYMMDDD etc from the result of clock.Monotonic.

Hmm, I want to disagree here. time.Time represents an instant in time, as time.Duration is the distance between two instants. Some other time libraries go further and call their type instant. In that context, or considering time.Time an alias for a theoretical time.Instant, I don't see why a clock that represents time by some other scale is an incorrect usage of the type. For example, in the clockskew example code, clock monotonic reports a real world time, it just happens to run behind the clock realtime if your machine goes to sleep. That doesn't make the time.Time it returns an invalid use of the type, it's just not accurate for telling the time now. At least it won't run backward.

Having said that you make a good argument for something like

package clock

type Timer interface { Elapsed() time.Duration }

To model things like process runtime, time since boot, etc. And those types should probably get strongly promoted in whatever API evolves.

>
>> If you were to define your own clock.Instant, what would make it
> sufficiently different to justify the cost of translating it to and from
> a time.Time
>
> I'd keep the underlying time.Time but define an alias like this
>
> type Instant struct {
> time.Time
> }

Hmm, that feels like the wrong hammer.

>
> You'd then be able to cast easily and cheaply to time.Time to use all
> the goodness in the stdlib, but the act of casting would remind you that
> you haven't actualy got a time.Time but something different so care is
> needed.

Being realistic, this package has to interoperable with the stdlib time package and adding requirements for conversion between types sounds like a hurdle that should be avoided.

>
> Another alternative would be to use time.Duration instead and have all
> the Clocks return Duration relative to the start of the runtime.

I agree, with reservation, see above.

[Dave Cheney]

PTAL. http://godoc.org/github.com/davecheney/junk/clock#Timer

[Nick Craig-Wood]

On 13/09/13 02:09, Dave Cheney wrote:
> PTAL. http://godoc.org/github.com/davecheney/junk/clock#Timer

I like the Timer interface.

> On Fri, Sep 13, 2013 at 10:19 AM, Dave Cheney <da...@cheney.net> wrote:
>> On 12/09/2013, at 22:28, Nick Craig-Wood <ni...@craig-wood.com> wrote:
>>> On 12/09/13 12:05, Dave Cheney wrote:
>>>> What specifically don't you like about returning a time.Time?
>>>
>>> I don't like the idea that you've got something called time.Time which
>>> doesn't represent real time as we know it. It makes no sense to print
>>> YYYYMMDDD etc from the result of clock.Monotonic.
>>
>> Hmm, I want to disagree here. time.Time represents an instant in time, as time.Duration is the distance between two instants. Some other time libraries go further and call their type instant. In that context, or considering time.Time an alias for a theoretical time.Instant, I don't see why a clock that represents time by some other scale is an incorrect usage of the type. For example, in the clockskew example code, clock monotonic reports a real world time, it just happens to run behind the clock realtime if your machine goes to sleep. That doesn't make the time.Time it returns an invalid use of the type, it's just not accurate for telling the time now. At least it won't run backward.

I'll just note that unlike linux the results of Windows and OS X calls
for a monotonic timer look nothing like time.Time so may require more
work to make them into a time.Time

QueryPerformanceCounter:
http://msdn.microsoft.com/en-us/library/ms644904(VS.85).aspx

mach_absolute_time:
https://developer.apple.com/library/mac/qa/qa1398/_index.html

>> Having said that you make a good argument for something like
>>
>> package clock
>>
>> type Timer interface { Elapsed() time.Duration }
>>
>> To model things like process runtime, time since boot, etc. And those types should probably get strongly promoted in whatever API evolves.

Yes I like that.

[Dave Cheney]

> I'll just note that unlike linux the results of Windows and OS X calls
> for a monotonic timer look nothing like time.Time so may require more
> work to make them into a time.Time
>
> QueryPerformanceCounter:
> http://msdn.microsoft.com/en-us/library/ms644904(VS.85).aspx

TBH - I wasn't expecting any of this to work on Windows.

> mach_absolute_time:
> https://developer.apple.com/library/mac/qa/qa1398/_index.html

Some days OS X makes me want to table flip. Bloody hell,
clock_gettime(2) isn't implemented on OSX and I haven't tried to call
mach traps under darwin.

I think the moral of this story is

1. Not all clocks and timers are available on all platforms (I should
see what boost does in this regard)

2. I'm still holding on to the idea of Clocks that report instants of
time, but I think Clocks are less useful than Timers.

[Dmitry Vyukov]

On Fri, Sep 13, 2013 at 5:21 AM, Dave Cheney <da...@cheney.net> wrote:
>> I'll just note that unlike linux the results of Windows and OS X calls
>> for a monotonic timer look nothing like time.Time so may require more
>> work to make them into a time.Time
>>
>> QueryPerformanceCounter:
>> http://msdn.microsoft.com/en-us/library/ms644904(VS.85).aspx
>
> TBH - I wasn't expecting any of this to work on Windows.
>
>> mach_absolute_time:
>> https://developer.apple.com/library/mac/qa/qa1398/_index.html
>
> Some days OS X makes me want to table flip. Bloody hell,
> clock_gettime(2) isn't implemented on OSX and I haven't tried to call
> mach traps under darwin.
>
> I think the moral of this story is
>
> 1. Not all clocks and timers are available on all platforms (I should
> see what boost does in this regard)

I would start with a minimal set of clocks:
1. real-time (current)
2. monotonic (the most useful and all OSes must provide it in some way
or another)
3. maybe high resolution (something better than current windows 15ms time)

[Kyle Lemons]

My main reason for suggesting that such a clock package have its own representation of a time coming from a clock is that for the time package to depend upon it, it wouldn't be able to use a time.Time.  It would also be weird to make it easy for clock monotonic (especially _RAW) to produce a human readable date/time string, because this is likely to be somewhat different from the actual wall clock time and humans are easily confused.

[Dave Cheney]

On Sat, Sep 14, 2013 at 2:45 PM, Kyle Lemons <kev...@google.com> wrote:
> My main reason for suggesting that such a clock package have its own
> representation of a time coming from a clock is that for the time package to
> depend upon it, it wouldn't be able to use a time.Time.

I think it is possible to use a time.Time in some cases, and a
time.Duration in others where it does not fit. Introducing a new type,
eg, clock.Instant would just require a lot of conversions back and
forth between time.Time and clock.Instant and in those cases it is
probably that what folks really wanted was a time.Duration of the
clock really has no baring on time in a human scale.

> It would also be
> weird to make it easy for clock monotonic (especially _RAW) to produce a
> human readable date/time string, because this is likely to be somewhat
> different from the actual wall clock time and humans are easily confused.

I disagree, you can store a time.Time in a database, or parse it from
a string, and people are quite happy to accept this as a valid use of
the time.Time type. Certainly time.Now() is the most natural use of
time.Time, but I don't think that other uses are so foreign that we
need restort to a new clock.Instant type.

[Jay Weisskopf]

While reading this thread, I found myself agreeing with everything Dmitry has said. Any function that takes a Duration as an input really ought to be using a monotonic clock. In the (probably few) cases where someone wants to use an absolute time rather than a relative duration, perhaps a time.Alarm(time.Time) function can be added. In my mind, the only thing that can't be updated transparently is the time.Since(time.Now()) idiom for measuring things. That may require a time.Stopwatch :-)

[Dave Cheney]

Sorry for reviving this old thread.

Is there any interest on developing this idea of an os/clock package
for the 1.3 development cycle ?

[Dmitry Vyukov]

I don't have plans to lead this/write design doc. If somebody else is
willing to, he must distill the requirements and write a design doc.

[Josh Bleecher Snyder]

> Sorry for reviving this old thread.
>
> Is there any interest on developing this idea of an os/clock package
> for the 1.3 development cycle ?

Yes! I'm game to help, but I don't think I can lead the effort.

-josh

[Jay Weisskopf]

As a user, I'm not sure I have a need for different clocks to be exposed by the stdlib API. The main change I desire is that SomeFuncs(d Duration) keep working if the wall time changes.

--
You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/D11F4zMs-E0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.

[Vaidhy Gopalan]

Is there a particular format for doing the design spec. I will try to distill the conversations in this thread into a spec.

-Vaidhy

[Dmitry Vyukov]

No, there is no.

[Dmitri Shuralyov]

Are there any updates on this?

I've recently found out that using time.Now() for timing has 15~ ms granularity on Windows, which doesn't work for network latency calculations and general physics calculations in a game.

Is there something in the standard package by now that I should be using instead of time.Now() if I want higher precision than 15~ ms?

[Dmitri Shuralyov]

> What version of Windows and Go are you using? I'm getting ~1ms precision with Go 1.4.2 on Windows 8.1 64-bit.

I was doing some tests on a Windows machine, and it happened to be Windows XP SP3 (32 bit) with Go 1.4.2 (32-bit).

I've noticed that the Go runtime sets timeBeginPeriod(1) at startup, and I can confirm via ClockRes.exe that it works:

Maximum timer interval: 15.600 ms

Minimum timer interval: 1.000 ms

Current timer interval: 1.000 ms

(When the Go program isn't running, Current timer interval is 15.600 ms)

But running a simple program like this clearly shows that time.Now() precision is 15~ ms, not 1 ms:

func main() {

    t0 := time.Now()

    for {

        fmt.Println(time.Since(t0).Seconds() * 1000)

    }

}

I wonder if it's a bug that affects Windows XP, or is it expected for Windows XP to have 15~ ms granularity while newer versions of Windows (presumably) have 1~ ms?

[Jay Weisskopf]

It looks like this is the current implementation at tip if you want to dig into it yourself. https://github.com/golang/go/blob/master/src/runtime/os1_windows.go#L313-L356

During the switch to monotonic clocks a year ago, I tried using QueryPerformenceCounter on Windows, but had to leave it out because it was causing problems on some machines. The current implementation, while relatively low resolution, is probably what works reliably everywhere.

--
You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/D11F4zMs-E0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[brainman]

On Monday, 15 June 2015 08:07:34 UTC+10, Dmitri Shuralyov wrote:

> ... Is there something in the standard package by now that I should be using instead of time.Now() if I want higher precision than 15~ ms?

There is nothing for that. But you can use any Windows API to measure time. For example Microsoft folks recomend you use QueryPerformanceCounter (http://blogs.msdn.com/b/oldnewthing/archive/2014/08/22/10551965.aspx). But it is not clear cut, for example QueryPerformanceCounter doesn't work well on my PC.

Alex

[brainman]

On Monday, 15 June 2015 08:43:41 UTC+10, Dmitri Shuralyov wrote:

> ... I wonder if it's a bug that affects Windows XP, or is it expected for Windows XP to have 15~ ms granularity while newer versions of Windows (presumably) have 1~ ms?

It sounds like it to me. This program http://play.golang.org/p/jNx-3eLOow prints:

[15.6241ms 15.6241ms 15.6241ms 15.6241ms 15.6241ms 15.6241ms 15.6241ms 15.6241ms 15.6241ms 15.6241ms]

here. I have windows xp.

Alex