time.monotonic() roll over

[ast]

Hello,

Does any body know when time.monotonic() rolls over ?

On python doc https://docs.python.org/3/library/time.html
it is said every 49.7 days on Windows versions older than
Vista. For more recent Windows, it is sais that monotonic()
is system-wide but they dont say anything about roll over,
probably it hasn't changed.

Anyway, if someone need a monotonic clock, he may think
to use either time.time() or time.monotonic().
There is no roll over problem with time.time() since the very
first one in planned far in the future, but time.time() can go
backward when a date update throught NTP server is done.

time.monotonic() is monotonic but roll over often (every 49.7
days)

So what to do ?

Using time.monotonic() and take some actions if a roll over
is detected ?

[Marko Rauhamaa]

"ast" <nom...@invalid.com>:

> Does any body know when time.monotonic() rolls over ?

Never, according to the documentation you linked.

Admittedly, the documentation confuses the reader by chatting about some
irrelevant internal Windows details.

Also, the tone of the documentation raises a suspicion that this code
might return a bad value on Windows:

def fifty_days():
a = time.monotonic()
time.sleep(50 * 24 * 3600)
return time.monotonic() - a

That is, the internal integer wrap is not guarded against between the
calls to time.monotonic(), maybe.


Marko

[Ian Kelly]

On Dec 4, 2014 8:56 AM, "Marko Rauhamaa" <ma...@pacujo.net> wrote:
>
> "ast" <nom...@invalid.com>:
>
> > Does any body know when time.monotonic() rolls over ?
>
> Never, according to the documentation you linked.
>
> Admittedly, the documentation confuses the reader by chatting about some
> irrelevant internal Windows details.

Not entirely irrelevant. The implication is that if you go more than 49 days without calling the function on old Windows systems, rollovers could be missed, which is good to know about. The result would still be monotonic, but it wouldn't accurately reflect the time elapsed.

[Chris Angelico]

On Fri, Dec 5, 2014 at 3:05 AM, Ian Kelly <ian.g...@gmail.com> wrote:
> On Dec 4, 2014 8:56 AM, "Marko Rauhamaa" <ma...@pacujo.net> wrote:
>>

>> "ast" <nom...@invalid.com>:
>>
>> > Does any body know when time.monotonic() rolls over ?
>>
>> Never, according to the documentation you linked.
>>
>> Admittedly, the documentation confuses the reader by chatting about some
>> irrelevant internal Windows details.
>

> Not entirely irrelevant. The implication is that if you go more than 49 days
> without calling the function on old Windows systems, rollovers could be
> missed, which is good to know about. The result would still be monotonic,
> but it wouldn't accurately reflect the time elapsed.

I don't know for sure about the newer Windowses, but I believe they
use a 64-bit counter instead of the 32-bit one used in previous
versions, so even if they do roll over, there'll be a much MUCH longer
time scale involved. Even if it stores time in nanoseconds, a 64-bit
counter would allow for hundreds of years between rollovers, which is
reasonably safe - much better than the month-and-a-bit of older ones,
which is shorter than quite a lot of my programs' uptimes. Hence the
lack of information about them; you don't have to worry. (Likewise on
Unix-derived systems, I believe, and for the same reason.)

ChrisA

[Marko Rauhamaa]

Ian Kelly <ian.g...@gmail.com>:

> The implication is that if you go more than 49 days without calling
> the function on old Windows systems, rollovers could be missed, which
> is good to know about.

The implication is all but clear, but that was my suspicion. It would be
so bad that the documentation should be much more explicit about it.

No, that would be a Python standard library bug, period.


Marko

[Chris Angelico]

It's not a Python issue. Python can't do anything more than ask the
system, and if the system's value rolls over several times a year,
Python can't magically cure that. The information has already been
lost.

ChrisA

[rand...@fastmail.us]

On Thu, Dec 4, 2014, at 10:50, Marko Rauhamaa wrote:
> That is, the internal integer wrap is not guarded against between the
> calls to time.monotonic(), maybe.

Looking at the code, it looks like it does guard against the rollover,
though if you let your program run for 49.7 days _without_ calling
time.monotonic during that time, it will not realize that it has passed
the original value multiple times. And of course, if it rolls over
before the first time you call monotonic in process B, process B's value
will be less than process A which called it before the rollover.

This is consistent with the documentation, and appears to be the reason
the documentation discusses this issue at all.

[Marko Rauhamaa]

Chris Angelico <ros...@gmail.com>:

> It's not a Python issue. Python can't do anything more than ask the
> system, and if the system's value rolls over several times a year,
> Python can't magically cure that. The information has already been
> lost.

Sure it could by having an invisible background thread occasionally call
time.monotonic(). It could even be done on the side without a thread.

Anyway, the idea of a clock is complicated:

* the program could be stopped by a STOP signal

* the program could be suspended from power management

* the program could be resurrected from a virtual machine snapshot

* the program could be migrated from a different physical machine

So, if I call

time.sleep(86400)

and the program is suspended for 24 hours, should time.sleep() return
right after it is resumed or after another 24 hours?


Marko

[Chris Angelico]

On Fri, Dec 5, 2014 at 5:09 AM, Marko Rauhamaa <ma...@pacujo.net> wrote:
> Sure it could by having an invisible background thread occasionally call
> time.monotonic(). It could even be done on the side without a thread.

No, it can't be solved by anything in that process, because...

> * the program could be stopped by a STOP signal

... what you said There's no way to guarantee to keep calling the
function. You have to depend on upstream.

ChrisA

[Marko Rauhamaa]

Chris Angelico <ros...@gmail.com>:

> ... what you said There's no way to guarantee to keep calling the
> function. You have to depend on upstream.

The caveats I listed are real concerns for the modern-day programmer.
However, they are of a different nature than the complaint against
time.monotonic().


Marko

[Ian Kelly]

On Thu, Dec 4, 2014 at 11:09 AM, Marko Rauhamaa <ma...@pacujo.net> wrote:
>
> Chris Angelico <ros...@gmail.com>:
>
> > It's not a Python issue. Python can't do anything more than ask the
> > system, and if the system's value rolls over several times a year,
> > Python can't magically cure that. The information has already been
> > lost.
>
> Sure it could by having an invisible background thread occasionally call
> time.monotonic(). It could even be done on the side without a thread.
>
> Anyway, the idea of a clock is complicated:
>
>  * the program could be stopped by a STOP signal
>
>  * the program could be suspended from power management
>
>  * the program could be resurrected from a virtual machine snapshot
>
>  * the program could be migrated from a different physical machine

This seems like a lot of effort to unreliably design around a problem that will matter to only a tiny fraction of users.

[Chris Angelico]

But they're also reasons you can't have Python code around the
problem. Anyway, it's only an issue for one platform, and only for
some versions of it; basically, if you're targeting anything other
than Windows XP, you should be able to use time.monotonic() without
concerns.

ChrisA

[Akira Li]

Ian Kelly <ian.g...@gmail.com> writes:

> On Thu, Dec 4, 2014 at 11:09 AM, Marko Rauhamaa <ma...@pacujo.net> wrote:
>>
>> Chris Angelico <ros...@gmail.com>:
>>
>> > It's not a Python issue. Python can't do anything more than ask the
>> > system, and if the system's value rolls over several times a year,
>> > Python can't magically cure that. The information has already been
>> > lost.
>>
>> Sure it could by having an invisible background thread occasionally call
>> time.monotonic(). It could even be done on the side without a thread.
>>
>> Anyway, the idea of a clock is complicated:
>>
>> * the program could be stopped by a STOP signal
>>
>> * the program could be suspended from power management
>>
>> * the program could be resurrected from a virtual machine snapshot
>>
>> * the program could be migrated from a different physical machine
>

> This seems like a lot of effort to unreliably design around a problem that
> will matter to only a tiny fraction of users.

- people's computers are mostly on batteries (laptops, tablets,
smartphones) -- "suspended from power management" use case
- corporations's computations are mostly virtualized -- possible
"ressurected", "migrated" use case

i.e., the opposite might be true -- non-virtualized PCs connected to AC
are (becoming) minority.


--
Akira

[Chris Angelico]

On Fri, Dec 5, 2014 at 7:24 AM, Akira Li <4kir...@gmail.com> wrote:
>> This seems like a lot of effort to unreliably design around a problem that
>> will matter to only a tiny fraction of users.
>
> - people's computers are mostly on batteries (laptops, tablets,
> smartphones) -- "suspended from power management" use case
> - corporations's computations are mostly virtualized -- possible
> "ressurected", "migrated" use case
>
> i.e., the opposite might be true -- non-virtualized PCs connected to AC
> are (becoming) minority.

That's a massive over-simplification, of course, but given that my
current desktop computer is running a dozen or so VMs for various
purposes (usually not more than 3-4 concurrently), I can't disagree
with you. However, there still are plenty of computers that are always
either fully running, or fully shut down; just because people _can_
suspend with applications running doesn't mean they _will_. (Quite a
few of my VMs, for instance, do not get saved/suspended - I shut them
down whenever I'm done with them. Even when I do suspend a VM, I often
terminate applications in it, and just use suspension to save having
to boot the OS every time. But that's partly because those VMs are the
ones running Windows for specific proprietary apps, and thus are
coping with the vagaries of those apps.)

In any case, it's not at all a problem to have the protection on
systems that won't actually need it. Much better than lacking the
protection on a system that does.

ChrisA

[Marko Rauhamaa]

Chris Angelico <ros...@gmail.com>:

> Even when I do suspend a VM, I often terminate applications in it, and
> just use suspension to save having to boot the OS every time.

One interesting detail is DHCP leases.

When it is resumed from suspension, a linux computer thinks it still has
the old IP address, which might have already been granted to some other
computer on the network. Thus, linux distros are equipped with hacky
scripts that get executed soon after the machine is resumed (and before
a lot of damage is done). One of them forces a DHCP renegotiation.

A similar scheme is missing from virtual machines: a VM that wakes up
from a snapshot doesn't automatically go through the resume scripts,
even though it arguably should.

Systemd probably shuffles the situation still more.


Marko

[Ian Kelly]

On Thu, Dec 4, 2014 at 1:24 PM, Akira Li <4kir...@gmail.com> wrote:

>
> Ian Kelly <ian.g...@gmail.com> writes:
> > This seems like a lot of effort to unreliably design around a problem that
> > will matter to only a tiny fraction of users.
>
> - people's computers are mostly on batteries (laptops, tablets,
>   smartphones) -- "suspended from power management" use case
> - corporations's computations are mostly virtualized -- possible
>   "ressurected", "migrated" use case
>
> i.e., the opposite might be true -- non-virtualized PCs connected to AC
> are (becoming) minority.

By "tiny fraction of users" I was referring to people who a) work with Python 3.3+, b) use time.monotonic in their code, and c) possibly have said code running on versions of Windows older than Vista; not people who suspend their systems or use VMs.

It's not clear to me whether those cases are relevant to the rollover concern anyway. I wouldn't be shocked if the GetTickCount() function simply stopped increasing while the system is suspended, since after all it's not "ticking" during that time.

[Marko Rauhamaa]

Ian Kelly <ian.g...@gmail.com>:

> It's not clear to me whether those cases are relevant to the rollover
> concern anyway. I wouldn't be shocked if the GetTickCount() function
> simply stopped increasing while the system is suspended, since after
> all it's not "ticking" during that time.

So, what's the semantics of time.sleep(), select.select() et al wrt
process or machine suspension?


Marko

[Steven D'Aprano]

Marko Rauhamaa wrote:

> So, if I call
>
> time.sleep(86400)
>
> and the program is suspended for 24 hours, should time.sleep() return
> right after it is resumed or after another 24 hours?

If the program is suspended, then no time should pass for that program.
Since sleep() is given in terms of a duration, not an absolute time ("sleep
until now + 24 hours"), if no time passes for that program, sleep() should
sleep for 24 hours *after the program resumes*.

Unfortunately a lot of systems get that wrong. E.g. I just ran "sleep 30"
from my Linux shell, immediately paused it using Ctrl-Z, waited a couple of
minutes, and used fg to continue. It returned immediately.

Why is this behaviour wrong?

Consider why people might call sleep. They're normally calling it to wait
for something else to complete. Often that something else is an external
process ("wait a few seconds to let the hard drive finish syncing, wait for
the web server to get less busy, wait for the user to catch up...") but it
might be another thread in the same process. If the process is suspended,
that other thread won't get to run, and so even though time on the outside
has continued to move forward, from the perspective of the process, it
hasn't.

The same applies even more so if the process is running in a virtual machine
which has just been suspended.

The same applies if the system clock is adjusted mid-sleep. If I call sleep
60, then immediately adjust the clock forward an hour (say, due to a
daylight savings adjustment), that shouldn't cause sleep to return. It
should still suspend for 60 seconds.



--
Steven

[Nobody]

On Thu, 04 Dec 2014 16:25:44 +0100, ast wrote:

> There is no roll over problem with time.time() since the very
> first one in planned far in the future, but time.time() can go
> backward when a date update throught NTP server is done.

> time.monotonic() is monotonic but roll over often (every 49.7
> days)
>
> So what to do ?
>
> Using time.monotonic() and take some actions if a roll over
> is detected ?

One possibility: use both and monitor the skew (the difference between the
two clocks).

The first time you read them, calculate and store the epoch for
time.monotic(). On subsequent calls, repeat the calculation and check that
the epoch is approximately the same. If it isn't, the difference between
the actual and expected values should be close to an exact multiple of
2**32 milliseconds, which tells you how many times time.monotonic() has
rolled over.

Adjustments to the clock on which time.time() is based shouldn't exceed a
few seconds per year, so there's not much risk that you won't be able to
figure out the correct adjustment.

[Dave Angel]

And I say you have it exactly backwards. People should NOT call sleep()
to measure something, they should only call it to release the processor
for "up to" a specified time. That time should normally be a major
fraction of the time that some external event is expected to take. Then
the code should check the external event and sleep some more if necessary.

If efficiency were not an issue, the code would just do a busy loop,
checking the external condition repeatedly.

Since the OS has no way of knowing whether the thing being waited for is
a thread, another process, a human being, a network operation, or the
end of the world, the interpretation of sleep needs to be the most
conservative one. There are many ways of suspending a process, and some
of them will also suspend the external event. Since the OS cannot know
which case is significant, it has to return control to the caller at the
soonest of the many possible interpretations.


--
DaveA

[Steven D'Aprano]

I never suggested you call sleep to measure something. You call sleep to
*wait* for something. I even used the word "wait" four times in the above.

> they should only call it to release the processor
> for "up to" a specified time.

That's silly. Zero is "up to" any positive value you want. Do you really
intend to say that this would be an acceptable implementation of sleep()?

def sleep(delay):
return

> That time should normally be a major
> fraction of the time that some external event is expected to take. Then
> the code should check the external event and sleep some more if necessary.
>
> If efficiency were not an issue, the code would just do a busy loop,
> checking the external condition repeatedly.

In many cases, you don't have an external condition that can be easily or
practically checked. So you estimate how long you need to wait, add a
safety margin, and write "sleep 5" (for some value of 5), and hope.

If you are a sys admin writing a script that runs during boot time, you do
that a lot. Or at least the ones I work with do :-)

> Since the OS has no way of knowing whether the thing being waited for is
> a thread, another process, a human being, a network operation, or the
> end of the world, the interpretation of sleep needs to be the most
> conservative one.

I agree! And returning immediately is not the most conservative one.

The most conservative approach is to assume that while you're suspended,
*everything else* is suspended too, so when you resume you still have to
sleep for the full N seconds.

> There are many ways of suspending a process, and some
> of them will also suspend the external event. Since the OS cannot know
> which case is significant, it has to return control to the caller at the
> soonest of the many possible interpretations.

/s/latest/



--
Steven

[Dave Angel]

I mentioned measurement because I couldn't imagine any other reason you
would be arguing the other side. You're certainly not waiting for the
event, wleep has no parameters that let you specify what event you might
be waiting for.

>
>> they should only call it to release the processor
>> for "up to" a specified time.
>
> That's silly. Zero is "up to" any positive value you want. Do you really
> intend to say that this would be an acceptable implementation of sleep()?
>
> def sleep(delay):
> return
>

Yes, of course. Not very machine efficient, but perfectly correct.
Because the testing code will make its check, and turn right around and
issue another sleep call. This is called a busy loop, and I mention it
below. It is the degenerate (if inefficient) form.

>
>> That time should normally be a major
>> fraction of the time that some external event is expected to take. Then
>> the code should check the external event and sleep some more if necessary.
>>
>> If efficiency were not an issue, the code would just do a busy loop,
>> checking the external condition repeatedly.
>
> In many cases, you don't have an external condition that can be easily or
> practically checked. So you estimate how long you need to wait, add a
> safety margin, and write "sleep 5" (for some value of 5), and hope.
>
> If you are a sys admin writing a script that runs during boot time, you do
> that a lot. Or at least the ones I work with do :-)
>
>
>> Since the OS has no way of knowing whether the thing being waited for is
>> a thread, another process, a human being, a network operation, or the
>> end of the world, the interpretation of sleep needs to be the most
>> conservative one.
>
> I agree! And returning immediately is not the most conservative one.
>
> The most conservative approach is to assume that while you're suspended,
> *everything else* is suspended too, so when you resume you still have to
> sleep for the full N seconds.
>
>
>> There are many ways of suspending a process, and some
>> of them will also suspend the external event. Since the OS cannot know
>> which case is significant, it has to return control to the caller at the
>> soonest of the many possible interpretations.
>
> /s/latest/
>

I guess we'll just have to agree to disagree. I can now see the
usefulness of a function like you describe, I just can't imagine it
being the sleep() we've all come to know and love.


--
DaveA

[Marko Rauhamaa]

Steven D'Aprano <steve+comp....@pearwood.info>:

> Unfortunately a lot of systems get that wrong. E.g. I just ran "sleep
> 30" from my Linux shell, immediately paused it using Ctrl-Z, waited a
> couple of minutes, and used fg to continue. It returned immediately.
>
> Why is this behaviour wrong?

I think the #1 thing is to specify the behavior clearly. I'm not seeing
that so it is impossible to say if the GNU coreutils sleep or
time.sleep() does what it was designed to do. I must admit I have
neglected to document that situation in some of my related designs.

Also, I think what is right or wrong depends on the use case. Ideally,
there are facilities to implement the desired semantics.

If a program spends long periods in an induced coma (which is becoming
more and more common nowadays), the whole design of the program is under
quite a bit of strain. What to do with time-based statistics collection?
Should periodic operations catch up with the lost time by repeating in a
tight loop for a zillion times? What to do with "impossible,"
out-of-order event sequences?


Marko

[Lele Gaifax]

Steven D'Aprano <steve+comp....@pearwood.info> writes:

> The most conservative approach is to assume that while you're suspended,
> *everything else* is suspended too, so when you resume you still have to
> sleep for the full N seconds.

That's an intriguing interpretation of what sleep() should do, but I
fail to see *how* the function could even attempt to do that: should it
keep its own notion of the elapsed time?

I mean, one simplicistic, naive (and wrong) implementation of Dave's
(and mine, FWIW) could be:

def sleep(secs):
now = time()
while time() < (now + secs):
pass # this is wrong, should really be some kind of "yield_cpu()"

How would it become if it should take into account the intervening
"coma"? Maybe something like

def sleep(secs):
nmillisleeps = secs * 1000
for i in range(nmillisleeps):
now = time()
while time() < (now + 0.001):
pass

?

Also, should it also assume that if the system suffers of a slowdown
instead (maybe simply due to another process using the same /inefficient
and wrong/ implementation of sleep() ;-) then everything else gets
slowed down too, and act accordingly?

just-for-the-sake-of-arguing-ly, lele.
--
nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri
real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia.
le...@metapensiero.it | -- Fortunato Depero, 1929.

[Akira Li]

Dave Angel <da...@davea.name> writes:

...many words about sleep()...

> Since the OS has no way of knowing whether the thing being waited for
> is a thread, another process, a human being, a network operation, or
> the end of the world, the interpretation of sleep needs to be the most

> conservative one. There are many ways of suspending a process, and

> some of them will also suspend the external event. Since the OS
> cannot know which case is significant, it has to return control to the
> caller at the soonest of the many possible interpretations.

That is why there is API (e.g., clock_nanosleep()) that allows us to
choose whether we need a relative delay (e.g., kill a subprocess if it
hasn't finished in 10 seconds) or an absolute deadline (e.g., these
lights should be on 9pm-6am local time).

*Both* use cases are valid.


--
Akira

[Steven D'Aprano]

Lele Gaifax wrote:

> Steven D'Aprano <steve+comp....@pearwood.info> writes:
>
>> The most conservative approach is to assume that while you're suspended,
>> *everything else* is suspended too, so when you resume you still have to
>> sleep for the full N seconds.
>
> That's an intriguing interpretation of what sleep() should do, but I
> fail to see *how* the function could even attempt to do that: should it
> keep its own notion of the elapsed time?

Provided that the operating system can provide a monotonic clock that only
ticks while the process calling it is running, that shouldn't be hard. The
only tricky part is if the monotonic clock rolls over.

I'm not sure how the OS would implement that, but most OSes supply a global
monotonic clock, so it's probably possible. I leave that as an exercise to
the reader :-)

Here's a discussion from the Factor programming language:

http://code-factor.blogspot.com.au/2009/11/monotonic-timers.html



--
Steven

[Ian Kelly]

Rather than continue to speculate, I just tested this. I ran the following script in a Windows 7 VM and paused the VM for part of the duration of the sleep call. I unpaused well before the sleep call would normally have ended.

>>> import time

>>> def test():

...   t1 = time.time()

...   m1 = time.monotonic()

...   time.sleep(60)

...   t2 = time.time()

...   m2 = time.monotonic()

...   return t2 - t1, m2 - m1

...

>>> test()

(84.92642459869386, 60.002399999999994)

As you can see, the sleep call and the monotonic time elapsed both appear to exclude the period when the VM was paused.

Then I tried the same with a MacBook and putting the system to sleep instead of pausing a VM:

>>> test()

(59.889225006103516, 60.00102640298428)

Reducing the sleep time to 10 seconds and suspending the process with Ctrl-Z produces a similar result on the MacBook:

(10.000508069992065, 10.000266332994215)

But the story is different in Linux. Here's what I got with the same Ctrl-Z test:

(16.000478506088257, 16.000478034024127)

I think the relative roundness of those numbers is just coincidence. I tried a second time and got:

(12.613622903823853, 12.613622067990946)

So it seems that it can go either way depending on the OS and perhaps the mechanism of suspension.

[Marko Rauhamaa]

Ian Kelly <ian.g...@gmail.com>:

> On Thu, Dec 4, 2014 at 3:44 PM, Marko Rauhamaa <ma...@pacujo.net> wrote:
>> So, what's the semantics of time.sleep(), select.select() et al wrt
>> process or machine suspension?
>
> Rather than continue to speculate, I just tested this.

The test, of course, can only have one useful result...

> So it seems that it can go either way depending on the OS and perhaps the
> mechanism of suspension.

... and you got it.

If the semantics had coincided, the question would remain unresolved; it
could just have been a coincidence.

However, the variance in the results means the semantics are
unspecified.

I think programmer's would need facilities to determine and control the
suspension behavior.

All things being equal, *early* wakeup is preferable to late wakeup
because the application can then check for a spurious wakeup and go back
to sleep. The reverse situation would be painful to deal with.


Marko