Preparing for the leap second

[Marco Marongiu]

Hi all

So, we have a leap second at the end of June. Before then, I'd like to
secure that our time-sensitive applications (e.g., a well known database
from a well-known software and hardware vendor) won't crash as they did
in 2006.

As far as I remember, in 2006 we found out that the system clock stepped
back one second, the database engine got mad, and crashed. Back then we
were using NTPv3 because xntpd on some Solaris host didn't know about v4.

But it was 6 years ago, and things seem to have changed in the meanwhile :)

I understand from "the NTP timescale and leap seconds" by Prof.Mills
that modern ntpd doesn't step back the clock but either "freezes" time
during the leap second, or it slightly increments it at each read until
the "real time" catches up.

Did I get it correctly?

If yes, then I would like to know:

- how can I tell if the operating system will freeze/slowdown, or step
the clock?

- how can I simulate a leap second, and see how the system reacts?

My understanding is that we have the freeze/slowdown behaviour
implemented from ntpd version 4 [1], but I don't find anything more
specific in terms of ntpd and OSs versions.

If you have detailed operating systems- and ntpd versions that are known
to work either way, could you please share?

Thanks a lot

Ciao
-- bronto

[1] http://www.eecis.udel.edu/~mills/ntp/html/release.html

[unruh]

It sounds like your old system may not have had the leap second
implimented. In that case ntp would suddenly find it was out by a
second. It would then make a jump in time ( sine the threshold is .128
sec) which with the inserted second would make it jump backward in time
by a second, which is what you saw happen.
Or you could run chrony, which does not jump (well you can have it jump
depending on the user selected threshold) but slews quickly.
(that only works on Linux/bsd not windows)

think you may be able to adjust that threshold in ntpd as well, and
make it greater than 1 second.

[Chuck Swiger]

On Jan 10, 2012, at 9:12 AM, Marco Marongiu wrote:
> I understand from "the NTP timescale and leap seconds" by Prof.Mills
> that modern ntpd doesn't step back the clock but either "freezes" time
> during the leap second, or it slightly increments it at each read until
> the "real time" catches up.
>
> Did I get it correctly?

I think so, yes.

> If yes, then I would like to know:
>
> - how can I tell if the operating system will freeze/slowdown, or step
> the clock?

Ask your operating system vendor, or look at the source code:

http://www.freebsd.org/cgi/cvsweb.cgi/~checkout~/src/sys/kern/kern_tc.c

> - how can I simulate a leap second, and see how the system reacts?

Change the system time to a few minutes before any particular leap second that your OS knows about (CF leapseconds file, if it has one) and see.

Regards,
--
-Chuck

[unruh]

No. Your os does not know about leapseconds, and the only way ntpd knows
is if the leapsecond flag in the packet exchanged with a server is set.
Thus you would have to simulate packets coming in to your system.
I believe that the leapsecond handling is actually in the kernel, not in
ntpd. When ntpd tells the kernel to insert a leapsecond at the day
turnover, then the system behaves as Mills decribes. (stops the clock
for a second but returns a monotonic time output if the kernel clock is
requested for time with a very small delta t.)

Thus you would have to ask your kernel vendor how it behaves. Not sure
what ntpd does on a Windows system.


>
> Regards,

[Chris Adams]

Once upon a time, Marco Marongiu <bront...@gmail.com> said:
>As far as I remember, in 2006 we found out that the system clock stepped
>back one second, the database engine got mad, and crashed. Back then we
>were using NTPv3 because xntpd on some Solaris host didn't know about v4.

Another problem I saw in 2009 was with older versions of Linux kernels
(such as that found in Red Hat Enterprise Linux 4). The kernel handled
inserting the leap second okay, but when it tried to print an
informative message about it, the system would deadlock if it was under
load.

That particular bug was easy to reproduce on a test system. I had a
script that set the time to a few seconds before midnight UTC, set the
leap flag with adjtimex(), count until a couple of seconds after
midnight, and loop. It would be fine as long as the system was idle,
but as soon as it was busy, the kernel deadlocked.

A similar setup might be used to test other problem software.
--
Chris Adams <cma...@hiwaay.net>
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.

[Dave Hart]

On Tue, Jan 10, 2012 at 19:47, unruh <un...@invalid.ca> wrote:
> No. Your os does not know about leapseconds, and the only way ntpd knows
> is if the leapsecond flag in the packet exchanged with a server is set.

There are two other ways ntpd learns of pending leap seconds which
override any upstream server leap bits. First you can install the
leap-seconds file and reference it in ntp.conf:

leapfile "/etc/leap-seconds"

Secondly, autokey clients will securely retrieve the leapseconds
values from autokey servers which use leapfile.

> I believe that the leapsecond handling is actually in the kernel, not in
> ntpd. When ntpd tells the kernel to insert a leapsecond at the day
> turnover, then the system behaves as Mills decribes. (stops the clock
> for a second but returns a monotonic time output if the kernel clock is
> requested for time with a very small delta t.)
> Thus you would have to ask your kernel vendor how it behaves.

The leapsecond support is typically present in both ntpd and the
kernel NTP extensions. If the kernel loop discipline is enabled, the
kernel is responsible for inserting the leap second as scheduled by
ntpd. Exactly how the kernel implements that insertion (such as
whether monotonicity is preserved) is implementation-dependent and can
be gleaned from system source code or documentation. On non-Windows
systems with the kernel loop discipline disabled (by -x or "tinker
step" [1] or "disable kernel") ntpd will step the clock back one
second sometime during the first second of the first of the month,
repeating part of 23:59:59 of the last day of the month.

> Not sure what ntpd does on a Windows system.

On Windows, there is no kernel leapsecond support, but the clock can
run at any rate. To avoid stepping backwards, the Windows port of
ntpd runs the system clock at 150% of the nominal rate for 2
unmodified seconds starting sometime during the first second of the
month, thus remains monotonic though with two nasty frequency steps.

Cheers,
Dave Hart

[Chuck Swiger]

On Jan 10, 2012, at 11:47 AM, unruh wrote:
>> Ask your operating system vendor, or look at the source code:
>>
>> http://www.freebsd.org/cgi/cvsweb.cgi/~checkout~/src/sys/kern/kern_tc.c
>>
>>> - how can I simulate a leap second, and see how the system reacts?
>>
>> Change the system time to a few minutes before any particular leap second that your OS knows about (CF leapseconds file, if it has one) and see.
>
> No. Your os does not know about leapseconds, and the only way ntpd knows
> is if the leapsecond flag in the packet exchanged with a server is set.
> Thus you would have to simulate packets coming in to your system.

In point of fact, ntpd will look for signed leapseconds data from ntpkey_leap-- see ntpd/ntp_crypto.c. I don't see any reason why this won't work with ntpd running in orphan mode to simulate a leap-second transition, even it isn't talking to an upstream server (or via simulated NTP traffic).

And, also in point of fact, zic used to compile the zoneinfo database can be told to pay attention to the NIST leapseconds table to figure out the TAI-UTC offset for each of the timezone files it processes.

> I believe that the leapsecond handling is actually in the kernel, not in
> ntpd.

Um. If you look through the cvsweb link above, you might notice that I was pointing to the kernel code for a particular OS which implements leap second handling.

Regards,
--
-Chuck

[Dave Hart]

On Tue, Jan 10, 2012 at 20:47, Dave Hart <ha...@ntp.org> wrote:
> ntpd.  Exactly how the kernel implements that insertion (such as
> whether monotonicity is preserved) is implementation-dependent and can
> be gleaned from system source code or documentation.  On non-Windows
> systems with the kernel loop discipline disabled (by -x or "tinker
> step" [1] or "disable kernel") ntpd will step the clock back one
> second sometime during the first second of the first of the month,
> repeating part of 23:59:59 of the last day of the month.

[1] http://www.eecis.udel.edu/~mills/ntp/html/miscopt.html#tinker

Cheers,
Dave Hart

[David J Taylor]

"unruh" <un...@invalid.ca> wrote in message
news:hB0Pq.35$sX2...@newsfe02.iad...
[]

> Not sure what ntpd does on a Windows system.

It behaved quite nicely last time. IIRC it either slows the clock by 50%
or speeds it up by 100%, so that it either takes 1 second to go from
00:00:00 to 00:00:02, or two seconds to go from 00:00:00 to 00:00:01, for
example.

The problem used to be that it only needed one rogue server announcing a
leap-second to trigger the change, but that was fixed for recent NTP
versions.

Cheers,
David

[Marco Marongiu]

So, wrapping up:

Il 10/01/2012 18:12, Marco Marongiu ha scritto:
> I understand from "the NTP timescale and leap seconds" by Prof.Mills
> that modern ntpd doesn't step back the clock but either "freezes" time
> during the leap second, or it slightly increments it at each read until
> the "real time" catches up.

This was confirmed, so I should expect the time on the host to be
monotonic. OK.

> - how can I tell if the operating system will freeze/slowdown, or step
> the clock?

This was not really answered, probably for my own fault. I'll rephrase
the question as follows:

Is there any system tool that will allow me to understand if the system
will freeze/slowdown, or step the clock in the event of a leap second?

> - how can I simulate a leap second, and see how the system reacts?

This was not fully answered. It could be my own fault again. Please let
me know if this could be a good way to test if a given combination
OS/ntpd will behave (doesn't apply to Windows tho).

I should shut down ntpd and configure the node as an "orphan" -- maybe
unplugging the network cable, or shutting down the interface, or setting
no "server" for it... whatever it is.

Then I should set the clock close to midnight, start ntpd again, and ask
the kernel to insert a leap second. Or maybe I should insert the leap
second _before_ starting ntpd?

I understand I could add a fake leap second by using "adjtimex -S", or
by using a fake leapseconds file. Is it correct? Any adjtimex command
line you may want to suggest, or discourage me to use?

Then, I should just watch the clock go, and see if it is monotonic when
midnight comes. Would this be a good way to do that?

while true ; do date +%H:%M:%S.%N ; done | tee time.log

Thanks

-- bronto

[unruh]

On 2012-01-11, Marco Marongiu <bront...@gmail.com> wrote:
> So, wrapping up:
>
> Il 10/01/2012 18:12, Marco Marongiu ha scritto:
>> I understand from "the NTP timescale and leap seconds" by Prof.Mills
>> that modern ntpd doesn't step back the clock but either "freezes" time
>> during the leap second, or it slightly increments it at each read until
>> the "real time" catches up.
>
> This was confirmed, so I should expect the time on the host to be
> monotonic. OK.
>
>
>> - how can I tell if the operating system will freeze/slowdown, or step
>> the clock?
>
> This was not really answered, probably for my own fault. I'll rephrase
> the question as follows:
>
> Is there any system tool that will allow me to understand if the system
> will freeze/slowdown, or step the clock in the event of a leap second?

If you told us what operating system you were using, more useful
suggestions might follow. YOu now have the answer for windows.
Whichoperating system are you using? Or do you want us to keep guessing?

[Marco Marongiu]

Il 11/01/2012 18:28, unruh ha scritto:
> If you told us what operating system you were using, more useful
> suggestions might follow. YOu now have the answer for windows.
> Whichoperating system are you using? Or do you want us to keep guessing?

I am mainly thinking of Linux, all 2.6 series; distributions may be
Debian Squeeze, some Ubuntu 10.04 LTS or higher, a very few Debian Lenny.

If possible, I'd be interested in Windows XP and higher (for desktops).
Nice to have, modern versions of Solaris -- like 11 and OpenIndiana.

Thanks

-- M

[Hal Murray]

In article <4F0DB4D6...@gmail.com>,

Marco Marongiu <bront...@gmail.com> writes:
>So, wrapping up:
>
>Il 10/01/2012 18:12, Marco Marongiu ha scritto:
>> I understand from "the NTP timescale and leap seconds" by Prof.Mills
>> that modern ntpd doesn't step back the clock but either "freezes" time
>> during the leap second, or it slightly increments it at each read until
>> the "real time" catches up.
>
>This was confirmed, so I should expect the time on the host to be
>monotonic. OK.

I wouldn't trust that. At least on the systems I've worked with,
the leap second fudge is done in the kernel rather than ntpd.

You can write some test code. Write a loop that prints the CPU's TSC
and the time, then spins a while to avoid too much logging. Collect
a lot of data over a leap second, then graph it.

(or google around and find timelog.c that Markus Kuhn wrote back in 2005)

--
These are my opinions, not necessarily my employer's. I hate spam.

[Marco Marongiu]

Hi again

Anybody willing to help with this procedure? There are some spots where
I could use some suggestions:

Il 11/01/2012 17:12, Marco Marongiu ha scritto:
> Please let me know if this could be a good way to test if a given
> combination OS/ntpd will behave (doesn't apply to Windows tho).
>
> I should shut down ntpd and configure the node as an "orphan" --
> maybe unplugging the network cable, or shutting down the interface,
> or setting no "server" for it... whatever it is.
>
> Then I should set the clock close to midnight, start ntpd again, and
> ask the kernel to insert a leap second. Or maybe I should insert the
> leap second _before_ starting ntpd?
>
> I understand I could add a fake leap second by using "adjtimex -S",
> or by using a fake leapseconds file. Is it correct? Any adjtimex
> command line you may want to suggest, or discourage me to use?
>
> Then, I should just watch the clock go, and see if it is monotonic
> when midnight comes. Would this be a good way to do that?
>
> while true ; do date +%H:%M:%S.%N ; done | tee time.log

As I said in another message, I am mainly thinking of Linux, all 2.6
series, or above; distributions may be Debian Squeeze, some Ubuntu 10.04

LTS or higher, a very few Debian Lenny.

If possible, I'd be interested in Windows XP and higher (for desktops).
Nice to have, modern versions of Solaris -- like 11 and OpenIndiana.

Thanks a lot

Ciao
-- M

[Martin Burnicki]

Marco Marongiu wrote:
> Hi again
>
> Anybody willing to help with this procedure? There are some spots where
> I could use some suggestions:
>
> Il 11/01/2012 17:12, Marco Marongiu ha scritto:
>> Please let me know if this could be a good way to test if a given
>> combination OS/ntpd will behave (doesn't apply to Windows tho).
>>
>> I should shut down ntpd and configure the node as an "orphan" --
>> maybe unplugging the network cable, or shutting down the interface,
>> or setting no "server" for it... whatever it is.
>>
>> Then I should set the clock close to midnight, start ntpd again, and
>> ask the kernel to insert a leap second. Or maybe I should insert the
>> leap second _before_ starting ntpd?

You should simply set up one machine as NTP server with the current leap
second file which announces a leap second for the end of June, 2012.
Depending on the NTP version on the server, you should use "orphan" or
configure only the "local clock" as reference time, so the server just
outputs its own system time. You should also fudge the server to a good
stratum for this test.

Then you should set UTC time on that server close to (maybe 1 or 2 hours
before) midnight for the correct leap second date, e.g. 22:00 UTC on
June 30, 2012, and start ntpd on the server.

Don't make the time interval before UTC midnight too short to give ntpd
some time to "settle" after startup.

If you try with a different date then ntpd might not accept the leap
second since (if I remember correctly) it does some plausibility checks
and tests if the leap second has been scheduled for June 30 or for
December 31.

Finally you can start all your client machines, run something like the
script you have proposed, maybe with an additional sleep of a few
milliseconds, on the clients and on the server, e.g.:

>> while true ; do date +%H:%M:%S.%N ; sleep 0.2; done | tee time.log

Keep in mind that your "date" command which I've quoted above prints
local time, not UTC, so when the leap second occurs the command may not
show midnight, but a time offset according to your timezone setting. Try
"date -u" to display your system's UTC time.

In the time interval before UTC midnight you can run "ntpq -c rv" on the
server and on the clients to see if the leap second warning has been
recognized on the server and propagated to the clients, in which case
you should see "leap=01" in the output.

>> I understand I could add a fake leap second by using "adjtimex -S",
>> or by using a fake leapseconds file. Is it correct? Any adjtimex
>> command line you may want to suggest, or discourage me to use?

I have never tried this, but using the leapsecond file on your server
should be straightforward.

>> Then, I should just watch the clock go, and see if it is monotonic
>> when midnight comes. Would this be a good way to do that?

Yes.

> As I said in another message, I am mainly thinking of Linux, all 2.6
> series, or above; distributions may be Debian Squeeze, some Ubuntu 10.04
> LTS or higher, a very few Debian Lenny.
>
> If possible, I'd be interested in Windows XP and higher (for desktops).
> Nice to have, modern versions of Solaris -- like 11 and OpenIndiana.

You can use any OS and any client you want to test in parallel if you
point them to the test server, and see how it works on each machine.


Martin
--
Martin Burnicki

Meinberg Funkuhren
Bad Pyrmont
Germany

[Marco Marongiu]

Il 24/01/2012 10:03, Martin Burnicki ha scritto:
[...]

I've just found out I missed this message. thanks a lot Martin, I'll try
out your procedure and see how it goes. I already have an old desktop PC
waiting for my tests :)

Ciao
-- bronto