Leap Seconds and IPv6

[David Malone]

Two small items.

1) I'm curious to know if anyone saw any unusual leap-second related
activity. There has been some talk of a Linux problem relating to
printk being called with too many locks held causing some machines
to lock up. I was wondering if anyone noticed anything else.

2) There's an Irish IPv6 Summit being held later this month, thay
may be of interest to some people on this list:

http://www.ipv6.ie/summit2009/

Early bird registration is still open I believe.

David.

[David Malone]

On Mon, Jan 05, 2009 at 10:57:09AM +0000, John P. Looney wrote:
> This year, we (Google) decided to ignore posix, and include the leap second
> gradually over a day, rather than all at once. It helped me sleep
> easier...so we saw nothing!

Big Ben did a similar thing I believe:

http://news.bbc.co.uk/2/hi/science/nature/7797818.stm

David.

[Rob Gallagher]

Hi David,

2009/1/5 David Malone <dwmalone=sysad...@maths.tcd.ie>:

> Two small items.
>
> 1) I'm curious to know if anyone saw any unusual leap-second related
> activity. There has been some talk of a Linux problem relating to
> printk being called with too many locks held causing some machines
> to lock up. I was wondering if anyone noticed anything else.
>

We haven't noticed any unusual side effects (yet!).

Our two stratum 1s, which are in the ie.pool.ntp.org rotation, peaked
at about 800kbit/s and 1.34mbit/s of incoming traffic. Although the
majority of this was several hours before midnight.

The NTP server that saw less traffic also drifted by 1s for a short
while, since it is MSF-backed I believe this is expected as the MSF
signal does not pre-announce the leap second.

rg

--
rob.gallagher (at) gmail.com || www.spoofedpacket.net || PK: 0x1DD13A78

[Gavin McCullagh]

Hi Dave,

On Mon, 05 Jan 2009, David Malone wrote:

> 1) I'm curious to know if anyone saw any unusual leap-second related
> activity. There has been some talk of a Linux problem relating to
> printk being called with too many locks held causing some machines
> to lock up. I was wondering if anyone noticed anything else.

I hadn't connected the two issues until now when you said it, but my Ubuntu
intrepid (8.10) desktop hung at almost exactly midnight. The last message
in /var/log/messages is:

Dec 31 23:56:49 ceartgoleor -- MARK --
Jan 5 09:34:34 ceartgoleor syslogd 1.5.0#2ubuntu6: restart.

and syslog:

Dec 31 23:59:01 ceartgoleor /USR/SBIN/CRON[16498]: (root) CMD ([ -x /usr/lib/sysstat/sa1 ] && { [ -r "$DEFAULT" ] && . "$DEFAULT" ; [ "$ENABL ED" = "true" ] && exec /usr/lib/sysstat/sa1 $SA1_OPTIONS 60 2 ; })
Jan 5 09:34:34 ceartgoleor syslogd 1.5.0#2ubuntu6: restart.

I've just noted it now, but I guess it may be what you're talking about.
We have a bunch of Ubuntu Hardy (8.04) servers all of whose ntpd instances
nicely logged the extra second in syslog and moved on with life as usual.
Perhaps something new in Intrepid has been broken.

Gavin

[John P. Looney]

 This year, we (Google) decided to ignore posix, and include the leap second gradually over a day, rather than all at once. It helped me sleep easier...so we saw nothing!

John

--
triad 238: Trí luchra ata mesa: luchra tuinde, luchra mná bóithe, luchra confoléimnige.
Three worst smiles: the smile of a wave, the smile of a lewd woman, thegrin of a dog ready to leap.

[Brian Boyle]

Hi Dave,

> 1) I'm curious to know if anyone saw any unusual leap-second related
> activity. There has been some talk of a Linux problem relating to
> printk being called with too many locks held causing some machines
> to lock up. I was wondering if anyone noticed anything else.

My Ubuntu desktop (Intrepid Ibex) took a dive at midnight. I've no
idea why - seconds a leaping or otherwise. My logs are unhelpful.

Do you know if it would be possible to create a test environment
to re-create the NTP conditions around a leap second? I'm guessing
I'd need some way of replaying a rigged GPS or MSF signal into
an ntpd server, while watching a subservient system for signs of
problems.

Brian.
--
Brian Boyle, Network Services Manager
HEAnet Limited, Ireland's Education and Research Network
1st Floor, 5 George's Dock, IFSC, Dublin 1
Registered in Ireland, no 275301 tel: +353-1-660 9040 fax: +353-1-660 3666
web: http://www.heanet.ie/

[David Malone]

On Mon, Jan 05, 2009 at 02:59:41PM +0000, Brian Boyle wrote:
> My Ubuntu desktop (Intrepid Ibex) took a dive at midnight. I've no
> idea why - seconds a leaping or otherwise. My logs are unhelpful.

Sounds like the same bug. According to what I've heard (though I'm
not sure the details are completly clear yet) Linux does a printk
when it inserts the leap second. However, sometimes some locks, if
locks are held at the time that prink occurs, this can lead to a
(dead?)lock. However, I'd wait until there is a proper analysis
before placing too much weight on that explaination.

> Do you know if it would be possible to create a test environment
> to re-create the NTP conditions around a leap second? I'm guessing
> I'd need some way of replaying a rigged GPS or MSF signal into
> an ntpd server, while watching a subservient system for signs of
> problems.

Here's how I'd recreate the leap second, if I had to:

1) Set up a machine with a recent ntpd as a server, configure
only the local clock as a refclock.

2) Configure the server with a leap seconds file, as described
at
http://support.ntp.org/bin/view/Support/ConfiguringNTP#Section_6.13.
make sure the file you grab is recent enough to have the Dec
2008 leap in it.

3) Set the date to be late in December 2008, so there's a
leap comming up.

4) Configure your test machine to use only your specially
configured NTP server. Once it is synced up, wait and see
what happens.

ntpd should do the right thing here, but I have to admit that I haven't
checked!

David.

[Brendan Minish]

I have a 'DIY' stratum 1 IPv6 timeserver that runs NanoBSD

It's based on one of these
http://www.febo.com/pages/soekris/
System clock is driven by a 'DIY' GPS disciplined(*) Rubidium source and
the precision timer is fed by 1PPS from a Motorola Oncore Jupiter GPS
module and the GPS feeds coarse time although it also tracks other
servers.

it's publicly available via IPv6 at balrog.home.minish.org unfortunately
at present it's a couple of jittery wireless hops and a sixxs tunnel
away from everywhere else.

I saw no issues with this at leapsecond time, the Leap second was
handled in a POSIX compliant way

thankfully none of the machines I admin had any issues at the leap
second. These are mostly running patched up to date centos 5.x and 4.x

My desktop (fedora 10) and the eeePC 901 (running Ubuntu 8.10) that I
was using to monitor the leap second also managed the leapsecond without
issue.

(*) from an NTP accuracy perspective there is no need for the system CPU
clock to be GPS disciplined, just very, very stable but the Rubidium
source is GPS disciplined here to allow for precise frequency
measurements.

.brendan
--
Brendan Minish, Westnet, Technology House, Castlebar, Co Mayo.
1850 930305 Registered in Ireland, Company no. 399770
Western Broadband Networks Limited, T/A WestNet

[Gavin McCullagh]

Hi,

On Mon, 05 Jan 2009, Brian Boyle wrote:

> My Ubuntu desktop (Intrepid Ibex) took a dive at midnight. I've no
> idea why - seconds a leaping or otherwise. My logs are unhelpful.

If you want to add your tuppence here, please do:

https://bugs.launchpad.net/ubuntu/+bug/314029

Sounds like we're seeing the same thing.

Gavin

[David Malone]

On Mon, Jan 05, 2009 at 03:23:17PM +0000, Colm MacC?rthaigh wrote:

> On Mon, Jan 5, 2009 at 3:15 PM, David Malone
> <dwmalone=sysad...@maths.tcd.ie> wrote:
> > ntpd should do the right thing here, but I have to admit that I haven't
> > checked!
>

> http://fortytwo.ch/mailman/pipermail/timekeepers/2005/001917.html

I think you can just do this with:

ntpq -c "rv 0 leap"

and check the "leap=" response. I've been collecting some stats on
this - there's a graph at:

http://www.maths.tcd.ie/~dwmalone/time/leap2008/ntp2008-leapbits.png

but I haven't had time to write it up yet.

David.

[Rob Gallagher]

2009/1/5 David Malone <dwmalone=sysad...@maths.tcd.ie>:

> Two small items.
>
> 1) I'm curious to know if anyone saw any unusual leap-second related
> activity. There has been some talk of a Linux problem relating to
> printk being called with too many locks held causing some machines
> to lock up. I was wondering if anyone noticed anything else.
>

There's also some interesting discussions on the nanog list about what
various people saw:

http://www.irbs.net/internet/nanog/0812/0705.html

Some anecdotal evidence of linux 2.6.21.5 machines and Oracle boxes
falling over.

[Nick Hilliard]

Rob Gallagher wrote:
> Some anecdotal evidence of linux 2.6.21.5 machines and Oracle boxes
> falling over.

... and yet more evidence that vendors / coders do not realise that leap
seconds or even leap years exist.

Nick

[David Malone]

> this - there's a graph at:
>
> http://www.maths.tcd.ie/~dwmalone/time/leap2008/ntp2008-leapbits.png
>
> but I haven't had time to write it up yet.

If anyone is interested, I now have some of this written up:

http://www.maths.tcd.ie/~dwmalone/time/leap2008.html

David.