How will GAE handle the leap second at 23:59:60 on 6/30/2012?

[Cesium]

Here's a summary of the problem:

The UNIX leap second mess.

How will App Engine's servers handle the leap second?

David

[Anand Mistry]

In general, Google handles leap seconds by smearing that second over the course of some time window. You can read about it on the official Google blog at http://googleblog.blogspot.com.au/2011/09/time-technology-and-leaping-seconds.html

[Cesium]

That's interesting reading from the Site Reliability Engineering Group (SREG).

I read:

Having accurate time is critical to everything we do at Google. 

Agreed.

I also read:

Usually when a leap second is almost due, the NTP protocol says a server must indicate this to its clients by setting the “Leap Indicator” (LI) field in its response. This indicates that the last minute of that day will have 61 seconds, or 59 seconds. (Leap seconds can, in theory, be used to shorten a day too, although that hasn’t happened to date.) Rather than doing this, we applied a patch to the NTP server software on our internal Stratum 2 NTP servers to not set LI, and tell a small “lie” about the time, modulating this “lie” over a time window w before midnight:

Huh? SREG thinks (Lies == accuracy).

And:

The leap smear is talked about internally in the Site Reliability Engineering group as one of our coolest workarounds, that took a lot of experimentation and verification, but paid off by ultimately saving us massive amounts of time and energy in inspecting and refactoring code. 

*Forehead slap!* 

SREG thinks (coolness > correctness) && (workarounds > hard work).

Looks like I'll have to get my time from the 'source'.

David

[Jeff Schnitzer]

I'm trying to imagine what kind of application intersects:

1) The set of applications which require fractional-second time precision

2) The set of applications which can be run on GAE, which makes no
guarantees WRT clock skew, let alone latency between components.

Seems to me that Google's solution is a pretty good one. Any solution
needs to balance two risks:

1) Chance an app will be negatively impacted by a second being
0.00001 longer than usual for a day. This chance is probably zero.
In fact, I'm willing to bet that many clocks naturally skew by this
amount in a day.

2) Chance that a piece of software will barf when confronted with a
timestamp like 23:59:60. This is probably quite high. I'll bet 90%
of time parsers ever written will fail.

I would be pissed if my app started throwing errors for one second.
Or wrote data into a database that subsequently couldn't be parsed
back out. Yuck.

Jeff

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/TAlU_-QPZ1QJ.
>
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengi...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

[Cesium]

Hey Jeff,

Think of, for example, a global network of sensors measuring geo-physical phenomena.

They need to be synchronized. The maximum uncertainly is determined by the phenomenon being measured.

It all boiled down to needing a time server on port 80 whose difference from UTC was less than

about 500ms.

I really didn't want to use the usual port 13 servers. So I wrote a GAE time server.

Three years have past since I last looked at this issue.

All the data I have collected show that GAE's server's are well within 500ms of UTC(NIST).

Even without 'guarantees.' That's good enough for me.

But today I learn that App Engine has a cool, clever workaround that slowly slews it 1 second off UTC(NIST).

It's not the length of the individual seconds (frequency) that counts. It's the integral of all these seconds (phase) that matters.

That integral in this case will be an error of 1 second. Next time the ERS demands a leap second, the error will be 2 seconds.

This leads to a time stamp that no longer accurately reflects statements like: "Number of seconds since 1 Jan 1970 00:00:00 UTC."

I'm just heart-broken to hear that Google decided to brush these leap seconds under the rug by changing the rate of their time-scale.

They are real points on the UTC time-scale, and important things happen on our planet during these leap seconds.

The punchline is that I did some poking around and found a buddy that serves UTC(NIST) on port 80.

Sweet!

David

[Simon Knott]

I'm really surprised by the servers being within 500ms of each other, to be honest.  

I've got a timesheet product on GAE and I've had clock skew of more than a minute between some instances, although it's fairly rare that it's anywhere near that large and it's a lot better now than it ever has been. 

Cheers,

Simon

[Cesium]

Hi Simon,

You wrote:

I'm really surprised by the servers being within 500ms of each other, to be honest.  

I've got a timesheet product on GAE and I've had clock skew of more than a minute between some instances, although it's fairly rare that it's anywhere near that large and it's a lot better now than it ever has been. 

Were you using time transfer methods to determine your error bars?

David 

[Cesium]

Yet more heartbreak due to engineers not handling the corner cases.

David

[Cesium]

Refactoring our code would be too expensive. Let's just ignore the leap second and hope for the best.

David