decisions about time

[Stefan Karpinski]

I had a very productive conversation with Jacob Quinn and Mike Nolta this afternoon about how to deal with dates and times in Julia and I'm sharing some (still tentative) decisions we reached while it's all fresh in my mind both for my own records and so that others can comment on the design.

===

Times and dates in base Julia will use Universal Time (UT). UT seconds are defined to be the amount of time it takes the Earth to rotate 1/86400 of a turn. This is not a fixed duration – the UT second is not really a unit of time, but rather an angle. The advantage of UT is that it is simple and intuitive – by definition, there are no leap seconds and most importantly, you can use dates and times as far into the past and the future as you want. This is how people expect dates and times to work.

The disadvantage of UT is, of course, that its seconds are not fixed durations – i.e. "time" is not measured in SI seconds. The SI second is the fundamental unit of the other major time standard: Terrestrial Time (TT). This is the kind of time you want if you are doing physics and computing velocities and accelerations and such. If you just want to work with SI seconds, you can load the SIUnits package (or its heir). Converting between UT and TT will require loading another package which implements ΔT – i.e the difference between UT and TT as a function of time (think about that for a second). The reason to separate the conversion from SIUnits is that the data to compute ΔT is quite extensive, and it's entirely possible to work with SI times without needing to do any translation between TT and UT. It may be possible to approximate ΔT more compactly, in which case we could consider just including the translation infrastructure in SIUnits.

UTC is a way of defining units of time so that at larger scales it matches TT, while at larger scales it matches UT. The basic unit of time in UTC is the SI second just like TT, and every time unit from seconds down means the same thing that it does in TT and represents a fixed temporal duration. Above the scale of seconds is where the complications set in. Most UTC minutes are 60 seconds, but every now and then, the last minute of a day is 61 seconds – it contains an extra "leap second". All hours are 60 minutes, but if one of those minutes contains a leap second, the hour is 3601 seconds long instead of 3600 seconds, and similarly, each day has 24 hours, but if one of those hours contains a leap second, the day is 86401 seconds instead of the usual 86400. Given two UTC dates and times, if you know how many leap seconds occurred between them, you can easily compute how many SI seconds the intervening duration consisted of. Although many time systems claim to be using UTC, they are usually actually using UT (e.g. Joda-Time). Among other things, UTC is only defined after 1972 and a few months into the future. If these systems were really using UTC, they would be unable to work with dates and times outside of that range. These systems also typically don't have leap seconds. That is, in fact completely correct behavior for working with UT but it is not UTC.

For system functions that return UNIX times, Julia will simply interpret their values as UT (with the appropriate linear offsets). This is incorrect behavior but *mandated* by POSIX, and it's what every system does. This means that differences between system times will be in terms of UT seconds – so they are technically angles, not durations, although we probably don't want to shove that in people's faces. In particular, you cannot convert a UT duration to SI seconds since that conversion depends on when the duration occurred. Instead, if you want a duration in SI seconds, you need to convert the start and end times to TT first and then subtract them, yielding a difference in SI seconds.

There will not be support for parsing or using political time zones in base Julia. When you call now() to get the current time, you will get a UT timestamps. Both UT and TT timestamps are completely divorced from location – they mean the same thing everywhere and don't imply anything about location. However, we may want to determine the system's current offset from Zulu time (i.e. UTC+0 – the zero-offset timezone, as opposed to the UTC time standard) and print timestamps in local time with the appropriate offset. For example, if I'm in New York and ask for the time, I might see it printed as "2014-02-11 23:48:57 -0500" (this is what Ruby prints), even though the time object that's being printed doesn't include any time zone. Rather, the offset comes from a global configuration that is automatically set at startup time. We may want to honor the TZ environment variable for this. It may seem questionable that the same time value will print differently on different systems, but the key is that if you parse any of those output forms back into a timestamp value, you will get identical values. If you want to default to printing timestamps in Zulu time, you can just set an environment variable (or change the global setting in your .juliarc.jl file).

The TimeZones package will provide zoned date/time types that include both a timestamp and a time zone (this should probably support both offsets and political time zones). The timestamp field is still an absolute UT or TT value (both should be supported) and means the same thing, irrespective of the time zone field; the time zone field is included only to modify how the date and time should be displayed. If you were, for example, storing timestamps with time zones in a data frame, you could just as well store the timestamp and the time zone separately. The main reason for combining the two into a single value is so that the values can be conveniently displayed in the appropriate time zone. This can also allow convenient behaviors like producing another zoned value with the same time zone when adding a duration to a zoned date time.

===

Immense thanks to both Mike and Jacob for this enlightening discussion – especially to Mike for steering us expertly through this tricky subject. Mike, please correct me if I've made any errors here.

[Stefan Karpinski]

A few words about terminology. This is a bit up in the air and I've tried to avoid committing too much to anything in particular in the above description. There's a bit of a tradition to call a specification of a point in time – whether in UT, TT or some other system – a DateTime. This is fine in UT – the definition of which makes it inherently straightforward to determine, for any possible value, via some basic arithmetic, what the date and the time are. When working with TT, on the other hand, the DateTime terminology becomes a bit strange. While it corresponds quite straightforwardly to a particular point in time, you need to know ΔT – which can only be determined empirically and with some error – to be able to say, for a given value, approximately what date or the time it corresponds to. In effect, you need to convert a UT value to TT before it really can be said to represent a date and a time of day. Moreover, regardless of time standard, the date and time depend on your time zone, so calling a value that is inherently independent of time zone a "DateTime" seems a bit wrong.

Instead, I'm to call these things "Timestamps". Others have suggested "Instant". The reason I like Timestamp is that a TT or UT value is an indicator of what moment in time something occurred – which is precisely the definition of a timestamp. "Instant", on the other hand, isn't nearly so specific, instead having connotations of an infinitesimally small interval of time. There's also the distinction between what a thing is and what it represents. The objects in our computer programs are not "instants" – that would imply that a value in the computer's memory is a moment in time, which is nonsensical. On the other hand, these objects are very much "timestamps" – i.e. they are indicators of when something happened. The term "DateTime" seems more appropriate for the composite type that combines a Timestamp with a TimeZone – that combination unambiguously determines both a date and a time.

[Kevin Squire]

UTC is a way of defining units of time so that at larger scales it matches TT, while at larger scales it matches UT. 

I assume you didn't mean to use larger for both here.  Which one should be "smaller"? 

This is an interesting read!

Kevin

[Stefan Karpinski]

On Feb 12, 2014, at 12:35 AM, Kevin Squire <kevin....@gmail.com> wrote:

UTC is a way of defining units of time so that at larger scales it matches TT, while at larger scales it matches UT. 

Yes, quite. The first should be smaller. At smaller scales it matches TT (seconds and below are SI units), at larger scales it matches UT.

[Mauro]

Wouldn't it be easier to make the earth turn at a constant rate?

On Wed, 2014-02-12 at 05:07, ste...@karpinski.org wrote:
> I had a very productive conversation with Jacob Quinn and Mike Nolta this
> afternoon about how to deal with dates and times in Julia and I'm sharing
> some (still tentative) decisions we reached while it's all fresh in my mind
> both for my own records and so that others can comment on the design.
>
> ===
>
> Times and dates in base Julia will use Universal

> Time<http://en.wikipedia.org/wiki/Universal_Time>(UT). UT seconds are

> defined to be the amount of time it takes the Earth to
> rotate 1/86400 of a turn. This is not a fixed duration – the UT second is
> not really a unit of time, but rather an angle. The advantage of UT is that
> it is simple and intuitive – by definition, there are no leap seconds and
> most importantly, you can use dates and times as far into the past and the
> future as you want. This is how people expect dates and times to work.
>
> The disadvantage of UT is, of course, that its seconds are not fixed
> durations – i.e. "time" is not measured in SI

> seconds<http://en.wikipedia.org/wiki/Second>.

> The SI second is the fundamental unit of the other major time
> standard: Terrestrial

> Time <http://en.wikipedia.org/wiki/Terrestrial_Time> (TT). This is the kind

> of time you want if you are doing physics and computing velocities and
> accelerations and such. If you just want to work with SI seconds, you can
> load the SIUnits package (or its heir). Converting between UT and TT will
> require loading another package which implements

> ΔT<http://en.wikipedia.org/wiki/DeltaT> –

> i.e the difference between UT and TT as a function of time (think about
> that for a second). The reason to separate the conversion from SIUnits is
> that the data to compute ΔT is quite extensive, and it's entirely possible
> to work with SI times without needing to do any translation between TT and
> UT. It may be possible to approximate ΔT more compactly, in which case we
> could consider just including the translation infrastructure in SIUnits.
>

> UTC <http://en.wikipedia.org/wiki/Coordinated_Universal_Time> is a way of

> variable<http://linux.die.net/man/3/tzset>for this. It may seem

> questionable that the same time value will print
> differently on different systems, but the key is that if you parse any of
> those output forms back into a timestamp value, you will get identical
> values. If you want to default to printing timestamps in Zulu time, you can
> just set an environment variable (or change the global setting in your
> .juliarc.jl file).
>
> The TimeZones package will provide zoned date/time types that include both
> a timestamp and a time zone (this should probably support both offsets and
> political time zones). The timestamp field is still an absolute UT or TT
> value (both should be supported) and means the same thing, irrespective of
> the time zone field; the time zone field is included only to modify how the
> date and time should be displayed. If you were, for example, storing
> timestamps with time zones in a data frame, you could just as well store
> the timestamp and the time zone separately. The main reason for combining
> the two into a single value is so that the values can be conveniently
> displayed in the appropriate time zone. This can also allow convenient
> behaviors like producing another zoned value with the same time zone when
> adding a duration to a zoned date time.
>
> ===
>
> Immense thanks to both Mike and Jacob for this enlightening discussion –
> especially to Mike for steering us expertly through this tricky subject.
> Mike, please correct me if I've made any errors here.

--
Sent with my mu4e

[Milan Bouchet-Valat]

Le mercredi 12 février 2014 à 00:07 -0500, Stefan Karpinski a écrit :

I had a very productive conversation with Jacob Quinn and Mike Nolta this afternoon about how to deal with dates and times in Julia and I'm sharing some (still tentative) decisions we reached while it's all fresh in my mind both for my own records and so that others can comment on the design.

Thanks for the detailed explanation. This sounds fine to me (disclaimer: I don't really care about leap seconds, more about hours and days, so I'm not competent on the technical details).

The small remark I have is that, as you say in your second message, it would make more sense to call the object storing UT Timestamp, and the object storing UT + time zone DateTime. This will make it much clearer what's the intent of these types, and will prevent many questions of people not using the right object and complaining about lack of time zone support.

My two cents

[Tim Holy]

On Wednesday, February 12, 2014 09:09:35 AM Mauro wrote:
> Wouldn't it be easier to make the earth turn at a constant rate?

I've got a pull request coming that will do just that. It's a bit tricky,
because if I mess it up, quite a lot of havoc could result.

--Tim

(More seriously, quite an enlightening post!)

[Tomas Lycken]

My first reaction when I read UT was "OK, what's this, and why not UTC?"

Then I followed the Wikipedia link, and thought "OK, now I know what UT is. Still, why not UTC?"

Then I read the following sentence:

"The disadvantage of UT is, of course, that its seconds are not fixed durations – i.e. 'time' is not measured in SI seconds."

and thought "WTF!?" (excuse my French...) "Why would anyone want to use this?"

Then I read the rest of the discussion, and realized that this would be a really great way to solve a lot of problems that usually arise in handling dates, datespans etc. Of course, it requires thorough and easy-to-read documentation, since it will be different than similarly named functionality in many other languages, but I do agree that this is a good path to follow. Especially naming things "Timestamp" for references to instances in time, and "DateTime" for references to how these instances in time are represented in various parts of the world.

// T

[Patrick O'Leary]

On Tuesday, February 11, 2014 11:07:21 PM UTC-6, Stefan Karpinski wrote:

There will not be support for parsing or using political time zones in base Julia. When you call now() to get the current time, you will get a UT timestamps. Both UT and TT timestamps are completely divorced from location – they mean the same thing everywhere and don't imply anything about location.

Since it's fresh on *my* mind, will TimeZones have enough information to get GPS time, which is TT-based, with an epoch in 1980? This works out to UTC-(number of leap seconds since the first SV went online), or TAI-19s. It would be nice not to have to pull in all of GPSTK for that.

[Jacob Quinn]

Great writeup Stefan!

I'm a bit confused on the naming scheme though; I didn't quite follow what exactly you want to name what. Can you clarify what you want to call the user-facing type that the user uses to construct a datetime instance vs. the underlying type that the datetime type wraps? My understanding from yesterday was something like

immutable UTInstant <: Instant

    ms::Int64

end

immutable DateTime{C<:Calendar} <: TimeType

    instant::UTInstant

end

Are you just saying we should rename `DateTime` here to `Timestamp`? Or `UTInstant` to `Timestamp`? Or do you just want to have a single `Timestamp` type that wraps a value directly? My vote, for simplicity, would be the following:

immutable Timestamp{P<:Period,C<:Calendar} <: TimeType

    utinstant::P

end

So I would have constructors like `ts = Timestamp(2014,2,12,12)` and could access the underlying utinstant by `ts.utinstant` for the number of angle-seconds my Timestamp represents. This also allows the underlying utinstant to be expressed in different precision levels (i.e. seconds, milliseconds, nanoseconds, etc.).

-Jacob

[Milktrader]

Great summary Stefan. I was gong to set some time aside to grok the Issue on this topic and this is lays it out well so far. 

Dan

[Stefan Karpinski]

On Wed, Feb 12, 2014 at 9:22 AM, Patrick O'Leary <patrick...@gmail.com> wrote:

I may not be understanding the question, but a TimeZone value would just be an indication of one of the political time zones (or an offset), so it wouldn't encode GPS time or anything like that. A TT timestamp, with or without a time zone, because it's based on a number of SI seconds since a fixed point in time in the reference frame of the Earth, will be trivially translatable to and from GPS time. (In fact, maybe storing offsets relative to the GPS base time makes the most sense since it's close to us).

The real question is how to get these values. If you have a pre-existing data set, you can of course just interpret it as GPS clock values, translate those to TT timestamps and then work with those. If you want to collect new data, e.g. from GPS hardware, then you'll need to have some Julia library that can do that – which should produce TT timestamps, rather than the UT timestamps that functions in Base produce. UT timestamps from libc functions are, of course, no good for doing precise physics work, but I think that it's already obvious.

[Patrick O'Leary]

On Thursday, February 13, 2014 8:32:38 AM UTC-6, Stefan Karpinski wrote:

On Wed, Feb 12, 2014 at 9:22 AM, Patrick O'Leary <patrick...@gmail.com> wrote:

Since it's fresh on *my* mind, will TimeZones have enough information to get GPS time, which is TT-based, with an epoch in 1980? This works out to UTC-(number of leap seconds since the first SV went online), or TAI-19s. It would be nice not to have to pull in all of GPSTK for that.

I may not be understanding the question, but a TimeZone value would just be an indication of one of the political time zones (or an offset), so it wouldn't encode GPS time or anything like that.

Maybe I should clarify by asking a more general question: can the timezone database be queried in such a way that I can obtain the number of leap seconds that have occurred between any two Timestamps/Instants, with the goal of interconverting between GPS and UTC while maintaining enough precision for GPS calculations? I think with the further discussion this can safely be considered an implementation detail, but it's an application that is near to me.

[Stefan Karpinski]

I'm still confused about how time zones enter into the picture.

[Mauro]

I think Patrick means, given a timestamp encoded in some local time, say
Pacific Time, is there enough information in to translate it to GPS/TT
time. I would think so.

I have another concern (maybe this was raised in the discussion already
but I don't want to read it a third time): I create a time stamp in the
future, which would be a UTtimestamp by default in Julia. How to I get
the time difference in SI seconds to another timestamp? I think that is
not possible as we don't know deltaT in the future (nor in the distant
past really, but for that we could just define one). We could make a
deltaT up but then, as real time progresses, the real deltaT and the
made up one will diverge. Thus the time difference in SI seconds
between my two timestamps will change as real time progresses. So,
probably the only consistent thing would be to throw an error if one
tries to get a time difference (in SI seconds) with a UTtimestamp in the
future. This would probably throw people off.

Then, is there a standard deltaT, which everyone uses to covert UT<->TT?
This one: http://maia.usno.navy.mil/ ? If there isn't, then at least
Julia should be very clear what is used to convert and what
interpolation is used between datapoints.

[Stefan Karpinski]

On Thu, Feb 13, 2014 at 10:36 AM, Mauro <maur...@runbox.com> wrote:

I think Patrick means, given a timestamp encoded in some local time, say
Pacific Time, is there enough information in to translate it to GPS/TT
time.  I would think so.

Ah, I see. In my proposed approach, there is no such thing as "encoded in Pacific Time" – the underlying timestamps are always stored in terms of TT or UT seconds since some fixed offset. The time zone is *only* used for presentation purposes.

I have another concern (maybe this was raised in the discussion already
but I don't want to read it a third time): I create a time stamp in the
future, which would be a UTtimestamp by default in Julia.  How to I get
the time difference in SI seconds to another timestamp?  I think that is
not possible as we don't know deltaT in the future (nor in the distant
past really, but for that we could just define one).  We could make a
deltaT up but then, as real time progresses, the real deltaT and the
made up one will diverge.  Thus the time difference in SI seconds
between my two timestamps will change as real time progresses.  So,
probably the only consistent thing would be to throw an error if one
tries to get a time difference (in SI seconds) with a UTtimestamp in the
future.  This would probably throw people off.

I'm not sure to what extent it's possible to accurately predict ΔT, but I'm sure you realize that the core issue here is that in such a case translation between TT and UT is fundamentally impossible. If we "made it up" we would be wrong. In such a case the only reasonable behavior is to raise an error. How did you get such a future UT timestamp in the first place? You certainly didn't measure it in the future. So you must have started from a current or past time and added some time to it. In which case, the right way to do this is to convert the current or past time from UT to TT and then add SI seconds to it.

 

Then, is there a standard deltaT, which everyone uses to covert UT<->TT?
This one: http://maia.usno.navy.mil/ ?  If there isn't, then at least
Julia should be very clear what is used to convert and what
interpolation is used between datapoints.

I'm not sure – I suspect that Mike can address this. Also, we don't need absolute agreement, just agreement down to the precision of our timestamps.

[Jacob Quinn]

Mauro,

I don't think throwing an error for converting a UTTimestamp in the future to TT will throw people off. As Stefan explained at the beginning of the thread, the thing that most people think about isn't SISeconds, it's UTAngleSeconds (or whatever we want to call them). What I mean is that if I'm dealing with a future UTTimestamp, 95% of the time I want the different between it and another UTTimestamp, which as Stefan explained is completely straightforward and won't throw any errors (though you're not getting SIseconds back).

I think the main takeaway Stefan and I had from our discussion with Mike is that dealing in TT time is ultimately unintuitive for the main general user, the least of which is the fact that outside the range 1972 - June 2014, TT time gets tricky define and/or impossible. That's what motivated the change to drop support for leap seconds in Base, because we don't want anyone to get the wrong idea who understands all of this that we are actually doing TT time. 

The idea is that the general user will use Base UTTimestamps for everything, date creation, arithmetic, durations, etc. and never think twice. Those who have a specialized need for SISeconds on the other hand, will be able to load the TT code from a package and deal with the particularities of TT time.

In response to Patrick's question, the conversion between TT and GPS will be trivial and will provide all the information you need. Another question we should address though is if we should have a `leapseconds(dt::DateTime)` in Base that would say how many leap seconds had occurred up to that DateTime. I think that would probably be OK since only people know what they're doing could roll their own TT if they wanted.

-Jacob

[Mauro]

On Thu, 2014-02-13 at 16:09, quinn....@gmail.com wrote:
> Mauro,
>
> I don't think throwing an error for converting a UTTimestamp in the future
> to TT will throw people off. As Stefan explained at the beginning of the

> thread, the thing that *most* people think about *isn't* SISeconds, it's

> UTAngleSeconds (or whatever we want to call them). What I mean is that if
> I'm dealing with a future UTTimestamp, 95% of the time I want the different

> between it and *another *UTTimestamp, which as Stefan explained is

> completely straightforward and won't throw any errors (though you're not
> getting SIseconds back).

Yes, I can see that that would be the case most of the time. But these
days with GPS data galore, there are a lot of SI seconds around, or not?
So, there is potential for the gotcha will be that adding seconds to a
UTtimestamp in the past (or present, how present is present, June 2014?)
works but if it is in the future it throws an error. We just need to
make sure that doesn't happen often.

> I think the main takeaway Stefan and I had from our discussion with Mike is
> that dealing in TT time is ultimately unintuitive for the main general
> user, the least of which is the fact that outside the range 1972 - June
> 2014, TT time gets tricky define and/or impossible. That's what motivated
> the change to drop support for leap seconds in Base, because we don't want
> anyone to get the wrong idea who understands all of this that we are
> actually doing TT time.

I agree with it being more intuitive and I think it is a good idea.

> The idea is that the general user will use Base UTTimestamps for
> everything, date creation, arithmetic, durations, etc. and never think
> twice. Those who have a specialized need for SISeconds on the other hand,
> will be able to load the TT code from a package and deal with the
> particularities of TT time.
>
> In response to Patrick's question, the conversion between TT and GPS will
> be trivial and will provide all the information you need. Another question
> we should address though is if we should have a `leapseconds(dt::DateTime)`
> in Base that would say how many leap seconds had occurred up to that
> DateTime. I think that would probably be OK since only people know what
> they're doing could roll their own TT if they wanted.

For conversion UT<->TT: what would Julia do if the astronomers figure
out that they calculated deltaT slightly wrong and update their
historical data table. Does that mean all SI-time differences in Julia
change? Or do we keep using the old standard? The former could lead to
some strange behaviour.

[Tomas Lycken]

> What I mean is that if I'm dealing with a future UTTimestamp, 95% of the time I want the different between it and another UTTimestamp, which as Stefan explained is completely straightforward and won't throw any errors (though you're not getting SIseconds back).

But if I create two UTTimestamps, and then get the difference between them, will it be impossible to convert this time span from UTSeconds to SISeconds? That, I think, would throw off users. I realize that to do this correctly, one would also have to require some kind of absolute time reference (i.e. some timestamp which relates to the start or end of the time span), and make sure that given the time span and the time stamp, the conversion is defined. What will the interface for this look like?

function sitimespan(ts::UTTimespan, start::UTTimestamp)
    # if start + ts is still within defined TT time
        # add appropriate number of leap seconds and return

    # else

        # throw error

end

Maybe one would also want an overload that takes an SITimestamp for the second argument.

// T

[Pierre-Yves Gérardy]

I'm not sure to what extent it's possible to accurately predict ΔT, but I'm sure you realize that the core issue here is that in such a case translation between TT and UT is fundamentally impossible.

Especially since TT is officially established retrospectively, by pooling the output of atomic clocks from around the earth and from satellites. Meanwhile, it is approximated using the output of local clocks.

But, with some margin of error, couldn't you match future UT and TT according to an astronomical model encompassing the rotation of the earth, the phase of the moon, the precession of the axis of rotation of the earth, and longitude/latitude (the precession means that the angular velocity of the surface relative to the sun is not the same everywhere), and other factors I can't think of? 

I suppose that UT is determined at Greenwich, and linearly extrapolated around the earth.

[Jacob Quinn]

Tomas,

Yes, I'm sure we could have some kind of interface like that, but we could also just say it's not supported. If you want a TT timespan (i.e. SISeconds), then you need to convert your UTTimestamps to TTTimestamps first, period. I don't think that's unreasonable since, as I mentioned, this is by far not what the general user is or should be looking for. In reality, UT and TT are fairly distinct in their uses/usage, and my gut says that there's really not a lot of conversion that needs to happen between the two (or should be happening between the two). If you need SISeconds or are dealing with TT Timestamps, you're living and coding in TT world. If you just need to work with regular dates and times and have never heard of TT, you live in UT world. I just don't see the common use involving much overlap between the two. I would say, as I mentioned in Patrick's case, the overlap we could support in Base would be returning the # of leapseconds that have occurred for a given UTTimestamp.

-Jacob

[Mauro]

No, the problem is not TT and atomic clocks are not running at the same
speed (those differences are tiny). The problem is that the earth's
rotation is unsteady. That is how deltaT comes about. And the problem
is that predicting deltaT into the future is error-prone: thus the
prediction is different from the hindcast.


Tomas:
Conversion UT->TT is possible if the timestamp lies in the past (or the
close future) but not if it is in the future. And yes, to translate a
UTtimespan to SI seconds, you need a start or end-time (and neither is
allowed to lie in the future).

[Stefan Karpinski]

On Thu, Feb 13, 2014 at 11:28 AM, Mauro <maur...@runbox.com> wrote:

On Thu, 2014-02-13 at 16:09, quinn....@gmail.com wrote:
> Mauro,
>
> I don't think throwing an error for converting a UTTimestamp in the future
> to TT will throw people off. As Stefan explained at the beginning of the

> thread, the thing that *most* people think about *isn't* SISeconds, it's

> UTAngleSeconds (or whatever we want to call them). What I mean is that if
> I'm dealing with a future UTTimestamp, 95% of the time I want the different

> between it and *another *UTTimestamp, which as Stefan explained is

> completely straightforward and won't throw any errors (though you're not
> getting SIseconds back).

Yes, I can see that that would be the case most of the time.  But these days with GPS data galore, there are a lot of SI seconds around, or not?

If you're working with GPS data, you should load support for SI seconds. Simple enough.

 

So, there is potential for the gotcha will be that adding seconds to a UTtimestamp in the past (or present, how present is present, June 2014?) works but if it is in the future it throws an error.  We just need to make sure that doesn't happen often.

If someone asks for a (UT timestamp + SI seconds) operation, the way to do the computation is to convert the UT timestamp to a TT timestamp and then do the now-trivial addition operation; the result is a TT timestamp. While it may be slightly jarring to start with a UT timestamp and get a TT timestamp, it will always work, and will only happen if the user loads the SI package and has an SI seconds quantity. If they're in base Julia, they'll be working with TT seconds and this isn't an issue. If the user subsequently converts that TT timestamp back to UT, *that* operation might fail if the timestamp is outside of the known range for ΔT, but I think that's not terribly shocking.

For what it's worth, I think that this is exactly the kind of situation where having a units system and typed quantities is extraordinarily helpful to prevent you from doing something subtly wrong.

For conversion UT<->TT: what would Julia do if the astronomers figure out that they calculated deltaT slightly wrong and update their historical data table.  Does that mean all SI-time differences in Julia change?  Or do we keep using the old standard?  The former could lead to some strange behaviour.

I think that we would update our ΔT computation. Any time you upgrade external dependencies, the behavior of your code may change – this is no different. If you need to replicate exact behavior, use the same version of the ΔT package. This issue does show that one must be careful about what form data is stored in. The basic rule is that you should store your data in the original form – whether TT or UT – rather than some derived form. That way, if you re-run the code that does translation with corrected ΔT, you will get correct answers instead of the composition of an old ΔT with a newer, slightly different  inverse of ΔT. This is really data science 101: the definitive version of your data should be the original form so that when you inevitably realize that you did something wrong in your data pipeline, you can re-derive corrected versions of all the subsequent transformed versions of your data.

I think this is a sufficiently niche concern that most people won't ever have to worry about it – and the difficulty is unavoidable. Systems that don't explicitly distinguish between TT and UT are not free of this problem – they just don't have a way of talking about the problem, let alone handling it correctly.

[Patrick O'Leary]

On Thursday, February 13, 2014 8:52:22 AM UTC-6, Stefan Karpinski wrote:

I'm still confused about how time zones enter into the picture.

tzdata knows about leap seconds.
https://github.com/eggert/tz/blob/master/leap-seconds.list

[Pierre-Yves Gérardy]

On Thursday, February 13, 2014 6:27:30 PM UTC+1, Mauro wrote:

On Thu, 2014-02-13 at 17:11, pyg...@gmail.com wrote:
>> I'm not sure to what extent it's possible to accurately predict ΔT, but
>> I'm sure you realize that the core issue here is that in such a case
>> translation between TT and UT is fundamentally impossible.
>>
>
> Especially since TT is officially established retrospectively, by pooling
> the output of atomic clocks from around the earth and from satellites.
> Meanwhile, it is approximated using the output of local clocks.
>
> But, with some margin of error, couldn't you match future UT and TT
> according to an astronomical model encompassing the rotation of the earth,
> the phase of the moon, the precession of the axis of rotation of the earth,
> and longitude/latitude (the precession means that the angular velocity of
> the surface relative to the sun is not the same everywhere), and other
> factors I can't think of?
>
> I suppose that UT is determined at Greenwich, and linearly extrapolated
> around the earth.

No, the problem is not TT and atomic clocks are not running at the same
speed (those differences are tiny).  The problem is that the earth's
rotation is unsteady. 

I know it is, but it is deterministic.

However, if you add geologic phenomena to the factors I listed above, I suppose that we end up with a chaotic process whose estimation drifts away, one floating point rounding at a time.

Amarite? 

[Stefan Karpinski]

Ok, but it is unclear from that document what leap seconds have to do with time zones. Perhaps I'm just being dense, but I'm having trouble understanding the issue.

[Patrick O'Leary]

I'm pretty sure there is no issue. There is a database with timezone data. The same database knows about leap seconds. That is useful when moving between UTC and GPS.

If you wanted to shoehorn it in, you could say that GPS sort of acts like a political timezone that changes relative to UTC every time a leap second occurs as opposed to when a political body says that it changes, but I wouldn't expect it to be handled that way, and it's orthogonal to whether I can get the package with the database which knows about leap seconds to tell me when the leap seconds were.

[Stefan Karpinski]

Ok, good. Thanks for clarifying.

[Fil Mackay]

Just to be clear: we are talking here about using UT and assuming that the Earth's rotation is perfectly constant (degrees per hour) right? ie. UT seconds are just slightly different from SI seconds, which is something that leap-seconds seek to converge.

On timezones: this is a really important piece to get right. Personally I think historical and current political timezone support is a must - how many times have I re-solved this problem for a new platform. Granted this may be in a package, but if you fundamentally look at what many (most?) "times" are - they are bound to a particular political zone. Not supporting this correctly has caused untold hassles for Microsoft and Apple over the years, when the Australian government has adjusted the "daylight savings" (GMT offset changes) due to one-off events such as the Olympics. Timezones are a mess, the US has had cities and states change their timezone membership, borders redrawn, and some disagreements over which is to be used before legislation. Dealing with historical data is really dangerous, with a great db.

Of course this db is present in all linux/osx/windows installations - in various formats. Do we want to bind with platform-specific db formats (registry for Windows from memory), of unknown quality - or ship our own.

While IANA is the most authoritative db, the following disclaimer exists: (we should suggest IANA move this database to github)

# This data is by no means authoritative; if you think you know better,

# go ahead and edit the file (and please send any changes to

# t...@iana.org for general use in the future).

When recording a time (in real-time), keep it in Zulu - easy. But when interpreting a given future or past time, just whacking on a GMT offset is a waste of time IMO, since it only works in some cases, and doesn't really add any value over and above storing a Zulu time. I would think we need to support the IANA timezone database (was Olson tz database), which is the only way to convert correctly between Zulu and political zones. The pytz package implements this database nicely. jutz anyone?

Making the distinction of UT and SI is helpful - we are mostly using UT but pretending it is SI-based. The IANA tz has leap-second data in it, so it would be easy to provide a conversion between UT and TT. I think this is fine, but we would want to be 'using' UT natively and then converting to TT when specifically required. TT based time would then be capable of having 61 seconds in a minute, but a UT based time wouldn't.

Potential type structure:

type Timespan

- nano-seconds (UT based)

type Timestamp: (raw point-in-time)

- Timespan since epoc, which could be a type parameter: Unix, 

type TimeZone:

- Name (eg. "Australia/Sydney") 

- Full history of timezone changes, as well as known future

type Date: 

- d/m/y

type Time

- h/m/s

type Datetime

- Date

- Time

- TimeZone

- UT or TT

[Stefan Karpinski]

On Thu, Feb 13, 2014 at 10:54 PM, Fil Mackay <f...@vertigotechnology.com> wrote:

Just to be clear: we are talking here about using UT and assuming that the Earth's rotation is perfectly constant (degrees per hour) right? ie. UT seconds are just slightly different from SI seconds, which is something that leap-seconds seek to converge.

No, we're not assuming that at all. UT is defined in terms of the Earth's rotation.

[Fil Mackay]

OK I see what you mean, you are right time will appear to warp (compared with SI) with large geological events..

[Fil Mackay]

No, we're not assuming that at all. UT is defined in terms of the Earth's rotation.

OK I see what you mean, you are right time will appear to warp (compared with SI) with large geological events..

Not that most of us care, but UT and TT shifted by ~1.6 microseconds (UT moved forward) due to the Japense tsunami in 2011. Enough to matter to some people.. though!

 

[Stefan Karpinski]

I'm suspect that Terrestrial Time did not change.

[Fil Mackay]

On Fri, Feb 14, 2014 at 5:55 PM, Stefan Karpinski <stefan.k...@gmail.com> wrote:

I'm suspect that Terrestrial Time did not change.

UT and TT are fairly unrelated at the micro level they converge with leap-seconds, so 1.5us is not going to create much stress. Regardless, they are both theoretical constructs anyway - UT can only be approximated by physical measurement (and is instantly out of date), and there are lots of TT's - one for each clock in fact!

The closest thing to the real "global time" is UTC but it's not a stationary target. It is gets slowly skewed by a global network of atomic clocks, that 'vote' on where they think UTC should head as these mater clocks attempt to cluster with each-other in their interpretation of time. For example in Australia the National Measurement Institute maintain the official atomic clock for Australia (legal time), but this differs from other countries equivalent since it's just impossible to synchronise them perfectly. NMI is one of the (200 or so) entities that vote into where it thinks UTC should skew over the coming months. These clocks should be within 2us of one another. So, UTC is really a theoretical thing - you can't actually get access to it. The only reasonably accurate global clock that you can actually access is GPS time - so that is kind of the "UTC you can actually use".

Regards, Fil.

[Mauro]

I think, using UT as default time stamps is a sound idea.

Also, I just stumbled over this document by the official time keepers.  It nice and short and I thought it was a good read giving clear definitions of most time standard discussed here:

http://www.bipm.org/cc/CCTF/Allowed/18/CCTF_09-27_note_on_UTC-ITU-R.pdf

And here one more short summary of the different time standards:

Of the different UTs, I think Julia would be using UT1.  Atomic time TAI, terrestrial time TT and GPS time are all in sync but with some constant offsets to each other (at least as far as I think we need to consider precision here). UTC is also in sync with TAI apart from leap seconds inserted to keep it in approximate sync with UT1 to within 0.9s.  And lastly deltaT=TT − UT1 (wikipedia shows a graph of deltaT vs time; it's complicated: http://en.wikipedia.org/wiki/DeltaT !)

[Fil Mackay]

Remember that the different time standards are only relevant for Datetime (d/m/y/h/m/s fields). Timestamp is just a number of seconds past a given epoc. It's only when this translation takes place that you need to make all these calls in terms of time type, timezones, leap seconds etc.

Julia could easily support these various types (I'd excluded the ones that were fairly simple offsets from UT1), but by default represent UTC which is the general expectation? UTC requires you to handle leap-seconds, it's the others that are easier :)

[Markus Kuhn]

Terms like UT and TT have specific meanings in astronomy. TT in particular is IMHO a somewhat unlikely choice for the library of a general-purpose programming language:

  - TT is a time definition the main purpose of which is to be backwards compatible with astronomical ephemeris data that had been collected before atomic clocks came online in the late 1950s. The modern leap-second free definition of time that everyone else usually refers to is called TAI (temps atomique international) and has an integer-offset to UTC. There is a fixed but non-integer offset between TT and TAI.

  - UT is an ambiguous term that can refer to a number of different definitions of time, all of which are well within one second of the mean solar time on the prime meridian: UT0, UT1, UT2, UTC, UTC-SLS, etc. So UT is a reasonable generic term if you don't want to state categorically how precisely you are synchronized to one of the many UT variants.


Regarding the issue of leap seconds, I would predict that one of two things is going to happen in the near future:

Option A: They get abolished. The global reference time that we now call UTC, and to which to most local timezones have an integer hour offset, is renamed and turned into TAI plus a fixed number of seconds offset.

Option B: UTC stays as it is, time protocols (PTP, NTP, etc.) will be extended to disseminate both UTC and TAI to computer operating systems, and operating-system API's on network synchronized computers will give applications a choice of at least two clocks: one being a version of UTC with a standardized safe way of navigating softly around leap seconds (e.g., UTC-SLS slows down by 0.1% for 1000 seconds before an inserted leap second), and the other being TAI.


A programming language doesn't have to know much about this, it just has to offer an API that passes on whatever different notions of clock the operating system supports (POSIX clock_gettime() has already a clockid_t argument for that). It also has to provide access to a library that can convert between different types of clock, including local timezones. Finally it has to provide conversion functions between different numerical representations of time. There are mainly three of these:

  a) scalar, fixed-point time: just a single numerical variable, usually 64-bit
  c) broken-down time in seconds+nanoseconds form, usually 64+32 bit
  b) broken-down time in YYYY-MM-DD hh:mm:ss.ssss form

The problem with scalar time is that a single 64-bit word does not scale well enough to cover all applications. You either have too low resolution for some applications (some networking people want to have less than 100 ns resolution these days to trace 100 Gbit/s packet timestamps) or you don't have a range that covers all of human history and beyond. That is one reason why time is commonly broken down into at least two integer words, such as seconds + nanoseconds in POSIX's "struct timespec". And finally, you need the year+day+... format, because that's what humans are used to and the Gregorian calendar conversion is not exactly trivial to implement.

So I think Julia should have three data types, for one of each of the above.

That choice of numeric representation is pretty orthogonal to definition of time you use (TAI, TT, UT0, UT1, UT2, UTC, UTC-SLS, any local time zone, time-since-boot, process time, thread time, etc.). You don't want to have to create for each element of the cross product of numerical representation and definition of time a dedicated language type of your own, because it would just be too many. Therefore, limit the standard types to just the numerical representations and let the application programmer choose what definition of time might be most useful to them. People steering satellite dishes have different requirements from people engaging in the insanity of real-time high-frequency trading.

The choice of definition of time is a bit like the choice of unit of measurement: too application specific to be easy to capture well in the type system of a core standard library of a programming language.


By the way, I wrote up a proposal for a revised C time API a few years ago, that some other API designers since have found an inspiring read:

  http://www.cl.cam.ac.uk/~mgk25/time/c/

Markus

[Stefan Karpinski]

Your input is timely ;-)

[stephe...@predict.com]

I'm less timely with my input, having been on vacation...

Thank you Stefan for getting this discussion going. Support for efficient universal time representations and operations is crucial for my work - the insanity of real-time trading, high frequency or not ;-). I generally have no need for TT, GPS time, and leap seconds, and so am happy to have it off in an optional package. Timezone support is also crucial for me, though given the portability and maintenance issues with any tz database such as IANA/Olsen, /usr/share/zoneinfo, etc. having this in a package could make sense; but so might at least providing a Timezone abstraction in Base, and leaving the details/implementation/database to a package.

Most of my work is in C++ where I use boost::date_time, which effectively allows one to specify the type of a time point (analogous to Stefan's "Timestamp") in terms of its resolution (e.g. seconds, milliseconds, microseconds, nanoseconds), its epoch (essentially the time represented by the 0 value), and the representation type (e.g. int32_t, int64_t, etc.) (those in the know will note I am grossly simplifying the actual architecture of boost::date_time). This means it is trivially possible to have multiple time point types, for example one with a range exceeding the lifetime of the earth with 1-second resolution, and another with nanosecond resolution with an appropriately convenient epoch, and to even support conversion between them and interoperation if so desired, while remaining compact in memory and without necessarily falling into cross-product explosion of methods. I am very wary of any hard-wired resolution - when I first started in this business, 1-second resolution was sufficient; over the years things have gradually moved down to the nanosecond-resolution level, causing painful bouts of reimplementation along the way.

Being parametrized on the epoch is very useful when trying to work with very fine resolution; while the usual unix epoch (1970/1/1 00:00:00 UTC) is generally usable for the modern financial markets with nanosecond resolution and Int64 (you get +/- 292 years), picosecond resolution will give you only +/- 3 months, so you need to be able to utilize an alternate epoch close to the time of interest. Making the epoch part of the type lets you automate various conversion and interoperations or prevent them from occurring, as appropriate.

W.r.t. timezones, I generally dislike implementations which fold the timezone into the type. To my mind a universal time is a universal time is a ...; timezones only matter when you need to convert between a universal time and something like a string or the "parts" of a date-time (i.e. year, month, day, hour, minute, second, ...), and at these junctures it is more likely something in the context or environment (what geographic location is a user's browser running in?) which will determine the appropriate timezone, rather than being part of the semantic of the program. In my experience most often the point in the code where the timezone is required is almost always on the fringes, near the UI, I/O, etc. and not in the core of the application. So my implementations always provide the appropriate APIs so that the desired timezone can be supplied to any operation which might require it, which of course default to some process-wide timezone setting, which in turn defaults from some environment variable or OS setting.

stephen.

[Scott Jones]

I just wanted to ask, are there any other languages / libraries / packages that have seconds that are of unpredictable length?

This seems utter insanity to me! (sorry, but that is my opinion)

All of the date/time formats that I've dealt with lately are all based on UTC (which uses SI seconds, of a nice, fixed length).

Yes, dealing with leap seconds can be a pain... but I don't see how going to seconds of no fixed length, which can change in length because of a tsunami,

is better...

Scott

On Wednesday, February 12, 2014 at 12:07:21 AM UTC-5, Stefan Karpinski wrote:

I had a very productive conversation with Jacob Quinn and Mike Nolta this afternoon about how to deal with dates and times in Julia and I'm sharing some (still tentative) decisions we reached while it's all fresh in my mind both for my own records and so that others can comment on the design.

===

Times and dates in base Julia will use Universal Time (UT). UT seconds are defined to be the amount of time it takes the Earth to rotate 1/86400 of a turn. This is not a fixed duration – the UT second is not really a unit of time, but rather an angle. The advantage of UT is that it is simple and intuitive – by definition, there are no leap seconds and most importantly, you can use dates and times as far into the past and the future as you want. This is how people expect dates and times to work.

The disadvantage of UT is, of course, that its seconds are not fixed durations – i.e. "time" is not measured in SI seconds. The SI second is the fundamental unit of the other major time standard: Terrestrial Time (TT). This is the kind of time you want if you are doing physics and computing velocities and accelerations and such. If you just want to work with SI seconds, you can load the SIUnits package (or its heir). Converting between UT and TT will require loading another package which implements ΔT – i.e the difference between UT and TT as a function of time (think about that for a second). The reason to separate the conversion from SIUnits is that the data to compute ΔT is quite extensive, and it's entirely possible to work with SI times without needing to do any translation between TT and UT. It may be possible to approximate ΔT more compactly, in which case we could consider just including the translation infrastructure in SIUnits.

UTC is a way of defining units of time so that at larger scales it matches TT, while at larger scales it matches UT. The basic unit of time in UTC is the SI second just like TT, and every time unit from seconds down means the same thing that it does in TT and represents a fixed temporal duration. Above the scale of seconds is where the complications set in. Most UTC minutes are 60 seconds, but every now and then, the last minute of a day is 61 seconds – it contains an extra "leap second". All hours are 60 minutes, but if one of those minutes contains a leap second, the hour is 3601 seconds long instead of 3600 seconds, and similarly, each day has 24 hours, but if one of those hours contains a leap second, the day is 86401 seconds instead of the usual 86400. Given two UTC dates and times, if you know how many leap seconds occurred between them, you can easily compute how many SI seconds the intervening duration consisted of. Although many time systems claim to be using UTC, they are usually actually using UT (e.g. Joda-Time). Among other things, UTC is only defined after 1972 and a few months into the future. If these systems were really using UTC, they would be unable to work with dates and times outside of that range. These systems also typically don't have leap seconds. That is, in fact completely correct behavior for working with UT but it is not UTC.

For system functions that return UNIX times, Julia will simply interpret their values as UT (with the appropriate linear offsets). This is incorrect behavior but *mandated* by POSIX, and it's what every system does. This means that differences between system times will be in terms of UT seconds – so they are technically angles, not durations, although we probably don't want to shove that in people's faces. In particular, you cannot convert a UT duration to SI seconds since that conversion depends on when the duration occurred. Instead, if you want a duration in SI seconds, you need to convert the start and end times to TT first and then subtract them, yielding a difference in SI seconds.

There will not be support for parsing or using political time zones in base Julia. When you call now() to get the current time, you will get a UT timestamps. Both UT and TT timestamps are completely divorced from location – they mean the same thing everywhere and don't imply anything about location. However, we may want to determine the system's current offset from Zulu time (i.e. UTC+0 – the zero-offset timezone, as opposed to the UTC time standard) and print timestamps in local time with the appropriate offset. For example, if I'm in New York and ask for the time, I might see it printed as "2014-02-11 23:48:57 -0500" (this is what Ruby prints), even though the time object that's being printed doesn't include any time zone. Rather, the offset comes from a global configuration that is automatically set at startup time. We may want to honor the TZ environment variable for this. It may seem questionable that the same time value will print differently on different systems, but the key is that if you parse any of those output forms back into a timestamp value, you will get identical values. If you want to default to printing timestamps in Zulu time, you can just set an environment variable (or change the global setting in your .juliarc.jl file).

The TimeZones package will provide zoned date/time types that include both a timestamp and a time zone (this should probably support both offsets and political time zones). The timestamp field is still an absolute UT or TT value (both should be supported) and means the same thing, irrespective of the time zone field; the time zone field is included only to modify how the date and time should be displayed. If you were, for example, storing timestamps with time zones in a data frame, you could just as well store the timestamp and the time zone separately. The main reason for combining the two into a single value is so that the values can be conveniently displayed in the appropriate time zone. This can also allow convenient behaviors like producing another zoned value with the same time zone when adding a duration to a zoned date time.

===

Immense thanks to both Mike and Jacob for this enlightening discussion – especially to Mike for steering us expertly through this tricky subject. Mike, please correct me if I've made any errors here.

[Markus Kuhn]

On 22/04/15 12:45, Scott Jones wrote:
> I just wanted to ask, are there any other languages / libraries / packages
> that have seconds that are of unpredictable length?

Well, pretty much every programming language implementation that inherits
from POSIX the (with hindsight very misleadingly named) concept of

"Seconds since the epoch"
http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap04.html#tag_04_15

which, as the current language of the above linked definition makes very clear, is not
actually intended to be any count of physical seconds, but is just an integer encoding
for a (year, month, day, hours, minutes, seconds) tuple as displayed on a UTC clock.

Note that NTP (and many other time sources, with the notable exception of GPS)
doesn't provide you anything other than that, so you have to go with what you get.
In that pragmatic sense, POSIX made a perfectly good choice.

With PTP, if you are lucky, you might get physical TAI seconds instead of UTC, but
that causes its own set of problems: if your local leapseconds table in the client
is out of sync, then your application trying to display local legal time
(defined based on UTC) or UTC may be a few seconds wrong.

Some people care about physical time intervals in second, others care about
accurate display of legal time (and most probably don't care at all).
You could say that the POSIX definition of time caters more for lawyers than
physicists. I would not call one option more correct than the other.

Do keep in mind, that no computer is perfectly synchronized to the world's
network of atomic clocks all the time. Some lose synchronization (by several
second per day) and then have to catch up, and during that time, their seconds
will also be of quite variable length: I believe Windows NT at some point simply
doubled or halved the length of a second to catch up. There are operational reasons
other than leap seconds for why computer clocks cannot count strictly SI seconds,
and applications have to cope with those all the time. Leap seconds probably
account for a small fraction of the cases when computer clocks do strange things.
But leap seconds get more press coverage, because they are widely announced in
advance, and therefore people can get more anxious about them than
about other clock accidents.

Markus

--
Markus Kuhn, Computer Laboratory, University of Cambridge
http://www.cl.cam.ac.uk/~mgk25/ || CB3 0FD, Great Britain

[Tim Holy]

The more proper question is, why don't any other languages/libraries/packages
advertise that the definition of the second is actually complicated? :-) Ours
is the same as everyone else's.

--Tim

On Wednesday, April 22, 2015 04:45:45 AM Scott Jones wrote:
> I just wanted to ask, are there any other languages / libraries / packages
> that have seconds that are of unpredictable length?
> This seems utter insanity to me! (sorry, but that is my opinion)
> All of the date/time formats that I've dealt with lately are all based on
> UTC (which uses SI seconds, of a nice, fixed length).
> Yes, dealing with leap seconds can be a pain... but I don't see how going
> to seconds of no fixed length, which can change in length because of a
> tsunami,
> is better...
>
> Scott
>
> On Wednesday, February 12, 2014 at 12:07:21 AM UTC-5, Stefan Karpinski
>
> wrote:
> > I had a very productive conversation with Jacob Quinn and Mike Nolta this
> > afternoon about how to deal with dates and times in Julia and I'm sharing
> > some (still tentative) decisions we reached while it's all fresh in my
> > mind
> > both for my own records and so that others can comment on the design.
> >
> > ===
> >
> > Times and dates in base Julia will use Universal Time

> > <http://en.wikipedia.org/wiki/Universal_Time> (UT). UT seconds are

> > defined to be the amount of time it takes the Earth to rotate 1/86400 of a
> > turn. This is not a fixed duration – the UT second is not really a unit of
> > time, but rather an angle. The advantage of UT is that it is simple and
> > intuitive – by definition, there are no leap seconds and most importantly,
> > you can use dates and times as far into the past and the future as you
> > want. This is how people expect dates and times to work.
> >
> > The disadvantage of UT is, of course, that its seconds are not fixed
> > durations – i.e. "time" is not measured in SI seconds

> > <http://en.wikipedia.org/wiki/Second>. The SI second is the fundamental

> > unit of the other major time standard: Terrestrial Time

> > <http://en.wikipedia.org/wiki/Terrestrial_Time> (TT). This is the kind of

> > time you want if you are doing physics and computing velocities and
> > accelerations and such. If you just want to work with SI seconds, you can
> > load the SIUnits package (or its heir). Converting between UT and TT will
> > require loading another package which implements ΔT

> > <http://en.wikipedia.org/wiki/DeltaT> – i.e the difference between UT and

> > TT as a function of time (think about that for a second). The reason to
> > separate the conversion from SIUnits is that the data to compute ΔT is
> > quite extensive, and it's entirely possible to work with SI times without
> > needing to do any translation between TT and UT. It may be possible to
> > approximate ΔT more compactly, in which case we could consider just
> > including the translation infrastructure in SIUnits.
> >

> > UTC <http://en.wikipedia.org/wiki/Coordinated_Universal_Time> is a way of

> > <http://linux.die.net/man/3/tzset> for this. It may seem questionable

[Scott Jones]

I asked if there are any languages etc. that have seconds of unpredictable length - and I disagree that what the Posix standard says means that seconds are UT seconds...

In fact, since they mention that it is based on UTC, it would imply that they mean SI seconds.

I'm not aware of any computer that would measure time in UT seconds... (since that is not even knowable in advance, and changes constantly)...

Using UT seconds seems to be a polite fiction... your computer is really measuring time in SI seconds (or at least as close as possible to them as it can get).

Trying to sweep the variability of UT seconds under the rug to avoid dealing with leap seconds ends up creating much larger problems...

I (and many others) think POSIX screwed up... I think that using TAI times (which can be converted easily to UTC times, and from there to legal times,

and always using physical SI seconds, makes much more sense.

Why doesn't Julia use TAI-10, there is a nice library, it is available for most platforms I believe...

I think you yourself suggested using TAI ;-)

Dr Markus Kuhn (University of Cambridge) lists the following arguments against leap seconds:

• Leap seconds could cause disruptions where computers are tightly synchronized with UTC.

• Leap seconds are a rare anomaly, which is a concern for safety-critical real-time systems (e.g. air-traffic control concepts entirely based on satellite navigation).

• Astronomical time (UT1), which is defined by Earth's rotation, is not significant in most people’s daily lives.

His arguments in favor of leap seconds include:

• There have been no credible reports about serious problems caused by leap seconds.

• Some computerized systems that work with leap seconds are costly to modify (eg. antennas that track satellites).

• Computer errors caused by leap seconds can be avoided simply by using International Atomic Time (TAI) instead of Universal Coordinated Time (UTC).

• Desktop computers and network servers have no trouble coping with leap seconds.

• Humankind has defined time by the Earth's rotation for over 5000 years – this tradition should not be given up because of unfounded worries of some air-traffic control engineers.

• Abandoning leap seconds would make sundials obsolete.

Scott

P.S. I loved the sundial argument!

[Scott Jones]

On Wednesday, April 22, 2015 at 9:17:24 AM UTC-4, Tim wrote:

The more proper question is, why don't any other languages/libraries/packages
advertise that the definition of the second is actually complicated? :-) Ours
is the same as everyone else's.

The languages/libraries/packages that I've dealt with generally talk about using UTC, and UTC *does* use fixed length seconds (SI) (hence the need for pesky leap seconds to sync up with the Earth's slowing rotation).

I disagree completely that Julia's definition is the same as everyone else's...  Julia is the only one that I've encountered that talks about using variable length seconds... to me, incredibly surprising in a language geared to the scientific/math community...

Scott

[Jacob Quinn]

Scott,

Great to hear some additional thoughts on this. For context on the current state of DateTime in Julia, see the following:

https://github.com/JuliaLang/julia/issues/3524

https://github.com/JuliaLang/julia/pull/7654

https://github.com/JuliaLang/julia/pull/7285

https://github.com/JuliaLang/julia/pull/5328

I think Tim Holy's point is valid; a lot of programming languages don't even seem to acknowledge the existence of leap seconds or in several cases say things like, "Language X supports UTC time through its DateTime type. Leap seconds are not supported." which is a total contradiction.

So tl;dr, we decided to base the core DateTime type on UT time which has very natural calculation properties (Date + Day, DateTime + Second) and is extremely simple to reason about for the majority of use cases for a general programming language. Also check out https://github.com/Keno/SIUnits.jl which supports a SISecond type. There have been several discussions on supporting a full TAI/UTC DateTime type as well that would be based on this SISecond type (I even have some old code lying around that incorporated leap seconds....)

Cheers. 

[Scott Jones]

On Apr 22, 2015, at 10:00 AM, Jacob Quinn <quinn....@gmail.com> wrote:

Scott,

Great to hear some additional thoughts on this. For context on the current state of DateTime in Julia, see the following:

https://github.com/JuliaLang/julia/issues/3524

https://github.com/JuliaLang/julia/pull/7654

https://github.com/JuliaLang/julia/pull/7285

https://github.com/JuliaLang/julia/pull/5328

I think Tim Holy's point is valid; a lot of programming languages don't even seem to acknowledge the existence of leap seconds or in several cases say things like, "Language X supports UTC time through its DateTime type. Leap seconds are not supported." which is a total contradiction.

I agree that this is a problem in lots of languages, propagated by the POSIX time definition… I’ve dealt with this in the past.

But just because other languages/libraries don’t deal with this correctly, I don’t think justifies variable length seconds…

I think that if you wanted to deal with some angular time, to deal with dates and not worry about leap seconds, it would have been better to have dealt with minutes…

UTC has just the last minute of the day when there is a leap second have 61 seconds instead of 60, and everything from there up is defined in terms of 60 minutes to the hour, 24 hours to the day,

and then the Gregorian calculations with leap years etc. to get from days to years.  This is analogous to February having 29 days in a leap year, the days themselves are not variable length, just

an extra day in one of the months every few years.

So, you’d store an integer value of the minutes from some epoch, such as the Unix epoch, and then a floating point value for the SI seconds.

So tl;dr, we decided to base the core DateTime type on UT time which has very natural calculation properties (Date + Day, DateTime + Second) and is extremely simple to reason about for the majority of use cases for a general programming language. Also check out https://github.com/Keno/SIUnits.jl which supports a SISecond type. There have been several discussions on supporting a full TAI/UTC DateTime type as well that would be based on this SISecond type (I even have some old code lying around that incorporated leap seconds….)

That’s what I would have much preferred, i.e. using TAI-10, and using the Olsen library and TZ databases to get valid UTC times and local “legal times”.

[Kevin Squire]

Hi Scott,

On Wed, Apr 22, 2015 at 6:27 AM, Scott Jones <scott.pa...@gmail.com> wrote:

Why doesn't Julia use TAI-10, there is a nice library, it is available for most platforms I believe...

One of the nice things about Julia is that all of this is easily implementable as a package--the current DateTime code even started out that way.  Since most people are pretty busy with other things, why don't you give it a shot and implement it?

Cheers!

   Kevin

[Stefan Karpinski]

Python: http://joda-time.sourceforge.net/faq.html#leapseconds

there is no notion of “leap seconds” here

NumPy: http://docs.scipy.org/doc/numpy/reference/arrays.datetime.html

Datetimes are always stored based on POSIX time (though having a TAI mode which allows for accounting of leap-seconds is proposed), with a epoch of 1970-01-01T00:00Z.

So no leap seconds there either, although there's been talk of allowing support of them. They interpret POSIX time as "no leap seconds" which seems pretty fair since POSIX just pretends that leap seconds don't exist.

Java: https://docs.oracle.com/javase/8/docs/api/java/util/Date.html

Although the Date class is intended to reflect coordinated universal time (UTC), it may not do so exactly, depending on the host environment of the Java Virtual Machine. Nearly all modern operating systems assume that 1 day = 24 × 60 × 60 = 86400 seconds in all cases.

Java punts to the OS but notes that almost all OSes don't do leap seconds.

JodaTime: http://joda-time.sourceforge.net/faq.html#leapseconds

Joda-Time does not support leap seconds.

Perl: http://perldoc.perl.org/functions/time.html

Returns the number of non-leap seconds since whatever time the system considers to be the epoch

And so on. In short, the typical approach seems to be to claim to use UTC while also pretending that leap seconds don't exist. Of course, this is inherently contradictory. A system that doesn't deal with leap seconds using an up-to-date leap second table, isn't really using UTC. And any system in which days match the Earth's rotation while being exactly 86400 seconds long is using variable-length seconds.

There are two quite different primary use cases for time measurements: calendars and physics. For calendars, you don't really care about leap seconds and it's ok that there are slight variations in the physical durations of seconds. For physics, you care very much that time is uniform, but you don't generally care what day or month it is. The solution is to use different notions of time for the two use cases: use UT for calendars and TT for physics. If you need to translate between the two, it's better to be clear about the distinction and honest that translation between them is non-trivial, rather than pretending they are the same thing, which is what systems that purport to implement UTC while ignoring leap seconds are effectively doing.

Since far more people use dates and times to keep track of events on Earth and its calendars than to do physics experiments, Julia's built in dates and times are in Universal Time. Most people also don't care about leap seconds and would rather just pretend they don't exist. This is exactly what UT gives you. There are exactly 86400 seconds in each day and your programs don't become incorrect when the International Earth Rotation and Reference Systems Service announces new leap seconds. The only issue is the jarring but inescapable fact that Earth days have non-uniform duration.

If you are doing physics experiments, of course, you should use SIUnits rather than DateTimes. Most people in this category will never need to convert between SIUnits nad DateTimes, but some few people do – i.e. astronomers. For this, you need a package that correctly implements ΔT. This is highly non-trivial computation based on a *lot* of historical data. We do need a package implementing ΔT – that would be a very welcomed contribution.

[Stefan Karpinski]

On Wed, Apr 22, 2015 at 12:57 PM, Stefan Karpinski <ste...@karpinski.org> wrote:

Python: http://joda-time.sourceforge.net/faq.html#leapseconds

there is no notion of “leap seconds” here

Sorry, link should be https://docs.python.org/2/library/datetime.html 

[Scott Jones]

In the end, it’s all relative anyway! ;-)

I wonder about writing code to deal with the relativistic time differences in satellites because they are further out of the Earth’s gravity well ;-)

(I believe GPS satellites do have to deal with that issue)

Scott

[Scott Jones]

Python actually says it will throw away the leap second, if it is getting a timestamp from a system that supports them, so you can have two timestamps, separated by 1 second, that return the same Python datetime.  So they do have a notion of leap seconds, and willfully throw them away... ;-)

Scott

On Wednesday, April 22, 2015 at 12:58:12 PM UTC-4, Stefan Karpinski wrote: