[Boost-users] Boost.DateTime and leap year/leap seconds

[Auer, Jens]

Hi,

 

I am considering using Boost.DateTime in one of my projects, and there have been some concerns about handling leap seconds on the customers side. The documentation says that other libraries do not handle leap seconds correctly, but it does not state that Boost.DateTime does. I found only one mail from 2009 on the mailing list asking about leap seconds, but this does not give an answer. Does anybody have some information on this?

 

The computations I intend to do are actually rather limited. I will be given a year and a number microseconds, and have to compute the day in the year that this describes from the start of the year, together with microseconds and milliseconds in the day. So what I want to implement is a function tuple<days, milliseconds, microsends> convertTime(year, microseconds).

 

Best wishes,

  Jens

 

--

Dr. Jens Auer | CGI | Software Engineer

CGI Deutschland Ltd. & Co. KG
Rheinstraße 95 | 64295 Darmstadt | Germany

T: +49 6151 36860 154

jens...@cgi.com

Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden Sie unter de.cgi.com/pflichtangaben.

 

CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to CGI Group Inc. and its affiliates may be contained in this message. If you are not a recipient indicated or intended in this message (or responsible for delivery of this message to such person), or you think for any reason that this message may have been addressed to you in error, you may not use or copy or deliver this message to anyone else. In such case, you should destroy this message and are asked to notify the sender by reply e-mail.

 

[Jeff Garland]

Hi Jens -

Leap second support never got added to the library -- most applications can ignore them.  In principle, of course, you can use the library to help you do the calculations.  Basically they are like leap years except that they are irregular -- so you have to use a collection to perform the adjustment instead of an algorithm.

Little confused about the use case though -- are the microseconds you receive including the leap seconds or not including them? Basically which way are you adjusting?

Jeff

_______________________________________________
Boost-users mailing list
Boost...@lists.boost.org
http://lists.boost.org/mailman/listinfo.cgi/boost-users

[Auer, Jens]

Hi Jeff,

 

The specification we got is a bit vague. Basically I am getting data packets from a server which puts a time stamp in the data which is defined as “microseconds in the current year”. From that, I have to create a time-stamp in a different format which is CCSDS day segmented time (epoch is 1.1.1958, 16-bit days, 32-bit milliseconds of the year and 16-bit microseconds of the millisecond). I can assume that both systems are synchronized via ntp, so I guess it is UTC. I am trying to get more details on that.

 

What I then need to do is to write a function that computes

convert(microseconds) -> (days since 1.1.1958, milliseconds, microseconds)

 

My idea was to:

1.    Decide which is the current year. When the two systems disagree, the data server takes precedence because it provides a more accurate clock internally.

2.    From this year, compute the number of days from 1.1.1958 to 1.1. This could something like
d = ptime(date( /* 1.1. of current year */) ) + microseconds(x)

3.    Compute the number of days in the current year from the number of microseconds.
days = d.date().day_of_year()

4.    Compute the milliseconds/microseconds in the current day from d.time_of_day().

 

The question about leap seconds came up when converting the microseconds in the current year to the day. I guess that the original microseconds value contains leap seconds, but I need to get a definite statement for this.

 

However, I am currently not implementing the conversion, but proposing how to do it (we need to write detailed design documents first…). I wanted to use Boost.DateTime and include a statement that when I convert the number of microseconds to days, leap seconds are taken into account.

 

Best wishes,

  Jens

 

--

Dr. Jens Auer | CGI | Software Engineer

CGI Deutschland Ltd. & Co. KG
Rheinstraße 95 | 64295 Darmstadt | Germany

T: +49 6151 36860 154

jens...@cgi.com

Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden Sie unter de.cgi.com/pflichtangaben.

 

CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to CGI Group Inc. and its affiliates may be contained in this message. If you are not a recipient indicated or intended in this message (or responsible for delivery of this message to such person), or you think for any reason that this message may have been addressed to you in error, you may not use or copy or deliver this message to anyone else. In such case, you should destroy this message and are asked to notify the sender by reply e-mail.

 

[Jeff Garland]

Ah yes, CCSDS -- well familiar with it working on satellite systems myself :) As for ntp -- obviously that can get *very sticky* since you really have no idea when it's going to adjust the clocks.  Here's an article from the leap second we just had on current state of some common platforms...ugly...

https://aws.amazon.com/blogs/aws/look-before-you-leap-the-coming-leap-second-and-aws/

Personally I feel it's so much smarter to have the timing systems run a 'steady_clock' in c++11 parlance --  a non-leap second adjusted like GPS or TAI and then allow software to convert to UTC for user consumption as needed.  But I realize you might not have control over that.  And note, you may be able to use a c++11 steady_clock

Getting back to the calculation at hand, you're on the correct course -- all I'd say is 'convert the leap second at the boundaries' and calculate in a monotonic, fixed time reference.  This makes the calculation just a standard calculation and the result can be trivially converted back to a leap second adjusted utc as desired.  By analogy, consider the 'time zone example' from the docs to calculate the 'local arrival time' of a flight from phoenix to new york.   Internally the code damps out all the complications associated with daylight savings time, etc by internally converting any local times to utc internally on construction -- then any calculations are simple adds and subtracts -- and then to get a new local reference you convert from the utc reference back out (see the 'just integer math here -- even though these are 'local times' internally utc is stored and all duration calculations just do ratio adjusted addding).

 time_zone_ptr phx_tz(new posix_time_zone("MST-07:00:00"));

  //local departure time in phoenix is 11 pm on april 2 2005 
  // (ny changes to dst on apr 3 at 2 am)
  local_date_time phx_departure(date(2005, Apr, 2), hours(23), 
                                phx_tz, 
                                local_date_time::NOT_DATE_TIME_ON_ERROR);

  time_duration flight_length = hours(4) + minutes(30); //just integer math here!!!
  local_date_time phx_arrival = phx_departure + flight_length;
  local_date_time nyc_arrival = phx_arrival.local_time_in(nyc_tz); //convert from utc to ny local

Exact same approach works with leap seconds -- convert your .60 second into a fixed reference system like gps -- do any duration math -- and convert back out to utc.  For me, I'd formalize all this in something like a leap second aware 'tm' struct -- something like this....

   struct leap_tm {

      leap_tm(int year, int milliseconds, int microseconds, bool leap_adjusted) :
      {

         //convert to leap or non leap ptime representation

         //use leap second 'table' to convert to utc

       }

      ptime as_non_leap_adjusted_utc() {...}
      ptime as_utc() {...}

    }

Note that there's a good chance that this will be a real feature in boost date-time 2.0, but time of arrival for bdt2 is not known.  Hope this helps...

Jeff

[Auer, Jens]

Hi Jeff,

 

Thanks for the help, but now I am getting a bit confused. First, I have a confirmation that what I will get is UTC-based, so I get the elapsed microseconds in the current year using the UTC scale with leap seconds adjusted. What I am now wondering is if Boost.DateTime supports the conversion into days and seconds.

 

CCSDStime fromTimeTag(microseconds micros)

{

    auto year = microsec_clock::universal_time().date().year();

    // create UTC time point

    auto utc = ptime(date(year, Jan, 1), micros);

   

      auto day = utc.date().day();

      auto t = utc.time_of_day();

      return CCSDtime{ year, t.total_seconds(), t.total_microseconds() }:

}

 

Reading the documentation I think that will not be correct when there were leap seconds.

 

I have found an example were the utc is converted to local time, and the different between local time and utc is used to calculate the number of leap seconds. I could use this value to correct the leap seconds, but it does look a little bit hackish.

[Howard Hinnant]

I’ve been following this thread with interest.

On Jul 22, 2015, at 8:59 AM, Auer, Jens <jens...@cgi.com> wrote:
> Reading the documentation I think that will not be correct when there were leap seconds.

Correct. As Jeff said earlier, leap second support is not in the current library, but might be a feature in a future boost date-time 2.0.

If you are using a C++11 or C++14 compiler, there is a library that will handle this problem with leap seconds taken into account for you:

http://howardhinnant.github.io/tz.html

I doubt you can use this library as the space industry equates up-to-date compilers with increased risk. However I mention it, as it is open source and perhaps you can crib some of its ideas onto the current boost date-time library.

In a nutshell, this C++11/14 library handles leap seconds along the lines of what Jeff suggested. Though in this case it creates a std::chrono - compatible clock which is leap second aware:

class utc_clock
{
public:
using duration = std::chrono::system_clock::duration;
using rep = duration::rep;
using period = duration::period;
using time_point = std::chrono::time_point<utc_clock>;
static constexpr bool is_steady = true;

static time_point now() noexcept;

template <class Duration>
static
std::chrono::time_point<utc_clock,
typename std::common_type<Duration, std::chrono::seconds>::type>
sys_to_utc(std::chrono::time_point<std::chrono::system_clock, Duration> t);

template <class Duration>
static
std::chrono::time_point<std::chrono::system_clock,
typename std::common_type<Duration, std::chrono::seconds>::type>
utc_to_sys(std::chrono::time_point<utc_clock, Duration> t);
};

and includes conversions to and from std::chrono::system_clock::time_point (which like boost date-time is based on Unix time). These conversions are simply lookups into an ordered list of leap second transitions, resulting in the addition or subtraction of the proper number of leap seconds.

This could be used like so:

struct CCSDS
{
std::uint16_t days;
std::uint32_t ms;
std::uint16_t us;
};

CCSDS
to_CCSDS(date::year y, std::chrono::microseconds us)
{
using namespace date;
using namespace std::chrono;
auto utc = utc_clock::sys_to_utc(day_point(y/jan/1)) + us;
auto sys = utc_clock::utc_to_sys(utc);
auto dp = floor<days>(sys);
auto d = dp - day_point(1958_y/jan/1);
us = utc - utc_clock::sys_to_utc(dp);
auto ms = duration_cast<milliseconds>(us);
us -= ms;
return {static_cast<std::uint16_t>(d.count()),
static_cast<std::uint32_t>(ms.count()),
static_cast<std::uint16_t>(us.count())};
}

The variable utc holds the “year + us” as a time point with microseconds precision. This time point counts microseconds, including leap seconds, since 1970-01-01 00:00:00 UTC.

The next step is find when the day started that is associated with utc. To do this one must convert utc back to “Unix time”, and then truncate that time point to a precision of days, resulting in the variable dp. dp is a count of days since 1970-01-01. Since your epoch is 1958-01-01, this is taken into account in creating “d”, the first value needed in your return type.

Now the number of microseconds since the start of the day needs to be computed. The start of the day, dp, is converted back into the leap-second aware system, and subtracted from the microsecond time point: utc. The variable us is reused to hold “microseconds since midnight”. Now it is a simple computation to split this into milliseconds since midnight, and microseconds since the last millisecond.

This has all been tested with:

void
test_to_CCSDS()
{
using namespace std::chrono;
using namespace date;
for (auto d = days{179}; d <= days{181}; ++d)
{
auto start = d + 86399s + 999999us + 0us;
for (auto us = start; us <= start + 3s; us += 1s)
{
auto t = to_CCSDS(2015_y, us);
std::cout << t.days << ' ' << t.ms << ' ' << t.us << '\n';
}
std::cout << '\n';
}
}

which yields:

20998 86399999 999
20999 999 999
20999 1999 999
20999 2999 999

20999 86399999 999
20999 86400999 999
21000 999 999
21000 1999 999

21000 86398999 999
21000 86399999 999
21001 999 999
21001 1999 999

Interpretation: On 2015-06-29 23:59:59.999999 we are 1us away from the next day. 1s later we are well into 2015-06-30.

24hrs later, we see a different pattern as the leap second is inserted. The last microsecond of the day is 1s later at 2015-06030 23:59:60.999999.

24hrs later we are back to days with only 86400s.

Assuming you have some conversion data to test against, this will hopefully match up with those test cases.

Howard

[Auer, Jens]

Hi Howard,

Thank you very much. That basically made my day!

We have no restrictions on the compiler, currently I'm using gcc 4.9 in C++-14 mode. I think I can also justify using a time handling library, so the chances are very good that we will use your library.

Many thanks and best wishes,

Jens

--
Dr. Jens Auer | CGI | Software Engineer
CGI Deutschland Ltd. & Co. KG
Rheinstraße 95 | 64295 Darmstadt | Germany
T: +49 6151 36860 154
jens...@cgi.com
Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden Sie unter de.cgi.com/pflichtangaben.

CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to CGI Group Inc. and its affiliates may be contained in this message. If you are not a recipient indicated or intended in this message (or responsible for delivery of this message to such person), or you think for any reason that this message may have been addressed to you in error, you may not use or copy or deliver this message to anyone else. In such case, you should destroy this message and are asked to notify the sender by reply e-mail.

-----Original Message-----
From: Boost-users [mailto:boost-use...@lists.boost.org] On Behalf Of Howard Hinnant
Sent: 22 July 2015 21:29
To: boost...@lists.boost.org
Subject: Re: [Boost-users] Boost.DateTime and leap year/leap seconds