Time values defined as time since noon minus 12h

[Tom Brown]

(This come out of discussion "New calendar_dates.txt exception_types to more compactly represent Daylight Saving Time exceptions?")

Summary of proposed change: interpret all time-of-day values (arrival_time, departure_time, start_time, end_time) as "time since noon minus 12h", breaking backwards compatibility, but bringing spec in line with all data I've seen and Google's internal parser.

Any comments on this proposal before I request it to be added to the official spec?

On Fri, Jul 24, 2009 at 3:57 PM, Tom Brown <tom.bro...@gmail.com> wrote:

You bring up some very good points about the DST transition. Looking at a few feeds in the USA (BART, Caltrain, San Diego, TriMet) I can't find anyone that has special service keys that run only the transition days in 2009 (begins on March 8 and ends on November 1). As Chas says SFMTA doesn't handle it specially. Are any GTFS producers correctly handling DST transitions according to seconds since midnight?

So why hasn't Google's trip planner returned trips off by one hour 2 days a year? Because when surrounded by time handling code someone had the good idea to interpret the HH:MM:SS in stop_times as seconds past 12h before midnight. In other words all stop times on transition days are treated as being in the new offset. This is convenient because most trips are scheduled at local times after the DST transition and don't need to be duplicated. Unlike adding special entries to the calendar files this interpretation doesn't require the data producer to pay attention to when the DST transition happens, except for the few trips scheduled at local times before the transition.

The problem is that we didn't go back and put this in the specification. So what are our options, as the community responsible for gtfs?

1) Leave spec is and require correct feeds to duplicate trips on transition days

2) Add new exceptions to service_id,date pairs which say that on these dates add or subtract one hour from stop_times. require correct feeds to add these exceptions to their data

3a) Continue interpreting stop_times as "seconds since midnight" unless a new exception marks a service as using "seconds since noon - 12h"

3b) Change specification to interpret all stop_times as "seconds since noon - 12h", breaking backwards compatibility.

3c) Change specification to interpret all stop_times as "seconds since noon - 12h", breaking backwards compatibility. Add new flag to force a particular service_id to use the old interpretation "seconds since midnight"

Assuming that most data today doesn't handle DST transitions (1,2,3a) require widespread changes which isn't very attractive. I think we should change the specification to document what is done in practice: stop_times values don't change on DST transition days for daytime service. The gtfs-changes welcome message says "Changes to the spec should be backwards-compatible. When adding features to the specification, we want to avoid making changes that will make existing feeds invalid." While this is a backwards incompatible change I think it actually makes existing feeds valid.

Details

[mostly based on DST leaps in the USA (spring 01:59:59 to 03:00:00; fall 01:59:59 to 01:00:00)]

 A quick skim of http://en.wikipedia.org/wiki/Daylight_saving_time_around_the_world finds other problems. In Egypt and Brazil there is no midnight when clocks are moved forward. I can't find anywhere that the leaps happen during the day so noon should be safe.

Feed producers who assume the HH:MM:SS are equivalent to local times are currently incorrect according to "s since midnight" for service days with a DST transition by one hour after the leap forward or backward, which is most of the day. Changing GTFS to "s since noon - 12h" will mean they are incorrect before the leap, a few hours in the night.

Feed producers who correctly produce HH:MM:SS as "s since midnight" will need to change the times forward or back an hour on transition days or move the service to the previous day and add 24h to the times. In the unlikely event that this is really hard we could offer a new flag to these services (3c).

Here is a little table of examples during transitions in the USA. Note that "s since midnight" is different from local time during most of transition days and times from the previous day are the same under "s since midnight" and "s since noon - 12h" interpretations.

          s since midnight         local time

calendar            s since noon - 12h

2009-03-08    00:00      01:00         0:00

2009-03-08    02:00      03:00         3:00

2009-03-08    07:00      08:00         8:00

2009-10-31    24:00      24:00        24:00

2009-11-01    00:00      *             0:00

2009-10-31    25:30      25:30         1:30 (1st 1:30)

2009-11-01    01:30      00:30         1:30 (1st 1:30)

2009-10-31    26:30      26:30         1:30 (2nd 1:30)

2009-11-01    02:30      01:30         1:30 (2nd 1:30)

2009-10-31    27:00      27:00         2:00

2009-11-01    03:00      02:00         2:00

2009-11-01    09:00      08:00         8:00

* can't represented on 2009-11-01 and needs to be moved to 2009-10-31

[Jehiah Czebotar]

+1 looks good to me.

[Brian Ferris]

-1

This feels like a hack to me.  More importantly, it won't solve the DST problem for all agencies.  King County Metro in Seattle, for example, has some routes that start before the 1 am time-shift that aren't affected by the DST shift and many routes that start after the 1 am time-shift that are affected by the shift.  The proposed change would incorrectly affect the start time for the pre-1am routes.  I know there is the assumed behavior of what we have now, but I think we're really just avoiding/delaying the fact that each agency needs to look at DST and decide how to handle it.

Brian

[Martin Akerman]

I think I'm with Brian on this one.

-Martin

[Brian Ferris]

Ok, since I don't want to sound TOO contrary on this issue...  I think my compromise position would look something like the following:

1) Support the current de-facto behavior of "service date start time" = time at noon on service date - 12 hours
2) Support the proposed additional calendar_date.txt exception_type additions of +1 and -1 for compactly adjusting the set of trips that need shifting because they start before the DST changeover point
3) Add language to the GTFS spec encouraging agencies to consider DST.  This will probably have to happen anyway to add clarity as to why the spec defines midnight as "noon-12h" in the first place.

Brian

[Tom Brown]

On Wed, Nov 4, 2009 at 2:22 PM, Brian Ferris <bdfe...@gmail.com> wrote:

Ok, since I don't want to sound TOO contrary on this issue...  I think my compromise position would look something like the following:

1) Support the current de-facto behavior of "service date start time" = time at noon on service date - 12 hours
2) Support the proposed additional calendar_date.txt exception_type additions of +1 and -1 for compactly adjusting the set of trips that need shifting because they start before the DST changeover point
3) Add language to the GTFS spec encouraging agencies to consider DST.  This will probably have to happen anyway to add clarity as to why the spec defines midnight as "noon-12h" in the first place.

I agree that agencies need to pay careful attention to trips around time shifts, but I think they need to do this independent of how times are defined. I think it is good to make trips that are away from the time shifts work with the trivial equivalence local time == hh:mm:ss in stop_times. I appreciate that you are taking care with King County Metro but Chas suggested that it will take a lot of effort for SF Muni, another large agency with 24h service, to break this assumption.

The advantage of using noon over anything relative to midnight is that noon is well defined. http://en.wikipedia.org/wiki/Daylight_saving_time_around_the_world says in Egypt and Brazil there is no localtime midnight when clocks are moved forward. There may be detailed information that actually defines a local midnight in these cases but it would be very nice to avoid it. 

I think we agree that saying time values are defined as time since noon minus 12h is better than what is in GTFS today. Also the spec will need to call out the complication of DST shfits to explain what is meant by noon on service date - 12 hours. It is these changes for which I'm drumming up support.

Re: exception_type

You said exception_type=DST-1 means subtract 1h on days that local time jumps forward. Should GTFS readers also add one when local time drops back? If we agree that defining time values relative to noon works what is the advantage of adding exception_type compared to making a service_id active on days with the most appropriate noon and + or - 24h from times? Will service_id with an exception_type not use noon as the fixed time point? What about timezones with a transition right at midnight, which day gets the 1h offset?

I wrote a junky little script to find timezones with DST transition near noon and midnight

Source: http://code.google.com/p/googletransitdatafeed/source/browse/trunk/python/misc/find_pytz_transition_times.py

Output:

Near noon

from 2010-03-28 03:59:59+04:00 to 2010-03-28 05:00:01+05:00 Asia/Baku

from 2009-10-25 04:59:59+05:00 to 2009-10-25 04:00:01+04:00 Asia/Baku

from 2010-03-28 02:59:59+02:00 to 2010-03-28 04:00:01+03:00 Asia/Istanbul

Near midnight

from 2009-10-25 00:59:59-04:00 to 2009-10-25 00:00:01-05:00 America/Havana

from 2009-10-25 00:59:59+00:00 to 2009-10-25 00:00:01-01:00 America/Scoresbysund

from 2009-10-30 00:59:59+03:00 to 2009-10-30 00:00:01+02:00 Asia/Amman

from 2009-06-19 22:59:59+06:00 to 2009-06-20 00:00:01+07:00 Asia/Dacca

from 2009-06-19 22:59:59+06:00 to 2009-06-20 00:00:01+07:00 Asia/Dhaka

from 2009-10-25 00:59:59+00:00 to 2009-10-25 00:00:01-01:00 Atlantic/Azores

from 2009-10-25 00:59:59-04:00 to 2009-10-25 00:00:01-05:00 Cuba

from 2009-09-24 23:59:59+03:00 to 2009-09-24 23:00:01+02:00 Africa/Cairo

from 2010-04-29 23:59:59+02:00 to 2010-04-30 01:00:01+03:00 Africa/Cairo

from 2009-05-31 23:59:59+00:00 to 2009-06-01 01:00:01+01:00 Africa/Casablanca

from 2009-08-20 23:59:59+01:00 to 2009-08-20 23:00:01+00:00 Africa/Casablanca

from 2010-04-04 01:59:59+02:00 to 2010-04-04 01:00:01+01:00 Africa/Windhoek

from 2009-11-01 01:59:59-09:00 to 2009-11-01 01:00:01-10:00 America/Adak

from 2009-11-01 01:59:59-08:00 to 2009-11-01 01:00:01-09:00 America/Anchorage

from 2009-10-17 23:59:59-03:00 to 2009-10-18 01:00:01-02:00 America/Argentina/Buenos_Aires

from 2010-03-20 23:59:59-02:00 to 2010-03-20 23:00:01-03:00 America/Argentina/Buenos_Aires

from 2009-10-17 23:59:59-03:00 to 2009-10-18 01:00:01-02:00 America/Argentina/Cordoba

from 2010-03-20 23:59:59-02:00 to 2010-03-20 23:00:01-03:00 America/Argentina/Cordoba

from 2009-10-17 23:59:59-04:00 to 2009-10-18 01:00:01-03:00 America/Argentina/San_Luis

from 2010-03-20 23:59:59-03:00 to 2010-03-20 23:00:01-04:00 America/Argentina/San_Luis

from 2009-10-17 23:59:59-03:00 to 2009-10-18 01:00:01-02:00 America/Argentina/Tucuman

from 2010-03-20 23:59:59-02:00 to 2010-03-20 23:00:01-03:00 America/Argentina/Tucuman

from 2009-10-17 23:59:59-04:00 to 2009-10-18 01:00:01-03:00 America/Asuncion

from 2010-03-13 23:59:59-03:00 to 2010-03-13 23:00:01-04:00 America/Asuncion

from 2009-11-01 01:59:59-09:00 to 2009-11-01 01:00:01-10:00 America/Atka

from 2009-11-01 01:59:59-06:00 to 2009-11-01 01:00:01-07:00 America/Boise

from 2009-10-17 23:59:59-03:00 to 2009-10-18 01:00:01-02:00 America/Buenos_Aires

from 2010-03-20 23:59:59-02:00 to 2010-03-20 23:00:01-03:00 America/Buenos_Aires

from 2009-11-01 01:59:59-06:00 to 2009-11-01 01:00:01-07:00 America/Cambridge_Bay

[Brian Ferris]

On Wed, Nov 4, 2009 at 3:58 PM, Tom Brown <tom.bro...@gmail.com> wrote:

Re: exception_type

You said exception_type=DST-1 means subtract 1h on days that local time jumps forward. Should GTFS readers also add one when local time drops back? If we agree that defining time values relative to noon works what is the advantage of adding exception_type compared to making a service_id active on days with the most appropriate noon and + or - 24h from times? Will service_id with an exception_type not use noon as the fixed time point? What about timezones with a transition right at midnight, which day gets the 1h offset?

So the following discussion applies to the (hopefully small) set of trips that start in the gap between midnight and the time at which DST kicks in.

I think there is no getting around the fact that you will need to define a separate service_id for these trips so that you can apply calendar date exceptions separate from other trips.  However, (correct me if I'm wrong on this) if we leave the spec as is, there is no easy way to specify that these trips should run an hour earlier or later as appropriate on DST days without replicating all of the trips and stop_times.

Walking through the issue in detail, agencies would need to first create a service_id + trips + stop_times to handle all service on normal dates along with an exception to disable service on DST days.  They'd then need to create a separate but almost direct copy of service_id + trips + stop_times that is active only on DST days and whose stop_times are incremented an hour in the proper direction.

The proposed +1/-1 (or DST+1/DST-1 if you prefer) calendar_date exception types mainly saves you the step of creating a near-duplicate set of trips + stop_times just to add or subtract an hour.  Instead, you just specify the appropriate +1/-1 on the DST days you care about.

Just to be formal, the DST-1 exception means that each stop_time will be scheduled relative to midnight defined as "noon-12h-1".  By the same token, DST+1 would define midnight as "noon-12h+1".

And just to walk through an example...

I am a transit agency and I have created a service_id "NIGHTOWL" for my late night / early morning service runs. These trips start running at midnight and on DST days they should keep running as schedule from midnight as if DST did not happen.  Last Sunday, Nov 1, we fell back an hour in the USA, so 1-2 am was repeated.  If we had let the proposed "noon-12h" set the start time for our night owl runs, they would have started at 1am (the first one) instead of 12am as we would have liked.  So we add the following calendar_date:

service_id,date,exception_type
NIGHTOWL,20091101,DST-1

For these trips, midnight will now be define properly as 12am.  I believe this scheme should work just the same for DST's that occur at 12 am (which is to say, 12 am is repeated).


So all definitions and examples aside, this proposed extension really only makes sense for reducing the amount of work for properly modeling the potentially small set of trips that have direct interaction with a DST shift and specifically addresses those agencies who want to keep running service as if a DST shift had not happened.  If agencies want to have custom service with an additional trip on -1 years and a missing trip on +1 years, then they will have to go the standard way of defining a separate service_id+trips+stop_times just for those days.  I mostly speak to how King County Metro handles DST, so perhaps other agencies are doing different things (or just not modeling it at all?).

[T Sobota]

More curious than anything, our system having only three buses/drivers
out on the campus so late in the day/early in the morning - but I am
having difficulty getting my head around the various points of
reference involved in this discussion:

From the bus driver/schedule perspective, let's say an operator has a
five-hour nightly shift... driving an hourly loop five times, Trip A
departing at 11pm (82000 seconds past midnight on their duty day),
Trip B at 12am... through Trip E at 3am. Shift ends after that last
hour loop trip is completed - at 4am (100800 seconds past midnight on
their duty day).

For this past weekend then (end of DST), operationally the Saturday
duty day operator could either drive their standard five hour shift...
which would mean that their fifth trip would have occured at
"2am" (per falling back one hour at 2am local time) - while from a
duty day perspective, it still departed at 97200 seconds past midnight
of the start of their duty day. Alternatively, the transit system
could extend the work shift on such a day, to accomdate at "3am"
departure time - meaning a sixth trip would be need to be operated at
100800 seconds past midnight of that duty day - which on a properly
set clock would be between 3am-4am (again due to falling back one hour
at 2am local time).

Our own system follows the operator perspective, rather than making
any accomodation for those looking at their watches - which leaves the
public to anticipate that the last scheduled trip at "3am" (97200 SPM)
will actually occur at "2am" when DST ends... and for that matter, at
"4am" when DST starts. Likewise, when DST starts, there would be two
trips at "1am" and none at "2am" - for those looking at their atomic
clock.

Where the larger transit properties are referencing what I would terms
as multiple duty days (for their 24 hour opertions)... part of me
wonders if the feed spec should somehow classify trip id or block id
according to it's respective duty day definition. If the default
definition is stop times are referenced as seconds past midnight (i.e.
duty day start is defined as 00:00:00 or zero seconds past midnight of
the calendar date) - and properties are otherwise able to supplement
additional duty days (i.e. duty day is defined as 12:00:00 or 43200
seconds past midnight of the calendar date) with the stop times of
those particular block or trip ids offset accordingly.


Again, my apologies being largely oblivious to the DST quirks of
transit scheduling - but nevertheless interested in the impacts (in
the sense of noting if these large transit properties are indeed
extending driver shifts, etc. for this one day a year)

[Tom Brown]

On Thu, Nov 5, 2009 at 01:52, Brian Ferris <bdfe...@gmail.com> wrote:

On Wed, Nov 4, 2009 at 3:58 PM, Tom Brown <tom.bro...@gmail.com> wrote:

Re: exception_type

You said exception_type=DST-1 means subtract 1h on days that local time jumps forward. Should GTFS readers also add one when local time drops back? If we agree that defining time values relative to noon works what is the advantage of adding exception_type compared to making a service_id active on days with the most appropriate noon and + or - 24h from times? Will service_id with an exception_type not use noon as the fixed time point? What about timezones with a transition right at midnight, which day gets the 1h offset?

So the following discussion applies to the (hopefully small) set of trips that start in the gap between midnight and the time at which DST kicks in.

I think there is no getting around the fact that you will need to define a separate service_id for these trips so that you can apply calendar date exceptions separate from other trips.  However, (correct me if I'm wrong on this) if we leave the spec as is, there is no easy way to specify that these trips should run an hour earlier or later as appropriate on DST days without replicating all of the trips and stop_times.

I agree that you generally need a special service_id for trips that run between midnight and the local time shift. I assume that by "leave the spec as is" you mean with the time values defined as time since noon minus 12h. I disagree that trips in the gap need to be duplicated to handle the simple case of a trip which always operates using pre-shift time or post-shift time.

 

I am a transit agency and I have created a service_id "NIGHTOWL" for my late night / early morning service runs. These trips start running at midnight and on DST days they should keep running as schedule from midnight as if DST did not happen.

A trip that runs early in morning as though DST did not happen should use time values that are relative to noon of the day before. For example, 27:00:00 is normally 3am but after a DST shift the local time would be 2am or 4am. These trips may need to use a special service_id active on dates one before the actual date the trips run.

For the simple case Tim from Madison described the GTFS creator won't need to add anything special for dates with a DST shifts. In fact, if a Sunday morning vehicle (operating around the time shift) is run as part of the Saturday duty day by a driver who leaves a watch on Saturday time they may already be doing the right thing because the trip needs a Saturday service_id to get the correct holiday exceptions. It is convenient that many trips around the gap can be modeled without the GTFS listing each shift date.

[Brian Ferris]

Ok, I see you what you mean and agree that this is within the realm of the spec.  It never occurred to me model it that way since the night service routes I've been looking at are all scheduled relative to midnight on Sunday in the native scheduling system.

Sorry for raising so much commotion.