Past-midnight trips during bundle switchover

[Frumin, Michael]

This email concerns the issue of switching bundles in OBA during active bus service (but at the transition point for agency scheduling periods).  This is important for us because we have active bus service 24/7.

 

Consider the following excerpts from a GTFS calendar.txt file, from a hypothetical GTFS file named Spring.zip:

 

service_id,monday,tuesday,wednesday,thursday,friday,saturday,sunday,start_date,end_date

20120408SAT,0,0,0,0,0,1,0,20120408,20120707

20120408SUN,0,0,0,0,0,0,1,20120408,20120707

 

This shows the service ID’s for, respectively, Saturday and Sunday service for the time period from April 8, 2012 through July 7, 2012.    Consider also some trips from the trips.txt file for this same hypothetical GTFS file below.  Note that, for convenience of exposition, trip_id is generated by (in pseudocode) sprintf(“%s_%s_%s_%s”, $trip->service_id, $trip->route_id, $trip->start_time, $trip->end_time) :

 

route_id,service_id,trip_id,trip_headsign,direction_id,block_id,shape_id

R1,20120408SAT,20120408SAT_R1_2350_2520,”Some Late Saturday Night Trip on R1”,0,,

R1,20120408SUN,20120408SUN_R1_0600_0730,”Some Sunday Morning Trip on R1”,0,,

 

During weekends entirely covered by this GTFS file, there will not be a problem with the bundle making available information on trip 20120408SAT_R1_2350_2520 (a trip on route R1 that starts late on Saturday night and continues into Sunday).  So, for example, at 00:15 on, say, April 15, the bundle will indicate that this trip is active on route R1.  At the same time, it would indicate that both of the following trips are active on route R2: 20120408SAT_R2_2350_2520 and 20120408SUN_R2_0001_0230.  So far, so good.

 

However, consider the case when the scheduling period ends and we need to swap bundles, at say the midnight between 7/7/2012 and 7/8/2012.  Consider an excerpt from the calendar.txt for the GTFS for the period July 8 through September 1, 2012, named Summer.zip:

 

service_id,monday,tuesday,wednesday,thursday,friday,saturday,sunday,start_date,end_date

20120709SAT,0,0,0,0,0,1,0,20120709,20120901

20120709SUN,0,0,0,0,0,0,1,20120709,20120901

 

And for trips.txt (basically the same as the previous service period, with new service ID’s):

 

route_id,service_id,trip_id,trip_headsign,direction_id,block_id,shape_id

R1,20120709SAT,20120709SAT_R1_2350_2520,”Some Late Saturday Night Trip on R1”,0,,

R1,20120709SUN,20120709SUN_R1_0600_0730,”Some Sunday Morning Trip on R1”,0,,

 

We swap the Summer bundle to replace the Spring bundle at 2012-07-09 00:01.  Then, at 00:15 on 7/9/2012 and we query for all the trips on R1.  We get back nothing, even though, in reality, trip 20120408SAT_R1_2350_2520 is still running from the previous scheduling period.

 

 

That’s the basic issue – some info simply falls through the cracks for trips taking place on Sunday but on a Saturday service_id when you have swapped a bundle at midnight.  Thoughts on anything we could do to help with this?  I have one hacky idea but I will keep it to myself for the time being so as not to muddy the waters.

 

 

Thanks,

Mike

[Brian Ferris]

I don't have a good solution to this, other than suggesting coming up
with a merged GTFS of some kind that would allow you to operate across
the schedule change more seamlessly. However, depending on the nature
of the changes to your GTFS feed across a service change, that might
not be a viable option.

Brian

> --
> You received this message because you are subscribed to the Google Groups
> "onebusaway-developers" group.
> To post to this group, send email to onebusaway...@googlegroups.com.
> To unsubscribe from this group, send email to
> onebusaway-devel...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/onebusaway-developers?hl=en.

[Frumin, Michael]

That is what I was afraid of. I don't think that a merged GTFS is really practical, because often across a pick/schedule change we will want to add/remove/change stops and routes and GTFS doesn't support saying "this stop/route should exist until such a date, but no longer".

How about this as a hack? The OBA bundle builder has a configurable flag that can tell it to take the start_date of all entries in calendar.txt and move them up one day. That way at least when you swap the bundle you will have the late-night trips for the Saturday service for the *new* bundle applying in that just-after-midnight window on Sunday right after you swap bundles.

Any obvious downsides to that?

Thanks,
Mike

> That's the basic issue - some info simply falls through the cracks for

[Brian Ferris]

I think it'd be easier to apply the hack as a GTFS modification that
produces a modified GTFS with the extra day of service. That way the
bundle builder can do what it's good at (building bundles) and the
GTFS modifier can do its thing.

Brian

[Frumin, Michael]

Fair enough.

It's not a trivial shell script to modify calendar dates but certainly something that could be done prior to building the bundle with a more sophisticated script.

[Frumin, Michael]

Brian, I've been thinking about this a bit more, especially since your description of how they currently manage bundle switchovers in Seattle (running 2 instances at once, and swap them at midnight).

After more consideration, I actually think that it makes sense for this to be an option on the main OBA bundle-builder. The reason is this -- anyone who wants to use OBA in a real production setting is going to have the issue I described in this thread, and so are likely to need the ability to mod the bundle's start date since that's the only manageable solution that currently exists.

Part of our work on onebusaway-enterprise has been to enable OBA to swap bundles mid-stream. This absolves people from having to run 2 instances at once as you described, but doesn't fix the problem of trips crossing midnight on the date of the schedule change. At this point, nobody has presented any other solution that makes sense to even partially remediate the issue.

Or am I way off base?

[Brian Ferris]

I guess I'm not opposed to hot bundle swapping.  I think that, combined with a better merged GTFS, could get you past the 24 hour operation problem potentially.

What does it take to implement hot-swapping?  You've already enabled this in the enterprise code, but does that require any changes to core?

[Frumin, Michael]

Brian, thanks.  My understanding of our in-person discussion was what I thought was a shared acknowledgement that GTFS doesn’t really currently support 2 picks at the same time.  At a minimum, it doesn’t allow one to say that certain stops or routes apply only for certain parts of the merged period.  When a route or a stop is removed or added, it will usually be on pick boundaries.

 

Can’t really speak about the details of how the hot bundle swapping was implemented.  Jeff/Sheldon?

 

Thanks,

Mike

[Jeff Maki]

Any of the changes required were already merged into OBA app mods. We
can move the hot swapping into the main repo, too, eventually--once
we've tested it fully over time.

-Jeff

Jeff Maki
OpenPlans Transportation
917-388-9088
jm...@openplans.org

[Brian Ferris]

I guess I'm arguing that we should do the work on GTFS to make that possible ; )  It's not just a problem for OBA, for example.

[Matt Conway]

At this time, there is a parallel discussion going on about defining transit using GTFS-realtime exclusively; it might be possible to have a bundle with the new GTFS and then publish a GTFS-realtime feed adding back the deleted stops and routes during the switchover (or vice-versa).

[Brian Ferris]

I think it doesn't need to be nearly so complicated.  I'm imagining something that simply merges the pre and post pick GTFS feeds in a way that stable trip / block ids can be maintained for matching to the realtime system.  The simplest way would just  be to merge the feeds, and if you have a trip that modifies its schedule across the pick, you rename the trip but add an additional field to indicate its original trip id so you can match it to realtime.

Brian

[Frumin, Michael]

That is clearly a critical requirement to make this work in a generalized way. Our GTFS has the convenience that all service_id and trip_id and block_id are prefixed with some bytes representing the pick they belong to (at least for our largest bus agency).

But what about the case where a new route is added in the new pick and we don't want it to show up in UI's and API's until the pick has actually happened? Under the scenario you describe, it would start to show up as soon as we loaded the transitionary 2-pick bundle. Similarly, a route that is being removed won't go away until we remove the 2-pick bundle. Same goes for stops.

Maybe the answer to the above is just that we tolerate that for what should be a very limited lifespan on the 2-pick bundle. But now we have created a pretty nightmarish operational/management situation where we need to:
- build a bundle for a given pick (bundle A)
- build a bundle for the new pick (bundle B)
- build a bundle for the transition (bundle AB)
- assuming we are in the given pick, replace bundle A with bundle AB as close as possible to midnight before the pick change
- replace bundle AB with bundle B as soon as possible after all blocks from bundle A are finished after the pick change.

It scares me to think about all of this in the context of our bundle building which also links GTFS to other data sources (eg operational schedules). Could you imagine doing this not at the GTFS level, but at the bundle level?

Thanks,
Mike

[Frumin, Michael]

One note: hot bundle swapping is useful not just for pick changes (which have all the issues described in this thread) but also for updates of the current pick which don't really have these issues, where everything is basically working flawlessly (knock on wood).

Thanks,
Mike

[Brian Ferris]

Service changes do complicate everything but I do think it's important to both come up with a merged GTFS across picks.  For 3rd party consumers of your GTFS and realtime data, having to synchronize their own updates to your bundle switch is tricky, if not impossible.  Having a reasonable merged GTFS can help mitigate this complication.

Either way, the merged GTFS issue is a bit orthogonal to the bundle swapping functionality.  I'm +1 on swapping support and I'd be curious to understand how you currently manage your workflow.

Brian

[Frumin, Michael]

Agreed re: orthogonality.

Workflow for bundle swapping at a high level. Sheldon understands the details a more than me at this point:
1. build the bundle
2. look for errors in our data indicated by build/validation outputs
3. rebuild
4. repeat 5 times until no errors
5. deploy bundle to Transit Data Manager (TDM component)
5a. Either as part of how the bundle is deployed to the TDM or how it is packaged before that deployment, it is given a date on which it should be applied to the actual system
6. Every 15 minutes, all bundle-needing components check in with the TDM to see if there are any bundles they should download to have ready to apply
7. At midnight on the date of a given bundle's applicability, all bundle-using components hot-swap their current bundle for the new one.
7a. If the date of the start of a bundle's applicability is in the past, the bundle-needing components swap in the new bundle at the start of the next hour.

(I guess 7 and 7a are really the same thing).

One thing to be clear about: the bundle applicability dates are distinct from the dates in the calendar.txt and calendar_dates.txt in the GTFS.

Make sense?

This needs to be documented on the currently content-free onebusaway-enterprise wiki.

Thanks,
Mike

[fruminator]

so, i didn't realize that the bundle builder had the whole pluggable pre-post task thing.  looks like that was a good way to make a plugin to munge the dates.  cool!

>> To post to this group, send email to onebusaway-developers@googlegroups.com.

>> To unsubscribe from this group, send email to

>> onebusaway-developers+unsub...@googlegroups.com.

>> For more options, visit this group at
>> http://groups.google.com/group/onebusaway-developers?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups "onebusaway-developers" group.

> To post to this group, send email to onebusaway-developers@googlegroups.com.
> To unsubscribe from this group, send email to onebusaway-developers+unsub...@googlegroups.com.

> For more options, visit this group at http://groups.google.com/group/onebusaway-developers?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups "onebusaway-developers" group.

> To post to this group, send email to onebusaway-developers@googlegroups.com.
> To unsubscribe from this group, send email to onebusaway-developers+unsub...@googlegroups.com.

> For more options, visit this group at http://groups.google.com/group/onebusaway-developers?hl=en.
>

--
You received this message because you are subscribed to the Google Groups "onebusaway-developers" group.

To post to this group, send email to onebusaway-developers@googlegroups.com.
To unsubscribe from this group, send email to onebusaway-developers+unsub...@googlegroups.com.