Real-time subway arrival estimates - we need your help

[Aaron Donovan]

All,

As you may have heard, the MTA and MTA New York City Transit are
designing a developers' web feed of subway arrival estimates
("countdown clock data") for the 1, 2, 3, 4, 5 and 6 subway lines. As
the MTA and NYC Transit design this web service, we would like to
receive feedback from developers on phase one, a web feed of GTFS-
realtime data, and phase two, a web feed using SIRI.

If you are interested in offering this kind of advice, and are willing
to agree to keep your observations and feedback under wraps until the
MTA goes live with the data, please send me an email, to: adonovan
(at) mtahq (dot) org.

Thanks,
Aaron

[Aaron Donovan]

Hi everyone,

Many thanks to all of those of you who volunteered to help us on this
project. We've re-evaluated the way we're going about our prep work
and have determined that the best way to solicit advice and feedback
from all of you is to do so right here on this board.

So, at the link below is a working draft GTFS-Realtime spec that we
have developed. GTFS Realtime users, please have a look at this
document.

http://httqa.mta.info/developers/pdfs/GTFS-Realtime-NYC-Subway.pdf

If you have any suggestions for how it could be improved, please let
us know. We are all ears. The purpose of this spec is to serve GTFS-RT
developers and ultimately the general public.

Folks from our GTFS-RT team will be monitoring this board and will
available to respond to questions and take down feedback.

Thanks again,

Aaron Donovan
Spokesman
Metropolitan Transportation Authority

[Sunny]

Given that the PA/CIS shows GO's completely, I would assume the feed will reflect them too? If it does then it will be a dramatic improvement over current apps and trip planners.

Is there a plan to implement this for BusTime-enabled routes?

[tomcola58]

30 minutes before the schedule departure time the RT feed will deliver
the adjusted timetable as modified because of ongoing planned work
(GOs). It is a significant improvement over the current subway plans
hence our excitement. Will leave the bus discussion to others.

Tom

[Sunny]

For the B division routes which are not expected to be in GTFS-realtime for a while, I think the MTA should release more detailed notice as to upcoming GO's. (first train/last train times, approximate bus headways for shuttles that are not load-and-go)

[Matt]

Hi,

I think there should be more details to Alert message than just "Train Delayed", such as "All northbound 4 trains are delayed due to signal problems"

[Adam Ernst]

I think that's a separate system, and while it would be great to have access to that data as well, it may beyond the immediate scope of this project. 

The MTA does already have an API for service alerts, and while it lacks the standardness of GTFS Realtime, it gets the job done. 

Adam

[Michael Justice]

The API for service alerts- are you talking about the ServiceStatus.txt file? Or some other API?

MJ

[tomcola58]

Thanks Adam. I could not have said it any better. The original scope
of this project is to mirror the automated train arrival information
that is published to the CIS signs. For example "1 South Ferry 4
minutes". If this train is determined to be stalled the sign will
read "1 South Ferry Delayed". We want to provide this trigger so that
consumers of our data could mirror this behavior. When the train
moves again the Delayed will be replaced with arrival predictions.
Current no Public Service Announcement or or Service Alerts are being
pushed across the interface. As Adam has stated that information is
available elsewhere.

Tom

On May 21, 8:18 pm, Adam Ernst <adamjer...@me.com> wrote:
> I think that's a separate system, and while it would be great to have access to that data as well, it may beyond the immediate scope of this project.
>
> The MTA does already have an API for service alerts, and while it lacks the standardness of GTFS Realtime, it gets the job done.
>
> Adam
>

[Adam Ernst]

Yes,  ServiceStatus.txt. It has it's shortcomings of course but it proves there is a system somewhere that is responsible for service alert info (and I doubt it's directly related to the one for GTFS Realtime)

Adam

[Michael B. Justice]

OK.  I see your point but it isn't a good substitute for a "legit" API that provides the information in advance.  

MJ

[Aaron Donovan]

Hi everyone,

At the link below, please find the current (beta) version of the MTA New York City Transit GTFS-RT spec (in pdf), and NYCT proto file (in txt). The revisions to these specifications were in response this discussion thread and one on the GTFS-realtime Google Group. 

This is the version we plan to develop, test and make available later this year. During our beta phase we would make additional changes as necessary (hopefully no major changes will be needed, if any at all).

Thank you to all who helped us arrive at this document, for your insight and support. Everyone on our team feels this approach worked very well and helped us reach what we expect will be a very useful spec.  

http://httqa.mta.info/developers/internal/GTFS-Realtime-NYC-Subway-Updated-Spec.pdf

http://httqa.mta.info/developers/internal/nyct-subway-proto.txt 

-Aaron

[Aaron Donovan]

Hey everyone,

The attached zip file has a 10-minute sample of real time train movement data from one afternoon last month on the 1, 2, 3, 4, 5 and 6 subway lines and the 42nd Street Shuttle, using the spec we'd all helped to develop earlier.

Please have a look and play around with it and let us know if there are any unforeseen issues. Your feedback here on this board would be very helpful to us. But please don't blog about this data release or launch any public demo projects without notifying us first. 

Thanks,

Aaron

On Tuesday, May 1, 2012 5:08:39 PM UTC-4, Aaron Donovan wrote:

[Jeremy Baron]

Hi,

On Fri, Jul 13, 2012 at 5:08 PM, Dan Jabbour <picn...@gmail.com> wrote:
> Are the individual gtfs files compressed somehow? I unzipped the main folder
> and when I open the files I get stuff that looks like this:
>
> n
>
> 1.0¡˝Ì˛ >`
>
> 1.0
[...]

They are serialized using protocol buffers into a binary format that's
not easily readable by humans. See Aaron's original mail on this
thread. (quoted below)

-Jeremy

[Aaron Donovan]

Hi All,

Attached please find the latest revised GTFS-RT specification and associated test data. The significant enhancements with this release include the inclusion of Vehicle Positioning data which can be used to determine a “stalled train” condition.  We also expanded track information to include both schedule and actual track.  This can be used to determine when a train is not operating according to it scheduled plan.

-Aaron

On Tuesday, May 1, 2012 5:08:39 PM UTC-4, Aaron Donovan wrote:

[Adam Ernst]

Great! I'll be taking a look when I can.

Thanks Aaron, for keeping us up-to-date on this exciting project.

Adam

[John Paul N.]

Would it be possible to report the (first/head) car number of the
train in service? Or, more pertinently, it would be great to know the
car-type that's operating (e.g. R62, R142...).

It appears that detailed messages such as those regarding the reasons
for delayed trains or diversions due to planned service changes will
not be supported in this project or version of the project.

For those who may not know, there is cross-posting at gtfs-realtime
group with a good discussion, but, appropriately, the discussion there
is about applying the NYCT-only extensions to the established gtfs-
realtime protocol: http://groups.google.com/group/gtfs-realtime/browse_thread/thread/2962619171f49018/84fb7385d18b6728

>  gtfs.2
> 56KViewDownload
>
>  gtfs.3
> 56KViewDownload
>
>  gtfs.4
> 56KViewDownload
>
>  gtfs.5
> 56KViewDownload
>
>  GTFS-Realtime-NYC-Subway 10 Sep.doc
> 156KViewDownload
>
>  NYCT-Subway.proto.txt
> 5KViewDownload

[tomcola58]

JP N

Currently that information (car numbers and car-type) is not available to data source that provides the feed.  Since it is not needed to drive our countdown clocks which is the source of our feed that information is not provided for distribution.  Sorry about that.

[nineonesevennyc]

Great to see progress on the reporting schema!

 

Any updates on gathering real-time information for Division B? (ie: the rest of the subway system, beyond the numbered trains)

 

Thanks..

[Aaron Donovan]

Hi there,

The ability to provide real-time information on different segments of the subway is largely dependent on the capabilities of the systems we use to control signals and train movement.

On the 1, 2, 3, 4, 5 and 6 lines and the 42nd Street Shuttle (collectively, the majority of the A Division or the former I.R.T.), we use the modern, computerized system known as Automatic Train Supervision (ATS), which was activated incrementally between 2005 and 2010 and allows our dispatchers to see precise train positions in real time. This is the system that feeds information into the countdown clocks that we are activating in stations along these lines.

On the A, B, C, D, E, F, G, J, M, N, Q, R and Z lines (collectively, the majority of the B Division, or the former IND. and B.M.T. systems), we use a fixed-block signaling system that uses technology that essentially dates back to the dawn of the subways. While we maintain the system continuously and regularly upgrade its components, the underlying technology has remained little changed since the nineteenth century. It is a time-tested, fail-safe system that allows our dispatchers to know a train's general position within fixed blocks of track, but it does not allow our dispatchers to see a train's precise position in real time. We have long-term plans in place to upgrade these lines to modern, computerized train control, pending availability of funding in future five-year capital programs. Our current five-year capital program runs through 2014.

Finally, the L line is controlled by our most sophisticated train dispatching system, Communications-Based Train Control (CBTC). Countdown clocks on the 24 stations of the L line are powered by CBTC.  MTA New York City Transit is currently installing CBTC on the 7 line.

-Aaron

[Sunny]

From what I understand, on the B division, a block can only be occupied by 1 train at at time and there are multiple blocks between two stations.

So why can't the arrival info be given that way? (coarser than the IRT, but finer than the station-by-station currently in use)

[Kurt Raschke]

Sunny,

The problem is that that information is not hooked into anything resembling a computer (just the model boards in the towers); that's precisely what the IRT ATS does.

-Kurt

[Samuel Wong]

Another problem is the designation of the train.  For example, the blocks don't tell us if the T/O punch their lineups for diversion or the main line.  A good example is the DeKalb switch.  You have four lines there going towards the Bridge and one line via the Montague Tunnel (or via 4th Avenue (Sea Beach, 4th Avenue, West End) and Brighton (Local/Express) in the opposite direction).  While the towers know the train lineups based on the boxes and communications relay, that info is not yet computerized.  We wouldn't know if which train is actually arriving at the station; thus at Pacific/Atlantic, you only get the Lcl/Exp train announcement and not the specific D/N or R annoucement in the IRT lines (minus 7).

--
Samuel Wong
samwo...@gmail.com

Google Voice (recommended): (646) 484-9142

(646) 648-2179
Please consider the environment before printing this message.

Government is a trust, and the officers of the government are trustees. And both the trust and the trustees are created for the benefit of the people. -Henry Clay

[nineonesevennyc]

Was curious if there were any updates on the optical character recognition (OCR) pilot mentioned a while back....  

http://secondavenuesagas.com/2012/06/26/photo-a-b-division-countdown-clock-in-trial/

[Sunny]

"Train 2 stations away" is certainly better than no announcements at all! But at Pacific on the northbound express platform, the announcements are useless because one station away is 36 St and there are multiple trains between 36 and Pacific.

I've heard audio arrival announcements at Jay St-MetroTech, how are those done?

[Samuel Wong]

Sunny,

Do you know if they Jay Street announcements are on the IND level or the R line and if they are automated? The automated version examples are on the A division (male) and Canarsie (female). I know sometimes the signal towers make manual arrival anmoucements through the PA system.

Sent from my Android Thunderbolt.

[Sunny]

These are on the IND level, in both directions. I never heard announcements on the R level. They appear to be manual announcements, but since whoever is reading the screen is able to identify a location and designation, it should be enough to input into a computer to generate automated announcements...

[tomcola58]

Hi DJ

Regarding your question (times that trains arrive and then leave each station along with an ID number of said train along). We will provide both arrival and departure times (see StopTimeUpdate message) but they will be the same unless there is a scheduled “Hold” for that trip.  Station dwells vary for a number of reasons and could range from a few seconds to much longer.  For example we may hold a local train (unplanned) for the arrival of an express to allow a smooth transfer of passengers.  Just too many variables and too granular to be of any real use.

Hope this helps

Tom

On Tuesday, September 25, 2012 12:34:02 AM UTC-4, Dave Jensen wrote:

Hi Aaron,

I'd first like to thank you helping coordinate this effort among developers.

I'd like to suggest something I haven't really seen here regarding what the API could offer which would be both simpler and more flexible.

Instead of having an API provide just countdown clock information for a few lines which, given it's inherent imprecision, isn't that flexible, why not provide...

--> the times that trains arrive and then leave each station along with an ID number of said train along

Having this corpus of data available both real-time and historical would provide third parties the ability to accurately forecast when trains will arrive where as well as the impact of delays.  Further, developers/statisticians could then use this mined data to inform customers as to the nature of whether a train is running "slow" or not.  One could even impute crowd levels on trains with the length of time spent at a stop.   Ultimately, this far simpler API would allow no shortage of value for end users as compared to a predigested imprecise feed of data.

Best,

DJ

[marcelo gobelli]

Hi,

I have a quick question. Is there an android app that is currently using GTFS realtime format for the subway lines? I would like to try it..

Thanks,

M

[Samuel Wong]

I haven't seen one yet on the Market, but I'm sure the developer for SchedNYC will incorporate some of the capabilities soon (at least for IRT ones).  SchedNYC is probably one of the most comprehensive NYCT apps on Android.

[John Paul N.]

Aww Sam, you don't have to keep plugging my app. :) But seriously, it took me at least a week just to learn about and install Protocol Buffers and the GTFS realtime and the MTA's proto.txt. And that latter file was the first version of the file from May. So long story short, I'm way behind in trying out the NYCT GTFS-realtime format.

As soon as the MTA announces an ETA for this service, I'll put my energy into it.

reydelcompas, the MTA has not released the API for public use yet, so there should be no apps using the GTFS-realtime for New York City. I did a search for "gtfs real time feeds" and there were some American transit systems that popped up. Hopefully you'll find apps for those cities that are using GTFS-realtime.

[tomcola58]

So I can catch up are looking at a realtime feed?  As you can imagine since we are a 24x7 operations, all of our maintenance and repairs are done during operating hours.  This will result in changes to the daily operating timetable the will deviate from the GTFS static feed.  Therefore the GTFS static feed schedules should be use for terminal planning purposes only.  

The realtime feed will push out current and/or possibly revised (caused by schedule changes) train information.  This begins between 25 and   30 minutes before the schedule departure time.  We also added an "is_assigned" bit to the NYCTTripDescripter.  This declares that a physical train is at the terminal and has been assigned a valid trip id with schedule information.  As stated above, there may be times where this deviate from the static data.  

Hope this helps

Tom C

So 

On Friday, November 16, 2012 12:30:18 PM UTC-5, Bowen Yu wrote:

Hi Aaron,

I have checked out this data and tried to correlate it with the MTA static data from http://httqa.mta.info/developers/ as a comparison between realtime train status and planned service. However I found that many trip instances in the realtime data cannot find a match in the planned service. What I'm doing is as follows:

For a trip_id, e.g. 079005_1..S03R, I know that in static data trip_id looks like A20120610WKD_0790000_1..S03R, so I check the week day of the start date of a realtime trip, add a prefix (WKD) to the time (079005) and get WKD_079005_1..S03R. Then I round up the second field as the static plan is scheduled on a 30 second interval, which means that the second digits can only be 00 or 50. I'm not sure if I shall round up or down so I try every possibilities, yielding WKD_079000_1..S03R, WKD_079050_1..S03R, WKD_079100_1..S03R, WKD_078950_1..S03R, etc. Finally I use these trip_id as keys to search in the MTA static data. What I can get are partial matches between realtime trips and planned service.

I wonder if I'm doing anything wrong here? or the static and realtime trips simply has certain amount of mismatches?

Thanks!

在 2012年9月10日星期一UTC-4上午11时28分45秒，Aaron Donovan写道：

[Sam Vermette]

Any updates on this? Does the MTA have any ETA of when it'll be releasing a production GTFS-realtime buffer for its subway lines?

[Aaron Donovan]

Hi Sam.  No word on a date yet.  Sorry, I know the uncertainty is frustrating.

-Aaron

[Nine Oneseven]

Any updates on real-time info for Division B?  (ie: A,B,C,D,E,F,G,M,N,Q,R,Z)?  

Understood the track circuitry is not in place to do things the way Division A does, but the MTA had at one point thrown out some intriguing ideas using OCR.  

[Aaron Donovan]

OK, the A Division data feed should be live at some point this morning.  Stay tuned for more.

-Aaron

On Thursday, December 20, 2012 3:17:46 PM UTC-5, Sam Vermette wrote:

[Aaron Donovan]

To answer your question, and nineonesevenNYC's question, about the B division, at this point all I can really do is recap what we've just announced to the public:

http://mta.info/news/stories/?story=921 

MTA Subway Time, like the countdown clocks at stations, is powered by a system called Automatic Train Supervision (ATS). This modern, digital computer system is used by train dispatchers and managers to monitor and control train movement. It is made possible by more than a decade of work and hundreds of millions of dollars' worth of investment in the MTA Capital Program to modernize signal and dispatching equipment along the seven subway lines that now provide real-time arrival information.

Subway tracks along these seven lines are electronically separated into segments between 30 and 1,000 feet long. Whenever a train enters a new segment, it updates the ATS system with the train's latest location, speed and route. ATS instantly shares the information with Subway Time, which automatically consults schedule data and feeds it through complex algorithms to create precision time-based arrival estimates.

“The existence of this sleek digital interface barely hints at the investment that had to be made in terms of hardware and infrastructure to make this enormous public benefit a reality,” said Thomas F. Prendergast, President of MTA New York City Transit. “Think of Subway Time as the small tip of a huge iceberg. For a product of this quality to be available on the lettered lines, we will need to commit hundreds of millions of dollars and years of dedicated effort.”

Automatic Train Supervision installation began in 1997. The system was activated in segments, with the substantial completion taking place in 2008. The project cost $20.8 million per year over 11 years, or $228.3 million in total.

The L Line is controlled through Communications Based Train Control (CBTC), a signaling system that is even more advanced than ATS and is capable of feeding real-time train arrival estimates to Subway Time as it does for countdown clocks at stations. The MTA plans to add the L to Subway Time over the next six to 12 months. Signaling on the 7  7Line is being upgraded to CBTC. The project is expected to be completed in 2016, and the 7 could be added to Subway Time thereafter.

The remaining subway lines use a fixed block signaling system that dates to the dawn of the subways. “The technology it uses has remained little changed since a time before computers, microprocessors, wireless telephones or handheld electronic devices,” Prendergast said. “It is a time-tested, fail-safe system that continues to flawlessly perform its vital intended function: preventing collisions. But it cannot offer us a digital feed.”

The MTA has long-term plans in place to upgrade these lines to ATS signaling, pending availability of funding in future capital programs. In the meantime, the MTA is exploring ways of providing real-time arrival estimates using other means. For example, New York City Transit has been experimenting with rudimentary countdown clocks at a number of stations on the lettered lines. And GPS could be used to provide estimates, at least for the elevated portions of routes.

On Friday, December 28, 2012 8:58:35 AM UTC-5, Team TransitUP wrote:

Congrats to the MTA for the impending launch of real-time Division A data!

This naturally begs the question of when real-time data for Division B  will arrive.  Since the advanced track/signal circuitry on Division A is not in place on Division B, it's time to look at rapidly deployable, less capital-intensive alternatives to capture and share real-time info for the A,B,C,D,E,F,G,J,M,N,Q,R,S, and Z trains that don't rely on traditional infrastructure.

Folks who are interested should drop a line to outr...@transitup.com

Cheers!

[Sunny]

Some questions about the (web) app:

-How do I know if a train is running local or express?

-How do I distinguish between the (6) and the <6>?

-What do the asterisks mean?

-How are battery runs shown?