Commuter Rail Feed Update

[Developer at MBTA]

Greetings developers! This space has been quiet lately, but we've been at work on some initiatives and will have some announcements coming soon, so keep watching this space for news and an upcoming meeting announcement. Today we're writing to tell you about a long-awaited update to our commuter-rail real-time data feed coming this Thursday, April 4th. 

The following improvements will go into effect: 

• Multi-Line Trips: Trips that operate partly on one line and partly on another, such as on the Franklin and Fairmount Lines or on the Lowell and Haverhill Lines, will now show up with the correct stations in the correct lines. For example, trip 212’s arrival at Ballardvale will appear in the Haverhill Line file, its arrival at Wilmington will appear in the Lowell Line file, and its arrival at North Station will appear in both. 
  
• JSON format changed to be smaller and adhere more closely to the intention of the standard. 
  
• “Duplicate” trains (the same trip being listed twice with two different vehicles) eliminated. 
  
• Numerous bugfixes and reliability improvements. 
  

Following is a sample of data in the new JSON format:

{
         "TimeStamp":"1327351837",
         "Trip":"175",
         "Destination":"Newburyport",
         "Stop":"Salem",
         "Scheduled":"1327351560",
         "Flag":"pre",
         "Vehicle":"1645",
         "Latitude":"42.49346",
         "Longitude":"-70.90718",
         "Heading":"34",
         "Speed":"65",
         "Lateness":"300"
}

Thanks for your attention, and if you have any questions please reply or send them to this address. 

Sincerely,

Developer at MBTA

[Developer at MBTA]

Matt, 

We had been planning on upgrading the existing JSON feed in place, but we were assuming that the existing JSON wasn't getting much use. We don't want to break your application for all your existing users if we can avoid it and are looking at alternatives now. We'll have an update shortly. 

Sincerely,

Developer at MBTA

On Monday, April 1, 2013 6:14:38 AM UTC-4, Matt Pietal wrote:

Are you planning to upgrade the existing JSON feeds in place?  The reason I ask is that I have a few thousand users of my commuter rail app and three days isn't enough time to rollout an upgrade to all users.  Ideally there'd be a small overlap period where both the old and new data feeds are in place so that I can roll over users and avoid an interruption in service.  This would also give developers a chance to test the new feeds first.

Thoughts?

Thanks,
Matt Pietal

[Matt Pietal]

Thanks for taking a look. I appreciate it.  If there's nothing you can easily do that's fine too.  Just trying to buy a bit of time to rollout an upgrade.

[Developer at MBTA]

All,

To accommodate developers using the current JSON feed, we have amended our real-time commuter rail transition plan.  

Thursday, 4/4/13: Old feeds at RTCR URLs, new feeds at temporary "RTCRnew" URLs. 

New feeds will be available at URLs with "RTCRnew" replacing "RTCR". So for example the existing Greenbush JSON feed will remain available at http://developer.mbta.com/lib/RTCR/RailLine_1.json , and the new-format JSON feed will become available at http://developer.mbta.com/lib/RTCRnew/RailLine_1.json . 

"New" versions of the JSON, CSV and XML files will be available at the RTCRnew URLs, but JSON is the only one for which the feed format itself is changing.  

Thursday, 4/25/13: New feeds at both RTCR and temporary "RTCRnew" URLs. 

At this time the new feeds will replace the old feeds but both URLs will remain functioning. Both http://developer.mbta.com/lib/RTCR/RailLine_1.json and http://developer.mbta.com/lib/RTCRnew/RailLine_1.json will have the new-format JSON. 

Thursday, 5/16/13: Temporary "RTCRnew" URLs retired. 

At this time JSON users will need to be using http://developer.mbta.com/lib/RTCR/RailLine_1.json . 

So if you're using CSV or XML you don't need to change URLs, and you'll start seeing the improvements in the data as of 4/25/13 (but you can preview it at the temporary RTCRnew URLs if you want to.) If you're using JSON you will need to make two changes: once to the new-format JSON at the new URL between 4/4/13 and 4/25/13, and once back to the old URL between 4/25/13 and 5/16/13. We know two changes are less convenient than one change but we don't want to keep multiple URLs around forever or permanently change to different ones. 

We hope that's clear and that it works for everyone! Let us know if you have any questions and we'll update this group once the RTCRnew URLs are working some time on Thursday. 

Sincerely, 

Developer at MBTA

[Developer at MBTA]

Alan, sorry we didn't send you a reply -- writing you an email off-list to follow up on this. 

-Developer at MBTA

On Tuesday, April 2, 2013 11:45:11 AM UTC-4, Alan wrote:

Can you please also look at making the JSON available to JavaScript applications via CORS? My posts to this group have gone ignored.

[Developer at MBTA]

Hi FlyerD. Nothing to announce about that at this time. We will say in a general way that we recognize that having different and unique feed formats for different modes is not ideal.  

Sincerely,

Developer at MBTA

On Tuesday, April 2, 2013 5:03:47 PM UTC-4, FlyerD4001 wrote:

I was wondering when Commuter Rail data would be included into the GTFS real - time feed? It would be a helpful addition to it and hopefully the other subway lines eventually.

[Developer at MBTA]

The new feeds are now live. To access them replace RTCR with RTCRnew in the URL. So for example the old-format JSON is currently available at http://developer.mbta.com/lib/RTCR/RailLine_8.json , new-format at http://developer.mbta.com/lib/RTCRnew/RailLine_8.json . 

In three weeks they'll both be new-format. In three more weeks the RTCRnew URL will go away. 

Please feel free to contact us at this address or to post to the group if you have any questions. 

Sincerely, 

Developer at MBTA

[Developer at MBTA]

Stefan, 

We'll consider your request for a future enhancement, but unfortunately we won't be able to accommodate the request right now. Right now we really just need to get the duplicate trips out of there, it has been the #1 request of developers using the feed (as you said your web app is probably a unique case.) 

Sincerely,

developer at MBTA 

On Friday, April 5, 2013 1:01:59 PM UTC-4, StefanW wrote:

Thanks for getting us the new JSON format and the multi-line trips!!

However, I'm actually very disappointed to learn that the duplicate trips will be filtered out! My web 'app' may be a unique case, but I'm more interested in any and all vehicles' data regardless of trip number duplication. The goal of my app is visualization & mapping and it's not necessarily intended for use as a commuter tool. (I do use my own app for my own commute every day though, and so do many of the people with whom I've shared it.)

Feel free to try my development / beta version and you'll see what I mean:

http://www.people.fas.harvard.edu/~wuensch/T/live-train-map-beta.html

(My original version was a painful full page load for each data pull but it's more polished with more explanation than the new version at the moment: http://www.people.fas.harvard.edu/~wuensch/T/live-train-map.cgi )

I **strongly** suggest that instead of filtering out duplicate trips, just add a flag / field to that vehicle's report which would let **us** filter it out. It's really important to me to be getting as many data points as possible. I made my app public on the MBTA forum at railroad.net and it's been very popular (for example) in tracking the new Hyundai Rotem coaches that are in acceptance testing. (The new coach 1800 has made many appearances in the data feed. Example: http://www.railroad.net/forums/viewtopic.php?f=65&t=51136&start=615#p1142477 ) If the duplicate trip info is gone, there will be a huge number of disappointed MBTA train fans out there!

I compare this additional flag suggestion to the Revenue / Non-Revenue flag on the 1.0 subway feed, or the "Note" field in the 2.0 subway feed. 

Please let the data be unfiltered, just tagged / flagged!!  Thanks!

-- Stefan Wuensch

[Developer at MBTA]

As a reminder, as of this Thursday both URLs will be available, and both will have the new data and the new JSON format. 

Three weeks later the "RTCRnew" URLs will become unavailable. 

Sincerely, 

Developer at MBTA

[Developer at MBTA]

This change is now in effect. Both URLs are available, and both have the new data and the new JSON format. 

Three weeks from now the "RTCRnew" URLs will become unavailable.

Sincerely,

Developer at MBTA

[StefanW]

I'm hoping to attend the event tomorrow (Tuesday May 7) but in case there's no time to discuss some Commuter Rail data specifics (or if it's not the appropriate venue), here's a few things I've noticed since your code push on April 25. 

• All "Lateness" values are now being rounded to zero or a multiple of 5 minutes. I would like to know why, and more importantly what algorithm you're using. I can understand that a train might not be considered late until / unless it's at least 5 min. late, but above that why not keep the detail in the data?

More important: if a train is **early** (yes it does happen) I hope that the rounding is being done down to the nearest 5 min. and NOT a conventional rounding operation. Why? If a train is 2 min. early and you round it in a conventional way, it would be then zero min. late, flagged as "On time", and someone could miss their train! Specific to the data format: if it's a Lateness value less than zero (meaning early) it should be rounded **away** from zero not towards zero.

• Since April 25 there's been a distinct mis-application of the Flag "arr" instead of "dep" - looks like a bug to me. I've been looking at historical data and calculating a rough distribution of the Flag values. Before April 25 there was a nice ratio of approach/arrive to depart flags. Since April 25 there's been virtually NO "dep" flags, and a huge number of "arr". This shows up in my app for example as a train that's showing a GPS position right at the Swampscott station still showing the status "arriving at Lynn". I can give you sample data and screen shots if you like.

• Also regarding the Flag data... Since April 25 there's been non-uniform use of Flag values for one train in the feed. (Probably related to above.) Here's a specific example right from the data feed, with epoch time converted to a human-readable time:

Thu May  2 09:34:16 2013,109,Rockport,Gloucester,Thu May  2 09:35:00 2013,app,1637,42.61789,-70.68197,88,11,

Thu May  2 09:34:16 2013,109,Rockport,Rockport,Thu May  2 09:44:00 2013,pre,1637,42.61789,-70.68197,88,11,0

Thu May  2 09:34:16 2013,109,Rockport,West Gloucester,Thu May  2 09:30:00 2013,arr,1637,42.61789,-70.68197,88,11,

Note that this one train is showing *three* different flags all at the same time. That's a bug! 

Hopefully I'll see you tomorrow!

-- Stefan

(this post edited to correct spelling)

[Developer at MBTA]

Transition reminder: This Thursday the temporary "RTCRnew" URLs will be shut down. 

Sincerely, 

Developer at MBTA