GTFS-Realtime Data Archive

[Ken Yoneda]

Hi All,

I decided to create a simple API that allows users to download GTFS realtime data from the past. The motivation to create this archive was because of unreliable data availability (see here).

For now, I just wanted to see if people find this useful. If there is a positive response, I will simplify the API endpoints (it's way too verbose at the moment), come up with a good name for the API (suggestions welcome), and change the throttling rate to a more reasonable limit. Please let me know what you guys think - I'm open to constructive criticism.

####################################################################################################

Description: Archive of MTA's GTFS-Realtime feeds. Packaged into archives every hour (one per feed).

Every archive contains 120 files as the feed is generated every 30 seconds.

For list of feeds, see:

http://datamine.mta.info/list-of-feeds

Archives are named in the following format:

{feed}-{year}-{month}-{day}-{hour}.tar.bz2

e.g. feed26-2018-04-28-00.tar.bz2

To access via API, use following URL structure:

https://2m9ldwhcmh.execute-api.us-east-2.amazonaws.com/gtfs_rt/historic.mta/{feed}/{year}/{month}/{day}/{archive}

e.g. https://2m9ldwhcmh.execute-api.us-east-2.amazonaws.com/gtfs_rt/historic.mta/feed26/2018/04/28/feed26-2018-04-28-00.tar.bz2

Archive Start: 16:00:00 EST, 27 April 2018

Throttling: Limited to 10 requests per second

####################################################################################################

Note: If this is against MTA's policy, I will gladly take the service down immediately.

[sm12...@gmail.com]

I got a 403 Forbidden reply, when I tried to access the archive. I used your example to test this out.

Is the archive still active or did you just for get to make it public?

Thanks.

[Ken Yoneda]

I took it down because nobody was using it. I can always put it back up though.

[Mary Buchanan]

Hey Ken,

I'd be interested in taking a look at your archive, too. Sounds like it could be really useful for something I'm working on. Can you put it back up?

[Ken Yoneda]

Hey Mary,

Sure! I'll try and get it back up soon, quite busy with work this week. Feel free to email and remind me if it's not up by next week.

[sm12...@gmail.com]

Hi Ken

Just a gentle reminder that several people are awaiting the archive revival with baited breath.

Thanks.

Steve

[Ken Yoneda]

Hi all, just put it back up. Let me know if there are any issues.

[sm12...@gmail.com]

Thanks.

I just started downloading. 

You may have a problem with the file naming because you are using local time. There's a local time change coming in a couple of weeks (EDT-->EST). It's fall behind. Therefore, you will have two sets of files with the same name between 2 and 3am during the official change. You could avoid this problem by using UTC time for naming the files.

Just a thought.

Steve

[Ken Yoneda]

Thanks for the suggestion Steve. Didn't think of that, I'll fix it soon!

--
You received this message because you are subscribed to a topic in the Google Groups "mtadeveloperresources" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mtadeveloperresources/aQqS_f9oYSY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mtadeveloperreso...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mikhail Venkov]

Sorry this is a bit off topic but how do I open these GTFS files? It doesn't look like it's a text file. When I open a file in text editor it's unreadable. Am I missing something? Is there a library for reading these files. Thanks!

To unsubscribe from this group and all its topics, send an email to mtadeveloperresources+unsub...@googlegroups.com.

[Alex S]

FWIW, these archives are an amazing resource.  Thanks very much for publishing them.

To unsubscribe from this group and all its topics, send an email to mtadeveloperreso...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mtadeveloperresources" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mtadeveloperreso...@googlegroups.com.

[Ken Yoneda]

It's serialized using protocol buffers, similar to XML but smaller and much faster. You can read about it here:

https://developers.google.com/transit/gtfs-realtime/#data_format

Feeel free to send me an email if you have any more question though.

[Ken Yoneda]

Update: From 2AM EST 11/4/2018, all file names are in UTC time. I decided to change it because of DST. Thanks Steve for the catch and suggestion.

I may decide to rename previous archives at a later point in time so that all file names are consistent.

[Dan O'Neil]

I know this is an old thread, but I attempted to pull some of the archived GTFS-RT data (22-Feb-2019) and the TRIP_ID values are not mapping to the corresponding GTFS static values. The GTFS-RT trip ID values are ~10 characters, for example "143950_H..N". Has anyone else run into the same issue and found a secret to translating these values?

[ZHENGWEN FAN]

Hi, Ken. The archived GTFS real-time is really valuable for me. But the API seems to be not valid now. Any chance to put it back?

[Ken Yoneda]

Hi Zhengwen,

I'm sorry but I'm no longer maintaining this data. Maybe I can help if you have specific dates in the past you need data for - let me know.

--

You received this message because you are subscribed to a topic in the Google Groups "mtadeveloperresources" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mtadeveloperresources/aQqS_f9oYSY/unsubscribe.

To unsubscribe from this group and all its topics, send an email to mtadeveloperreso...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/mtadeveloperresources/c2348def-569e-4598-93f1-270934cf8b29n%40googlegroups.com.