Errors in the latest GTFS data for Metro North and comments on the data explosion
Brett Bergquist
North. For example, in trips.txt there is the following trips:
5,2,2983912,South Norwalk - Danbury,1810,0,,21,1
5,2564,3028925,South Norwalk - Danbury,1810,0,,21,1
5,2568,3037649,South Norwalk - Danbury,1810,0,,21,1
Schedule 2 is tuesday, wednesday, thursday, and schedule 2564 and
2568, add in days 3/22/2011 and 3/31/2011 respectively. So from this
data, train 1810 only runs on tuesday, wednesday, and thursday, but
the paper schedule does not show this.
The same thing for other Danbury trips.
---
Also, I am very troubled by the way that exceptions are now
represented. For example, I see exceptions for schedule 1 remove a
series of dates:
1,20110405,2
1,20110406,2
1,20110412,2
...
and then see schedule 2578 add these dates right back in. Analyzing
the data, I can see that there are only 20 trips that are in schedule
2578 that are not in schedule 1:
trip_short_name,trip_id,schedule
7611,3060105,2578
7613,3060106,2578
7721,3060117,2578
7725,3060119,2578
7115,3060042,2578
7119,3060044,2578
7121,3060046,2578
7123,3060047,2578
7602,3060100,2578
7124,3060048,2578
7126,3060049,2578
7117,3060043,2578
7128,3060050,2578
7312,3060066,2578
7510,3060089,2578
7514,3060091,2578
7516,3060093,2578
7501,3060085,2578
7823,3060131,2578
7827,3060132,2578
This is determined by the query:
SELECT
A.trip_short_name, A.trip_id, S.service_id as schedule
from trips A join calendar S on A.service_id = S.service_id
where (S.service_id = 2578)
and A.trip_short_name not in
(
SELECT
B.trip_short_name
from trips B join calendar S on B.service_id = S.service_id
where S.service_id = 1
)
which finds those trips that exists in schedule 2578 and are not
present in schedule 1. So only those 20 trips are really exceptions
to the schedule 1 trips and need to be handled with exceptions. A
similar thing for the schedule 2579 for example.
What this does is explode the trips data. In the delivered file,
there are 5189 trips defined. If I clean up the data manually, I can
reduce this to 1147 (I also removed trips that occurred in the past
like those scheduled for 3/22, 3/31, 4/2 since these were previous to
when I would distribute the new data). Now for each of these extra
trips defined, you need the extra stop times. In the data as
delivered, there are 64997 stop times. Cleaned up, this has 14416.
As you can see a substantial difference. This has an effect on data
size, processing speed, etc. It also has a big effect on looking at
the data to match it to the paper schedules. The paper schedules sure
do not have 5189 trips.
Previous to the new process, the data would look like my cleaned up
data. The new process seems to be very inefficient in producing the
scheduling exceptions. So before I go writing an algorithm to do what
I did manually, I am wondering if the process to produce the data
could be improved to provide data in a more efficient manner: standard
schedules like mon-fri, sat/sun, fri, sun, etc. and then just the few
exceptions to these schedules.
I can write an algorithm to rebuild the data in this format and remove
about a 4/5'ths of the data for myself, but probably all could benefit
if the delivered data were made more efficient and straight forward.
Brett
John L
The exceptions listed are the baseball and/or football schedules. In our current scheduling system, there is no way to determine if a trip has changed a time or stop, so to be conservative, the whole thing is added. I would also point out that most, if not all, Hudson line expresses from Croton Harmon have Yankees-East 153rd Street added as a stop so there are more than just trips being added. This is the way we will be adding special and one day schedules going forward. This would include holidays and getaways as well.
In our next iteration of scheduling software, this should not be an issue.
You can choose to do what you want with the data set, but if you eliminate a trip of the same name that has added/removed stops or a time change, you run the risk of eliminating legitimate trips.
For sure there should be a weekday 1810 and we will investigate this. I am sure this is related to bussing. However, from the above, 1810 will still be added for special and one day schedules.
I will say this, we of all developers, know about the large duplication that occurs with special schedules and one day schedules. We took great care to not provide a Monday, Tuesday through Thursday, Friday, Saturday and Sunday base schedules. The same logic for removing duplicated trips cannot be simply added to the one off schedules and have to compare stops and time differences for reach train in the schedule. We know the what and how, it is the when that is the question. The information is not wrong, does not break trip logic and conforms to the GTFS standard.
If this poses a systemic problem to apps, well the risk gets elevated. So if it does cause problems delivering accurate information in a timely, please let me know. I can use that....
Brett Bergquist
Thanks for the response, John,
Before my algorithm removes anything, it does compare all stops and times because you are correct, if a trip changes in its stops or times then it truly is not the same trip. So those cannot be removed.
And yes, the trip information is not wrong and does conform. It does add a burden to low memory and storage constrained devices, so that is why I am cleaning up the data. When I come up with a programmatic solution that provides the same data in a more condensed format, taking into all of the considerations that you have listed, would it be useful to provide the application back to you for possible use? This will probably be done in Java but possibly in Perl or PHP.
Joe Hughes
wrote:
> burden to low memory and storage constrained devices, so that is why I am
> cleaning up the data. When I come up with a programmatic solution that
> provides the same data in a more condensed format, taking into all of the
> considerations that you have listed, would it be useful to provide the
> application back to you for possible use? This will probably be done in
> Java but possibly in Perl or PHP.
GTFS data would be interesting not only to mtadeveloperresources, but
also to the larger transit-developers community [1]—do share whatever
you come up with!
Cheers,
Joe
[1] http://groups.google.com/group/transit-developers
John L
Most definitely, we do have some developed code but any thing to push it along is helpful. Pst it to me if need be but need the justification from the post more so to push it so thanks for the posts!
Still looking at the Danbury stuff and may repost tomorrow latest Monday.
Brett Bergquist
Looking at the New Haven Weekend Yankee PDF, I see trip 7533 which is the first trip for a 1:05 weekend game. If I look in the data, I see:
MacbookPro:google_transit brett$ grep 7533 trips.txt
3,2577,3059567,New Haven - Yankees-E153 St.,7533,1,,6,1
MacbookPro:google_transit brett$
So this trip has the 2577 service Id. Now if I look at:
MacbookPro:google_transit brett$ grep 2577 calendar.txt
2577,0,0,0,0,0,0,0,20110403,20111007
and
MacbookPro:google_transit brett$ grep 2577 calendar_dates.txt
2577,20110403,1
2577,20110501,1
2577,20110522,1
2577,20110612,1
MacbookPro:google_transit brett$
So my understanding is that trip 7533 only runs on 4/3, 5/1, 5/22, and 6/12. But I look at the actual Yankee's schedule and see home games on 4/16 and 6/11 (Saturday's) that are at 1:05pm but do not seem to have references in the data. Is this correct? It does not seem to match the paper where there is no reference to exceptions for Saturday's for example.
Chris Schoenfeld
GTFS consumers to sit down and throw our heads together about
processes, distribution, changes, and QA.
I've been trying to solve this problem since 2008, but right now I
feel like a lot of us are all having similar issues and need to have a
working meeting to try to streamline the process.
There is no doubt in my mind that if we were to spend any amount of
time doing this, it would save John's team and GTFS consumers like
myself a lot of anxiety, frustration, and confusion moving forward.
Chris
John L
Different schedules for weekday day and evenings, weekend day and evenings.
John L
I am tooling with this on the weekends now seeing the interest in it ;)
Brett Bergquist
John L
Brett Bergquist
Wayne
I think Chris has a good idea and it could be very beneficial to both sides if we have a well defined agenda, and clearly defined expectations, before meeting; what's in-scope, out-of-scope, take-aways, etc.
Part of the struggle, IMHO, sometimes stems from not having a good understanding as to "why" data is tagged the way it is; and I realize it may depend heavily on legacy processes and current limitations - not looking to point fingers, just a better understanding. The point is, if as developers we have a better understanding and/or comfort that the data format will remain consistent AND we better understand the decisions that determine the data set, then we should be able to reduce the time it takes to get new sets implemented.
We definitely want to contribute in streamlining the process.
Other thoughts?