Trouble building bundle with WMATA GTFS

[Kurt Raschke]

I'm trying to build a bundle with the latest WMATA GTFS (from
http://gtfs.s3.amazonaws.com/wmata-archiver_20120123_0405.zip)

The builder seems to go into a loop and eventually exhaust the heap no
matter how much I give it; the most I tried was 25 GB of heap on an
Amazon EC2 extra-large instance.

Here is the traceback:

2012-01-29 20:25:44,044 INFO [TripEntriesFactory.java:97] : trips to
process: 18
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.onebusaway.quickstart.bootstrap.BootstrapMain$1.run(BootstrapMain.java:199)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.onebusaway.quickstart.bootstrap.BuildBootstrapMain.main(BuildBootstrapMain.java:32)
... 6 more
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2746)
at java.util.ArrayList.toArray(ArrayList.java:296)
at java.util.ArrayList.<init>(ArrayList.java:151)
at org.onebusaway.transit_data_federation.bundle.tasks.transit_graph.DistanceAlongShapeLibrary.recursivelyConstructAssignments(DistanceAlongShapeLibrary.java:325)
(29 more stack trace elements identical to the above line snipped...)

The whole log is at https://gist.github.com/1700619.

I know the builder is expected to use a considerable amount of RAM,
but the stack trace makes me wonder if there isn't some runaway
recursion at work.

Is there any way to increase the verbosity of the builder so I might
be able to see what it is in the feed that is causing this behavior?
Any other ideas?

Thanks,

Kurt Raschke

[Brian Ferris]

There definitely seems to be an issue with OBA's handling of the feed.
Specifically, OneBusAway performs a step where it attempts to match
each stop along a trip to the underlying shape data. There are a few
shapes in the WMATA GTFS that seem to be a bit malformed, which is
causing the shape-matching code to choke. I've checked in a quick fix
that attempts to be more flexible in handling bad shapes. For now, it
will spit out a bunch of errors for the WMATA feed and then fail. If
you want to use the bundle anyway, ignoring the bad shape data,
specify the following command-line option to the bundle builder:

-P tripEntriesFactory.throwExceptionOnInvalidStopToShapeMappingException=false

Let me know if that helps. Again, my apologies about not responding
sooner. This got caught up in mailing list moderation.

Brian

> --
> You received this message because you are subscribed to the Google Groups "onebusaway-developers" group.
> To post to this group, send email to onebusaway...@googlegroups.com.
> To unsubscribe from this group, send email to onebusaway-devel...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/onebusaway-developers?hl=en.
>

[Kurt Raschke]

Trouble with WMATA's shapes would not surprise me at all; apparently
it is known that some of them are just wrong:

https://github.com/MobilityLab/TransitNearMe/issues/17

I have not yet tried a new build of OBA, but once I get my dev
environment working I'll do that and report back.

If there were a reliable source of corrected shapefiles, then I'd
consider patching the GTFS transformer to support substituting
defective shapes, as the TransitNearMe tool does, but honestly I would
much prefer that WMATA fix their feed.

-Kurt

[Kurt Raschke]

So, I've finally gotten around to trying the latest version of the
quickstart to build a bundle for the WMATA GTFS. As expected, it
fails, but at least this time it fails quickly, with copious
diagnostics. Since there were many, many failures, I picked one at
random to look at. Following the stop-to-shape-matching debugging
instructions on the wiki, I pasted the stops and shape into the map
tool, and I didn't see anything obviously wrong (that is, all of the
stops are more or less on the shape).

https://github.com/OneBusAway/onebusaway-application-modules/wiki/Stop-to-Shape-Matching
http://developer.onebusaway.org/maps/

That said, the shape does look a bit complex...there are several
one-way loops, and part of the shape seems to have no stops on it.
So, I wonder if this would be helped by the recent fix in a83b395260,
or if this is a separate issue (or if WMATA's data is lacking).

-Kurt

On Tue, Feb 7, 2012 at 4:36 PM, Brian Ferris <bdfe...@onebusaway.org> wrote:

[Brian Ferris]

I think the WMATA shapes are still just weird.  Check out this example from shape 813:



There's just too many weird artifacts, random jumps, and loopbacks to figure out what the bus is actually doing there.

I think your best is to pass the:

-P tripEntriesFactory.throwExceptionOnInvalidStopToShapeMappingException=false

flag to the bundle builder to ignore bad shape mappings.

Brian

[Kurt Raschke]

Brian,

I did eventually disable the exception to build a bundle despite the bad shapes.

Having done so, what will the impact on OBA be? I built bundles for WMATA and Montgomery County Ride On this way, and they seem to work fine (including with GTFS-realtime for Ride On), but I'd like to have a better understanding of what I'm doing.

Thanks again,

Kurt

Sent from my iPad

On Apr 30, 2012, at 8:03, "Brian Ferris" <bdfe...@onebusaway.org> wrote:

> I think the WMATA shapes are still just weird. Check out this example from shape 813:
>

> <image.png>

[Brian Ferris]

I think the impact should be negligible in practice.  Shapes will look weird when displayed to users, but beyond that.

Brian