Any work done on mulithreaded GTFS imports?

[Stefan de Konink]

Hi,

We've got a fairly big GTFS feeds on our hands. About 700MB. The single threaded processing speed is just slow if you happen to have a Core i7. Has there any work been done on multithreading the gtfsdb_to_scheduled_edges generator for example?

Stefan

[Brandon Martin-Anderson]

I don't think so, no. What transit system in the world is 700 MB? Are
you sure it doesn't have a lot of duplicate records?

-B

> --
> You received this message because you are subscribed to the Google Groups
> "Graphserver" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/graphserver/-/uPMGT2FbkrQJ.
> To post to this group, send email to graph...@googlegroups.com.
> To unsubscribe from this group, send email to
> graphserver...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/graphserver?hl=en.

[Stefan de Konink]

On 31-05-12 00:58, Brandon Martin-Anderson wrote:
> I don't think so, no. What transit system in the world is 700 MB? Are
> you sure it doesn't have a lot of duplicate records?

We call it our Dutch legacy transitplanner 9292. Which materialized all
their service dates, then again they did collect all transit agencies in
The Netherlands.

Anyway, in order to solve this thing I tried different approaches, but
failed due to the objects not being able to be picked (references to
SQLite connection objects don't help). In essence I want
gtfsdb_to_scheduled_edges to be in a paralel form.

...so after a very long wait I did get my graph. But I hope someone
could give a hand on how to get that function to perform better.


Stefan