Extraction of required trip information from OTP using Jython/Python

[Etikaf Hussain]

Hi,

I am working on a research project and using trip information given by OTP in the analysis. The number of expected OD pairs are approx. 40million. I have written a python script which passes an Http api request to OTP. In the result, I get a dictionary of plan that includes a whole lot of information on each leg of the trip (in case of PT). I extract the required information and store it in a separate file.

The problem is, that I have to run it for a large number of trips between OD pairs and I know that this is not an efficient way to work with a large dataset. Different online articles suggest that we should go for python/Jython. 

I have gone through https://github.com/rafapereirabr/otp-travel-time-matrix example. It works fine, but even after almost searching for two months, I am unable to get the required datasets using Jython scripting. Any help in this regard will be highly appreciated. For reference, I have attached the snap of the required result that I get from Http request.

Thanks,

Etikaf 

[Leonard Ehrenfried]

If I understand your problem correctly then querying 40 million trips is too slow, correct?

Routing algorithms are by their nature quite CPU and memory intensive. From my personal experience, I've been able to get roughly 10 requests per second on moderate hardware which means that 40 million queries take more than a month. However, by throwing hardware at the problem I've been able to scale up a cluster to more than 150 requests per second, which would mean a significant speed up. Is using more server capacity an option?

Otherwise, you could give the currently unreleased OTP 2 a go, which has a new routing algorithm which is supposed to be faster. However, since it's still in pre-release state, there isn't a lot of information around.

Best

Leonard

--
You received this message because you are subscribed to the Google Groups "OpenTripPlanner Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to opentripplanner-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/opentripplanner-users/7826f6cb-ab11-4e3b-b1ab-cbd5cfc1b358%40googlegroups.com.

[Etikaf Hussain]

Thanks, Leonard for your reply.

I think you have perceived my problem correctly. I agree it is really CPU intensive. As per my experience, on a normal PC with 8 cores, with threading, it will take more than 3 months. I would love to implement the way you have scale up a cluster to get 150 requests per second. Can you please share your experience?

On the use of more server capacity, I have run the code on high-performance centre (HPC-cluster) and included threading into it. It takes around 7-10 days to complete the task with 32 cores availability. However, I am planning to make the process executable on a normal PC with reasonable process time.

Because of the same reason you stated, I am hesitating to shift to OTP 2. Since I am a civil engineer and do not have much background in working with programming.

Thanks again,

Etikaf

To unsubscribe from this group and stop receiving emails from it, send an email to opentripplanner-users+unsub...@googlegroups.com.

[Leonard Ehrenfried]

What we did was to run several OTP instances behind a load balancer and this way we could scale the number of requests to a very high number. My memory is a little hazy but I think to get to 150 requests per we used 6 instances with 8 cores each. If you dont't care about costs I can see this being indefinitely scalable.

I'm not sure you can go to the hundreds of requests that you require on an ordinary PC.

However, you could try the following to make the computation less CPU intensive:

Firstly, cut out the walking parts by routing from station to station.

If you need the walking part then remove smaller roads from the OSM file and leave only main ones. This way there are less possible ways and the computation should be quicker. For this you can try a tool like osmium.

Best

Leonard

To unsubscribe from this group and stop receiving emails from it, send an email to opentripplanner-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/opentripplanner-users/7826f6cb-ab11-4e3b-b1ab-cbd5cfc1b358%40googlegroups.com.

--

You received this message because you are subscribed to the Google Groups "OpenTripPlanner Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to opentripplanner-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/opentripplanner-users/41c182a9-bd50-49bd-a342-965826576579%40googlegroups.com.

[Etikaf Hussain]

Great Leonard. 

I don't think I am familiar with the instances. I will try now to explore that.

I have already taken out the walking part, and the route taken is from stop to stop. However, it is possible walking may exist in the final result. Since the analysis is from many to many, there is high chances that no direct transit services run between two stops/stations or the optimum path may walk from one stop (origin stop) to another, take transit service from there and reach the destination stop and vice versa.

I will give it a try, lets see how it goes.

Thanks

Etikaf

To unsubscribe from this group and stop receiving emails from it, send an email to opentripplanner-users+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/opentripplanner-users/7826f6cb-ab11-4e3b-b1ab-cbd5cfc1b358%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "OpenTripPlanner Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to opentripplanner-users+unsub...@googlegroups.com.

[Thomas Gran]

 You do not have RT updates, I would look at OTP2, give it a day. The API very similar to OTP1 and the OTP2-Migration-Guide.md should explain the differences. Also, if you only want the "fastest" journey you should be able to modify the OTP2 to only run just Raptor or RangeRaptor (not Multi-Criteria-RangeRaptor with the generalized-cost). Another tip is to look at the SpeedTest (A Java program in the code) skipping the Web container overhead and network latency - it is used to performance test the different variations of the algorithm. You could modify that to run your test and I would guess you can get the query time down to 50 ms, maybe lower (depending on the size of your graph). The Speed test run queries in seriell so you would have to modify it to run requests in parallell to boost the performance. 

For the Norwegian graph (|V|=2,151,156 |E|=5,102,955) it takes 60-70 ms per query for a plain Raptor search.


Thomas

[Etikaf Hussain]

Thanks, Thomas for your comment. I have successfully migrated my code to OTP2 (thanks for providing link to migration guide). Can you please further elaborate on how I can select to run only Raptor or RangeRaptor as mentioned? Is there any command for it or any documentation that is helpful to understand how to implement it?

Thanks in advance,

Etikaf

[Thomas Gran]

In the org.opentripplanner.routing.algorithm.raptor.transit.mappers.RaptorRequestMapper the profile is hardcoded to run RaptorProfile.MULTI_CRITERIA, you could change that to STANDARD. Thomas

[Prateek Agarwal]

Hi Thomas. I was looking at OTP2. You said OTP2 has Raptor and Range-Raptor. Does it also have McRaptor? If yes can you please point me to its location in the repository. I want to look at it source code. Also Multi-Criteria-RangeRaptor is combination of McRapot +Rraptor right? 

[t2g...@gmail.com]

You will find the Raptor implementation here: https://github.com/opentripplanner/OpenTripPlanner/tree/dev-2.x/src/main/java/org/opentripplanner/transit/raptor

So, the OTP implementation support McRangeRaptor, but only one cost-based-criteria in addition to arrival-time and transfers. It also support reverse-search, witch is used to build heuristics to optimize the McRaptor forward search. The Raptor implementation is quite generic, and allows you to chose between StdRaptor and McRaptor, run a single iteration or multiple iterations (Range Raptor minutes). There is also 3 different state implementations, one for McRaptor, and 2 implementations for the StdRaptor: BestTimes and Standard. The BestTimes don't keep path information, only the best arrival-times. There are also a few optimizations - the most important is the heuristic destination pruning and the ability to ignore wait time. When OTP do ONE planning request (arriveBy=false), then OTP runs 3 Raptor searches: The first is a OneIterationRaptor search to calculate a suitable search-window(this can be omitted if the search-window is passed in), then the second search is a Reverse Raptor search to compute heuristics (minNumberOfTransfers, minTravelTime, minGeneralizedCost for each stop). The last, the third raptor search is a McRangeRaptor search with heuristic destination pruning. I recommend playing with the SpeedTest (https://github.com/opentripplanner/OpenTripPlanner/blob/dev-2.x/src/test/java/org/opentripplanner/transit/raptor/speed_test/package-info.java) to try out different Raptor configurations/optimizations. The way OTP is using Raptor is limited, the different Raptor configuration options are not exposed by the OTP APIs.