Beginner Having Problems with Batch Scripting in 0.19.0-SNAPSHOT-shaded

[Ross K]

Hello, I am completely new to Open Trip Planner, Java, and Jython and pretty new to Python.  

I have had success using Open Trip Planner in the browser and would like to move on to doing analytical work in batch mode.

I am starting from scratch trying to follow the batch accessibility analysis example here (specifically, the grocery store script here) working with Windows 7 and Java Development Kit 1.8.0_60.  

I understand the example analysis was done with an earlier version (the version mentioned in the comments is 0.12.1/master), but I am hoping I might eventually be able to get that kind of analysis running in my environment with a newer version.

Trying to follow the "Getting OTP" instructions here, I downloaded Git and Maven (apache-maven-3.3.3) and can successfully clone the master from Github and build OTP JAR files.

I tried using Eclipse, but I think that is beyond me for now.

So, for example, in C:\OpenTripPlanner\target I now have:

otp-0.19.0-SNAPSHOT.jar

otp-0.19.0-SNAPSHOT-shaded.jar

otp-0.19.0-SNAPSHOT-sources.jar

I also downloaded the standalone version of Jython 2.7.0 based on the recommendations in the "Launching a script from OTP" section of the Scripting documentation and copied that into C:\OpenTripPlanner as well:

jython-standalone-2.7.0

(Also tried with jython-standalone-2.5.4-rc1.jar, but that didn't seem to matter)

In addition, I put the accessibility script ("accessibility.py") in the same folder (C:\OpenTripPlanner) and have a default Graph.obj there of Chicago.

I have versions of the Chicago Census Blocks shapefile and the grocery stores' XY coordinates, but haven't reached the point of using those files yet due to other issues. 

For reference, Environment Variables and Values:

JAVA_HOME: C:\Program Files\Java\jdk1.8.0_60

JYTHON_HOME: C:\jython2.7.0*

*Should JYTHON_HOME be set to the path to the folder location of jython-standalone-2.7.0.jar instead or does that defeat the purpose of the standalone version?

When trying to run the script with a Java command (I have not had luck with the suggested Jython command like "jython -Dpython.path=otp-x.y.z-shaded.jar myscript.py") that includes the OTP and Jython JAR files in the classpath, I get an error saying, among other things, that 'OtpsEntryPoint' is not defined:

C:\OpenTripPlanner\target>java -cp otp-0.19.0-SNAPSHOT-shaded.jar;jython-standalone-2.5.4-rc1.jar org.opentripplanner.standalone.OTPMain --script accessibility.py

16:26:22.185 INFO (OTPServer.java:38) Wiring up and configuring server.

16:26:22.202 INFO (BSFOTPScript.java:60) Running script C:\OpenTripPlanner\target\accessibility.py

16:26:22.206 DEBUG (BSFManager.java:399) BSFManager:declareBean

16:26:22.206 DEBUG (BSFManager.java:752) BSFManager:registerBean

16:26:22.207 DEBUG (BSFManager.java:483) BSFManager:exec

16:26:22.207 DEBUG (BSFManager.java:674) BSFManager:loadScriptingEngine

16:26:22.208 DEBUG (BSFManager.java:556) BSFManager:getClassPath

16:26:22.208 DEBUG (BSFManager.java:548) BSFManager:getClassLoader

16:26:23.219 ERROR (BSFManager.java:499) Exception :

java.security.PrivilegedActionException: null

        at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0

_60]

        at org.apache.bsf.BSFManager.exec(BSFManager.java:491) ~[otp-0.19.0-SNAP

SHOT-shaded.jar:1.1]

        at org.opentripplanner.scripting.impl.BSFOTPScript.run(BSFOTPScript.java

:66) [otp-0.19.0-SNAPSHOT-shaded.jar:1.1]

        at org.opentripplanner.standalone.OTPMain.run(OTPMain.java:153) [otp-0.1

9.0-SNAPSHOT-shaded.jar:1.1]

        at org.opentripplanner.standalone.OTPMain.main(OTPMain.java:88) [otp-0.1

9.0-SNAPSHOT-shaded.jar:1.1]

Caused by: org.apache.bsf.BSFException: exception from Jython:

Traceback (most recent call last):

  File "<string>", line 19, in <module>

NameError: name 'OtpsEntryPoint' is not defined

        at org.apache.bsf.engines.jython.JythonEngine.exec(JythonEngine.java:146

) ~[otp-0.19.0-SNAPSHOT-shaded.jar:1.1]

        at org.apache.bsf.BSFManager$6.run(BSFManager.java:493) ~[otp-0.19.0-SNA

PSHOT-shaded.jar:1.1]

        ... 5 common frames omitted

Exception in thread "main" java.lang.RuntimeException: org.apache.bsf.BSFExcepti

on: exception from Jython:

Traceback (most recent call last):

  File "<string>", line 19, in <module>

NameError: name 'OtpsEntryPoint' is not defined

        at org.opentripplanner.standalone.OTPMain.run(OTPMain.java:159)

        at org.opentripplanner.standalone.OTPMain.main(OTPMain.java:88)

Caused by: org.apache.bsf.BSFException: exception from Jython:

Traceback (most recent call last):

  File "<string>", line 19, in <module>

NameError: name 'OtpsEntryPoint' is not defined

        at org.apache.bsf.engines.jython.JythonEngine.exec(JythonEngine.java:146

)

        at org.apache.bsf.BSFManager$6.run(BSFManager.java:493)

        at java.security.AccessController.doPrivileged(Native Method)

        at org.apache.bsf.BSFManager.exec(BSFManager.java:491)

        at org.opentripplanner.scripting.impl.BSFOTPScript.run(BSFOTPScript.java

:66)

        at org.opentripplanner.standalone.OTPMain.run(OTPMain.java:153)

        ... 1 more

Out of curiosity, I wanted to see how far I could get submitting lines one by one in interactive Jython.

In interactive Jython, I am able to successfully import packages and classes (the lines below all work)... 

import sys

sys.path.append('C:\\OpenTripPlanner\\target\\otp-0.19.0-SNAPSHOT-shaded.jar')

sys.path.append('C:\\OpenTripPlanner\\target\\jython-standalone-2.7.0.jar')

from org.opentripplanner.scripting.api import *

otp = OtpsEntryPoint.fromArgs([ "--graphs", "." ])

...but I am unable to assign values to functions.

For example, "r.setModes('WALK')" gives this error:

"TypeError: setModes(): 1st arg can't be coerced to org.opentripplanner.routing.core.TraverseModeSet"

Same result when trying "r.setModes('WALK,TRANSIT')" which I have seen elsewhere.

Some questions:

• Do I need to somehow incorporate the opentripplanner-jython bindings into this process before running the accessibility script?
• What have I set up incorrectly that is making the Java command unable to use OtpsEntryPoint, but allows interactive Jython to use OtpsEntryPoint with no problem?
• Could anyone point me to a guide to batch scripting for beginners in a recent version of Open Trip Planner in a Windows environment?

Any advice on what I am missing would be greatly appreciated!  

I'm hoping other beginners might learn from my mistakes and your knowledge.

[Laurent GRÉGOIRE]

Hi Ross,

On 7 October 2015 at 23:28, Ross K <geomap...@gmail.com> wrote:
> Do I need to somehow incorporate the opentripplanner-jython bindings into
> this process before running the accessibility script?
> What have I set up incorrectly that is making the Java command unable to use
> OtpsEntryPoint, but allows interactive Jython to use OtpsEntryPoint with no
> problem?
> Could anyone point me to a guide to batch scripting for beginners in a
> recent version of Open Trip Planner in a Windows environment?

There is two distinct and mostly incompatible scripting mechanisms,
that explain most of your problems:

1) The "Chicago grocery store" uses the (probably deprecated?) batch
processor, is based on a rather old version of OTP, and rely on
accessing OTP internal classes that are not meant to be stable. So
using this script with the latest version of OTP w/o digging into the
inner details of the code to check how it can work is probably doomed
to fail. If you do not want to delve into OTP yourself, I won't
recommend you this.

2) The "Scripting Jython API" is based on a dedicated Java API layer
in OTP that is meant to be more stable and do not rely on the
BatchProcessor inner class. It has probably more chances to run; but
the API is not the same as the other, so any Jython code from 1) won't
work. If you want to use this, better use the provided demo scripts
[1] and base your work upon this. However OTP move fast those days,
and the Jython API sometimes breaks; but this is meant to be fixed.

Having said that, 2) is probably slower and more limited than 1)
because BatchProcessor is multi-threaded and all the processing is
done in Java, which is faster than Jython were each Jython call is
somehow expensive. To alleviate that, you can run multiple scripts on
the same server, and most heavy-processing calls are made entirely in
Java, to minimize expensive operations (for example the point set
evaluate function perform the evaluation on a collection of
destinations at the same time).

[1] https://github.com/opentripplanner/OpenTripPlanner/blob/master/src/test/resources/scripts/test.py

HTH,

--Laurent

[Ross K]

Thank you for explaining this, Laurent!  With a Graph.obj built from an OSM PBF file and GTFS data, that Graph.obj saved at C:\OpenTripPlanner\target\graphs\[myrouter], and two CSV files with XY coordinates as well as the test.py script saved in C:\OpenTripPlanner\target, I am now able to run scripts with your 2nd approach ("Scripting Jython API") after changing the directory to C:\OpenTripPlanner\target and running commands like "jython -J-Xmx8192m -Dpython.path=otp-0.19.0-SNAPSHOT-shaded.jar test.py" on a Windows 7 machine with 16GB of RAM.

Unfortunately, I am now having trouble getting a script to do what I need.

If possible, would you please direct me to a guide to analyzing the transit network travel times between individual pairs (no summarization needed) of source and destination XY points?

I have simple datasets to get started, each with 2 unique entries, so I am just trying to find 4 travel times in all.

I think I am misunderstanding how to do this kind of analysis (below is an attempt).  If I run "result = spt.eval(s)", I am getting identical (and impossibly fast - 22s and 60s, respectively) results for the source, regardless of the destination.  If I run "result = spt.eval(d)", I get a result of "None."

In particular, I seem to not understand how spt.eval() is working.  As a beginner, I guess I thought that that function would be called on the source-destination pair, rather than on a single point.    

Any guidance would be greatly appreciated!

Setup above...

srcs = otp.loadCSVPopulation('sources.csv', 'Y', 'X')

dests = otp.loadCSVPopulation('dests.csv', 'Y', 'X') 

for s in srcs:

req.setOrigin(s) 

for d in dests:

print "Processing: ",s," and ",d

req.setDestination(d)

spt = router.plan(req) 

print "SPT: ", spt

if spt is None: continue

result = spt.eval(s)

print "Result: ", result

[Laurent GRÉGOIRE]

Hi Ross,

On 8 October 2015 at 19:55, Ross K <geomap...@gmail.com> wrote:
> In particular, I seem to not understand how spt.eval() is working. As a
> beginner, I guess I thought that that function would be called on the
> source-destination pair, rather than on a single point.

OtpsSPT::eval() can be used to evaluate a single point (aka
"individual" in OTP parlance) OR a collection of points at once (that
is one of the optimization I was talking about in my previous mail
about reducing the number of expensive calls Jython/Java). When called
with a collection, it returns a collection of "evaluated"
points/individuals, where each evaluated contains the
time/boarding/walk distance for the SPT, and the original point,
including the exact resolved location (where street snapping occurs
and the destination is not exactly on a street). You can see the Java
OtpsSPT.java class to see what it accepts. See also
OtpsEvaluatedIndividual.java to see what you get in return (and more
generally, all classes in the "org.opentripplanner.scripting.api"
package).

In your case, evaluating the SPT at the origin will indeed return a
trivial small time (a few seconds), but this is not relevant and is
expected.

Below your original code modified, but *not tested* (there could be
some bugs or syntax issues):
--------------------------------------------------------
req = ...

srcs = otp.loadCSVPopulation('sources.csv', 'Y', 'X')
dests = otp.loadCSVPopulation('dests.csv', 'Y', 'X')

for s in srcs:
req.setOrigin(s)

spt = router.plan(req)

if spt is None: continue

result = spt.eval(dests)
for r in result:
print r.getTime(), r.getIndividual().getLocation(),
r.getIndividual().getSnappedLocation()
--------------------------------------------------------

HTH,

--Laurent

[Ross K]

Great, thank you so much, Laurent!

This script (saved as 'C:\OpenTripPlanner\target\test.py') is working for me, based on your suggestions above, to calculate the travel time between respective origin and destination point pairs and print the result.

from org.opentripplanner.scripting.api import *

otp = OtpsEntryPoint.fromArgs([ "--graphs", "C:\\OpenTripPlanner\\target\\graphs", "--router", "myrouter"])

router = otp.getRouter('myrouter')

req = otp.createRequest()

req.setDateTime(2015, 9, 15, 10, 00, 00)

req.setModes('TRANSIT,WALK') 

#The file 'C:\OpenTripPlanner\target\sources.csv' has 3 columns (X, Y, and NAME) for the XY coordinates and an identifier (for example, 'Work' or 'Home')

srcs = otp.loadCSVPopulation('sources.csv', 'Y', 'X')

#The file 'C:\OpenTripPlanner\target\dests.csv' has the same 3 columns above

dests = otp.loadCSVPopulation('dests.csv', 'Y', 'X')

for s in srcs:

req.setOrigin(s)

spt = router.plan(req)

if spt is None: continue

result = spt.eval(dests)

for r in result:

print s.getStringData('NAME'), r.getIndividual().getStringData('NAME'), r.getTime(), r.getWalkDistance(), r.getIndividual().getLocation(),r.getIndividual().getSnappedLocation()

[Rafael Pereira]

Hi Lauren,

Would it be possible to share in the repository those files* used to run the test.py script in order to make it a reproducible example ? 

* colleges.csv, insee.csv , GTFS

cheers,

Rafael

[Laurent GRÉGOIRE]

Hi Rafael,

It's a bit awkward and unnatural to put a large GTFS in the repository, and anyway I would not have the rights to use the one I used specifically for those tests.

What can be done however is that I put an example CSV file with 2 or 3 dummy rows whose position (lat, lon) can be easily modified to match the router area used for the test. That way the demo can be used with any router, the user would just have to use or download any GTFS.

HTH,

--Laurent

--
You received this message because you are subscribed to the Google Groups "OpenTripPlanner Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to opentripplanner-...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ross K]

Hello Laurent and Rafael, I wondered if one of you could help me understand how to perform a series of one-to-one origin-destination travel time calculations using the Jython approach.

In other words, if I wanted to know the travel time between the 1st origin in a CSV file and the 1st destination in a destination CSV file, then the travel time between the 2nd origin and 2nd destination, etc., rather than the travel time from the 1st origin to all destinations.

As a beginner, I thought I would be able to use an index, but I am getting the error "TypeError: 'org.opentripplanner.scripting.api.OtpsPopulation' object is unsubscriptable".

Given source, destination, and output files, how would you change the code below to run this kind of one-to-one analysis?  Or is already example code you would be kind enough to direct me to?

srcs = otp.loadCSVPopulation('sources.csv', 'Y', 'X')

dests = otp.loadCSVPopulation('dests.csv', 'Y', 'X')

matrixCsv = otp.createCSVOutput()

matrixCsv.setHeader(['Origin', 'Destination', 'Walk_distance', 'Travel_time'])

for s in srcs:

req.setOrigin(s)

spt = router.plan(req)

if spt is None: continue

result = spt.eval(dests)

for r in result:

matrixCsv.addRow([ s.getStringData('name'), r.getIndividual().getStringData('name'), r.getWalkDistance() , r.getTime()])

matrixCsv.save('traveltime_matrix.csv')

Thank you,

Ross

To unsubscribe from this group and stop receiving emails from it, send an email to opentripplanner-users+unsub...@googlegroups.com.

[Rafael Pereira]

Hi Ross,

I think I would go around this problem by creating a loop which reads  sources.csv' and dests.csv' one row at a time > perform the query for that pair of rows > save to the output > go back to the loop in the next row etc.

I'm not sure how you would do this, since I'm not much familiar with Python myself but I think this shouldn't be complicated and you might find the answer here: 

http://stackoverflow.com/questions/30964244/read-specific-line-in-csv-file-python

http://stackoverflow.com/questions/12774622/python-stuck-at-reading-a-specific-row-from-a-csv

https://gist.github.com/nsonnad/7598828

http://stackoverflow.com/questions/13428318/reading-rows-from-a-csv-file-in-python

best wishes,

Rafael Pereira

To unsubscribe from this group and stop receiving emails from it, send an email to opentripplanner-...@googlegroups.com.

[Laurent GRÉGOIRE]

Hi Ross, Rafael,

In python you can use the builtin "zip" function to create a list of
pair from two lists:

a = [1,2,3]
b = [7,8,9]
print zip(a,b) # => [(1, 7), (2, 8), (3, 9)]

So your example becomes something like (not tested), assuming the same
number of sources and destinations:

srcs = otp.loadCSVPopulation('sources.csv', 'Y', 'X')
dests = otp.loadCSVPopulation('dests.csv', 'Y', 'X')

for s,d in zip(srcs,dests):

req.setOrigin(s)
spt = router.plan(req)
if spt is None: continue

r = spt.eval(d)

matrixCsv.addRow([ s.getStringData('name'),
r.getIndividual().getStringData('name'), r.getWalkDistance() ,
r.getTime()])

However since scripting will do a undirected path search this solution
is not optimal. It could be speed up by adding to the scripting API a
directed path search function. If you know a bit of Java that should
not be that complex, you can have a look at the scripting.api package
in OTP.

HTH,

--Laurent