GTFS feeds for The Netherlands

[Thomas Koch]

Hello all,

We would like to announce that we are supplying a GTFS feed with transit data for virtually all bus/metro/train/tram and some ferry services in the Netherlands, it also contains most international connections towards Belgium, France, Germany and the UK. The feed is licensed under CC0 license without tedious constraints about what you can and cannot do with the data.

The transit data used is supplied by transit agencies via the legal framework "OpenGeo ND-OV loket", in a transmodel-based format. The transmodel-data is then converted to GTFS by us.

Some technical details: the feed contains shapes for most operators and we've developed a workflow that generate shapes for all rail, tram and metro lines when missing. Currently the stop-locations are directly set by the operator but we're going to switch over to the IFOPT data-set when that becomes available, this will provide a very high quality, maintained stopplace clustering (GTFS: parent stations).

We're also looking to incorporate fare information, but we're currently constrained by both the unavailability of this data and the lack of suitable structure in GTFS to model the fare-structure in the Netherlands. We're optimistic about the availability of fare information changing and the flexibility of GTFS.

The feed also contains transfer information supplied by Dutch railways on specific trip-to-trip transfers, indicating whether a transfer is (im)possible on trip-to-trip detail.

We're also offering a GTFS-realtime feed, currently under as-is, with tripupdates, vehicle-positions and alerts. The vehicle-positions and trip-updates contain almost all bus and tram lines and the Metro in Rotterdam (Amsterdam is in the works), the only trips not covered are vehicles not equipped with the necessary hardware. Real-time train information will also be available in the future but will likely require some new additions the GTFS-realtime standard such as platform changes.

The feeds are available here:

http://gtfs.ovapi.nl/nl/gtfs-nl.zip

http://gtfs.ovapi.nl/nl/alerts.pb

http://gtfs.ovapi.nl/nl/vehiclePositions.pb

http://gtfs.ovapi.nl/nl/tripUpdates.pb

Kind regards,

Thomas Koch

OVapi

[steiner...@gmail.com]

This is great! So OVapi is the first agency in Europe that published their realtime information as open data, right? Or is there another country in Europe that does so?

[Thomas Koch]

We're technically not the agency ourselves but we take the (transmodel) datasets, integrate and convert it to GTFS (realtime).
That the agencies themselves do not publish GTFS themselves is actually a good thing as it allows developers to work with datasets the agencies are actually working with. This avoids issues seen with for example the RATP feed where there are unresolved issues and expired data. Furthermore the continuity of the transmodel datasets is guaranteed by law.

But yes, I think the Netherlands has the first GTFS-realtime feed in Europe. I hope this will set an example for our neighboring countries.

Kind regards,

Thomas Koch
OVapi.

[steiner...@gmail.com]

Thanks for the reply, so does each agency in the Netherlands provide realtime data for all trips or are there only a few of them that cover particular routes at this time?
Best regards

[Thomas Koch]

Practically all trips, there are some exclusions. Those exclusions consist almost entirely out of transport-on-demand busses, buurtbussen (small busses supported by volunteers, buergerbussen in German) and bus-services on 4 out of 5 of the Frisian Islands (Waddeneilanden). Also ferry services are not realtime.

You can get an idea of the coverage by looking at http://www.ovradar.nl/ which uses the same feed as the vehicle-positions feed. Ignore the small hole around Amsterdam, vehicle positions will become available in the next few days there,the tripupdates do contain those trips already.

Realtime train data will be there in the course of this year but will likely require extensions to the current GTFS-realtime spec.

Op vrijdag 14 februari 2014 18:01:56 UTC+1 schreef steiner...@gmail.com:

[Thomas Koch]

Note that ovradar.nl is a solution that directly visualizes the GPS positions as given by the AVL system in the vehicles instead of a time-interpolation system such as Raildar.fr and Zugradar(?).

Op vrijdag 14 februari 2014 19:42:05 UTC+1 schreef Thomas Koch:

[Joa]

On Tuesday, February 11, 2014 4:45:19 PM UTC-8, Thomas Koch wrote:

We're technically not the agency ourselves but we take the (transmodel) datasets, integrate and convert it to GTFS (realtime).

I have a question around agency/operator curation: The agency.txt file lists the feed originators following the spec, and based on feed validation, is expected to be technically valid.
Beyond that, I see a challenge in the way the agencies/operators are integrated: As far as exposing the data to travelers is concerned, it appears to me that contractors are not necessarily adequately differentiated towards a particular service area. As an example: Drilling down into the feed of ARR (Arriva), I find multiple routes with short names 1, 2, 3, 4 and so forth; all of course common across the service areas that Arriva happens to contract for. With that, it does not seem to be possible to key Arriva's (and possibly others') data towards individual cities or regions. Any thoughts on that?
JP

[Thomas Koch]

The multiple routes with the same route_short_name are identifiable by different route_long_names.

4107,ARR,1,Leiderdorp Rijnland Ziekenhuis - Leiden Stevenshof,,3,,,

4083,ARR,1,Stadsdienst Alphen a/d Rijn Ridderveld Oost en Wes,,3,,,

4129,ARR,1,Stadsdienst Gouda via Plaswijck en Groenhoven,,3,,,

50,ARR,1,Ruwaard - Centraal Station,,3,,,

211,ARR,1,Busstation - Westeinde,,3,,,

188,ARR,1,Lelystad-Haven - Lelycentre,,3,,,

But i'm sensing you're looking for a way to obtain just the routes around Amsterdam? Best way would be to transform data by removing stops from stops.txt and the subsequent relationships to stop_times,trips and routes.

Op zaterdag 15 februari 2014 17:24:37 UTC+1 schreef Joa:

[Joa]

On Saturday, February 15, 2014 8:47:34 AM UTC-8, Thomas Koch wrote:

The multiple routes with the same route_short_name are identifiable by different route_long_names.

4107,ARR,1,Leiderdorp Rijnland Ziekenhuis - Leiden Stevenshof,,3,,,

4083,ARR,1,Stadsdienst Alphen a/d Rijn Ridderveld Oost en Wes,,3,,,

4129,ARR,1,Stadsdienst Gouda via Plaswijck en Groenhoven,,3,,,

50,ARR,1,Ruwaard - Centraal Station,,3,,,

211,ARR,1,Busstation - Westeinde,,3,,,

188,ARR,1,Lelystad-Haven - Lelycentre,,3,,,

Developers who have been processing a variety feeds are likely familiar with such an arrangement. Baltimore's MARC service is modeled in this way. However, and this is the distinction - MARC service forms an entity that travelers can readily identify. ARR's routes do not have much commonality other than they happen to have the same route name.

 

But i'm sensing you're looking for a way to obtain just the routes around Amsterdam?

Best way would be to transform data by removing stops from stops.txt and the subsequent relationships to stop_times,trips and routes.

I'm actually interested in the whole data set. I've used ARR as an example only, being the feed where this problem appears to be most prevalent.
Something can be put together of course trying to rationalize the feeds based on various heuristics. But chances are the result is just off enough to indignate or amuse locals enough for them to discount an otherwise valid app. Hence my question about curation of the feeds.

[Thomas Koch]

I think you're running into the way the Dutch public transit is modeled. The Netherlands as of a whole is a dense network without a clear division of entities. There are administrative divisions (concession area's) but those are not entities a ordinary traveler except. For example the concession area "Zuid-Holland-Noord" owned by Arriva contains three of these Arriva (ARR) lines with number 1, each part of a (small) city-network in the cities in Alphen a/d Rijn, Leiden en Gouda. Even in the timetable-book published by Arriva for this concession area, these lines are shoved together

Op zondag 16 februari 2014 20:56:57 UTC+1 schreef Joa:

[steiner...@gmail.com]

I am currently trying to include this information into OpenTripPlanner (OTP). The GTFS information works well there, but how is it possible to include the realtime feed into OTP? 

For this, I tried to expand my application-context.xml like this:

<bean id="periodicGraphUpdater" class="org.opentripplanner.api.servlet.PeriodicGraphUpdater">
  <property name="updaters">
   <list>
     <bean class="org.opentripplanner.updater.GtfsRealtimeUpdater">
      <property name="url" value="http://gtfs.ovapi.nl/new/" />
      <property name="defaultAgencyId" value="ARR" />
     </bean>
   </list>
  </property>
 </bean>

 

but there is no additional information when executing the start-server.bat and calculating a route. Is this the correct URL to insert?

Does GTFS Realtime generally work in the stable branch or do I need the OTP master branch?

Thanks for any help!

[Thomas Koch]

It should work find with OTP, not sure whether the necessary changes have been committed to the master branch. If it's not working you should try the mmri branch. In any case this feed is being consumed by the OTP instance behind opentripplanner.eu, the settings are:

#building

wget http://gtfs.ovapi.nl/nl/gtfs-nl.zip -P build

java -server -Xmx24G -jar otp-core/target/otp.jar -a --transitIndex -l -b build

#running

java -Xmx20G -server -jar otp-core/target/otp.jar -s -a -l -g build -p 9050

We have the following Graph.properties file:

myalert.type = real-time-alerts
myalert.frequencySec = 60
myalert.url = http://gtfs.ovapi.nl/new/alerts.pb
myalert.earlyStartSec = 3600
myalert.defaultAgencyId = ARR
websocket.type = websocket-gtfs-rt-updater
websocket.defaultAgencyId = ARR
websocket.url = ws://localhost:8089/tripUpdates
websocket.tripTimeUpdater = continuesDelay

Op dinsdag 18 februari 2014 19:27:53 UTC+1 schreef steiner...@gmail.com:

[steiner...@gmail.com]

No I just want to try to set up a single instance for a research for university, including GTFS realtime information. At the end I want to analyze some previous realtime data using locally stored realtime feeds from different dates.

What I have done so far is using the Trimet OTP.zip (http://maps.trimet.org/otp-dev/otp.zip), downloading OVapi GTFS data for Netherlands as well as OSM data for Netherlands and changed the graph-builder.xml file to be able to build a graph from Netherlands. The routing graph that was created is about 1,8GB and works well if I start the http://localhost:8080/opentripplanner-webapp.
The next step should include the RT information as well. Because of this, I found the OTP Documentation (https://github.com/opentripplanner/OpenTripPlanner/wiki/GTFS-Realtime), where it is recommended to change the application-context.xml, which can be found in the root directory of the otp itself:

<bean id="periodicGraphUpdater" class="org.opentripplanner.api.servlet.PeriodicGraphUpdater">
    <property name="updaters">
        <list>
            <bean class="org.opentripplanner.updater.GtfsRealtimeUpdater">
                <property name="url" value="YOUR URL" />
                <property name="defaultAgencyId" value="YOUR AGENCY ID" />
            </bean>
       </list>
   </property>
</bean>

When entering your URL (http://gtfs.ovapi.nl/nl/) of OVapi (where all realtime information is contained) and executing the start-server.bat command, there is still no realtime information included when calculating a route.
Or what URL has to be entered here? (I can't find any information about this...) 


I am a little bit confused about the stable and the master branch... Is it true that for the master branch a graph.properties file has to be created that includes the URL of the alerts.pb file, right? What about the tripUpdates.pb? And where does OTP get information about the gtfs-realtime-OVapi.proto file? 
Besides, the graph.properties has to be saved in the same path where the  the graph is located, right?


Furthermore, is there also a standalone version of the master branch like the Trimet OTP.zip where only configuration files need to be changed and a simple start-server.bat can be executed? Or is it only accessible via Eclipse or Netbeans?
I mean, I have already loaded the mmri branch into eclipse and successfully created a graph after adapting the graph-config.xml, but I don't know how to start the server to be able to access the webinterface. 
 
Sorry for this bunch of questions, but everything seems very complicated without any documentation...Would be nice to get some information about this. Thank you!

[Thomas Koch]

The new version of OTP is easier to deploy, without messing with XML files and such. It automatically loads GTFS en OSM files from a specified folder and if a Graph.properties is present also realtime feeds. 

I think there are no binaries present of the mmri-rt branch or master branch but it's not that hard to create those yourselves using git and maven.

git clone https://github.com/opentripplanner/OpenTripPlanner.git

git checkout mmri-rt (for mmri-rt branch)

mvn package -DskipTests

And you should be all set.

Op woensdag 19 februari 2014 21:30:02 UTC+1 schreef steiner...@gmail.com:

[steiner...@gmail.com]

oh thank you very much, now I got it, with the help of the 2min tutorial (https://github.com/opentripplanner/OpenTripPlanner/wiki/TwoMinutes)

everything works fine with the mmri-rt branch, routes can be calculated, but there is an error concerning the GtfsRealtimeAlertsUpdater in the console:

15:27:26.608 ERROR (GraphUpdaterManager.java:172) Error while running graph writer org.opentripplanner.updater.alerts.GtfsRealtimeAlertsUpdater$1:

java.lang.NullPointerException: null

        at org.opentripplanner.routing.patch.AlertPatch.apply(AlertPatch.java:66) ~[otp.jar:1.1]
        at org.opentripplanner.routing.impl.PatchServiceImpl.apply(PatchServiceImpl.java:87) ~[otp.jar:1.1]
        at org.opentripplanner.updater.alerts.AlertsUpdateHandler.handleAlert(AlertsUpdateHandler.java:144) ~[otp.jar:1.1]
        at org.opentripplanner.updater.alerts.AlertsUpdateHandler.update(AlertsUpdateHandler.java:63) ~[otp.jar:1.1]
        at org.opentripplanner.updater.alerts.GtfsRealtimeAlertsUpdater$1.run(GtfsRealtimeAlertsUpdater.java:112) ~[otp.jar:1.1]
        at org.opentripplanner.updater.GraphUpdaterManager$2.run(GraphUpdaterManager.java:170) ~[otp.jar:1.1]
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [na:1.7.0_21]
        at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) [na:1.7.0_21]
        at java.util.concurrent.FutureTask.run(Unknown Source) [na:1.7.0_21]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source) [na:1.7.0_21]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [na:1.7.0_21]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.7.0_21]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.7.0_21]
        at java.lang.Thread.run(Unknown Source) [na:1.7.0_21]

I am calling the webinterface via http://localhost:8080/, in the graph.properties the websocket is on port 8089. I tried to start the server with default setting (port 8089), but there was an error. So I changed the port in the graph.properties file to ws://localhost:8080/tripUpdates. Is this right or is there another mistake why the real-time information doesn't work?

[steiner...@gmail.com]

This error also occurs occasionally:

15:43:29.838 ERROR (WebsocketGtfsRealtimeUpdater.java:135) Could not connect to ws://localhost:8080/tripUpdates: Invalid handshake response

[Thomas Koch]

My bad i gave a Graph.properties that only works local with the experimental websockets differential feed, this should be the correct one, but i haven't tested it.

 

myalert.type = real-time-alerts
myalert.frequencySec = 60
myalert.url = http://gtfs.ovapi.nl/new/alerts.pb
myalert.earlyStartSec = 3600
myalert.defaultAgencyId = ARR 

gtfsrt.type = stop-time-updater 

gtfsrt.sourceType = gtfs-http

gtfsrt.defaultAgencyId = ARR
gtfsrt.url = http://gtfs.ovapi.nl/new/tripUpdates.pb
gtfsrt.tripTimeUpdater = continuesDelay

For the alert nullpointer, i've run in to that one but afaik that was patched and fixed in the mmri-rt branch.. Are your sure you're building the graph with a transitIndex?

Should be like:

java -server -Xmx24G -jar otp-core/target/otp.jar -a --transitIndex -l -b build

java -Xmx20G -server -jar otp-core/target/otp.jar -s -a -l -g build -p 9050

Op donderdag 20 februari 2014 21:49:29 UTC+1 schreef steiner...@gmail.com:

[Andrew Byrd]

On 02/21/2014 12:15 AM, Thomas Koch wrote:
> For the alert nullpointer, i've run in to that one but afaik that was
> patched and fixed in the mmri-rt branch.. Are your sure you're building
> the graph with a transitIndex?
> Should be like:
> java -server -Xmx24G -jar otp-core/target/otp.jar -a --transitIndex -l
> -b build
> java -Xmx20G -server -jar otp-core/target/otp.jar -s -a -l -g build -p 9050

On a related note, in a branch I've got code that builds a transit index
on demand, so we don't have to worry about whether or not the index
builder module was used on a particular graph. Hopefully that will make
it into version 1.0.

-Andrew

[Andrew Byrd]

... and I just realized this conversation is being held on the
transit-developers list. I guess this thread is not entirely
off-subject, but it is OTP-specific and technical enough that any
further discussion should probably be moved to the opentripplanner-dev
or opentripplanner-users list.

-Andrew

[steiner...@gmail.com]

Thank you Thomas, this Graph.properties works much better, but there is still one little error. Perhaps you guys could also explain in a short summary on the wiki the meaning of all these shortcuts "-s -a -l", just for understanding.

Thank you Andrew, I just moved the discussion to the following thread: https://groups.google.com/forum/#!topic/opentripplanner-users/F1MTMQW1B1Y

[Andrew Byrd]

Hi Daniel,

Just run otp.jar or the otp shell script with the --help switch. See
below for reference. Howver, remember that none of this is yet part of
an official OTP release and so is subject to change over the next month
or so.

-Andrew


~/git/OpenTripPlanner$ ./otp --help
Usage: java -Xmx[several]G -jar otp.jar [options] files
Options:
-a, --analyst enable OTP Analyst extensions
Default: false
-b, --build build graphs at specified paths
-c, --cache the directory under which to cache
OSM and
NED tiles
-e, --elevation download and use elevation data for
the graph
Default: false
-g, --graphs path to graph directory
-h, --help Print this help message and exit
Default: false
-m, --inMemory pass the graph to the server
in-memory after
building it, without saving to disk
Default: false
-l, --longDistance use an algorithm tailored for
long-distance
routing
Default: false
--noEmbedConfig Skip embedding config in graph
(Embed.properties)
Default: false
--noParentStopLinking skip linking of stops to parent
stops (GTFS)
Default: false
--noStreets skip all street input files (OSM)
Default: false
--noTransit skip all transit input files (GTFS)
Default: false
--parentStationTransfers create direct transfers between the
constituent stops of each parent station
Default: false
-p, --port server port
-r, --router Router ID, first one being the default
-s, --server run a server
Default: false
-t, --static path to static content
--transitIndex build a transit index for GTFS data
Default: false
--useTransfersTxt use transfers.txt file for the
gtfsBundle
(GTFS)
Default: false
-v, --verbose Verbose output
Default: false
-z, --visualize open a debugging graph visualizer
Default: false

> --
> You received this message because you are subscribed to the Google
> Groups "Transit Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to transit-develop...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[steiner...@gmail.com]

Thank you!!

> an email to transit-developers+unsub...@googlegroups.com.

[steiner...@gmail.com]

Thomas, is there also a shapefile from the OVapi feed available that includes the trips?
I tried to parse the feed into a kml file using the kmlwriter.exe, but there were some memory issues, I think it is maybe too big... (http://code.google.com/p/googletransitdatafeed/issues/detail?id=373&start=100)

Thanks,
Daniel

[Thomas Koch]

Check, http://data.openov.nl/lijnnetkaart/

Op vrijdag 21 maart 2014 18:51:25 UTC+1 schreef steiner...@gmail.com:

[steiner...@gmail.com]

Thanks, but unfortunately the shapefile doesn't contain any information about trip_id's.
I found a script that is able to extract GTFS data into shapefiles on http://lin-ear-th-inking.blogspot.com/2011/10/extracting-gtfs-data-using-jeql.html and adapted it to my needs.The output is a GTFS-stops shapefile and a GTFS-trips shapefile:

CSVReader tshape hasColNames: file: "C:\\shp ovapi\\test\\ovapi\\shapes.txt";
CSVReader ttrip hasColNames: file: "C:\\shp ovapi\\test\\ovapi\\trips.txt";
CSVReader troutes hasColNames: file: "C:\\shp ovapi\\test\\ovapi\\routes.txt"; 
Mem troutes;
crs_WSG84 = "EPSG:4326";
trips = select distinct route_id, trip_id, realtime_trip_id, trip_headsign, shape_id
    from ttrip;
tpt = select shape_id, Val.toInt(shape_pt_sequence) seq,
        Geom.createPoint(Val.toDouble(shape_pt_lon),
                                 Val.toDouble(shape_pt_lat) ) pt
    from tshape
    order by shape_id, seq;
tline = select shape_id,
     CRS.project(Geom.toMulti(geomConnect(pt)), crs_WSG84) geom
    from tpt
    group by shape_id;
trouteLines = select r.route_id, r.agency_id,
    r.route_short_name, r.route_long_name,
    r.route_type, r.route_url,
    trip_id, realtime_trip_id, trip_headsign, l.*
        from tline l
        join trips t on t.shape_id == l.shape_id
        join troutes r on r.route_id == t.route_id;
ShapefileWriter trouteLines file: "C:\\shp ovapi\\test\\gtfs_trips.shp";
CSVReader tstopsRaw hasColNames: file: "C:\\shp ovapi\\test\\ovapi\\stops.txt";
tstop = select stop_id,stop_code,stop_name,
        CRS.project(Geom.createPoint(
        Val.toDouble(stop_lon),
        Val.toDouble(stop_lat) ), crs_WSG84) pt,
        zone_id
        from tstopsRaw;
ShapefileWriter tstop file: "C:\\shp ovapi\\test\\gtfs_stops.shp";

[steiner...@gmail.com]

I have another question regarding the GTFS realtime feed. When observing the tripUpdates in plain text, a trip contains the given information.
How is it possible to estimate the delay for the future stops? I mean, if there was a slightly increase or decrease of the delay over time it would make sense to me, but the delay is going up and down, so I can't see any pattern in it. This is completely incomprehensible for me. Do you use the current position of the vehicle or do you also use other vehicle positions, or even some additional traffic information?

trip {
  trip_id: "10258650"
  start_date: "20140321"
  schedule_relationship: SCHEDULED
  1003: "\n\017RET:M007:316868"
}
stop_time_update {
  stop_sequence: 1
  arrival {
    delay: -244
    time: 1395427856
  }
  departure {
    delay: 66
    time: 1395428166
  }
  stop_id: "132544"
}
stop_time_update {
  stop_sequence: 2
  arrival {
    delay: 0
    time: 1395428265
  }
  departure {
    delay: 59
    time: 1395428324
  }
  stop_id: "132520"
}
stop_time_update {
  stop_sequence: 3
  arrival {
    delay: 0
    time: 1395428355
  }
  departure {
    delay: 58
    time: 1395428413
  }
  stop_id: "132496"
}
stop_time_update {
  stop_sequence: 4
  arrival {
    delay: 0
    time: 1395428445
  }
  departure {
    delay: 48
    time: 1395428493
  }
  stop_id: "132498"
}
stop_time_update {
  stop_sequence: 5
  arrival {
    delay: 0
    time: 1395428520
  }
  departure {
    delay: 50
    time: 1395428570
  }
  stop_id: "132565"
}
stop_time_update {
  stop_sequence: 6
  arrival {
    delay: 0
    time: 1395428595
  }
  departure {
    delay: 46
    time: 1395428641
  }
  stop_id: "132494"
}
stop_time_update {
  stop_sequence: 7
  arrival {
    delay: 0
    time: 1395428670
  }
  departure {
    delay: 54
    time: 1395428724
  }
  stop_id: "132511"
}
stop_time_update {
  stop_sequence: 8
  arrival {
    delay: 0
    time: 1395428745
  }
  departure {
    delay: 55
    time: 1395428800
  }
  stop_id: "132514"
}
stop_time_update {
  stop_sequence: 9
  arrival {
    delay: 21
    time: 1395428841
  }
  departure {
    delay: 64
    time: 1395428884
  }
  stop_id: "132516"
}
stop_time_update {
  stop_sequence: 10
  arrival {
    delay: 19
    time: 1395428914
  }
  departure {
    delay: 63
    time: 1395428958
  }
  stop_id: "132518"
}
stop_time_update {
  stop_sequence: 11
  arrival {
    delay: 0
    time: 1395428970
  }
  departure {
    delay: 53
    time: 1395429023
  }
  stop_id: "132524"
}
stop_time_update {
  stop_sequence: 12
  arrival {
    delay: 0
    time: 1395429090
  }
  departure {
    delay: 55
    time: 1395429145
  }
  stop_id: "132528"
}
stop_time_update {
  stop_sequence: 13
  arrival {
    delay: 0
    time: 1395429240
  }
  departure {
    delay: 38
    time: 1395429278
  }
  stop_id: "132532"
}
stop_time_update {
  stop_sequence: 14
  arrival {
    delay: 0
    time: 1395429390
  }
  departure {
    delay: 47
    time: 1395429437
  }
  stop_id: "132530"
}
stop_time_update {
  stop_sequence: 15
  arrival {
    delay: 0
    time: 1395429510
  }
  departure {
    delay: 50
    time: 1395429560
  }
  stop_id: "132542"
}
stop_time_update {
  stop_sequence: 16
  arrival {
    delay: 0
    time: 1395429585
  }
  departure {
    delay: 68
    time: 1395429653
  }
  stop_id: "132540"
}
stop_time_update {
  stop_sequence: 17
  arrival {
    delay: 0
    time: 1395429675
  }
  departure {
    delay: 55
    time: 1395429730
  }
  stop_id: "132538"
}
stop_time_update {
  stop_sequence: 18
  arrival {
    delay: 0
    time: 1395429780
  }
  departure {
    delay: 38
    time: 1395429818
  }
  stop_id: "132536"
}
stop_time_update {
  stop_sequence: 19
  arrival {
    delay: 0
    time: 1395429885
  }
  departure {
    delay: 36
    time: 1395429921
  }
  stop_id: "132534"
}
stop_time_update {
  stop_sequence: 20
  arrival {
    delay: 0
    time: 1395429960
  }
  departure {
    delay: 32
    time: 1395429992
  }
  stop_id: "132558"
}
stop_time_update {
  stop_sequence: 21
  arrival {
    delay: -15
    time: 1395430035
  }
  departure {
    delay: 29
    time: 1395430079
  }
  stop_id: "132550"
}
stop_time_update {
  stop_sequence: 22
  arrival {
    delay: -19
    time: 1395430121
  }
  departure {
    delay: 33
    time: 1395430173
  }
  stop_id: "132552"
}
stop_time_update {
  stop_sequence: 23
  arrival {
    delay: -37
    time: 1395430253
  }
  departure {
    delay: 0
    time: 1395430920
  }
  stop_id: "132582"
}
vehicle {
  id: "RET:0"
  label: "0"
}
timestamp: 1395430253

[Thomas Koch]

Based on the fact that i don't see a StopTimeUpdate.ScheduleRelationship.SCHEDULED explicitly given, these delays are in fact the actual achieved delays based on the difference between timestamp in the arrival/departure message and scheduled time. Pretty sure the 0sec arrival-delay's in this case are just missing a arrival trigger message though.

The delay going up and down is an expected behavior due to the planned extra time with regard to drive times and dwell times. 

-

To answer the question about how delays are propagated to the stops in the future: The vehicle provides us with a arrival-punctuality for the next stop. 

If that punctuality is negative and the next stop is a waitpoint[1], the departure-punctuality is set to 0. Otherwise the negative punctuality is decayed over time/distance (when there are links with a very slow planned velocity).

If the arrival-punctuality is positive, it is assumed the trip-punctuality will not increase, but instead will decrease. To achieve that the difference between the scheduled drivetime and theoretical maximum speed are subtracted from the punctuality at each following stop until the punctuality approaches 0.

[1] Waitpoint is the NeTEX name, also known as timingpoint. Vehicle does not departure before scheduled deperaturetime.

Op maandag 24 maart 2014 19:36:30 UTC+1 schreef steiner...@gmail.com:

[steiner...@gmail.com]

Am Montag, 24. März 2014 17:32:46 UTC-4 schrieb Thomas Koch:

Based on the fact that i don't see a StopTimeUpdate.ScheduleRelationship.SCHEDULED explicitly given, these delays are in fact the actual achieved delays based on the difference between timestamp in the arrival/departure message and scheduled time.

Does it mean that there is a route guidance system running in the background that permanently calculates the delay between the current vehicle position and the next stop in the trip? Or how is it possible to get the delay information?
 

Pretty sure the 0sec arrival-delay's in this case are just missing a arrival trigger message though.

The delay going up and down is an expected behavior due to the planned extra time with regard to drive times and dwell times. 

Ok so there is a lack of trigger messages that won't be written into the tripUpdates.pb file, which is, after taking a look at the whole decoded tripUpdates.pb file, an enormous amount of missing information. So, does OTP use every single stop arrival and departure delays for the calculation of the overall delay of a particular trip? Or does it just take the arrival delay information from the last stop_id?

If the arrival-punctuality is positive, it is assumed the trip-punctuality will not increase, but instead will decrease. To achieve that the difference between the scheduled drivetime and theoretical maximum speed are subtracted from the punctuality at each following stop until the punctuality approaches 0.

So because of the assumption that the trip-punctuality will decrease over time, it also means that, because of unexpected incidents, the delay information gets more and more inaccurate over time if the trip has plenty of stops, right?

[Thomas Koch]

Op dinsdag 25 maart 2014 22:30:14 UTC+1 schreef steiner...@gmail.com:

Am Montag, 24. März 2014 17:32:46 UTC-4 schrieb Thomas Koch:

Based on the fact that i don't see a StopTimeUpdate.ScheduleRelationship.SCHEDULED explicitly given, these delays are in fact the actual achieved delays based on the difference between timestamp in the arrival/departure message and scheduled time.

Does it mean that there is a route guidance system running in the background that permanently calculates the delay between the current vehicle position and the next stop in the trip? Or how is it possible to get the delay information?

There is a route guidance system that sends a message when it approaches the stop and opens the door (ARRIVAL) and when it leaves the stop. (DEPARTURE). This gives a timestamp which is used to give ETA and delay's for stops visited. 

 

 

Pretty sure the 0sec arrival-delay's in this case are just missing a arrival trigger message though.

The delay going up and down is an expected behavior due to the planned extra time with regard to drive times and dwell times. 

Ok so there is a lack of trigger messages that won't be written into the tripUpdates.pb file, which is, after taking a look at the whole decoded tripUpdates.pb file, an enormous amount of missing information. So, does OTP use every single stop arrival and departure delays for the calculation of the overall delay of a particular trip? Or does it just take the arrival delay information from the last stop_id?

OTP takes delays from each stoptimeupdate, note GTFS-RT has the delay-propagation thing where the same delay is propagated to subsequent stops, see https://developers.google.com/transit/gtfs-realtime/trip-updates

 

If the arrival-punctuality is positive, it is assumed the trip-punctuality will not increase, but instead will decrease. To achieve that the difference between the scheduled drivetime and theoretical maximum speed are subtracted from the punctuality at each following stop until the punctuality approaches 0.

So because of the assumption that the trip-punctuality will decrease over time, it also means that, because of unexpected incidents, the delay information gets more and more inaccurate over time if the trip has plenty of stops, right?

Yes it's very hard to predict unexpected event's ;)

[steiner...@gmail.com]

Ok so there is a lack of trigger messages that won't be written into the tripUpdates.pb file, which is, after taking a look at the whole decoded tripUpdates.pb file, an enormous amount of missing information. So, does OTP use every single stop arrival and departure delays for the calculation of the overall delay of a particular trip? Or does it just take the arrival delay information from the last stop_id?

OTP takes delays from each stoptimeupdate, note GTFS-RT has the delay-propagation thing where the same delay is propagated to subsequent stops, see https://developers.google.com/transit/gtfs-realtime/trip-updates

This also means that, due to some missing delay information in some stops, OTP may misinterpret the overall delay for a particular route? Because the overall delay is calculated for one trip as a whole.
Or is it just assumed that, like in the example below, the trip is on time if the last stop contains a delay information of 0? In conclusion, I just don't get it how OTP calculates the overall delay that is visualized in the web interface as a green or red number.

trip {
  trip_id: "10348596"
  start_date: "20140325"
  schedule_relationship: SCHEDULED
  1003: "\n\016RET:640:327655"
}
stop_time_update {
  stop_sequence: 1
  departure {
    delay: 122
    time: 1395767042
  }
  stop_id: "52302"
}
stop_time_update {
  stop_sequence: 2
  arrival {
    delay: 111
    time: 1395767153
  }
  departure {
    delay: 127
    time: 1395767169
  }
  stop_id: "50957"
}
stop_time_update {
  stop_sequence: 3
  arrival {
    delay: 122
    time: 1395767212
  }
  departure {
    delay: 122
    time: 1395767212
  }
  stop_id: "49724"
}
#####cut#######
stop_time_update {
  stop_sequence: 25
  arrival {
    delay: 96
    time: 1395769356
  }
  departure {
    delay: 96
    time: 1395769356
  }
  stop_id: "49903"
}
stop_time_update {
  stop_sequence: 26
  arrival {
    delay: 0
    time: 1395769560
  }
  departure {
    delay: 0
    time: 1395769560
  }
  stop_id: "50923"
}
vehicle {
  id: "RET:1061"
  label: "1061"
  1003: "\b\001\022\023MAN Lion\'s City T\303\234"
}
timestamp: 1395769505

[Thomas Koch]

No it's correctly done in OTP, see https://github.com/opentripplanner/OpenTripPlanner/issues/1222

Op woensdag 26 maart 2014 18:03:27 UTC+1 schreef steiner...@gmail.com:

[steiner...@gmail.com]

Ok thank you for this hint, so the information is simply redundant in this feed, you can either use the "delay" or "time" information.

One other question, I am currently storing the RT information into a postgres database where I want to analyze it afterwards. For this, I'm requesting the vehiclePositions.pb file for a pre-defined time range. Today I saw that at after 06:19:03 pm CEST  (14.04.), suddenly no vehiclePosition had a vehicleDescriptor any more.

So, before this time, there were 3571 vehiclePositions available (whereas 142 were without a vehicleDescriptor):

Vehicle: trip {
  trip_id: "10919359"
  start_date: "20140414"
  schedule_relationship: SCHEDULED
  1003: "\n\016ARR:17183:1027"
}
position {
  latitude: 52.161125
  longitude: 4.532298
}
current_stop_sequence: 43
current_status: IN_TRANSIT_TO
timestamp: 1397492217
stop_id: "81833"
vehicle {
  id: "ARR:8755"
  label: "8755"
}
1003: "\b\261\001"

Vehicle: trip {
  trip_id: "10092500"
  start_date: "20140414"
  schedule_relationship: SCHEDULED
  1003: "\n\fVTN:10:10080"
}
position {
  latitude: 51.647602
  longitude: 4.861387
}
current_stop_sequence: 5
current_status: STOPPED_AT
timestamp: 1397492242
stop_id: "26098"
vehicle {
  id: "VTN:5812"
  label: "5812"
}
1003: "\b\376\002"

And after this timestamp up to now there were up to 715 vehiclePositions available (whereas 715 were without a vehicleDescriptor):

Vehicle: position {
  latitude: 52.50574
  longitude: 4.984042
}
current_stop_sequence: 4
current_status: IN_TRANSIT_TO
stop_id: "13677"
1003: "\bB"


Vehicle: position {
  latitude: 53.10354
  longitude: 6.100555
}
current_stop_sequence: 1
current_status: STOPPED_AT
1003: "\b\000"

Is this the case because the RT service is being disabled over night? If yes, is there a certain time range where the vehiclePositions RT service is working?

Thanks in advance,
Daniel

[Thomas Koch]

Thanks for the message. That's when a newer version was deployed. The TripDescriptor missing seems a regression error. We'll look into it and deploy a newer version tonight.

Op dinsdag 15 april 2014 04:29:08 UTC+2 schreef steiner...@gmail.com:

[steiner...@gmail.com]

You're welcome. Please let me know when the new version is launched. Thanks!

[Thomas Koch]

It's planned to deploy at around midnight.

Op dinsdag 15 april 2014 17:32:41 UTC+2 schreef steiner...@gmail.com:

[steiner...@gmail.com]

Yeah it works, the only thing I realized is that there nearly half of all vehicles don't have a position information (vehiclePosition.hasPosition()):

Timestamp: 2014-04-16 12:01:34.556 EDT
There are 7464 vehiclePositions in this file.
There are 0 vehiclePositions without a tripID in this file.
There are 3530 vehiclePositions without position information in this file.

[Thomas Koch]

UnfortunatelyGVB (Amsterdam) vehicles are (currently) not capable to transmit their GPS position, hence we're only able to indicate whether they are stopped/driving to a stop.

Op woensdag 16 april 2014 18:06:09 UTC+2 schreef steiner...@gmail.com:

[steiner...@gmail.com]

Today I realized that a vehicle position was described by a "NaN" as lat and lon. Whatever this means, it's no float so an error was thrown...

trip {
  trip_id: "10376835"
  start_date: "20140421"
  schedule_relationship: SCHEDULED
  1003: "\n\rCXX:X080:7009"
}
position {
  latitude: NaN
  longitude: NaN
}
current_stop_sequence: 24
current_status: IN_TRANSIT_TO
stop_id: "43865"
1003: "\b\223\001"

[steiner...@gmail.com]

I also have a question regarding the new version, did you change the logic behind the current_stop_sequence after a vehicle has the status "IN_TRANSIT_TO"? In the former version, when a vehicle had the status "IN_TRANSIT_TO", the current_stop_sequence was automatically the highest stop sequence (last stop) from that particular trip. Since the new version, I see that the current stop sequence in case of the status "IN_TRANSIT_TO" is the same as from the last stop. Is this considered as the final version or do you plan to change it to "IN_TRANSIT_TO" where the current_stop_sequence is the following one? In my mind it would make more sense.

Regards,
Daniel

[Thomas Koch]

I'm not sure what you mean, With IN_TRANSIT_TO it should state the next following stop_id. A quick sample tells me that it does exactly that.

Op maandag 21 april 2014 20:01:48 UTC+2 schreef steiner...@gmail.com:

[steiner...@gmail.com]

Yeah I'm aware of that, what I meant is that when you look at the right picture, these are the data collected before the change of version. There you can see that always when the current status is "IN TRANSIT TO", the assigned "current stop sequence" was the sequence of the last stop of that particular trip. After the change (left picture), you can see that at each position where the current status is "IN TRANSIT TO", the current stop sequence is the same as the current stop sequence from the the last stop that was departed.

I'm just following my logic when thinking about the assignment of the current stop sequence at status "IN TRANSIT TO" like:

The bus driving the tripID 123456:

|   stopID    |      stop sequence    |     current status   |      timestamp      |      position
    2000                    0                      STOPPED  AT            00:00:00             x,y
    3000                    1                      IN TRANSIT TO           00:01:00             x,y
    3000                    1                      IN TRANSIT TO           00:02:00             x,y
    3000                    1                      STOPPED AT             00:03:00             x,y
    3000                    1                      STOPPED AT             00:04:00             x,y
    4000                    2                      IN TRANSIT TO           00:05:00             x,y
    4000                    2                      IN TRANSIT TO           00:06:00             x,y
    5000                    2                      STOPPED AT             00:07:00             x,y


Instead of the left picture, where the status IN TRANSIT TO is assigned to the stop before. But this is just my opinion :)

[steiner...@gmail.com]

I also realized that a large amount of trips won't be finished with the status "STOPPED AT" the last stop, which means that it isn't possible to filter a whole trip out of the real-time information. Is there a certain reason why this is the case?

[Thomas Koch]

Could be for a lot of reasons, such as the end stop not being detected, bandwith issues around the busstation, etc etc. 

Op maandag 28 april 2014 19:14:57 UTC+2 schreef steiner...@gmail.com:

[Thomas Koch]

Could be for a lot of reasons, such as the end stop not being detected, bandwith issues around the busstation, etc etc. 

Op maandag 28 april 2014 19:14:57 UTC+2 schreef steiner...@gmail.com:

I also realized that a large amount of trips won't be finished with the status "STOPPED AT" the last stop, which means that it isn't possible to filter a whole trip out of the real-time information. Is there a certain reason why this is the case?

[Bastiaan]

I found this post as I'm trying to install OTP for realtime planning in the Netherlands. I tried to insall OTP using following commands:

git clone https://github.com/opentripplanner/OpenTripPlanner.git

git checkout mmri-rt (for mmri-rt branch)

mvn package -DskipTests

but after: 

git checkout mmri-rt

I get this error: fatal: Not a git repository (or any of the parent directories): .git

In the manual on github I also get a lot of references to non existing pages. Also the manual on github seems very outdated. What would be the best manual to follow for setting up OTP for all transport in the netherlands?

Op donderdag 20 februari 2014 11:41:27 UTC+1 schreef Thomas Koch:

The new version of OTP is easier to deploy, without messing with XML files and such. It automatically loads GTFS en OSM files from a specified folder and if a Graph.properties is present also realtime feeds. 

I think there are no binaries present of the mmri-rt branch or master branch but it's not that hard to create those yourselves using git and maven.

git clone https://github.com/opentripplanner/OpenTripPlanner.git

git checkout mmri-rt (for mmri-rt branch)

mvn package -DskipTests

And you should be all set.

Op woensdag 19 februari 2014 21:30:02 UTC+1 schreef steiner...@gmail.com:

No I just want to try to set up a single instance for a research for university, including GTFS realtime information. At the end I want to analyze some previous realtime data using locally stored realtime feeds from different dates.

What I have done so far is using the Trimet OTP.zip (http://maps.trimet.org/otp-dev/otp.zip), downloading OVapi GTFS data for Netherlands as well as OSM data for Netherlands and changed the graph-builder.xml file to be able to build a graph from Netherlands. The routing graph that was created is about 1,8GB and works well if I start the http://localhost:8080/opentripplanner-webapp.
The next step should include the RT information as well. Because of this, I found the OTP Documentation (https://github.com/opentripplanner/OpenTripPlanner/wiki/GTFS-Realtime), where it is recommended to change the application-context.xml, which can be found in the root directory of the otp itself:

<bean id="periodicGraphUpdater" class="org.opentripplanner.api.servlet.PeriodicGraphUpdater">
    <property name="updaters">
        <list>
            <bean class="org.opentripplanner.updater.GtfsRealtimeUpdater">
                <property name="url" value="YOUR URL" />
                <property name="defaultAgencyId" value="YOUR AGENCY ID" />
            </bean>
       </list>
   </property>
</bean>

When entering your URL (http://gtfs.ovapi.nl/nl/) of OVapi (where all realtime information is contained) and executing the start-server.bat command, there is still no realtime information included when calculating a route.
Or what URL has to be entered here? (I can't find any information about this...) 


I am a little bit confused about the stable and the master branch... Is it true that for the master branch a graph.properties file has to be created that includes the URL of the alerts.pb file, right? What about the tripUpdates.pb? And where does OTP get information about the gtfs-realtime-OVapi.proto file? 
Besides, the graph.properties has to be saved in the same path where the  the graph is located, right?


Furthermore, is there also a standalone version of the master branch like the Trimet OTP.zip where only configuration files need to be changed and a simple start-server.bat can be executed? Or is it only accessible via Eclipse or Netbeans?
I mean, I have already loaded the mmri branch into eclipse and successfully created a graph after adapting the graph-config.xml, but I don't know how to start the server to be able to access the webinterface. 
 
Sorry for this bunch of questions, but everything seems very complicated without any documentation...Would be nice to get some information about this. Thank you!

Am Mittwoch, 19. Februar 2014 09:18:16 UTC-5 schrieb Thomas Koch:

It should work find with OTP, not sure whether the necessary changes have been committed to the master branch. If it's not working you should try the mmri branch. In any case this feed is being consumed by the OTP instance behind opentripplanner.eu, the settings are:

#building

wget http://gtfs.ovapi.nl/nl/gtfs-nl.zip -P build

java -server -Xmx24G -jar otp-core/target/otp.jar -a --transitIndex -l -b build

#running

java -Xmx20G -server -jar otp-core/target/otp.jar -s -a -l -g build -p 9050

We have the following Graph.properties file:

myalert.type = real-time-alerts
myalert.frequencySec = 60
myalert.url = http://gtfs.ovapi.nl/new/alerts.pb
myalert.earlyStartSec = 3600
myalert.defaultAgencyId = ARR
websocket.type = websocket-gtfs-rt-updater
websocket.defaultAgencyId = ARR
websocket.url = ws://localhost:8089/tripUpdates
websocket.tripTimeUpdater = continuesDelay

Op dinsdag 18 februari 2014 19:27:53 UTC+1 schreef steiner...@gmail.com:

I am currently trying to include this information into OpenTripPlanner (OTP). The GTFS information works well there, but how is it possible to include the realtime feed into OTP? 

For this, I tried to expand my application-context.xml like this:

<bean id="periodicGraphUpdater" class="org.opentripplanner.api.servlet.PeriodicGraphUpdater">
  <property name="updaters">
   <list>
     <bean class="org.opentripplanner.updater.GtfsRealtimeUpdater">
      <property name="url" value="http://gtfs.ovapi.nl/new/" />
      <property name="defaultAgencyId" value="ARR" />
     </bean>
   </list>
  </property>
 </bean>

 

but there is no additional information when executing the start-server.bat and calculating a route. Is this the correct URL to insert?

Does GTFS Realtime generally work in the stable branch or do I need the OTP master branch?

Thanks for any help!

[Andrew Byrd]

Hi Bastiaan,

Most Git commands only make sense within a git repository directory. You cloned the OTP repository but you didn’t move into the newly cloned directory before trying to act upon it.

If by “the manual on GitHub” you are referring to the project wiki, as explained on the landing page of that wiki it contains only outdated / legacy documentation. All current documentation is version controlled within the OTP repository and automatically published to readthedocs at http://opentripplanner.readthedocs.org/en/latest/. There is no need to build OTP from source using Git and Maven if you just want to deploy it rather than modify the source code. 

You are welcome to direct any further questions to the OpenTripPlanner developers’ or users’ mailing lists.

Regards,

Andrew

-- 
You received this message because you are subscribed to the Google Groups "Transit Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to transit-develop...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[bido...@gmail.com]

Thank you for this information. It is great! If the fare-structure is integrated with the GTFS, that will be excellent source of information for Urban Planning Students and professionals.

On Monday, January 20, 2014 at 1:01:41 PM UTC+1, Thomas Koch wrote:

Hello all,

We would like to announce that we are supplying a GTFS feed with transit data for virtually all bus/metro/train/tram and some ferry services in the Netherlands, it also contains most international connections towards Belgium, France, Germany and the UK. The feed is licensed under CC0 license without tedious constraints about what you can and cannot do with the data.

The transit data used is supplied by transit agencies via the legal framework "OpenGeo ND-OV loket", in a transmodel-based format. The transmodel-data is then converted to GTFS by us.

Some technical details: the feed contains shapes for most operators and we've developed a workflow that generate shapes for all rail, tram and metro lines when missing. Currently the stop-locations are directly set by the operator but we're going to switch over to the IFOPT data-set when that becomes available, this will provide a very high quality, maintained stopplace clustering (GTFS: parent stations).

We're also looking to incorporate fare information, but we're currently constrained by both the unavailability of this data and the lack of suitable structure in GTFS to model the fare-structure in the Netherlands. We're optimistic about the availability of fare information changing and the flexibility of GTFS.

The feed also contains transfer information supplied by Dutch railways on specific trip-to-trip transfers, indicating whether a transfer is (im)possible on trip-to-trip detail.

We're also offering a GTFS-realtime feed, currently under as-is, with tripupdates, vehicle-positions and alerts. The vehicle-positions and trip-updates contain almost all bus and tram lines and the Metro in Rotterdam (Amsterdam is in the works), the only trips not covered are vehicles not equipped with the necessary hardware. Real-time train information will also be available in the future but will likely require some new additions the GTFS-realtime standard such as platform changes.

The feeds are available here:

http://gtfs.ovapi.nl/nl/gtfs-nl.zip

http://gtfs.ovapi.nl/nl/alerts.pb

http://gtfs.ovapi.nl/nl/vehiclePositions.pb

http://gtfs.ovapi.nl/nl/tripUpdates.pb

Kind regards,

Thomas Koch

OVapi