How to process GTFS Realtime v1.0 feed data?

[jdkwa...@gmail.com]

Hi, I have a used the code referenced at https://developers.google.com/transit/gtfs-realtime/examples/nodejs-sample to successfully read in the GTFS realtime feed posted at https://www.mbta.com/developers/gtfs-realtime

However, what I really need to do is read the feed posted here https://gtfsrt.api.translink.com.au/, which is v1.0 realtime. I have tried to just read it in with my existing code, but I get the following error:

====================

C:\gtfs\gtfs_realtime\node_modules\protobufjs\ProtoBuf.js:1987
                            throw(e);
                            ^

Error: Illegal value for Message.Field .transit_realtime.TripDescriptor.schedule_relationship: 5 (not a valid enum value)
    at Field.ProtoBuf.Reflect.Field.verifyValue (C:\stuff\_clients\david apelt\gtfs\gtfs_realtime\node_modules\protobufjs\ProtoBuf.js:2321:27)
    at Message.set (C:\stuff\_clients\david apelt\gtfs\gtfs_realtime\node_modules\protobufjs\ProtoBuf.js:1713:50)
    at Message.ProtoBuf.Reflect.Message.decode (C:\stuff\_clients\david apelt\gtfs\gtfs_realtime\node_modules\protobufjs\ProtoBuf.js:2139:29)
    at Field.ProtoBuf.Reflect.Field.decode (C:\stuff\_clients\david apelt\gtfs\gtfs_realtime\node_modules\protobufjs\ProtoBuf.js:2594:46)
    at Message.ProtoBuf.Reflect.Message.decode (C:\stuff\_clients\david apelt\gtfs\gtfs_realtime\node_modules\protobufjs\ProtoBuf.js:2139:51)
    at Field.ProtoBuf.Reflect.Field.decode (C:\stuff\_clients\david apelt\gtfs\gtfs_realtime\node_modules\protobufjs\ProtoBuf.js:2594:46)
    at Message.ProtoBuf.Reflect.Message.decode (C:\stuff\_clients\david apelt\gtfs\gtfs_realtime\node_modules\protobufjs\ProtoBuf.js:2139:51)
    at Field.ProtoBuf.Reflect.Field.decode (C:\stuff\_clients\david apelt\gtfs\gtfs_realtime\node_modules\protobufjs\ProtoBuf.js:2594:46)
    at Message.ProtoBuf.Reflect.Message.decode (C:\stuff\_clients\david apelt\gtfs\gtfs_realtime\node_modules\protobufjs\ProtoBuf.js:2137:51)
    at Function.Message.decode (C:\stuff\_clients\david apelt\gtfs\gtfs_realtime\node_modules\protobufjs\ProtoBuf.js:1982:41)

====================

I guess this is because of missing fields in the v1.0 feed header. Specifically, the v2.0 feed header on the https://www.mbta.com/developers/gtfs-realtime site has, for example:

header {
  gtfs_realtime_version: "2.0"
  incrementality: FULL_DATASET
  timestamp: 1549585866
}

The version 1.0 feed header on the https://gtfsrt.api.translink.com.au/ site has, for example:

header {
  gtfs_realtime_version: "1.0"
  timestamp: 1549426599
}

So "incrementality" is missing.

How do I modify my code to read in v1.0 realtime feed data? It is obviously breaking when it gets to the following line:

var feed = GtfsRealtimeBindings.FeedMessage.decode(body)

Thanks for any help.

[Andrew Byrd]

Hello,

I don’t think the problem is decoding a v1.0 versus v2.0 GTFS-RT.

If you look at the file https://developers.google.com/transit/gtfs-realtime/gtfs-realtime.proto and search for ScheduleRelationship, you’ll see that the ScheduleRelationship enum does not have a value 5. This file also has a comment stating "Version of the feed specification. The current version is 2.0.” so it’s reflecting version 2.0 of the specification.

It appears that the file you are consuming contains a nonstandard enum value, so it can’t be consumed by standard software. Are you perhaps using an experimental version of the feed? 

A quick search I turned up this repo, whose name mentions MBTA and has a modified protobuf defintion that includes a value REPLACEMENT = 5: https://github.com/dvreed77/mbta/blob/master/gtfs-realtime.proto#L482

You may be able to substitute this protobuf definition (or another one from MBTA) into your GTFS-RT reading application, and the repo I linked to above may provide some ideas on how to read / interpret the feed.

I don’t know all the details, but I hope this helps a bit.

-Andrew

--
You received this message because you are subscribed to the Google Groups "GTFS-realtime" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gtfs-realtim...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gtfs-realtime/7a72a75e-0118-40b2-94cf-1e45eaad9212%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tony Laidig]

The NSW gtfs-rt feeds use a particular header Authenticaton which most other feeds don’t use. Have you accounted for that?

[Sean Barbeau]

tl;dr - if you add REPLACEMENT = 5 to the TripDescriptior.ScheduleRelationship entity in your .proto file and re-genereate your code to read GTFS-rt it should work.

It looks like this is a result of the ScheduleRelationship REPLACEMENT values being removed from the GTFS-rt spec in 2014:

https://groups.google.com/forum/#!topic/gtfs-realtime/77c3WZrGBnI

Instead of being removed from the .proto file, it should have been left in the .proto and marked as deprecated for backwards compatibility as described here:

https://github.com/google/transit/issues/109#issuecomment-442476116

The NSW feeds must still be using an older version of the .proto file for their producing code.

Sean

[jwalker]

Thanks for your help everyone.

I noticed that some people were talking about NSW data in the replies. I am actually trying to process translink v1.0 data from QLD - which is giving me problems.

Since I am using javascript/nodejs, I installed all packages via npm (i.e. npm install gtfs-realtime-bindings), I wasn't quite sure which .proto file to edit and how to regenerate the code. So I just modified the code in gtfs-realtime.js and appended:

{

"name": "REPLACEMENT",

"id": 5

}

to the TripDescriptior.ScheduleRelationship section. It works now. Thanks for the input on this everyone!

I do however also get the following error on occasion:

===============

C:\gtfs\gtfs_realtime\node_modules\protobufjs\ProtoBuf.js:1987

                            throw(e);

                            ^

Error: Missing at least one required field for Message .transit_realtime.FeedMessage: header

===============

My feel is that this may indicate that it is trying to process garbage data at times? Maybe the quickest way to fix this is to catch the error in my code and just ignore it.

Cheers.

[Ken Tsang]

How frequently are you hitting the server?  Perhaps it's rate limiting you and returning an error instead?

Might be worth catching that exception and storing a copy of the pbf that comes through.

[j.walker]

I'm hitting the server every 5 sec. I'm guessing that is a little excessive? I'm finding it mostly occurs when I console.log() data like lat/long etc.

I'll do as you suggested and capture the pbf that causes the error and investigate further when I get back into work. Thanks again for the suggestion.

[Ken Tsang]

Yeah, you can probably reduce the frequency. The SEQ and CNS feeds only update every ~10-15s.

The TransLink Google Groups might be the best forum to discuss issues specific with that feed: https://groups.google.com/forum/#!categories/translink-australia-opendata/gtfs-and-gtfs-rt

[j.walker]

On Sunday, 10 February 2019 19:56:48 UTC+10, Ken Tsang wrote:

Yeah, you can probably reduce the frequency. The SEQ and CNS feeds only update every ~10-15s.

Ok thanks for that, I'll reduce it to 10s and see how I go. Much appreciate all the help!