GTFS RT Newbie - Unexpected end-group tag error

[Emma]

Hello,

I am new to GTFS-realtime and after struggling for a while with different url libraries, I finally managed to open a .proto file (uploaded on Google Drive) which seemed ok to me, but I keep getting this error :

RuntimeWarning: Unexpected end-group tag: Not all data was converted
at line feed.ParseFromString(response.content)

First I got it with the test file I wrote myself, so I thought I might have had done it wrong.

Thing is, I then basically copy-pasted the sample code from the Service Alert Example provided by Google (as in the code below), and got the very same error.

I am working with Python 3.5 on Windows (though I don't think this will be very useful in my case).

Can anyone help me ?

Thanks a lot !

My code :

from google.transit import gtfs_realtime_pb2
import requests

feed = gtfs_realtime_pb2.FeedMessage()
response = requests.get('https://drive.google.com/open?id=155uHbnQINCh_h-xOlrW1w_WoO7apAilG')     # uploaded on my own Drive for testing
feed.ParseFromString(response.content)      # the line that throws the error

for entity in feed.entity:
  if entity.HasField('trip_update'):
    print(entity.trip_update)

[Nikhil VJ]

Hi Emma,

The google drive link is not direct : it doesn't give the actual data, instead it serves up the same HTML page that you see when you click on it. If you want to access an online .pb file, it needs to be directly accessible, no middle download pages.

Instead I found this way to read a .pb file from local disk:

with open('VehiclePositions_93.pb','rb') as f:
   buff = f.read()
feed.ParseFromString(buff) # so buff is in place of 'response.content' in your code

And I got that after searching online for protocol buffer handling, which is a broader field than GTFS itself.

Regards

Nikhil VJ, 

Pune, India

[Emma]

Hello Nikhil,

Thank you very much for your answer, I now understand what was the issue. I tried your solution, it indeed doesn't throw the error I got first, but instead I get this one :

google.protobuf.message.DecodeError: Error parsing message on the feed.ParseFromString(buff)line.

Since I get a different error when typing a wrong path or filling in a non-existing file path, I believe the program managed to access my file ; however, it still cannot parse it. Is this a problem with the file I am using ? It is named exp.proto (not .pb as in your example, but maybe it doesn't matter ?) and contains only the copy-paster of the example I mentioned earlier.

Thanks in advance,

Emma

[Nikhil VJ]

Hi Emma,

Ah! It does matter! .proto isn't the protocol buffer file holding the data, it's a template.

If you want to open up ANY .pb file, grab one from here. No api, login etc needed.

https://mbta.com/developers/gtfs-realtime

----------

It took me a long time to wrap my head around this process too, will try to lay it out..

1. .proto files are human-written templates that define how we're going to structure our data.

2.  from that, in python, a "xyz_pb2.py" library / module is created which acts like a machine-made template. There's no data in there. It's just instructing how to work with the data. You can say it's the "compiled" version of the template.

3. When we write "from google.transit import gtfs_realtime_pb2" at the top, we're including the GTFS-RT's protocol buffer template. At this point, the template we need, to read and write GTFS-RT feeds, is already loaded. So no need to bother with .proto etc now.

4. So our program is now using the GTFS-RT protocol buffer template to read and write .pb files, which hold the data.

I got to learn this stuff from outside the GTFS-RT world. This article explaining protocol buffers helped: 

https://www.datadoghq.com/blog/engineering/protobuf-parsing-in-python/ 

Regards,

Nikhil VJ

Pune, India

[Emma]

Hello Nikhil,

Thank you, I think I got it ! I indeed thought it worked kind of similarly to XML, but I visited your link and now I understand why it wasn't working.

Thanks again for your help !

Best regards,

Emma