Announcing gtfs-realtime-bindings: making GTFS-realtime easier for developers

[Brian Ferris]

tl;dr gtfs-realtime-bindings is an open-source collection of pre-built language bindings for parsing GTFS-realtime data in many common programming languages.  I'd like to make it an official part of the GTFS-realtime documentation site.

A major stumbling block for many developers when first getting started with GTFS-realtime is generating classes for their favorite programming language from the gtfs-realtime.proto protocol buffer schema.  After seeing the same questions repeated on developer mailing lists and StackOverflow posts time and time again, I decided to do something about it.

I've put together the gtfs-realtime-bindings open-source project, which provides pre-built classes generated from gtfs-realtime.proto for the following languages:

.NET

Java

JavaScript / Node.js

Python

Ruby

What's more, those classes have been packaged and published to the appropriate module repository for each language.  I've also put together "Hello world!" code-snippets for each language showing how to get started.  Hopefully, these efforts will make it super easy for developers to get started with GTFS-realtime.  Try out the modules and the getting-started code snippets in your favorite language and let me know how it goes!

So, all well and good, but now a question for the GTFS-realtime community...

I'd like to propose that these bindings become "officially" part of the GTFS-realtime spec page, in that they would be mentioned explicitly in the documentation on how to use GTFS-realtime, possibly with a page dedicated to a quick-start tutorial.  The bindings would always be kept in sync with the latest version of gtfs-realtime.proto on the site.

I know there are other GTFS-realtime class packages out there, but I think the community would benefit from having one official set of libraries to start from.  Developers are of course free to generate their own bindings as needed (for extensions, different languages, etc) but hopefully this would meet the default needs of most developers.  And of course, I'm happy to open up access to the project for contributions as the community sees fit.

Thoughts?

Brian

P.S. "Why isn't my favorite language included?!"

C++, Objective C, PHP... there are a few popular programming languages that I skipped because either (a) it wasn't clear that enough people are parsing GTFS-realtime using these languages or (b) it wasn't clear how I could package the code into an easily reusable library.  If you feel differently, let me know and we'll see what we can do.

[Stefan de Konink]

If you suggest to open a ticket, it would be handig to have issues enabled
on the repository.

Stefan

[Brian Ferris]

I didn't realize that wasn't enabled by default.  I don't think it's possible to write code without issues of some kind ;)

[Sean Barbeau]

Thanks Brian, this is great!

I'd like to propose that these bindings become "officially" part of the GTFS-realtime spec page, in that they would be mentioned explicitly in the documentation on how to use GTFS-realtime, possibly with a page dedicated to a quick-start tutorial.  The bindings would always be kept in sync with the latest version of gtfs-realtime.proto on the site.

+1

It's great that this is on Github as well.  Do you have any thoughts about how Github issues/pull requests factor into the proposal and implementation of new/experimental fields?  For example, experimental fields could exist in a WIP pull request until they are officially accepted, at which point they would be merged.  Would make it easier for others to test new features, and there would be a canonical implementation of the proposal to test with.

Sean

[Brian Ferris]

I had not considered taking it that far.  I'll admit that tracking proposal text via email can get a little murky.  That said, I think I prefer email for extended discussion of proposals.  So short answer is I don't know ; ) but always curious what others think.

--
You received this message because you are subscribed to the Google Groups "GTFS-realtime" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gtfs-realtim...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gtfs-realtime/8d3d6749-0a4a-4dc3-9554-69ec50d369e9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Brian Ferris]

Just wanted to follow up to see if anyone had any concerns about my original proposal:

I'd like to propose that these bindings become "officially" part of the GTFS-realtime spec page, in that they would be mentioned explicitly in the documentation on how to use GTFS-realtime, possibly with a page dedicated to a quick-start tutorial.  The bindings would always be kept in sync with the latest version of gtfs-realtime.proto on the site.

I know Sean gave a +1 so I will probably move forward with this unless I hear an opposing view.

Brian

On Fri Jan 23 2015 at 7:50:26 PM Brian Ferris <bdfe...@google.com> wrote:

I had not considered taking it that far.  I'll admit that tracking proposal text via email can get a little murky.  That said, I think I prefer email for extended discussion of proposals.  So short answer is I don't know ; ) but always curious what others think.

On Fri Jan 23 2015 at 10:42:13 AM Sean Barbeau <bar...@cutr.usf.edu> wrote:

Thanks Brian, this is great!

I'd like to propose that these bindings become "officially" part of the GTFS-realtime spec page, in that they would be mentioned explicitly in the documentation on how to use GTFS-realtime, possibly with a page dedicated to a quick-start tutorial.  The bindings would always be kept in sync with the latest version of gtfs-realtime.proto on the site.

+1

It's great that this is on Github as well.  Do you have any thoughts about how Github issues/pull requests factor into the proposal and implementation of new/experimental fields?  For example, experimental fields could exist in a WIP pull request until they are officially accepted, at which point they would be merged.  Would make it easier for others to test new features, and there would be a canonical implementation of the proposal to test with.

Sean

On Friday, January 23, 2015 at 10:45:51 AM UTC-5, Brian Ferris wrote:

I didn't realize that wasn't enabled by default.  I don't think it's possible to write code without issues of some kind ;)

On Fri Jan 23 2015 at 12:34:47 AM Stefan de Konink <ste...@konink.de> wrote:

If you suggest to open a ticket, it would be handig to have issues enabled
on the repository.

Stefan

--
You received this message because you are subscribed to the Google Groups "GTFS-realtime" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gtfs-realtime+unsubscribe@googlegroups.com.

[andrew byrd]

On Fri, Jan 23, 2015, at 09:03, 'Brian Ferris' via GTFS-realtime wrote:
> I'd like to propose that these bindings become "officially" part of
> the GTFS-realtime

> spec page <https://developers.google.com/transit/gtfs-realtime/>, in that

> they would be mentioned explicitly in the documentation on how to use
> GTFS-realtime, possibly with a page dedicated to a quick-start tutorial.
> The bindings would always be kept in sync with the latest version of
> gtfs-realtime.proto on the site.

+1. And the GTFS-RT spec page should definitely mention that the code
for all these different languages is auto-generated from the same
protobuf specification to help people grasp how this works.

Thanks for your work on this. This should help people jump in and begin
working with GTFS-RT.

-Andrew

[Sean Barbeau]

One item that hasn't been discussed, but I think is worth explicitly mentioning, is the topic of versioning.  If production projects are going to depend on this library, it will be very important to have a clear versioning policy.

From a quick look at the code, looks like Brian has been good at cutting new releases of the bindings each time the protobuf specification is updated.  This is definitely critical.

Given that breaking changes may be introduced into future versions of the library (e.g., experimental fields that get moved/renamed/removed), I think its also important to document a summary of the changes for each release in release notes, especially for experimental fields.  Additionally, it would be nice to see an explicit reference to the version of the protobuf spec that was used in these same release notes (though, granted, you can dig through the code/Git history to find this).

On this note - one concern I have is that the FeedHeader.gtfs_realtime_version seems to be stuck at 1.0, despite a few changes to the spec (and bindings).  IIRC, these changes have all been experimental fields (OccupancyStatus and TripUpdate.delay). 

Are the experimental field changes intentionally not reflected in gtfs_realtime_version? 

Sean

[Brian Ferris]

Versioning is a little tough.  For the most part, we try to document every change to the spec at https://developers.google.com/transit/gtfs-realtime/changes#RevisionHistory  Like GTFS before it, we've also tried to avoid backwards incompatible changes to the official spec (can we avoid this forever?  I don't know).  However, it certainly helpful to track which instance of gtfs-realtime.proto you've built against.  We could keep it simple and just the use last-revision date.

But I've struggled with how to encode that in the version numbers of the gtfs-realtime-bindings releases.  Each language (and it's corresponding package repo) has it's versioning kinks.  What's more, I can't guarantee I won't make mistakes during the release process that won't require incremental releases independent of spec changes (notice how the Ruby module is already out of step with the others).

Since each module is version X.Y.Z, we could try something like:

X = 1 until we have some criticial reason to change it

Y = YYYYMMDD of the GTFS-realtime proto version

Z = patch version for when we need to fix bugs with a release unrelated to the underlying proto

Thoughts?

--

You received this message because you are subscribed to the Google Groups "GTFS-realtime" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gtfs-realtime+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gtfs-realtime/898788bf-2f3e-48ef-a808-7ea2e74b1851%40googlegroups.com.

[Ivan Egorov]

I don't believe in any versioning being beneficial here besides got commit. These files only define bindings to protobuf, that is syntax not semantics. Given pb's code and binary backward and forward compatibility (when not reusing tags), there is no need in explicit version. Any version of the file in this repository will be able to parse any version of binary representation.

I don't have opinion on gtfs_realtime_version field apart from that we ignore it completely as feeds' consumer at the moment.

To unsubscribe from this group and stop receiving emails from it, send an email to gtfs-realtim...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gtfs-realtime/CAG9YwWD%3D4YaqnE_fFGHeSaFTTOkY2n-qEpsTqVAsQAU2SsFOKQ%40mail.gmail.com.

[Brian Ferris]

Ok, I've added a "Code Samples" section to the GTFS-realtime site:

https://developers.google.com/transit/gtfs-realtime/code-samples

I'd appreciate any feedback you might have on the new page.

Brian

--
You received this message because you are subscribed to the Google Groups "GTFS-realtime" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gtfs-realtime+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gtfs-realtime/1422539137.634903.220482561.524F7D7D%40webmail.messagingengine.com.

[Brian Ferris]

Now with PHP support:

https://developers.google.com/transit/gtfs-realtime/code-samples#php

This is the first PHP I've ever written ;)

[Mathieu Méa]

Hi Brian,

Any way to have Java binding that doesn't depend on Protocol Buffers, especially for Android?

The 0.5 MB jar file is not that big but it feels like a big overhead just to read a web service... don't need a library to read XML/JSON feeds.

Maybe some advice on how to use Proguard to remove the useless stuff from the jar?

Thanks

Mathieu Méa (MonTransit)

[Brian Ferris]

GTFS-realtime data is encoded using Protocol Buffers, so tricky to remove the dependency ;)

I think it's worth noting that GTFS-realtime, like GTFS, is a raw firehose data format that is optimized for use by a backend-server, not a mobile app.  The typical usage pattern is to have your own server that consumes GTFS and GTFS-realtime data and then provides an optimized API to your mobile application tailored to your application needs.  For example, this is how the OneBusAway backend-server and mobile apps works.

That said, if you really want GTFS-realtime on mobile, there are alternative Protocol Buffer implementations optimized for Android (just one example from LMGTFY: https://github.com/square/wire).

--
You received this message because you are subscribed to the Google Groups "GTFS-realtime" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gtfs-realtim...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gtfs-realtime/0b3f1c82-4607-48b9-b6ae-2347370a468d%40googlegroups.com.

[Mathieu Méa]

Thanks for pointing out alternative Protocol Buffer implementations, I'll look into it.

I currently don't have any backend-server as my app is not doing anything that can't be done on today's mobile devices (with multi-core CPU and GB(s) of RAM).