Proposal: Make FeedHeader.timestamp a required field

[Sean Barbeau]

Hi all,
Inspired by recent discussions, I'd like to take a shot at tightening up the GTFS-rt spec a bit.  This one seems like low-hanging fruit.

Currently, FeedHeader.timestamp is an optional field.  Given that this is "the moment when the content of this feed has been created (in server time)", its a critical piece of information for consumers.

Can we agree that FeedHeader.timestamp should be required, instead of optional?

Sean

[Jorden Verwer]

Personally, I don't see the need. I'm not against tightening things up, but I don't see this as an issue that needs fixing. I don't think we use the timestamp field for anything here at GoAbout.

Regards,

Jorden

[Ivan Egorov]

Sorry, this should never happen. In fact, non of fields should have been set up required in the first place. The reason for using "optional" (rather than "required") in the protocol buffer definition is to permit better forward-and-backward compatibility to changes in the protocol buffer definition, as "required" fields lead to incompatibilities that cannot be overcome at the application level.

[andrew byrd]

On Fri, Jan 16, 2015, at 10:06, Ivan Egorov wrote:
> Sorry, this should never happen. In fact, non of fields should have been
> set up required in the first place.

This is a good point and seems to be a valid point of view. Cap'n Proto,
which is in some ways a successor to Protobuf, does not allow required
fields:

http://kentonv.github.io/capnproto/faq.html#how-do-i-make-a-field-required-like-in-protocol-buffers.

"the 'required' keyword in Protocol Buffers turned out to be a horrible
mistake."

-Andrew

[Sean Barbeau]

Sorry, this should never happen. In fact, non of fields should have been set up required in the first place. The reason for using "optional" (rather than "required") in the protocol buffer definition is to permit better forward-and-backward compatibility to changes in the protocol buffer definition, as "required" fields lead to incompatibilities that cannot be overcome at the application level.

Ok, that's definitely a valid point.  So, the cardinality field in the GTFS-rt spec is driven by the protobuf implementation.  Which means, going forward, we will never have any "required" fields.

So, it seems a better solution is stated at the bottom of the link that Andrew shared:

The right answer is for applications to do validation as-needed in application-level code. If you want to detect when a client fails to set a particular field, give the field an invalid default value and then check for that value on the server.

We should have an "official" open-source GTFS-rt validation tool that checks feeds and verifies at the application level whether or not particular feed is populated correctly, based on some set of rules that we define.  Is such a tool planned by Google?  And, how do we document in the spec what the failure and pass conditions are, especially if we can't/shouldn't change the "cardinality" field?

Some background for my interest in tightening up the GTFS-rt spec - I've been having discussions with transit agencies, vendors, and other application developers about setting up GTFS-rt feeds.  The spec is currently being viewed as an application-level spec that defines what should be in a GTFS-rt feed.  With almost all fields being optional, it's very easy for a vendor to say they are compliant with the spec and drop fields that they really should be populating.  This leads to sub-standard GTFS-rt feeds.  It also leads to disagreements between producers and consumers in what should be in a GTFS-rt feed, with agencies stuck somewhere in the middle.  Better tools and documentation that are agreed upon within the GTFS-rt community seem like the best way to fix this.

[Joachim Pfeiffer]

"The spec is currently being viewed as an application-level spec that defines what should be in a GTFS-rt feed.  With almost all fields being optional, it's very easy for a vendor to say they are compliant with the spec and drop fields that they really should be populating.  This leads to sub-standard GTFS-rt feeds.  It also leads to disagreements between producers and consumers in what should be in a GTFS-rt feed, with agencies stuck somewhere in the middle. "

CAD vendors are up in the air as well. If the requirements of the contract they have bid on state "compliance", that's what they will provide. You have the right idea though: The requirements need to be upped to "compatibility". This necessitates a validator that can be used to test the vendor product all the way through to acceptance testing. With that established as a requirement, agencies and operators can provide a level playing field and vendors will happilly bid on that and roll it out. Right now the situation reminds of where TCIP once stood - everybody can populate fields "optionally" and be "compliant".

--
You received this message because you are subscribed to the Google Groups "GTFS-realtime" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gtfs-realtim...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gtfs-realtime/896a8eee-b76a-4085-9fc8-dc2cd1e5dfd2%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Eric Andresen]

These are all very valid points, thank you for bringing it up. I'll see if we can't propose some replacement wording to describe fields in terms of their _semantic_ cardinality, rather than their protobuffer-defined one. Watch this space :-)

To view this discussion on the web visit https://groups.google.com/d/msgid/gtfs-realtime/CACVRw0MgyQcfH11J6xOxEwQcjQDnwV-i3JLhPw48fWJ%2B0WpVKg%40mail.gmail.com.

[Sirinya Matute]

Hello – Sirinya Matute writing here from Big Blue Bus in Santa Monica, Calif. I am a marketer and not a developer. I will share a concern of mine here. There are some vendors who very much embrace and want to enable their clients to participate fully in pushing out whatever data they have per the spec. Those vendors embody the joys and ideals of transit as a mode. And then there are those who literally will do the very bare minimum. I don’t know enough to advise this group on how to close loopholes. It sounds like some people here are flagging the risks of a loophole with optional fields. I am trying to release a feed of my agency’s data in the gtfs-rt standard using an add-on module to my cad/avl system. I don’t fret that I will not have any muscle to ensure that module continues to push data out as the spec evolves -- if the spec is mostly comprised of ‘optional’ fields.

 

I am curious as to why there has been limited/no adoption of TCIP, which I learned of this past weekend at Transportation Camp (remember, I’m not a developer…these things are cool to me but all new). That would be appropriate for an off-line discussion or on a different group. If you could message me privately, I would be most appreciative.

 

Gratefully,

Sirinya Matute

Santa Monica Big Blue Bus

Sirinya.matute at smgov. net

To view this discussion on the web visit https://groups.google.com/d/msgid/gtfs-realtime/CACVRw0MgyQcfH11J6xOxEwQcjQDnwV-i3JLhPw48fWJ%2B0WpVKg%40mail.gmail.com.

[Barbeau, Sean]

Thanks, that would be great! J

 

Sean

 

From: gtfs-r...@googlegroups.com [mailto:gtfs-r...@googlegroups.com]
Sent: Friday, January 16, 2015 12:24 PM
To: gtfs-r...@googlegroups.com
Subject: Re: [GTFS-realtime] Re: Proposal: Make FeedHeader.timestamp a required field

 

These are all very valid points, thank you for bringing it up. I'll see if we can't propose some replacement wording to describe fields in terms of their _semantic_ cardinality, rather than their protobuffer-defined one. Watch this space :-)

--
You received this message because you are subscribed to a topic in the Google Groups "GTFS-realtime" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/gtfs-realtime/wm3W7QIEZ9Y/unsubscribe.
To unsubscribe from this group and all its topics, send an email to gtfs-realtim...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gtfs-realtime/CAFLeBcGdyj%3DciHzDhSo48tWuuFu8YsVsgidBWnqu3uZ0mAqEvg%40mail.gmail.com.

[Tony Laidig]

Hey Sirinya,
As for your question about TCIP, I think it's a good topic but this is not the place. If you'd like to start a discussion over at transit-developers, I'd be happy to chime in with my misgivings about the standard, how I've worked around them, and how we've used it in NYC.

Tony

[Sirinya Matute]

Hi Tony – I’m sorry to be mass replying to the group. Tony, please feel free to message me off line.

Sirinya...@smgov.net.

To the others, apologies.

To view this discussion on the web visit https://groups.google.com/d/msgid/gtfs-realtime/48857020-194a-4ab5-9636-378239b6fc86%40googlegroups.com.

[Sean Barbeau]

These are all very valid points, thank you for bringing it up. I'll see if we can't propose some replacement wording to describe fields in terms of their _semantic_ cardinality, rather than their protobuffer-defined one. Watch this space :-)

Eric,
Any further movement on the above?

There are a few discussions going on about GTFS-rt validators (including a possible GSoC project), and it would be good to have official Google input on semantic cardinality.

Thanks,
Sean

[Sean Barbeau]

Eric,
Just pinging again on this - any progress towards outlining semantic cardinality of GTFS-rt fields?  Is this something you'd like me or others to propose?

Sean

[Ivan Egorov]

Hi Sean,

Don't jump on this just yet. This proposal requires significant update to the specification overall, and we'd like to freeze it for couple more weeks while we work out a more convenient editorial process (teaser: we are likely to export the specification to a github repository).

Ivan.

[Ivan Egorov]

Sean,

Feel free to jump on it now :)

Ivan.

[Barbeau, Sean]

Ivan,
Ok, thanks. I guess we'll tackle semantic cardinality field-by-field, then, each in separate PRs?

Sean

-----Original Message-----
From: gtfs-r...@googlegroups.com [mailto:gtfs-r...@googlegroups.com] On Behalf Of Ivan Egorov
Sent: Wednesday, July 29, 2015 4:32 PM
To: GTFS-realtime
Subject: Re: [GTFS-realtime] Re: Proposal: Make FeedHeader.timestamp a required field

--
You received this message because you are subscribed to a topic in the Google Groups "GTFS-realtime" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/gtfs-realtime/wm3W7QIEZ9Y/unsubscribe.
To unsubscribe from this group and all its topics, send an email to gtfs-realtim...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gtfs-realtime/ba22d79e-b17b-4297-b34d-aafa79760a5f%40googlegroups.com.

[Ivan Egorov]

I'd create a single PR related to one or a couple of fields and from there draw a conclusion whether we want to extend it to the full spec or (more probably) to start a series of PRs.

[Barbeau, Sean]

Ok, sounds good.

Sean

-----Original Message-----
From: gtfs-r...@googlegroups.com [mailto:gtfs-r...@googlegroups.com] On Behalf Of Ivan Egorov

--
You received this message because you are subscribed to a topic in the Google Groups "GTFS-realtime" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/gtfs-realtime/wm3W7QIEZ9Y/unsubscribe.
To unsubscribe from this group and all its topics, send an email to gtfs-realtim...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gtfs-realtime/1cc71424-c613-4dfc-a8ac-312c55d60d81%40googlegroups.com.