GTFS RT Extension ID request

[david....@mtahq.org]

LIRR would like to add track to the StopTimeUpdate in the form of

message LirrStopTimeUpdate {

  optional string track = 1;

  //can add additional fields here without having to

  //extend StopTimeUpdate again

}

You may contact me at David....@mtahq.org for any inquires regarding this request.

Thank you,

David

[Stefan de Konink]

Isn't this already possible in the default GTFS-RT messaging?

--
Stefan

[Jym Dyer]

>> message LirrStopTimeUpdate {
>> optional string track = 1;
>> //can add additional fields here without having to
>> //extend StopTimeUpdate again
>> }

> Isn't this already possible in the default GTFS-RT messaging?

Not as far as I can tell. Indeed, Metro-North has requested
and received extension ID 1005 for exactly the same field:

https://groups.google.com/d/msg/gtfs-realtime/_sEV6_60AsM/wPN7hB6UxfEJ

I know of some other agencies who've wanted this as well,
either track IDs or platform IDs, and if we go all the way
back to extension ID 1001, the NYC Subway, there is the
concept of scheduled_track and actual_track:

message NyctStopTimeUpdate {
optional string scheduled_track = 1;
optional string actual_track = 2;
}

Maybe when the dust settles on GTFS-realtime v2.0 we can think
about adding these oft-requested fields?
<_Jym_>

[Stefan de Konink]

On Tuesday, October 3, 2017 8:39:24 PM CEST, Jym Dyer wrote:
>>> message LirrStopTimeUpdate {
>>> optional string track = 1;
>>> //can add additional fields here without having to
>>> //extend StopTimeUpdate again
>>> }
>
>> Isn't this already possible in the default GTFS-RT messaging?
>
> Not as far as I can tell. Indeed, Metro-North has requested
> and received extension ID 1005 for exactly the same field:

Google allows to publish the stop-id for the actual track (as track
change), as long this stop-id is part of the same parent stop.

--
Stefan

[Rachael Morgan]

Sounds like there's demand for this - should we just make it part of the spec?

[Stefan de Konink]

On donderdag 5 oktober 2017 04:14:27 CEST, 'Rachael Morgan' via

GTFS-realtime wrote:
> Sounds like there's demand for this - should we just make it
> part of the spec?

Isn't it already ;) Or should we document how users should do it?

--
Stefan

[Stefan de Konink]

To be clear: Google Maps accepts publishing track changes using a different
stop_id, given that that stop is under the same parent_stop. We publish
this for The Netherlands in production.

--
Stefan

[David....@mtahq.org]

Understood. however at the present, that approach is at best problematic for us implement.

We are under a bit of a time squeeze as well to have it available for internal use.

[Stefan de Konink]

On donderdag 19 oktober 2017 14:21:22 CEST, David....@mtahq.org wrote:
> Understood. however at the present, that approach is at best
> problematic for us implement.
> We are under a bit of a time squeeze as well to have it
> available for internal use.

How can it be *more* problematic to implement an existing field which is
already exported in both GTFS and GTFS-RT, than introducing a complete new
field without any relationship to the scheduled data?

--
Stefan

[David....@mtahq.org]

Our stops contain stop_id of our stations. We do not (yet) have a data source of all tracks at each station. If the stop_id of the record specifying parent_station requires a matching entry in stops then presently we would not be able to fulfill that. To create a source data set that could populate stops with the supplemented track would involve resources that would exceed the time necessary to have the changes in place by. 

Using an extension, I can quickly create the class extension needed to generate the custom Protobuf class files and populate the track_id value by modifying our existing application.

[Stefan de Konink]

On donderdag 19 oktober 2017 14:46:24 CEST, David....@mtahq.org wrote:
> Our stops contain stop_id of our stations. We do not (yet) have
> a data source of all tracks at each station. If the stop_id of
> the record specifying parent_station requires a matching entry
> in stops then presently we would not be able to fulfill that. To
> create a source data set that could populate stops with the
> supplemented track would involve resources that would exceed the
> time necessary to have the changes in place by.
> Using an extension, I can quickly create the class extension
> needed to generate the custom Protobuf class files and populate
> the track_id value by modifying our existing application.

Consider you do have the knowledge about the number of tracks at a station
[this should be easy to collect from your current realtime], but don't know
the exact positions, (or: how they are scheduled), you could:

- GTFS: duplicate the stop_id into stop_id:0

- GTFS: rename stop_id in stop_times.txt to stop_id:0

- GTFS: for the original stop_id set the location_type = 2

- GTFS: for each station generate duplicated stops for the highest
platform code you have observed,
parent_station = stop_id
stop_id = stop_id:platform_code
platform_code = platform_code
location_type = 0

(keep the coordinates the same, until you fix that in a later
release, ignore the warnings they are at the same location)

- GTFS-RT: for every realtime track available in your system you
will send stop_id:platform_code as its stop_id.


In five steps you have solved the entire problem in a stateless way all
journey planners understand.

--
Stefan

[Dave Barker, MBTA]

Although Google allows the stop_id in a StopTimeUpdate to be a different child stop from what had been specified in GTFS, the GTFS-realtime .proto doesn't say anything about that. So I would be concerned that taking that approach might break some other feed consumers, who would assume that either using a different stop _id was not allowed (which had been my read) or make a different assumption about how to treat a stop_id that appears in a StopTimeUpdate but doesn't have a corresponding entry in stop_times (and how to treat the corresponding stop_id that appears in stop_times but has no StopTimeUpdate). 

I think it would be a cleaner model to introduce the assigned track as a new field anyway. This way the StopTimeUpdate can more easily be matched by consumers; there is no ambiguity in situations where a trip has multiple stop_times with different stop_id's that share the same parent*; and it draws attention to the fact that this track assignment is distinct additional information. 

-Dave Barker, MBTA

* i.e. a loop route that begins and ends at the same station, but different platforms.

[Stefan de Konink]

On donderdag 19 oktober 2017 22:16:11 CEST, Dave Barker, MBTA wrote:
> Although Google allows the stop_id in a StopTimeUpdate to be a
> different child stop from what had been specified in GTFS, the
> GTFS-realtime .proto doesn't say anything about that.

How would you ever communicate a deviation of the pattern to a consumer the
standard would not allow to enter a different stop_id sequence. Please
apply some common sense. The stop_id must exist in stops.txt, that is the
only requirement from the GTFS-RT standard. Google Transit narrows this
down and requires the changes of the stop_id to fall under the same parent
station (for algoritmic reasons concerning transfers). Other
implementations OpenTripPlanner, RRRR, do not have this requirement and
always accept arbitrary stop sequences as communicated via StopTimeUpdates.

> I think it would be a cleaner model to introduce the assigned
> track as a new field anyway.

We have a field for that, it is called platform_code, and should be
specified in stops.txt.

https://developers.google.com/transit/gtfs/reference/gtfs-extensions#stopstxt_1

The rest of your arguments do not make any sense. If a producer fails to
understand it must provide a stop_id for each platform lets fix that
understandig first. Because that producer would also fail to understand
that for a different track a different transfer time would apply.

--
Stefan

[Sean Barbeau]

So I would be concerned that taking that approach might break some other feed consumers, who would assume that either using a different stop _id was not allowed (which had been my read) or make a different assumption about how to treat a stop_id that appears in a StopTimeUpdate but doesn't have a corresponding entry in stop_times (and how to treat the corresponding stop_id that appears in stop_times but has no StopTimeUpdate).

I agree with Dave - there is definitely a number of parties (including myself) that assume right now the GTFS-realtime spec doesn't allow a stop_sequence/stop_id pairing that differs from the GTFS data.  We actually had a discussion in the last few days about the larger issues of changing trip geometry on-the-fly, and IMHO this should be part of that change.

Also, please keep the discussion civil - we're all working towards common goals here and we'll get to those faster if we respect others comments/proposals.

Sean

[Stefan de Konink]

On Friday, October 20, 2017 6:59:42 PM CEST, Sean Barbeau wrote:
> I agree with Dave - there is definitely a number of parties
> (including myself) that assume right now the GTFS-realtime spec
> doesn't allow a stop_sequence/stop_id pairing that differs from
> the GTFS data.

I'm sorry, but that is nowhere in the spec mandatory. Not even an
assumption that sequences must align, the requirements are clear the stop
must be present in stops.txt and the StopTimeUpdates should be provided in
sequence; No change required in the standard. Especially since major
realtime consumers have implemented it in the correct way.

And if you even want to disagree on that, take a peak at the code
facilitating the original ScheduleRelationship added, replacement and
unscheduled.

<http://developer.onebusaway.org/modules/onebusaway-gtfs-realtime-api/1.1.1/apidocs/com/google/transit/realtime/GtfsRealtime.TripDescriptor.ScheduleRelationship.html#ADDED>

> We actually had a discussion in the last few
> days about the larger issues of changing trip geometry
> on-the-fly, and IMHO this should be part of that change.

shapes.txt are currently not part of GTFS-RT, hence it is a completely
different subject and should not confuse the reader in this thread with
arguments that are not related.

> Also, please keep the discussion civil - we're all working
> towards common goals here and we'll get to those faster if we
> respect others comments/proposals.

A discussion is acted on relateble facts. People, including you, don't get
'right' if they assert unrelated issues. The proposal here is from my
perspective so off that some vendors want to push their inability to play
by the current standard and create forks, even when they are sincerely
helped with an implementation guidelines and best practices.

If we go into the discussion, nobody countered the argument I made on the
platform_code. Hence solely on that I would be against any proposal that
includes track changes bypassing that field. Because if you want to
disagree on the platform_code extension, we should have a different
discussion on a different mailinglist.

--
Stefan

[Ken Tsang]

I don't think the two ideas are mutually exclusive.  I think there's a case to be made for improving the way track changes are conveyed.

As Stefan said, some operators currently implicitly convey platform/track changes by simply changing the stop_id with one sharing the same parent_station.

Introducing a new, optional field to convey what the scheduled stop_id and/or stop_sequence (e.g. scheduled_stop_id and scheduled_stop_sequence) allows feed consumers to link the stop variations to static GTFS in a more concrete manner, especially in loop scenarios.

Since we'd specify the scheduled stop_id/stop_sequence in a new field, this will not break any existing implementations which already caters for parent_station linking.

The challenge in loop scenarios (and imo, the hole in the spec today) is that operators aren't required to list all stops and can omit the stop_sequence.  Once the trip passes the first of the two stops which share the same parent_station, the matching to static GTFS is ambiguous without state-based logic to determine if a stop has already been passed.

And why link to static GTFS in the first place? Because not all information is conveyed through GTFS realtime.  Simple examples are: does the service pick up / set down at this stop?  What's the stop_headsign?  IMO, these (and all other GTFS corresponding static fields) should also be able to be optionally conveyed in GTFS-realtime for service change and addition purposes.  That's a discussion for another thread though.

--

Ken

[Stefan de Konink]

On maandag 23 oktober 2017 15:53:12 CEST, Ken Tsang wrote:
> Since we'd specify the scheduled stop_id/stop_sequence in a new
> field, this will not break any existing implementations which
> already caters for parent_station linking.

But my problem at this point is; this will require multiple downstream
implementations that should handle multiple ways of describing the same
thing. Since we are here at GTFS(-RT) and not a bloated standard like NeTEx
and/or SIRI avoiding these mistakes are key.

> The challenge in loop scenarios (and imo, the hole in the spec
> today) is that operators aren't required to list all stops and
> can omit the stop_sequence. Once the trip passes the first of
> the two stops which share the same parent_station, the matching
> to static GTFS is ambiguous without state-based logic to
> determine if a stop has already been passed.

Maybe the best thing with the above is forcing all parts of a trip (incl.
sequence) to be communicated. But I would ike to have a separate issue on
this point.

> And why link to static GTFS in the first place? Because not all
> information is conveyed through GTFS realtime. Simple examples
> are: does the service pick up / set down at this stop? What's
> the stop_headsign? IMO, these (and all other GTFS corresponding
> static fields) should also be able to be optionally conveyed in
> GTFS-realtime for service change and addition purposes. That's
> a discussion for another thread though.

Agreed. But the 'why link' is also an algorithmic choice. When major
journey planners use preprocessing where overlapping service journey
patterns are key in their succes, claiming that "realtime is enough" is not
going to help (nor will it scale).

--
Stefan

[Sean Barbeau]

To clarify my earlier comment - when I said "this should be part of that change", I was suggesting that a separate proposal should cover the topic of referencing stop_id/stop_sequence pairs in GTFS-rt that don't appear in GTFS.  The current GTFS-rt spec is silent on this issue hence the different interpretations.  I agree with Ken that there can be improvements in how track changes are handled outside of the stop_id/stop_sequence pairing issue.

 People, including you, don't get 'right' if they assert unrelated issues.

Stefan - please take a deep breath :).  No one is saying you or I are right or wrong.  When the spec is silent on something we all have our own interpretations and opinions for the best way to handle the scenario based on our own experiences and implementations.  Hopefully we can talk through these different viewpoints and come to a consensus on the best way to collectively move forward.

Sean

[Stefan de Konink]

Hi Sean,

On Monday, October 23, 2017 6:06:54 PM CEST, Sean Barbeau wrote:
> Hopefully we can talk through these different
> viewpoints and come to a consensus on the best way to
> collectively move forward.

What would be your prososal to speed up this proces? Create separe issues
on github outlining them, or start different threads on this mailinglist?

--
Stefan

[Sean Barbeau]

David's original request here was for an extension ID, which is different from a proposal for adopting a new feature into OBA.

David - on this note, I believe MTA already has an extension of 1001?  See https://github.com/google/transit/blob/master/gtfs-realtime/CHANGES.md#extension-registry.  AFAIK extensions are specific to agencies, not features, so you could add the track under your existing extension ID.  However, you'd obviously need to get consumers to adopt it.

For addition/changes to the spec, we should follow the official process outlined at https://github.com/google/transit/blob/master/gtfs-realtime/CHANGES.md.

My personal preference would be to see pull requests with specific proposals for changes/additions instead of Github issues or discussions on this list, because it's easier to comment on features when you can see the diff in the pull request.

Sean

[Jym Dyer]

> David's original request here was for an extension ID,
> which is different from a proposal for adopting a new

> feature ...

I'm afraid my comments sidetracked this a bit, but yes, David
needs an extension ID well before new features can be hashed
out.

> David - on this note, I believe MTA already has an extension
> of 1001?

The 1001 extension is for the NYCT subway, and it introduces
two new fields for original track assignment and updated track
assignment. Metro-North also has extension 1005 to add exactly
the same "track" field that LIRR wants:

https://groups.google.com/d/msg/gtfs-realtime/_sEV6_60AsM/wPN7hB6UxfEJ

<_Jym_>

[Brenner, David M]

Thank you Sean and Jym,

Jym - you beat me to the punch there in replying to Sean regarding the extension ID assignments. :)

--
You received this message because you are subscribed to the Google Groups "GTFS-realtime" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gtfs-realtim...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gtfs-realtime/Jym.23Oct2017.59ee3ee9%40goswift.ly.
For more options, visit https://groups.google.com/d/optout.

[Andrew Byrd]

> On 23 Oct 2017, at 17:37, Stefan de Konink <ste...@konink.de> wrote:
>
> On maandag 23 oktober 2017 15:53:12 CEST, Ken Tsang wrote:
>> Since we'd specify the scheduled stop_id/stop_sequence in a new field, this will not break any existing implementations which already caters for parent_station linking.
>
> But my problem at this point is; this will require multiple downstream implementations that should handle multiple ways of describing the same thing. Since we are here at GTFS(-RT) and not a bloated standard like NeTEx and/or SIRI avoiding these mistakes are key.

I have not been able to follow all the detail of this discussion, but I do want to say I agree strongly with this point: the success of GTFS and GTFS-RT comes from its simplicity. Multiple standards already exist for describing every minute detail of transit operations, which often allow expressing the same thing in many different ways.

Most software will never be able to fully support these massive specifications. One of the advantages of GTFS and GTFS-RT is the ability of smaller / open source projects to fully implement the entire spec, and unambiguously know how and where certain pieces of information are expressed.

It is indeed key for the continued success of these formats to avoid uncontrolled growth with multiple ways of expressing the same thing.

Andrew

[David....@mtahq.org]

Sean, 

  Thank you. Approximately how long does it take to get an ID assignment?

[Sean Barbeau]

David,

Google would be the one to assign this if a new Extension ID is needed - it looks like it would be 1008 based on the current extension registry:

https://github.com/google/transit/blob/master/gtfs-realtime/CHANGES.md#extension-registry

Google can reply here to do that.

I haven't worked with extensions much myself, but my understanding is that the Extension ID is assigned once to the developer, and then the developer uses that Extension ID for all custom messages.  You can see the extisting MTA Subway .proto at:

http://datamine.mta.info/sites/all/files/pdfs/nyct-subway.proto.txt

I guess the main question here is if MTA LIRR should be considered a separate developer from MTA Subway.  In the Extension Registry the "Developer" just says "New York City MTA". Practically speaking I believe you'd need to merge your new message definition into the same .proto above.

Also, if I've got any of the above wrong someone please correct me.

Thanks,

Sean

--
You received this message because you are subscribed to a topic in the Google Groups "GTFS-realtime" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/gtfs-realtime/BZOfsVeI2Cc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to gtfs-realtime+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gtfs-realtime/afe4463c-ad7c-4dab-8ab6-24c9e20b1f32%40googlegroups.com.

[Rachael Morgan]

Hi David,

Are you able to use extension 1005? It looks identical to your proposal.

Cheers,

Rachael

To unsubscribe from this group and all its topics, send an email to gtfs-realtim...@googlegroups.com.

[John L]

Stefan, only if your stop id''s have track represented.

For example, Grand Central Terminal would then have 48 locations. It does not.

On Sep 25, 2017 8:51 AM, <david....@mtahq.org> wrote:

LIRR would like to add track to the StopTimeUpdate in the form of

message LirrStopTimeUpdate {

  optional string track = 1;

  //can add additional fields here without having to

  //extend StopTimeUpdate again

}

You may contact me at David....@mtahq.org for any inquires regarding this request.

Thank you,

David

--
You received this message because you are subscribed to the Google Groups "GTFS-realtime" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gtfs-realtime+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gtfs-realtime/81b76745-c487-4282-a7d3-17b4391d2a81%40googlegroups.com.

[Stefan de Konink]

On maandag 30 oktober 2017 13:22:40 CET, John L wrote:
> Stefan, only if your stop id''s have track represented.

Exactly, but this is already part of the specification.

> For example, Grand Central Terminal would then have 48
> locations. It does not.

If you would check Amsterdam Central Station, you would see that it has
more even more. For GTFS we might want to add topology, durations and paths
from entrance to the stop_ids.

--
Stefan

[John L]

So in order to use gtfs or gtfs rt, transit properties need to change the way they do business?

Think again.

Give the extension ID to LIRR already.

I also beg to differ between pedestrian way finding and vehicle trip planning.

A rider wants to know how to get from point A to B. At the beginning of their trip, their origin is from their current location to station A and their final destination from station B. Way finding within a station was not the General Transit Feed Specification otherwise it would be gwfs.

Again, grant the extension ID already.

--
You received this message because you are subscribed to the Google Groups "GTFS-realtime" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gtfs-realtime+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gtfs-realtime/abe7ad80-149f-4bc1-af77-41617408f4e9%40konink.de.

[Stefan de Konink]

On maandag 30 oktober 2017 13:38:56 CET, John L wrote:
> So in order to use gtfs or gtfs rt, transit properties need to
> change the way they do business?

Have you found one seriously sized operator to use GTFS datamodels in their
internal systems opposed to Transmodel/NeTEx/SIRI?

> Think again.

We have defined a standard here, if this standard specifies that each
location to board has an unique ID, you either play by the rules or your
data is poor to inform your consumers with.

> I also beg to differ between pedestrian way finding and vehicle
> trip planning.
>
> A rider wants to know how to get from point A to B. At the
> beginning of their trip, their origin is from their current
> location to station A and their final destination from station
> B. Way finding within a station was not the General Transit Feed
> Specification otherwise it would be gwfs.

In 2017 we do multimodel travel information. From address to address.
Having the entrance of a station, to the correct boarding positions is key.
If an operator insist of not giving track information, transfer times
become guess work.

More and more I am start to question if some of the people on this mailing
list won some kind of a tender on producing GTFS-RT, but their datamodels
do not align with how things are modelled in GTFS/GTFS-RT and they do not
want to take te trivial actions (look at the simple steps I gave) to make
it compliant.

--
Stefan

[John L]

Who is this 'we' Stephan? The specification is clear, it allows for the way that Metro-North Railroad and Long Island Railroad do business and most of the fields that you speak of are optional.

Keep your conspiracy theories to yourselves as I am referring to the 'we' you speak of. To refute your claims that it is not compliant (YOU ARE WRONG) see that the fields are optional.

I will argue your points to say that the feeds allow for this and we demand that it be done, so it should just be done already.

"To facilitate the testing of new features and to allow developers to add extra information to a GTFS Realtime feed, we will take advantage of the Extensions feature of Protocol Buffers. Extensions allow us to define a namespace in a Protocol Buffer message where third-party developers can define additional fields without the need to modify the original proto definition.

When a developer is interested in extending the GTFS Realtime Specification, they should contact the GTFS Realtime mailing list and we will assign them the next available extension id, picked incrementally from a list of numbers starting at 1000 and going up and documented in the Extension Registry section found below."

https://developers.google.com/transit/gtfs-realtime/guides/extensions

Is this a LIE?!? Is Google LYING to us that we cannot extend the RT feed? I think not but rather an over zealous developer who thinks that their way is the right way, which it may be for you but not for others. 

--
Stefan

--
You received this message because you are subscribed to the Google Groups "GTFS-realtime" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gtfs-realtime+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gtfs-realtime/ab7323f9-ce09-4315-a7f7-dafd8c2b18f7%40konink.de.

[Stefan de Konink]

On maandag 30 oktober 2017 14:46:18 CET, John L wrote:
> Who is this 'we' Stephan?

Everyone that works on these two mailinglists to help to write the
documentation, produce consumer and producer implementations, and answers
questions.

> The specification is clear, it allows
> for the way that Metro-North Railroad and Long Island Railroad
> do business and most of the fields that you speak of are
> optional.

And the request for this Extension ID explicitly mentions that the use will
be to bypass these optional fields, and introduce new fields to bypass the
existing standardisation. [If you want to have a discussion, please counter
this argument]

--
Stefan