Fwd: Data product published

[Matthew Burdett]

Noticed Gemini is live, has anyone received data yet?  Peter published an S class feed yesterday and I couldn't get any data through...

---------- Forwarded message ---------
From: Rail Data Marketplace <rail.data....@notifications.service.gov.uk>
Date: Thu, 18 Sept 2025, 14:09
Subject: Data product published
To: <matthewbu...@gmail.com>

Hello, A data product has been published on Rail Data Marketplace. Product name: NWR Passenger Train Allocation and Consist Product description: This product provides Passenger Train Allocation and Consist Interface derived from Gemini and other passenger…

	Hello, A data product has been published on Rail Data Marketplace. Product name: NWR Passenger Train Allocation and Consist Product description: This product provides Passenger Train Allocation and Consist Interface derived from Gemini and other passenger train allocation and consist systems to Consuming System via the Rail Data Marketplace. This product use a feed of data from WebGemini, a passenger operator rail industry system which manages the allocation of resources (rolling stock) to train services. The data is updated by train operators who are responsible for its accuracy. No guarantee is given on whether this information will match the rolling stock used on a particular service, particularly in times of disruption and data may be updated after a train has run.This product was delivered in collaboration with OpenTrainTimes, whose technical contribution was instrumental to its successful publication on the Rail Data Marketplace. Sign in and view the product details here: https://raildata.org.uk/dashboard/dataProduct/P-3a2ccb58-e1f9-416b-a40e-0614d0269ecf/overview Thank you, Rail Data Marketplace Team Do not reply to this email For assistance, submit a support request here: https://raildata.org.uk/helpAndInformation/contactSupport You can sign in for more support options.

[Peter Davies]

I have just subscribed to the NWR Passenger Train Allocation and Consist data product.

However there is no documentation and though I am provided with some sort of Kafka(?) SASL credentials there is no indication as to how you query to look at the data.  It is a pretty opaque interface/API.

Maybe to those in the know it is all very obvious.

There IS a Preview button but that does not work so the answer is no I have not received any data.

Would be very happy to be pointed at some info on how to access the data.

I would try reading up on Kafka/SASL but since the Preview does not work I am not confident that this product does either.

[Peter Hicks]

All,

Messages should be flowing on the feed soon/in a few hours/later tonight/by tomorrow morning - there is still a bit of work I need to do to publish the data!

Peter

--
You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/f4ec90d4-639f-4228-b57e-58d8c6b238b4n%40googlegroups.com.

[Matthew Burdett]

You need a Kafka consumer to receive the data just like with the train movement and Darwin pushport feeds.

Without knowing your programming language I can't comment on how to access the data but I've had no problems using Rdkafka and php.

Doubt there will be any documentation any time soon, best thing to do is dump it into a file and work through it.

Many thanks to Peter who's worked hard to get this up with rdg and NR. 

[lava...@gmail.com]

Initially I tend to use the command line - curl - as a quick way to
investigate a data product.
Then I write code in Java to actually work with the data.

I will do some reading up on Apache Kafka consumers once I have managed
to play with the RDM Preview.

I repeat the thank you to Peter (I apologised directly for being in such
a rush to get a look at the data).

On 18/09/2025 15:13, Matthew Burdett wrote:
> You need a Kafka consumer to receive the data just like with the train
> movement and Darwin pushport feeds.
>
> Without knowing your programming language I can't comment on how to
> access the data but I've had no problems using Rdkafka and php.
>
> Doubt there will be any documentation any time soon, best thing to do is
> dump it into a file and work through it.
>
> Many thanks to Peter who's worked hard to get this up with rdg and NR.
>
> On Thu, 18 Sept 2025, 14:59 Peter Davies, <lava...@gmail.com

> <mailto:lava...@gmail.com>> wrote:
>
>
> I have just subscribed to the NWR Passenger Train Allocation and
> Consist data product.
>
> However there is no documentation and though I am provided with some
> sort of Kafka(?) SASL credentials there is no indication as to how
> you query to look at the data.  It is a pretty opaque interface/API.
> Maybe to those in the know it is all very obvious.
>

> There IS a Preview button but *that does not work* so the answer is

> no I have not received any data.
>
> Would be very happy to be pointed at some info on how to access the
> data.
> I would try reading up on Kafka/SASL but since the Preview does not
> work I am not confident that this product does either.
>
> On Thursday, 18 September 2025 at 14:12:42 UTC+1 Matthew Burdett wrote:
>
> Noticed Gemini is live, has anyone received data yet?  Peter
> published an S class feed yesterday and I couldn't get any data
> through...
>
> ---------- Forwarded message ---------
> From: *Rail Data Marketplace*
> <rail.data....@notifications.service.gov.uk>
> Date: Thu, 18 Sept 2025, 14:09
> Subject: Data product published
> To: <matthewbu...@gmail.com>
>
>

> __

> Hello, A data product has been published on Rail Data
> Marketplace. Product name: NWR Passenger Train Allocation and
> Consist Product description: This product provides Passenger
> Train Allocation and Consist Interface derived from Gemini and
> other passenger…

> Rail Delivery Group
>
>
>
>
>
> Hello,
>
> A data product has been published on Rail Data Marketplace.
>
> Product name: NWR Passenger Train Allocation and Consist
>
> Product description: This product provides Passenger Train
> Allocation and Consist Interface derived from Gemini and other
> passenger train allocation and consist systems to Consuming
> System via the Rail Data Marketplace. This product use a feed of
> data from WebGemini, a passenger operator rail industry system
> which manages the allocation of resources (rolling stock) to
> train services. The data is updated by train operators who are
> responsible for its accuracy. No guarantee is given on whether
> this information will match the rolling stock used on a
> particular service, particularly in times of disruption and data
> may be updated after a train has run.This product was delivered
> in collaboration with OpenTrainTimes, whose technical
> contribution was instrumental to its successful publication on
> the Rail Data Marketplace.
>
> Sign in and view the product details here:
>
> https://raildata.org.uk/dashboard/dataProduct/P-3a2ccb58-

> e1f9-416b-a40e-0614d0269ecf/overview <https://raildata.org.uk/
> dashboard/dataProduct/P-3a2ccb58-e1f9-416b-a40e-0614d0269ecf/
> overview>

>
> Thank you,
> Rail Data Marketplace Team
>

> ------------------------------------------------------------------------

>
>
> Do not reply to this email
>
> For assistance, submit a support request here:
> https://raildata.org.uk/helpAndInformation/contactSupport
> <https://raildata.org.uk/helpAndInformation/contactSupport>
> You can sign in for more support options.
>
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "A gathering place for the Open Rail Data community" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to openraildata-t...@googlegroups.com

> <mailto:openraildata-t...@googlegroups.com>.

> To view this discussion, visit https://groups.google.com/d/msgid/
> openraildata-talk/f4ec90d4-639f-4228-

> b57e-58d8c6b238b4n%40googlegroups.com <https://groups.google.com/d/
> msgid/openraildata-talk/f4ec90d4-639f-4228-
> b57e-58d8c6b238b4n%40googlegroups.com?
> utm_medium=email&utm_source=footer>.
>
> --
> You received this message because you are subscribed to a topic in the

> Google Groups "A gathering place for the Open Rail Data community" group.

> To unsubscribe from this topic, visit https://groups.google.com/d/topic/
> openraildata-talk/JfLMAm_PyzQ/unsubscribe <https://groups.google.com/d/
> topic/openraildata-talk/JfLMAm_PyzQ/unsubscribe>.
> To unsubscribe from this group and all its topics, send an email to
> openraildata-t...@googlegroups.com <mailto:openraildata-
> talk+uns...@googlegroups.com>.

> To view this discussion, visit https://groups.google.com/d/msgid/

> openraildata-talk/CA%2B2-
> wYGihUvWTBJGMjbN2zdg5WU9HFpsFDYG2tpyrR5ipuPgqQ%40mail.gmail.com
> <https://groups.google.com/d/msgid/openraildata-talk/CA%2B2-
> wYGihUvWTBJGMjbN2zdg5WU9HFpsFDYG2tpyrR5ipuPgqQ%40mail.gmail.com?
> utm_medium=email&utm_source=footer>.

[David Wheatley]

I've put together a basic consumer in Go which will dump messages from the feed into flat files broken up per-hour.

It's based on what I have running to store RDM Darwin push port messages in S3-compatible storage, so also retains that functionality here too, and I will be storing Gemini data as soon as it starts arriving (and some messages already have!).

Hopefully it's of some use to people that don't know where to start, even if the documentation is rather bare.

https://github.com/davwheat/gemini-push-port

David

To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/edf81e41-b318-4078-8872-50a189f19f25%40gmail.com.

[Peter Hicks]

You’re too quick, I have another product on the RDM where I will publish an hourly batch of updates.  Should go live in a day or two when I catch my breath.

To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/CAGZsNiin0Ay9QY_RmQnvbW1r0gC9tPZ3Dgoaiu0ObmhHtygAPg%40mail.gmail.com.

[Gaelan Steele]

Hi Peter,

Thanks for everything you’ve done here - this is very exciting stuff!

On LinkedIn, you posted:

> The GB rail industry is publishing train consist data for passenger trains (limitations apply) via the Rail Data Marketplace under an open licence.

Would you be able to elaborate on what those limitations are?

Best wishes,

Gaelan

On Sep 18, 2025, at 6:50 PM, 'Peter Hicks' via A gathering place for the Open Rail Data community <openrail...@googlegroups.com> wrote:



To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/b6C8VRiJaqrr8bCQbk_WhsnymuGVSGBIHajr-R5aqLGIlHT5mnnUqpDEE7T_2fgPi8G9LXMgq-a-NsQ0dgB-6Jg-Kv1-eyf_hFDMdEnzVZw%3D%40poggs.co.uk.

[Peter Hicks]

Limitations include:

Not all operators update GEMINI so won’t show up here.  London Overground are one, and there isn’t necessarily a benefit to them doing so as all their trains are the same length on a route I believe.

Operators may not update in real time or during disruption, or when stock swaps occur.  Don’t bet your house and car on 800123 being on the service reported during or after disruption.

Units may show the wrong orientation or the carriages in the wrong order so treat it with a pinch of salt.

Basically: it’s a darned good start and can be made even better, but we need to take the first step.

To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/9114F15A-1604-4DBC-9164-FB1799B6C9B7%40canishe.com.

[Matthew Burdett]

Hello

Just thought I'd share i have not gotten any messages yet, just incase nobody is aware of a potential issue. Could just be me and my RDM keys need rolling or something and if so I'll contact support.

(Nothing comes on the preview on RDM), I did refresh this a few times between 3.30 this morning and 6 before work and no change. 

(I'm using same set up as my push port so json message and the XML contained within the msg itself assuming this is the format?)

Best wishes 

To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/HurpXHH940fkFEEmHhEuzbBCkIHy0XVSCqX1D3s0gIcP9rjX63dvlMiifCqjyrKBnpMYdlGjEJK5W5cUbpMj6eJ2xIdtS6OmBt8K1q24VT4%3D%40poggs.co.uk.

[Peter Hicks]

Hi Matthew

On Friday, 19 September 2025 at 11:12, Matthew Burdett <matthewbu...@gmail.com> wrote:

Just thought I'd share i have not gotten any messages yet, just incase nobody is aware of a potential issue. Could just be me and my RDM keys need rolling or something and if so I'll contact support.

(Nothing comes on the preview on RDM), I did refresh this a few times between 3.30 this morning and 6 before work and no change.

I'm not aware of any issue - plenty of messages are coming through when I connect:

The messages are published as XML as that's what's produced by the source system.  There may be the option to convert them in to JSON too, but really you should be using an XSD and generated classes otherwise you'll be in for pain if/when the format changes.

Do you have sample client code I can try here with my credentials?

Peter

[David Wheatley]

The feed is working fine for me, and has been since my reply to this thread yesterday. The preview functionality on the RDM site doesn't work, though. Whether that's expected with Pub/Sub data or something else, I'm not too sure.

Double-check your configuration, but failing that I suppose your suggestion of contacting support is the next step if the same setup works fine for Darwin (assuming that's via RDM too).

It is XML within the Kafka message. You can see examples I've stored here: https://ilovetrains.co.uk/gemini-push-port/2025-09-19

David

To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/CA%2B2-wYHywKdrNFJKSbiTdE94Fdo24PQEGBTfEzvmAZZS_p4ikQ%40mail.gmail.com.

[Ben Woodward]

Mine seems to be working, make sure you've enabled SASL and SSL/TLS, and set the username/password.

On Fri, 19 Sept 2025 at 11:12, Matthew Burdett <matthewbu...@gmail.com> wrote:

To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/CA%2B2-wYHywKdrNFJKSbiTdE94Fdo24PQEGBTfEzvmAZZS_p4ikQ%40mail.gmail.com.

[Matthew Burdett]

Sorry guys I've got it now. Lazy of me to reuse some existing code. Didn't realise the whole message was in XML. Thanks. 

To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/CAE0U5-Z60b6k2U6nvhHReq_89rtdqqhp-Dd7Ssf7%2B_%2BjR%3DtVWw%40mail.gmail.com.

[Cameron Bird]

Not sure if I've missed them, but will the XSDs be published? I can't see them on the RDM page.

[Peter Hicks]

On Friday, 19 September 2025 at 12:28, Cameron Bird <bird...@gmail.com> wrote:

Not sure if I've missed them, but will the XSDs be published? I can't see them on the RDM page.

Already on it: https://www.whatdotheyknow.com/request/linx_xml_schema_definitions

I had a quick search and Google comes up with just one hit - the WDTK request above.  Google AI helpfully suggests that "GB_TAF_TAP_TSI_complete.xsd is an XML Schema Definition file that defines the structure and rules for XML documents related to TAF (Technical Aspects of Freight) and TAP (Technical Aspects of Passenger)'.  The "TA" stands for "Telematics Applications", but I guess that doesn't matter when AI comes up with such a robustly confident answer.

Peter

[Jez Smith]

David and all,

Firstly, a huge thanks for Peter in pushing and supporting the publication of this data set.  I hope that it will be enormously helpful in opening up data and spurring innovation from the community, and without Peter, it would not have been possible.

On the point about the preview not working; it isn't and it should.  Preview works in other pub/sub products (albeit of a slightly different flavour), we are exploring two potential reasons why it isn't working, and those will hopefully give us some options for getting it sorted.  Grateful if you could bare with us for a few days while we work it through.

Cheers

Jez

[Peter Hicks]

On Monday, 22 September 2025 at 11:30, Jez Smith <jezinwo...@gmail.com> wrote:

Firstly, a huge thanks for Peter in pushing and supporting the publication of this data set. I hope that it will be enormously helpful in opening up data and spurring innovation from the community, and without Peter, it would not have been possible.

Well, that's clearly made my Monday - thank you!

On the point about the preview not working; it isn't and it should. Preview works in other pub/sub products (albeit of a slightly different flavour), we are exploring two potential reasons why it isn't working, and those will hopefully give us some options for getting it sorted. Grateful if you could bare with us for a few days while we work it through.

To add in some technical detail, the reason I think it's not working is because we're publishing XML straight in to Kafka, and not JSON.  From the looks of other feeds such as Darwin, I think the UI expects to have JSON, and it might silently fail.  It was the cause of some stress on Thursday evening/Friday when there were reports of "It doesn't work" coming in!

Peter