Fwd: Data product published

[Matthew Burdett]

Noticed Gemini is live, has anyone received data yet?  Peter published an S class feed yesterday and I couldn't get any data through...

---------- Forwarded message ---------
From: Rail Data Marketplace <rail.data....@notifications.service.gov.uk>
Date: Thu, 18 Sept 2025, 14:09
Subject: Data product published
To: <matthewbu...@gmail.com>

Hello, A data product has been published on Rail Data Marketplace. Product name: NWR Passenger Train Allocation and Consist Product description: This product provides Passenger Train Allocation and Consist Interface derived from Gemini and other passenger…

	Hello, A data product has been published on Rail Data Marketplace. Product name: NWR Passenger Train Allocation and Consist Product description: This product provides Passenger Train Allocation and Consist Interface derived from Gemini and other passenger train allocation and consist systems to Consuming System via the Rail Data Marketplace. This product use a feed of data from WebGemini, a passenger operator rail industry system which manages the allocation of resources (rolling stock) to train services. The data is updated by train operators who are responsible for its accuracy. No guarantee is given on whether this information will match the rolling stock used on a particular service, particularly in times of disruption and data may be updated after a train has run.This product was delivered in collaboration with OpenTrainTimes, whose technical contribution was instrumental to its successful publication on the Rail Data Marketplace. Sign in and view the product details here: https://raildata.org.uk/dashboard/dataProduct/P-3a2ccb58-e1f9-416b-a40e-0614d0269ecf/overview Thank you, Rail Data Marketplace Team Do not reply to this email For assistance, submit a support request here: https://raildata.org.uk/helpAndInformation/contactSupport You can sign in for more support options.

[Peter Davies]

I have just subscribed to the NWR Passenger Train Allocation and Consist data product.

However there is no documentation and though I am provided with some sort of Kafka(?) SASL credentials there is no indication as to how you query to look at the data.  It is a pretty opaque interface/API.

Maybe to those in the know it is all very obvious.

There IS a Preview button but that does not work so the answer is no I have not received any data.

Would be very happy to be pointed at some info on how to access the data.

I would try reading up on Kafka/SASL but since the Preview does not work I am not confident that this product does either.

[Peter Hicks]

All,

Messages should be flowing on the feed soon/in a few hours/later tonight/by tomorrow morning - there is still a bit of work I need to do to publish the data!

Peter

--
You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/f4ec90d4-639f-4228-b57e-58d8c6b238b4n%40googlegroups.com.

[Matthew Burdett]

You need a Kafka consumer to receive the data just like with the train movement and Darwin pushport feeds.

Without knowing your programming language I can't comment on how to access the data but I've had no problems using Rdkafka and php.

Doubt there will be any documentation any time soon, best thing to do is dump it into a file and work through it.

Many thanks to Peter who's worked hard to get this up with rdg and NR. 

[lava...@gmail.com]

Initially I tend to use the command line - curl - as a quick way to
investigate a data product.
Then I write code in Java to actually work with the data.

I will do some reading up on Apache Kafka consumers once I have managed
to play with the RDM Preview.

I repeat the thank you to Peter (I apologised directly for being in such
a rush to get a look at the data).

On 18/09/2025 15:13, Matthew Burdett wrote:
> You need a Kafka consumer to receive the data just like with the train
> movement and Darwin pushport feeds.
>
> Without knowing your programming language I can't comment on how to
> access the data but I've had no problems using Rdkafka and php.
>
> Doubt there will be any documentation any time soon, best thing to do is
> dump it into a file and work through it.
>
> Many thanks to Peter who's worked hard to get this up with rdg and NR.
>
> On Thu, 18 Sept 2025, 14:59 Peter Davies, <lava...@gmail.com

> <mailto:lava...@gmail.com>> wrote:
>
>
> I have just subscribed to the NWR Passenger Train Allocation and
> Consist data product.
>
> However there is no documentation and though I am provided with some
> sort of Kafka(?) SASL credentials there is no indication as to how
> you query to look at the data.  It is a pretty opaque interface/API.
> Maybe to those in the know it is all very obvious.
>

> There IS a Preview button but *that does not work* so the answer is

> no I have not received any data.
>
> Would be very happy to be pointed at some info on how to access the
> data.
> I would try reading up on Kafka/SASL but since the Preview does not
> work I am not confident that this product does either.
>
> On Thursday, 18 September 2025 at 14:12:42 UTC+1 Matthew Burdett wrote:
>
> Noticed Gemini is live, has anyone received data yet?  Peter
> published an S class feed yesterday and I couldn't get any data
> through...
>
> ---------- Forwarded message ---------
> From: *Rail Data Marketplace*
> <rail.data....@notifications.service.gov.uk>
> Date: Thu, 18 Sept 2025, 14:09
> Subject: Data product published
> To: <matthewbu...@gmail.com>
>
>

> __

> Hello, A data product has been published on Rail Data
> Marketplace. Product name: NWR Passenger Train Allocation and
> Consist Product description: This product provides Passenger
> Train Allocation and Consist Interface derived from Gemini and
> other passenger…

> Rail Delivery Group
>
>
>
>
>
> Hello,
>
> A data product has been published on Rail Data Marketplace.
>
> Product name: NWR Passenger Train Allocation and Consist
>
> Product description: This product provides Passenger Train
> Allocation and Consist Interface derived from Gemini and other
> passenger train allocation and consist systems to Consuming
> System via the Rail Data Marketplace. This product use a feed of
> data from WebGemini, a passenger operator rail industry system
> which manages the allocation of resources (rolling stock) to
> train services. The data is updated by train operators who are
> responsible for its accuracy. No guarantee is given on whether
> this information will match the rolling stock used on a
> particular service, particularly in times of disruption and data
> may be updated after a train has run.This product was delivered
> in collaboration with OpenTrainTimes, whose technical
> contribution was instrumental to its successful publication on
> the Rail Data Marketplace.
>
> Sign in and view the product details here:
>
> https://raildata.org.uk/dashboard/dataProduct/P-3a2ccb58-

> e1f9-416b-a40e-0614d0269ecf/overview <https://raildata.org.uk/
> dashboard/dataProduct/P-3a2ccb58-e1f9-416b-a40e-0614d0269ecf/
> overview>

>
> Thank you,
> Rail Data Marketplace Team
>

> ------------------------------------------------------------------------

>
>
> Do not reply to this email
>
> For assistance, submit a support request here:
> https://raildata.org.uk/helpAndInformation/contactSupport
> <https://raildata.org.uk/helpAndInformation/contactSupport>
> You can sign in for more support options.
>
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "A gathering place for the Open Rail Data community" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to openraildata-t...@googlegroups.com

> <mailto:openraildata-t...@googlegroups.com>.

> To view this discussion, visit https://groups.google.com/d/msgid/
> openraildata-talk/f4ec90d4-639f-4228-

> b57e-58d8c6b238b4n%40googlegroups.com <https://groups.google.com/d/
> msgid/openraildata-talk/f4ec90d4-639f-4228-
> b57e-58d8c6b238b4n%40googlegroups.com?
> utm_medium=email&utm_source=footer>.
>
> --
> You received this message because you are subscribed to a topic in the

> Google Groups "A gathering place for the Open Rail Data community" group.

> To unsubscribe from this topic, visit https://groups.google.com/d/topic/
> openraildata-talk/JfLMAm_PyzQ/unsubscribe <https://groups.google.com/d/
> topic/openraildata-talk/JfLMAm_PyzQ/unsubscribe>.
> To unsubscribe from this group and all its topics, send an email to
> openraildata-t...@googlegroups.com <mailto:openraildata-
> talk+uns...@googlegroups.com>.

> To view this discussion, visit https://groups.google.com/d/msgid/

> openraildata-talk/CA%2B2-
> wYGihUvWTBJGMjbN2zdg5WU9HFpsFDYG2tpyrR5ipuPgqQ%40mail.gmail.com
> <https://groups.google.com/d/msgid/openraildata-talk/CA%2B2-
> wYGihUvWTBJGMjbN2zdg5WU9HFpsFDYG2tpyrR5ipuPgqQ%40mail.gmail.com?
> utm_medium=email&utm_source=footer>.

[David Wheatley]

I've put together a basic consumer in Go which will dump messages from the feed into flat files broken up per-hour.

It's based on what I have running to store RDM Darwin push port messages in S3-compatible storage, so also retains that functionality here too, and I will be storing Gemini data as soon as it starts arriving (and some messages already have!).

Hopefully it's of some use to people that don't know where to start, even if the documentation is rather bare.

https://github.com/davwheat/gemini-push-port

David

To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/edf81e41-b318-4078-8872-50a189f19f25%40gmail.com.

[Peter Hicks]

You’re too quick, I have another product on the RDM where I will publish an hourly batch of updates.  Should go live in a day or two when I catch my breath.

To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/CAGZsNiin0Ay9QY_RmQnvbW1r0gC9tPZ3Dgoaiu0ObmhHtygAPg%40mail.gmail.com.

[Gaelan Steele]

Hi Peter,

Thanks for everything you’ve done here - this is very exciting stuff!

On LinkedIn, you posted:

> The GB rail industry is publishing train consist data for passenger trains (limitations apply) via the Rail Data Marketplace under an open licence.

Would you be able to elaborate on what those limitations are?

Best wishes,

Gaelan

On Sep 18, 2025, at 6:50 PM, 'Peter Hicks' via A gathering place for the Open Rail Data community <openrail...@googlegroups.com> wrote:



To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/b6C8VRiJaqrr8bCQbk_WhsnymuGVSGBIHajr-R5aqLGIlHT5mnnUqpDEE7T_2fgPi8G9LXMgq-a-NsQ0dgB-6Jg-Kv1-eyf_hFDMdEnzVZw%3D%40poggs.co.uk.

[Peter Hicks]

Limitations include:

Not all operators update GEMINI so won’t show up here.  London Overground are one, and there isn’t necessarily a benefit to them doing so as all their trains are the same length on a route I believe.

Operators may not update in real time or during disruption, or when stock swaps occur.  Don’t bet your house and car on 800123 being on the service reported during or after disruption.

Units may show the wrong orientation or the carriages in the wrong order so treat it with a pinch of salt.

Basically: it’s a darned good start and can be made even better, but we need to take the first step.

To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/9114F15A-1604-4DBC-9164-FB1799B6C9B7%40canishe.com.

[Matthew Burdett]

Hello

Just thought I'd share i have not gotten any messages yet, just incase nobody is aware of a potential issue. Could just be me and my RDM keys need rolling or something and if so I'll contact support.

(Nothing comes on the preview on RDM), I did refresh this a few times between 3.30 this morning and 6 before work and no change. 

(I'm using same set up as my push port so json message and the XML contained within the msg itself assuming this is the format?)

Best wishes 

To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/HurpXHH940fkFEEmHhEuzbBCkIHy0XVSCqX1D3s0gIcP9rjX63dvlMiifCqjyrKBnpMYdlGjEJK5W5cUbpMj6eJ2xIdtS6OmBt8K1q24VT4%3D%40poggs.co.uk.

[Peter Hicks]

Hi Matthew

On Friday, 19 September 2025 at 11:12, Matthew Burdett <matthewbu...@gmail.com> wrote:

Just thought I'd share i have not gotten any messages yet, just incase nobody is aware of a potential issue. Could just be me and my RDM keys need rolling or something and if so I'll contact support.

(Nothing comes on the preview on RDM), I did refresh this a few times between 3.30 this morning and 6 before work and no change.

I'm not aware of any issue - plenty of messages are coming through when I connect:

The messages are published as XML as that's what's produced by the source system.  There may be the option to convert them in to JSON too, but really you should be using an XSD and generated classes otherwise you'll be in for pain if/when the format changes.

Do you have sample client code I can try here with my credentials?

Peter

[David Wheatley]

The feed is working fine for me, and has been since my reply to this thread yesterday. The preview functionality on the RDM site doesn't work, though. Whether that's expected with Pub/Sub data or something else, I'm not too sure.

Double-check your configuration, but failing that I suppose your suggestion of contacting support is the next step if the same setup works fine for Darwin (assuming that's via RDM too).

It is XML within the Kafka message. You can see examples I've stored here: https://ilovetrains.co.uk/gemini-push-port/2025-09-19

David

To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/CA%2B2-wYHywKdrNFJKSbiTdE94Fdo24PQEGBTfEzvmAZZS_p4ikQ%40mail.gmail.com.

[Ben Woodward]

Mine seems to be working, make sure you've enabled SASL and SSL/TLS, and set the username/password.

On Fri, 19 Sept 2025 at 11:12, Matthew Burdett <matthewbu...@gmail.com> wrote:

To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/CA%2B2-wYHywKdrNFJKSbiTdE94Fdo24PQEGBTfEzvmAZZS_p4ikQ%40mail.gmail.com.

[Matthew Burdett]

Sorry guys I've got it now. Lazy of me to reuse some existing code. Didn't realise the whole message was in XML. Thanks. 

To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/CAE0U5-Z60b6k2U6nvhHReq_89rtdqqhp-Dd7Ssf7%2B_%2BjR%3DtVWw%40mail.gmail.com.

[Cameron Bird]

Not sure if I've missed them, but will the XSDs be published? I can't see them on the RDM page.

[Peter Hicks]

On Friday, 19 September 2025 at 12:28, Cameron Bird <bird...@gmail.com> wrote:

Not sure if I've missed them, but will the XSDs be published? I can't see them on the RDM page.

Already on it: https://www.whatdotheyknow.com/request/linx_xml_schema_definitions

I had a quick search and Google comes up with just one hit - the WDTK request above.  Google AI helpfully suggests that "GB_TAF_TAP_TSI_complete.xsd is an XML Schema Definition file that defines the structure and rules for XML documents related to TAF (Technical Aspects of Freight) and TAP (Technical Aspects of Passenger)'.  The "TA" stands for "Telematics Applications", but I guess that doesn't matter when AI comes up with such a robustly confident answer.

Peter

[Jez Smith]

David and all,

Firstly, a huge thanks for Peter in pushing and supporting the publication of this data set.  I hope that it will be enormously helpful in opening up data and spurring innovation from the community, and without Peter, it would not have been possible.

On the point about the preview not working; it isn't and it should.  Preview works in other pub/sub products (albeit of a slightly different flavour), we are exploring two potential reasons why it isn't working, and those will hopefully give us some options for getting it sorted.  Grateful if you could bare with us for a few days while we work it through.

Cheers

Jez

[Peter Hicks]

On Monday, 22 September 2025 at 11:30, Jez Smith <jezinwo...@gmail.com> wrote:

Firstly, a huge thanks for Peter in pushing and supporting the publication of this data set. I hope that it will be enormously helpful in opening up data and spurring innovation from the community, and without Peter, it would not have been possible.

Well, that's clearly made my Monday - thank you!

On the point about the preview not working; it isn't and it should. Preview works in other pub/sub products (albeit of a slightly different flavour), we are exploring two potential reasons why it isn't working, and those will hopefully give us some options for getting it sorted. Grateful if you could bare with us for a few days while we work it through.

To add in some technical detail, the reason I think it's not working is because we're publishing XML straight in to Kafka, and not JSON.  From the looks of other feeds such as Darwin, I think the UI expects to have JSON, and it might silently fail.  It was the cause of some stress on Thursday evening/Friday when there were reports of "It doesn't work" coming in!

Peter

[Jez Smith]

The Preview on NWR Passenger train consist and allocation should now be working.

[Peter Hicks]

Hi Jez

I thought about replying on the list, but decided against it as haters will take every possible opportunity to hate :-)

On Tuesday, 30 September 2025 at 10:26, Jez Smith <jezinwo...@gmail.com> wrote:

The Preview on NWR Passenger train consist and allocation should now be working.

I'm not sure this is quite what users need.  On the left are the messages as consumed by a Kafka client, and on the right are the same messages in the 'preview' window - and the RDM web interface converts the data in to JSON rather than leaving it in XML!

There is a risk that users will see the data in JSON format, write their code against this (ignoring the XSDs which are to be published soon), and when they come to test it, find out that all their work has been in vain, as the data is actually in a different format.

Peter

[Peter Hicks]

On Tuesday, 30 September 2025 at 10:40, 'Peter Hicks' via A gathering place for the Open Rail Data community <openrail...@googlegroups.com> wrote:

I thought about replying on the list, but decided against it as haters will take every possible opportunity to hate :-)

Well that's embarrassing.  The only way I could have made this worse was to send a "xxx would like to recall the message titled...", which is the corporate way of making everyone open the message to see what you did!

Still, it's noting I wouldn't say in public, it just came out a little louder than I wanted.

Peter

[Jez Smith]

Ha!  Haters?!?  I am not sure I know what you mean......our biggest critics these days are referred to as 'reluctant consumers'! ;-)

We certainly find your informal role as 'critical friend' extremely helpful, Peter, and thanks for raising the potential pitfall above.

I will raise it with the developers and see what we can do to address it.

Cheers

Jez

[David Wheatley]

Ha!  Haters?!?  I am not sure I know what you mean......our biggest critics these days are referred to as 'reluctant consumers'! ;-)

I don't want to turn this discussion into an RDM-hate conversation, but seeing comments like this coming from the person that leads the RDM project is concerning and only reminds me that railway open data is only becoming less open.

When data is released exclusively via the RDM platform, there's not much choice to be anything other than a "reluctant consumer". You might see that as a win, but I most certainly don't. Using the RDM platform isn't a choice for many, but rather a necessity.

I don't see any reason why a consumer would want to be made to use a platform where if they criticise RDG/RDM or do something that the industry doesn't like then their account could be terminated after 30 days and they would lose access to all data feeds. And for those that might think that sort of idea is nothing but unsubstantiated hypothesis, some organisations have already had that threat (and worse) made against them already.

Improvements to RDM have been made since launch, such as amending the terms for sign-up, and I can respect that, but you still have a long way to go before myself and other members of the open data community view RDM as something that isn't simply a data cartel with excessive control over the accessibility of data within the platform.

Jez, you can give us promises on mailing lists all you would like, but at the end of the day this is a public service run by an organisation on behalf of the public sector using taxpayer funds. RDM has decided to add bureaucracy and contractual wording to obtaining open data which never (NROD) or barely (NRE feeds) existed on the various open data portals that existed prior to RDM. RDM needs to update its agreements to include these promises if they are worth the pixels they're written on.

All the best,

David

These views are strictly my own.

--

You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.

To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/12e2bd63-bbba-425b-ab12-fe31b7f013e2n%40googlegroups.com.

[Matthew Burdett]

Appreciate the new archive of messages, thanks Peter.

Having logged these myself into hour by hour files, they do get very big uncompressed, some even 200mb in size.

To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/CAGZsNiiFwGD1VQYznTeLMzS-F2xgvQr%3Dqp6HZfO%3DJKfbd-SzVQ%40mail.gmail.com.

[Jez Smith]

David,

My apologies if my levity on a Tuesday afternoon missed the mark.

Whilst I can see that you don't agree, my view is that a level of bureaucracy and contractual control is necessary on RDM because, in many cases, it is not public data which is being made available.  The platform needs to cope with both public and private publishers, open publishing and chargeable datasets.  It provides a level of control for any publisher (public or private) and deals with that complexity in a consistent way.

Consumers are subscribing via RDM to data which may be published by another organisation, so those 3 parties are bound using platform agreements and data sharing agreement.  We did receive feedback about the platform agreements restricting criticism of RDM and reworded them as a result so now access can only be restricted if a Consumer "deliberately seeks to unfairly defame RDM or RDG, mislead users of RDM or provides information which is demonstrably incorrect and, having been given the opportunity to do so, the Data Consumer fails to correct this.".  That is a pretty high bar and I would not expect any professional to cross it.  I am unaware of anyone having 'had that threat (and worse)' but please do get in touch privately and I will look into it.  

Decisions around the data sharing agreement are left to publishers, and for public sector organisations OGL3 is frequently used which is extremely open.  Private sector publishers sometimes put conditions on use.

So, yes, there are contractual responsibilities on all parties to act professionally, and perhaps that seems like an imposition, however it is a free service to consumers and takes no direct public-funding (and quite how you would calculate indirect funding is anyone's guess).  RDM does give access to over 200 data products in one place / one login, has a pretty good record on availability and has a responsive support desk (questions about data products take a little longer).  RDM has championed the opening up of data more broadly and works closely with organisations across the industry to do just that, evidenced by the number of new datasets which appear on RDM.  We continue to try to improve the service, listening to feedback and trying to respond whilst balancing the needs of publishers and consumers and providing protections for all.

So swings and roundabouts.  I know that you will make your own judgement about whether it is all worthwhile.

All the best,

Jez

These views are also strictly my own.

[Gaelan Steele]

Hi Jez,

Thanks, genuinely, for everything you’ve done - for all my complaints about the RDM, it has been a catalyst in getting a bunch of interesting data released for the first time, and it’s absolutely been a net positive.

> On Sep 30, 2025, at 5:09 PM, Jez Smith <jezinwo...@gmail.com> wrote:
>
> Whilst I can see that you don't agree, my view is that a level of bureaucracy and contractual control is necessary on RDM because, in many cases, it is not public data which is being made available. The platform needs to cope with both public and private publishers, open publishing and chargeable datasets. It provides a level of control for any publisher (public or private) and deals with that complexity in a consistent way.

Obviously this makes sense for data released under commercial arrangements. But for the vast majority of us hobbyists who would never consider entering into such an agreement - I think it’s safe to say they’re quite out of reach for us - all the bureaucracy does us absolutely no good. Hobby projects often happen on a whim; if I want to have a play with some data, and it looks like a pain to get at, I likely won’t bother - and if I do bother filling out the form, if/when I’m approved a few days later the moment may be gone.

I hope, as you look at longer-range improvements to the RDM, you consider moving to a two-tier design where data under the OGL (or similarly permissive terms) is available on a 100% self-service basis, and we complete the bureaucracy (the manual approval process, any arguably-onerous agreements) as/when we are interested in pursing a commercial agreement. Ideally data files could be downloaded with no signup process at all, although that’s harder to do with APIs and pubsub data.

Of course I recognize there’s a cost there in terms of complexity on your side, and inevitably such things must be balanced against everything else you’re spending development time on. All I ask is that you count this among the many other feature requests you’re inevitably getting from everyone involved :)

Best wishes,
Gaelan Steele

[David Wheatley]

Hi all,

Has anyone else seen a sudden decline in messages this evening?

There's still one or two, but significantly less than I'd usually expect for this time of day.

No telling where any potential issue may lie in the chain of providers, though!

Cheers,

David

--
You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.

To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/B61FCF53-4685-4F97-A6A6-43FABED7D117%40canishe.com.

[Matthew Burdett]

Also seeing fewer messages over longer periods. I remember it did this the other day possibly the 12th. Although could have just been me.

To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/CAGZsNihperz%2BB83pOhijNqd9m367-wj017NiME8waXDJuKoYGA%40mail.gmail.com.

[Matthew Burdett]

Has now picked up for me, must have been bad timing lol 

To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/CAGZsNihperz%2BB83pOhijNqd9m367-wj017NiME8waXDJuKoYGA%40mail.gmail.com.

[David Wheatley]

Thanks for giving it a kick Matthew... 😜

To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/CA%2B2-wYFRZ2tOb9OKtKGXmTf5TqcheTArYcG%2BjBPwk%2BLb9h%2BZMw%40mail.gmail.com.

[Peter Hicks]

Hi David

On Friday, 21 November 2025 at 17:54, 'David Wheatley' via A gathering place for the Open Rail Data community <openrail...@googlegroups.com> wrote:

Has anyone else seen a sudden decline in messages this evening?

No alerts at OTT Towers, and yes I have alerts :). The consist messages are extracted from GEMINI every five minutes, but at 16:51, the next batch was received at 17:01, then 17:31, then 18:01.  Another batch came through at 18:06.

I suspect a problem upstream from here, possibly with the extraction side of things as everything downstream from that is fairly linear and uncomplicated.

Peter

[Matthew Burdett]

Hello

There Must be some sort of upstream issue, there's been no data since midnight.

Not sure if I need to raise a ticket anywhere.

Kind regards 

--
You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.

To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/79MJqjzcuVHCvc9T39WlAwQQtxXewn2-9snYIp-IT7Ur8z-P_FnmeNEe0FBhcXo9CTwCL441OZMXcFDwBQUWBFS-I94e11r8Dzw_DcEQ5Xk%3D%40poggs.co.uk.

[Phil Wieland]

Yes, I've had nothing since 01:14 this morning.  Client restart hasn't helped.

Phil

[David Wheatley]

Yes, same on my end. Stopped around 0130 on two of my consumers.

Hopefully it'll be fixed soon.

David

To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/CA%2B2-wYEHb4Cf52mo5ynChdA_L%3Dsgpi%3D9G4ZoHo5cF5vN_OAWMA%40mail.gmail.com.

[Phil Wieland]

Ticket RDM0003080 raised.

[Peter Hicks]

On Tuesday, 10 February 2026 at 09:07, Phil Wieland <philw...@gmail.com> wrote:

Ticket RDM0003080 raised.

Fixed - an embarrassingly straightforward case of a lack of disk space on a server causing a process to terminate.  Sorry everyone :(

Messages were lost between 01:14:55 and 08:33:40 - I will see if I can pick these up elsewhere and get them added to the archive product on RDM (https://raildata.org.uk/dashboard/dataProduct/P-fac2476d-e83c-4f71-b49d-2d14b9053ef2/overview) for tomorrow.

The feed is back up and running now.  Can anyone confirm?

Peter

[Phil Wieland]

Yes, now receiving messages.  Thanks.

[Rob West]

Thanks Peter

[Peter Hicks]

On Tuesday, 10 February 2026 at 09:48, 'Peter Hicks' via A gathering place for the Open Rail Data community <openrail...@googlegroups.com> wrote:

Messages were lost between 01:14:55 and 08:33:40 - I will see if I can pick these up elsewhere and get them added to the archive product on RDM (https://raildata.org.uk/dashboard/dataProduct/P-fac2476d-e83c-4f71-b49d-2d14b9053ef2/overview) for tomorrow.

Unfortunately, the outage was not 'clean' and message rates dropped in the minutes prior to the outage.  This means it's not just a simple case of "paste in replacement messages from another archive".  Yesterday's data has therefore not been uploaded yet.

If anyone needs the missing data from yesterday, please drop me an email offlist and I can send you the data kindly provided by Tom Cairns from another source.

I'll be back in touch when I've sorted out the archive properly.

Peter

[Adam Williams]

I definitely want to be clear from the outset that I am appreciative we have gotten to a point where this data is in the public domain and can be consumed, but....

> Fixed - an embarrassingly straightforward case of a lack of disk space on a server causing a process to terminate.  Sorry everyone :(

It sounds like the way this works today is .. rather fragile. I had thought "we" - as a profession - had mostly moved away from relying on single instances of basically anything in pursuit of reliability and being able to do deploys without any risk of downtime.

I would've expected a node to have been recycled as soon as the process terminated, to be honest, and picked up by a standby or a new application node spawned ready to continue work. As to how this works, I suppose that's really a subjective choice, but there are many technical approaches of achieving that same goal.

Is there any work planned here to improve resilience and provide more of a production-suited service? I'm going to assume there was monitoring in place but the timing of this happening was simply unfortunate and meant that it didn't get an immediate response (understandable if it's just you Peter!).

We'll follow-up with Tom for the missing data.

Cheers,

Adam

[Peter Hicks]

Hi Adam

That’s a bit harsh.  Considering I’ve stepped in to get this data out a lot quicker than the industry was able to do, and I’m not being financially compensated for it, I think many people would call your post really rather rude, inconsiderate and wholly ungrateful.

Peter

Sent from Proton Mail for iOS.

-------- Original Message --------

--
You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.

To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/e9065630-fdef-4ce1-875f-3820526c46a5n%40googlegroups.com.

[Adam Williams]

I was clear in the first line of my response that I wasn't attacking trying to attack your efforts, quite the opposite. Nor do I think it's right that this work has had to be done as effectively a "goodwill gesture" in order to get the data product over the line and out the door. 

But if we can't have a sensible, adult conversation about the architecture behind these products and potential incremental improvements that could be made then it's going to continue to be less reliable than I think consumers would want and reasonably expect. And it's going to consume yet more of your time picking up the pieces afterwards when things like this do go wrong [and frankly it's a fact of life that they will at some point], which is especially unfair on you given you've now stated you've not been financially compensated for this work. I would like to understand why that's the case, as it's clear that the likes of Tata Consultancy Services have done quite well out of RDM and the platform is far from perfect today.

Equally, if companies are meant to be consuming this data via some other means where an SLA can be provided and it's not reliant on a single box, I'd be interested to learn more, but I'm not aware that any such facility exists for wider industry participants. So in absence of that, consumers who want to be able to use this data really have no choice, do they?

KR,

Adam

[Peter Hicks]

Hi Adam

On Wednesday, 11 February 2026 at 13:22, 'Adam Williams' via A gathering place for the Open Rail Data community <openrail...@googlegroups.com> wrote:

I was clear in the first line of my response that I wasn't attacking trying to attack your efforts, quite the opposite. Nor do I think it's right that this work has had to be done as effectively a "goodwill gesture" in order to get the data product over the line and out the door.

Rather than steam in and discredit my several decades worth of IT experience, it might have been a little nicer to hold back a little bit and ask if I'd already planned work to make it robust (because honestly, I didn't think I'd still be doing this some months on... ), which is:

1. Commissioned two of the three feeds in to a production environment, pending some software upgrades soon
2. Started logging the raw data from these feeds on independent servers so it's easy to replay or rebuild data should it be lost
3. Moved forward plans to split out RDM publishing in to separate microservices

I was going to announce, without much fanfare, exactly the above later today once I'd had a bit of a rest from a 6am start and very full-on day in healthcare.

Considering that I am doing this because I want this data to be open, rather than because I'm getting anything tangible in return, I think that's a pretty swift turnaround.  Yes, it could be better (as everything could be), but for a 'first not insignificant outage' where I've been upfront on the root cause... maybe a bit of kindness, yeah?

But if we can't have a sensible, adult conversation about the architecture behind these products and potential incremental improvements that could be made then it's going to continue to be less reliable than I think consumers would want and reasonably expect. And it's going to consume yet more of your time picking up the pieces afterwards when things like this do go wrong [and frankly it's a fact of life that they will at some point], which is especially unfair on you given you've now stated you've not been financially compensated for this work. I would like to understand why that's the case, as it's clear that the likes of Tata Consultancy Services have done quite well out of RDM and the platform is far from perfect today.

TCS have done incredibly well at releasing a mostly-bespoke product (even though contractually it might not be seen as such) that does a few things reasonably well.  For example, it's great that I can have somebody else handling billing for me should I want to release a paid-for API, and that I can protect my own servers by having  the RDM's presumably resilient infrastructure proxying HTTP requests, at the expense of any HTTP errors being passed-through, and other things like observability.  But for everything else, it's pretty mediocre and feels like an MVP that's pushed to production.

For the most part of the last decade and a half, I've been working my butt off for free running openraildata-talk and giving out help and assistance to people - even competitors - because if this group didn't exist, we'd all be poorer in knowledge and there would be little puddles of discussion without a central focal point.  I've also been running the Open Rail Data Wiki for a similar duration without any kind of financial compensation because not everyone wants to read a mailing list.  Pages from the wiki are regularly quoted in documents that industry suppliers sent to Network Rail.

None of those things - especially my time - has been compensated by industry in any financially meaningful way for any sustained period - which sucks, quite frankly.  I've offered my time for a less-than-normal day rate on many occasions to try to move things forward faster, better and with more engagement with users - but nobody seems to have the budget to do it - or worse, wants to get my ideas for free.  However there's loads of money to get the Big Four consultancies in to write slick strategy papers referenced as "The XYZ Report" which don't change much.

Equally, if companies are meant to be consuming this data via some other means where an SLA can be provided and it's not reliant on a single box, I'd be interested to learn more, but I'm not aware that any such facility exists for wider industry participants. So in absence of that, consumers who want to be able to use this data really have no choice, do they?

There is the LINX External Interface, but that is an enterprise-level solution which has a long setup-time per connection, and a tangible cost to scale.  One of the things I've always talked about with industry is that you absolutely need to make your data easy to consume via a self-service platform, as well as having a robust platform for those running things that need an SLA, even if no money changes hands.

Gripe over.

Peter

[Adam Williams]

Evening Peter,

I've re-read my initial email and I don't see anything that attempted to discredit you as a person or your experience. What I do see is:

• A suggestion that the architecture that we ended up with is not ideal. I do not think that that is controversial.

• A suggestion that best practice is not reliant on a single node that can fail, because with more mature DevOps and approaches to service reliability we've moved past that sort of architecture. If this came across as patronising, I sincerely apologise - but I also know there are less technical people that consume the messages sent on this group, hence setting out exactly how I would've expected an "ideal" (or "more ideal") set-up to recover, minimising downtime in this sort of scenario as opposed to what it sounded like what happened.

• A question as to whether anything was planned to improve resilience - this is where you would've been entirely within your rights to reply back and say "Yes, it's already been planned since the start - but because it's been unpaid work and I wanted the data available sooner rather than later, this was the compromise".

I'm glad you had already planned improvements in these areas, and (as I said at the very start) I am appreciative of the work you have done - and continue to do - on this data product. All of the things you have suggested sound like they will be helpful.

I know dealing with outages sucks - trust me, I have been there, but I'm a big believer at work in blameless post-mortems wherever possible after an incident to try and learn from and improve on areas that contributed to an incident. Criticising the architecture or specific pieces of code as much as you like is welcomed in pursuit of trying to build something better for customers; digging up authors from the git blame, bringing up names when discussing technical decisions or personal attacks are not acceptable. That was my intention here, nothing else.

KR,

Adam

--
You received this message because you are subscribed to a topic in the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/openraildata-talk/JfLMAm_PyzQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to openraildata-t...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/tXpOzeC2-wyPD82Y3wH2Y8KwMH37QlurBHag0vBoMpMMoLWhLMcX8p7n7GF0Y8Hbylricrx5mZxVRHiUfaZDzUpB41y-LOxaG5p6cVcp2ag%3D%40poggs.co.uk.

[Lawrence Job]

Hi,

Reflecting on the above, I was wondering if there was any way I (or anyone else) can contribute and share the burden?

I'm really happy with what I've been able to build on open rail data, and it would have taken me much longer without the support of the Open Rail Data Wiki and various works from the broader community. I'm very grateful for that.

Is there any way one can meaningfully help out as a software engineer/technical architect/technical writer to any of design, implementation, documentation?

Kind regards,

Lawrence Job